Network unavailable after worker joins control-plane

harriotj

I am two running VMs on VirtualBox (network set as bridge-adapter) for the worker and control-plane nodes

I am having problems with the first lab in creating a cluster

After running the join command on the worker node I notice it joins the cluster but never becomes ready. The describe node command indicates the worker node has stopped publishing status.

At that point I noticed I could no longer ping from either node to the other. Any clues?

FYI:
worker-node IP: 10.0.0.55
cp-node IP: 10.0.0.139

$sudo ip route
default via 10.0.0.1 dev enp0s3 proto dhcp metric 100 
10.0.0.0/24 via 10.0.0.171 dev cilium_host proto kernel src 10.0.0.171 
10.0.0.0/24 dev enp0s3 proto kernel scope link src 10.0.0.139 metric 100 
10.0.0.171 dev cilium_host proto kernel scope link 
169.254.0.0/16 dev enp0s3 scope link metric 1000

Name:               kube4
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kube4
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 29 Jun 2024 16:03:16 +0930
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.cilium.io/agent-not-ready:NoSchedule
                    node.kubernetes.io/unreachable:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  kube4
  AcquireTime:     <unset>
  RenewTime:       Sat, 29 Jun 2024 16:03:46 +0930
Conditions:
  Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----             ------    -----------------                 ------------------                ------              -------
  MemoryPressure   Unknown   Sat, 29 Jun 2024 16:03:26 +0930   Sat, 29 Jun 2024 16:04:29 +0930   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure     Unknown   Sat, 29 Jun 2024 16:03:26 +0930   Sat, 29 Jun 2024 16:04:29 +0930   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure      Unknown   Sat, 29 Jun 2024 16:03:26 +0930   Sat, 29 Jun 2024 16:04:29 +0930   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready            Unknown   Sat, 29 Jun 2024 16:03:26 +0930   Sat, 29 Jun 2024 16:04:29 +0930   NodeStatusUnknown   Kubelet stopped posting node status.
Addresses:
  InternalIP:  10.0.0.55
  Hostname:    kube4
Capacity:
  cpu:                2
  ephemeral-storage:  19947120Ki
  hugepages-2Mi:      0
  memory:             4005976Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  18383265762
  hugepages-2Mi:      0
  memory:             3903576Ki
  pods:               110
System Info:
  Machine ID:                 138ec1e0ef054b269929f06da17de685
  System UUID:                224bb1df-fdf8-064a-bdc1-d79a8e4d8d50
  Boot ID:                    46eddc31-2179-4f0b-b882-38afe5c7425f
  Kernel Version:             5.15.0-113-generic
  OS Image:                   Ubuntu 20.04.6 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.18
  Kubelet Version:            v1.30.1
  Kube-Proxy Version:         v1.30.1
PodCIDR:                      20.0.1.0/24
PodCIDRs:                     20.0.1.0/24
Non-terminated Pods:          (2 in total)
  Namespace                   Name                CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                ------------  ----------  ---------------  -------------  ---
  kube-system                 cilium-2psgw        100m (5%)     0 (0%)      10Mi (0%)        0 (0%)         62m
  kube-system                 kube-proxy-trgz7    0 (0%)        0 (0%)      0 (0%)           0 (0%)         62m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (5%)  0 (0%)
  memory             10Mi (0%)  0 (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:              <none>

harriotj

Perhaps it has something to do with the Cilium PodCIDR conflicting with the host CIDR. If so how does one change the Cilium PodCIDR

chrispokorni

Hi @harriotj,

Please provide the outputs of the following commands:

kubectl get nodes -o wide
kubectl get pods -A -o wide

Regards,
-Chris