3.2 Node Not Ready

ilmx

I have followed the steps in the PDF to add a node to the cluster but I get a status "NotReady" in the node.

Reason          Message
------          -------
KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

What can be the issue? I am using local VMs and the network communication between them works, as the node could be added in the cluster.

chrispokorni

Hi @ilmx,

There could be several reasons why the node remains in a NotReady state. What hypervisor are you using, what are the sizes (cpu, mem, disk) of the VMs, what guest OS is running the VMs (distribution, release), what type of network are VMs attached to, is all ingress traffic enabled at the hypervisor level?

Regards,
-Chris

kishorevaishnav

Is kubelet is running?

ilmx

Hi,

I am using VMware Workstation. I have created a Host-Only network for them, and assigned IPs statically to each machine. No firewall in place. They can communicate with each other (ping, netcat, etc).

I have created one CP and two working nodes. The VMs are exactly the same:

2 CPU / 4 GB RAM / 500 GB disc
Distributor ID: Ubuntu
Description:    Ubuntu 22.04 LTS
Release:        22.04
Codename:       jammy

The cilium pods in the CP run well, but the ones in the WN don't:

cilium-operator-788c7d7585-592lg   0/1     ContainerCreating   0   5h15m   10.10.10.4      wn2
cilium-operator-788c7d7585-hr5tk   0/1     ContainerCreating   0   5h15m   10.10.10.3      wn1
cilium-tk29d                       0/1     Init:0/6            0   5h15m   10.10.10.3      wn1
cilium-vth84                       0/1     Init:0/6            0   5h13m   10.10.10.4      wn2
cilium-zjqdb                       1/1     Running             0   5h13m   10.10.10.2      cp 
coredns-5d78c9869d-lpnbq           1/1     Running             0   5h13m   192.168.0.214   cp 
coredns-5d78c9869d-n8xd2           1/1     Running             0   5h13m   192.168.0.246   cp 
etcd-cp                            1/1     Running             0   2d23h   10.10.10.2      cp
kube-apiserver-cp                  1/1     Running             2   2d23h   10.10.10.2      cp
kube-controller-manager-cp         1/1     Running             17  2d23h   10.10.10.2      cp
kube-proxy-26bvd                   1/1     Running             0   2d23h   10.10.10.2      cp
kube-proxy-lzf86                   0/1     ContainerCreating   0   43h     10.10.10.4      wn2
kube-proxy-r8pz6                   0/1     ContainerCreating   0   5h13m   10.10.10.3      wn1
kube-scheduler-cp                  1/1     Running             16  5h13m   10.10.10.2      cp

I tried to create a second node to start from scratch but the same issue happened.

I did a change in the YAML file provided (cilium-cni.yaml) because I realised there was a network cidr defined inside, so I changed it to 10.10.10.0/16 to match my network. Then I re-applied but nothing changed.

I am not sure how to investigate further.

ilmx

@kishorevaishnav said:
Is kubelet is running?

Yes, kubelet is running in every machine:

$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Tue 2023-10-24 20:36:24 UTC; 2 days ago

chrispokorni

Hi @ilmx,

Please use the OS release recommended by the lab guide - Ubuntu 20.04 LTS.
The CPU seems adequate per VM, memory could be increased to 8 GB per VM, and 20-50 GB disk space should suffice for the purpose of the lab exercises.
All VMs should have access to/from the internet, to/from each other. All ingress traffic to the VMs from all sources, to all ports and all protocols should be allowed. Please ensure the network type selected satisfies these requirements.
Defining a Cilium Pod network CIDR that is distinct from your existing VM network will help to understand the different networks that take part in the Kubernetes traffic management.

Regards,
-Chris

ilmx

Thanks. I will try also with Ubuntu 20, but I would like to understand how to solve the problem. The network complies except for the Internet access (although I can hack into this to e.g. pull a Pod). But I believe the lack of Internet should not be an issue for my specific problem, should it?

Is there anything specific that I can do to research the issue cni plugin not initialized?