LF258 Unable to join worker to control plane on Lab 3.2

skopos

chrispokorni

Hi @skopos,

This initial error is one of the most common issues reported in the forum, with cluster VMs running in the cloud or on local hypervisors.

The health checks seem to fail because the health check port does not accept the connection. This is typical for a firewall blocking access to that particular port. What type of infrastructure is hosting your cluster? Local hypervisor or cloud? In each case please ensure that the infrastructure level firewall allows all incoming traffic from all sources, all protocols, to all ports. In addition, the guest OS of your VM instances should have firewalls disabled.

Also, ensure that /etc/hosts includes the correct control plane node information.

Regards,
-Chris

skopos

Both VMs are hosted virtualbox.

Firewalls are disabled and interfaces are set in "promiscuous mode".

We had a meeting during office hours today.

skopos

chrispokorni

Hi @skopos,

What is the value of the controlPlaneEndpoint from the kubeadm-config.yaml file, and what was the entire kubeadm join command used on the worker node?

Also, what is the size of the worker node (CPU, mem, disk)?

Regards,
-Chris

skopos

Command to join worker node:
kubeadm join --token fx8hwt.i02nwpzk3iqpesc5 kmaster:6443 --discovery-token-ca-cert-hash sha256:5de8cf806d593ce7e28ce382cb9b32fe91acf10f3a60e6f8a954469edcadc18c

size of worker: 2CPU, 3GB, 12GB disk

chrispokorni

Hi @skopos,

Thank you for the detailed output.
Occasionally, I noticed that kubelet complaining (indirectly) about the size of its nodes as well, more precisely about the available CPU and memory. Also, from our earlier conversation, I remember that we had tried to bootstrap the cluster several times on the same two VirtualBox VMs, each time with different configuration attempts.

Considering the "history" of these two VMs, I would recommend a fresh install, starting with two new VirtualBox VMs. Please ensure that the control plane VM has 2 CPU cores, 4 to 8 GB of memory, 15-20 GB virtual disk, and the worker VM has 2 CPU cores, 4+ GB memory, 15-20 GB virtual disk. Each VM with one Bridged network interface, promiscuous mode - allow all, with private IP addresses assigned by the default DHCP server of VirtualBox (they will be from 192.168.x.0/24 range, no need to assign manually during the provisioning process).

After the guest OS installation (Ubuntu 20.04 LTS), please proceed with the bootstrapping steps of the Kubernetes cluster, while paying close attention to the following details:
- calico.yaml - IP range should be modified to 10.200.0.0/16 (to avoid overlapping IP addresses between nodes and pods)
- kubeadm-config.yaml - podSubnet: 10.200.0.0/16 (to match the pod network from calico.yaml)
- kubeadm-config.yaml - controlPlaneEndpoint: "k8scp:6443" - use the alias provided in the lab guide, NOT the hostname of the control plane VM
- /etc/hosts - on both VMs, to include the private IP of the control plane VM (192.168.x.y) and the k8scp alias (if desired, it can include the hostname as well)

Regards,
-Chris

oleksazhel

@skopos Hi, obviously you have an issue with kubelet on worker's node it can be seen from the first output you provided. So I would check if kubelet is running at all with the command: systemctl status kubelet and in case it is not in Active state I would check what is wrong with it using command journalctl -xeu kubelet. And as @chrispokorni already noticed your nodes' subnet is overlapped with pods' subnet I would fix it as well.

By the way, I had the same output from kubelet and it was related to the fact that kubelet cgroup driver was different from docker cgroup driver. You can compare them using following commands:

• sudo docker info | grep "Cgroup Driver"
• sudo cat /var/lib/kubelet/config.yaml | grep cgroupDriver

skopos

hi @oleksazhel , yes indeed kubelet was not working.
I am redeploying the lab environment, and I will keep updating you all about the progress

skopos

@chrispokorni I have reinstalled all the environment as suggested without any progress.

@oleksazhel , I think you are right as the cgropdriver is different .
Which one should be configured? (systemd or cgroupfs)

oleksazhel

@skopos I used systemd, but I believe you can configure one of them, the main thing that they were the same.

chrispokorni

Hi @skopos,

There are steps in the lab guide where the docker daemon gets configured with the correct cgroup driver. This needs to be done on each node. If the steps are missed, these errors will persist.

Regards,
-Chris

skopos

which step?
I already did in lab 3.1
step 15 (a) on control plane

I do not find any instruction for configuring the worker

chrispokorni

Hi @skopos,

In Exercise 3.2 the steps under "1(c)i for Docker. The content of the daemon.json file should be the same as presented earlier in Exercise 3.1 step 15(a).

Regards,
-Chris

skopos

ok I have done as suggested but I have errors in my kubelet service

chrispokorni

Hi @skopos,

I would be interested to see what exactly "failed to load..." Can you attach the kubelet logs from the worker VM syslog, instead of a screenshot?
What Ubuntu version is running on the worker VM? How much CPU, memory and disk space is the worker provisioned with?

Regards,
-Chris

skopos

Ubuntu 20.04
2 CPU
2GB RAM
20 GB

I am attaching the output of journalctl -u kubelet, tell me if you need anything else

oleksazhel

@skopos From your logs: May 09 20:08:54 worker kubelet[9031]: E0509 20:08:54.029745 9031 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\"" -- that is time when you tried to join worker to cluster. I cannot see more attempts to join.

Then I can see error like May 11 20:45:44 worker kubelet[1693]: E0511 20:45:44.491487 1693 server.go:206] "Failed to load kubelet config file" err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml" , and it is okay, because config will be created during joining to cluster with kubeadm join command. So just try to rejoin worker.

chrispokorni

Hi @skopos,

Thank you for the detailed feedback. I agree with @oleksazhel that your logs show that kubelet panicked about the cgroup driver at first, and then repeatedly about the missing kubelet/config.yaml file, which would need to be generated part of the join phase. Other than that there is nothing else in there to indicate what may be wrong.

However, something that typically is not obvious is the fact that kubelet may misbehave due to insufficient resources - in this case I suspect the 2GB of RAM causing the errors. The worker VM runs the guest OS, the container runtime - docker, and eventually needs to run the kubelet node agent, a proxy node agent, a networking node agent Calico, all this before it will run any containerized workload. The OS and the runtime alone may use up a lot of the 2GB of RAM, so eventually when kubelet is started up, it cannot fully activate the worker node because it no longer has enough working memory.

I recommend, again, that the worker be provisioned with 2 CPU cores and 4+ GB memory to be able to build the cluster. If the physical resources of the host system cannot accommodate two VMs of the required sizes, you could complete many of the lab exercises on a single node cluster, but do expect differences in outputs and in the behavior of certain applications.

Regards,
-Chris

skopos

Yes, that was it
It is fixed!

Thank you @oleksazhel and @chrispokorni for the kind help.