LF258 Unable to join worker to control plane on Lab 3.2

Answers
-
Hi @skopos,
This initial error is one of the most common issues reported in the forum, with cluster VMs running in the cloud or on local hypervisors.
The health checks seem to fail because the health check port does not accept the connection. This is typical for a firewall blocking access to that particular port. What type of infrastructure is hosting your cluster? Local hypervisor or cloud? In each case please ensure that the infrastructure level firewall allows all incoming traffic from all sources, all protocols, to all ports. In addition, the guest OS of your VM instances should have firewalls disabled.
Also, ensure that
/etc/hosts
includes the correct control plane node information.Regards,
-Chris0 -
Both VMs are hosted virtualbox.
Firewalls are disabled and interfaces are set in "promiscuous mode".
We had a meeting during office hours today.
0 -
0
-
Hi @skopos,
What is the value of the
controlPlaneEndpoint
from the kubeadm-config.yaml file, and what was the entirekubeadm join
command used on the worker node?Also, what is the size of the worker node (CPU, mem, disk)?
Regards,
-Chris0 -
Command to join worker node:
kubeadm join --token fx8hwt.i02nwpzk3iqpesc5 kmaster:6443 --discovery-token-ca-cert-hash sha256:5de8cf806d593ce7e28ce382cb9b32fe91acf10f3a60e6f8a954469edcadc18csize of worker: 2CPU, 3GB, 12GB disk
0 -
Hi @skopos,
Thank you for the detailed output.
Occasionally, I noticed that kubelet complaining (indirectly) about the size of its nodes as well, more precisely about the available CPU and memory. Also, from our earlier conversation, I remember that we had tried to bootstrap the cluster several times on the same two VirtualBox VMs, each time with different configuration attempts.Considering the "history" of these two VMs, I would recommend a fresh install, starting with two new VirtualBox VMs. Please ensure that the control plane VM has 2 CPU cores, 4 to 8 GB of memory, 15-20 GB virtual disk, and the worker VM has 2 CPU cores, 4+ GB memory, 15-20 GB virtual disk. Each VM with one Bridged network interface, promiscuous mode - allow all, with private IP addresses assigned by the default DHCP server of VirtualBox (they will be from 192.168.x.0/24 range, no need to assign manually during the provisioning process).
After the guest OS installation (Ubuntu 20.04 LTS), please proceed with the bootstrapping steps of the Kubernetes cluster, while paying close attention to the following details:
-calico.yaml
- IP range should be modified to10.200.0.0/16
(to avoid overlapping IP addresses between nodes and pods)
-kubeadm-config.yaml
-podSubnet: 10.200.0.0/16
(to match the pod network from calico.yaml)
-kubeadm-config.yaml
-controlPlaneEndpoint: "k8scp:6443"
- use the alias provided in the lab guide, NOT the hostname of the control plane VM
-/etc/hosts
- on both VMs, to include the private IP of the control plane VM (192.168.x.y) and the k8scp alias (if desired, it can include the hostname as well)Regards,
-Chris0 -
@skopos Hi, obviously you have an issue with kubelet on worker's node it can be seen from the first output you provided. So I would check if kubelet is running at all with the command:
systemctl status kubelet
and in case it is not in Active state I would check what is wrong with it using commandjournalctl -xeu kubelet
. And as @chrispokorni already noticed your nodes' subnet is overlapped with pods' subnet I would fix it as well.By the way, I had the same output from kubelet and it was related to the fact that kubelet cgroup driver was different from docker cgroup driver. You can compare them using following commands:
sudo docker info | grep "Cgroup Driver"
sudo cat /var/lib/kubelet/config.yaml | grep cgroupDriver
1 -
hi @oleksazhel , yes indeed kubelet was not working.
I am redeploying the lab environment, and I will keep updating you all about the progress0 -
@chrispokorni I have reinstalled all the environment as suggested without any progress.@oleksazhel , I think you are right as the cgropdriver is different .
Which one should be configured? (systemd or cgroupfs)0 -
@skopos I used systemd, but I believe you can configure one of them, the main thing that they were the same.
0 -
Hi @skopos,
There are steps in the lab guide where the docker daemon gets configured with the correct cgroup driver. This needs to be done on each node. If the steps are missed, these errors will persist.
Regards,
-Chris0 -
which step?
I already did in lab 3.1
step 15 (a) on control planeI do not find any instruction for configuring the worker
0 -
Hi @skopos,
In Exercise 3.2 the steps under "1(c)i for Docker. The content of the
daemon.json
file should be the same as presented earlier in Exercise 3.1 step 15(a).Regards,
-Chris0 -
ok I have done as suggested but I have errors in my kubelet service
0 -
Hi @skopos,
I would be interested to see what exactly "failed to load..." Can you attach the kubelet logs from the worker VM syslog, instead of a screenshot?
What Ubuntu version is running on the worker VM? How much CPU, memory and disk space is the worker provisioned with?Regards,
-Chris0 -
@skopos From your logs:
May 09 20:08:54 worker kubelet[9031]: E0509 20:08:54.029745 9031 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
-- that is time when you tried to join worker to cluster. I cannot see more attempts to join.Then I can see error like
May 11 20:45:44 worker kubelet[1693]: E0511 20:45:44.491487 1693 server.go:206] "Failed to load kubelet config file" err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml"
, and it is okay, because config will be created during joining to cluster withkubeadm join
command. So just try to rejoin worker.0 -
Hi @skopos,
Thank you for the detailed feedback. I agree with @oleksazhel that your logs show that kubelet panicked about the cgroup driver at first, and then repeatedly about the missing kubelet/config.yaml file, which would need to be generated part of the join phase. Other than that there is nothing else in there to indicate what may be wrong.
However, something that typically is not obvious is the fact that kubelet may misbehave due to insufficient resources - in this case I suspect the 2GB of RAM causing the errors. The worker VM runs the guest OS, the container runtime - docker, and eventually needs to run the kubelet node agent, a proxy node agent, a networking node agent Calico, all this before it will run any containerized workload. The OS and the runtime alone may use up a lot of the 2GB of RAM, so eventually when kubelet is started up, it cannot fully activate the worker node because it no longer has enough working memory.
I recommend, again, that the worker be provisioned with 2 CPU cores and 4+ GB memory to be able to build the cluster. If the physical resources of the host system cannot accommodate two VMs of the required sizes, you could complete many of the lab exercises on a single node cluster, but do expect differences in outputs and in the behavior of certain applications.
Regards,
-Chris1 -
1
Categories
- All Categories
- 49 LFX Mentorship
- 102 LFX Mentorship: Linux Kernel
- 553 Linux Foundation Boot Camps
- 296 Cloud Engineer Boot Camp
- 119 Advanced Cloud Engineer Boot Camp
- 52 DevOps Engineer Boot Camp
- 53 Cloud Native Developer Boot Camp
- 4 Express Training Courses
- 4 Express Courses - Discussion Forum
- 1.9K Training Courses
- 18 LFC110 Class Forum
- 6 LFC131 Class Forum
- 25 LFD102 Class Forum
- 150 LFD103 Class Forum
- 17 LFD121 Class Forum
- LFD137 Class Forum
- 61 LFD201 Class Forum
- LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 598 LFD259 Class Forum
- 102 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- LFD273 Class Forum
- 2 LFS145 Class Forum
- 24 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 3 LFS203 Class Forum
- 69 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 34 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 140 LFS253 Class Forum
- LFS254 Class Forum
- 1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 92 LFS260 Class Forum
- 130 LFS261 Class Forum
- 32 LFS262 Class Forum
- 79 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 17 LFS267 Class Forum
- 17 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS281 Class Forum
- 221 LFW211 Class Forum
- 167 LFW212 Class Forum
- SKF100 Class Forum
- 901 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 208 Networking
- 101 Printers & Scanners
- 85 Storage
- 761 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 141 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 356 Ubuntu
- 477 Linux System Administration
- 41 Cloud Computing
- 69 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 77 Network Management
- 108 System Management
- 49 Web Management
- 66 Mobile Computing
- 23 Android
- 29 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 536 Off Topic
- 131 Introductions
- 216 Small Talk
- 21 Study Material
- 817 Programming and Development
- 275 Kernel Development
- 508 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 59 All In Program
- 59 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)