LFS258 Lab 3.1 error in step 21

eboyko

Hi, i get an error on step 21:

root@cp:~# kubeadm init --config=kubeadm-config.yaml --upload-certs \

| tee kubeadm-init.out

[init] Using Kubernetes version: v1.27.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cp k8scp kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.22.23.32]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [cp localhost] and IPs [10.22.23.32 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [cp localhost] and IPs [10.22.23.32 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.

I checked kubelet status (journalctl shows the same error):

root@cp:~# systemctl status kubelet

в—Џ kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
в””в”Ђ10-kubeadm.conf
Active: active (running) since Mon 2023-10-09 08:09:19 UTC; 37min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 115676 (kubelet)
Tasks: 12 (limit: 19088)
Memory: 38.6M
CGroup: /system.slice/kubelet.service
в””в”Ђ115676 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9

РѕРєС‚ 09 08:45:50 cp kubelet[115676]: E1009 08:45:50.591912 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:45:55 cp kubelet[115676]: E1009 08:45:55.592907 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:00 cp kubelet[115676]: E1009 08:46:00.595090 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:05 cp kubelet[115676]: E1009 08:46:05.596712 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:10 cp kubelet[115676]: E1009 08:46:10.597993 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:15 cp kubelet[115676]: E1009 08:46:15.599536 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:20 cp kubelet[115676]: E1009 08:46:20.601409 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:25 cp kubelet[115676]: E1009 08:46:25.603476 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:30 cp kubelet[115676]: E1009 08:46:30.605489 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
РѕРєС‚ 09 08:46:35 cp kubelet[115676]: E1009 08:46:35.607091 115676 kubelet.go:2760] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'

My VM for CP:
VMware ESXi Hypervisor
4 CPUs
16 GB memory
200 GB HDD
Ubuntu 20.04.6 LTS
1 network adapter

I tried to repeate all previous steps, but it didn't help. Your help will be appeciated. Thanks in advance.

malloc_failed

If I'm remembering this section correctly, you'll have to edit the file cilium-cni.yaml and edit one of the subnets to AVOID any subnets you're already using. IIRC, this was 192.168.-something, which I edited to 172.16.-something.

Use "kubeadm reset" to clean up the failed node and start fresh.

Use this to test beforehand:
kubectl apply --dry-run --validate='strict' -f /path/to/cilium-cni.yaml

chrispokorni

Hi @eboyko,

I would also recommend checking the ESXi firewall to ensure it allows all inbound traffic to the VM - all protocols, to all ports, from all sources.

Regards,
-Chris

eboyko

I initialized the cp with this and it worked:
root@cp:~# kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.27.1 --node-name k8scp --upload-certs

Maybe there is some mistake in kubadm-config.yaml provided from LFS258V2023-09-14SOLUTIONS.tar.xz?

eboyko

I reinstalled Ubuntu, and it worked. Thank you.

chrispokorni

Hi @eboyko,

I am glad to know that the cluster bootstrapping was successful.

However, keep in mind that "k8scp" was intended to be an alias to the control plane host (supplied to kubeadm init by the controlPlaneEndpoint attribute of the kubeadm-config.yaml manifest or the --control-plane-endpoint flag), and not an actual node name set with the --node-name flag. So the --node-name is set to "cp" while the --control-plane-endpoint is set to the "k8scp" alias.

The --control-plane-endpoint is set to an alias in preparation for the high availability lab exercise in chapter 16.

Regards,
-Chris