Lab 2.2 cp is showing NotReady due to cni plugin not initialized

When checking the status of the cp and worker nodes
- kubectl get node
- NAME STATUS ROLES AGE VERSION
- cp NotReady control-plane 135m v1.24.1
- worker Ready <none> 50m v1.24.1
and found cp is NotReady. I then tried a
- kubectl describe node cp
- Name: cp
- Roles: control-plane
- Labels: beta.kubernetes.io/arch=amd64
- beta.kubernetes.io/os=linux
- kubernetes.io/arch=amd64
- kubernetes.io/hostname=cp
- kubernetes.io/os=linux
- node-role.kubernetes.io/control-plane=
- node.kubernetes.io/exclude-from-external-load-balancers=
- Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
- node.alpha.kubernetes.io/ttl: 0
- volumes.kubernetes.io/controller-managed-attach-detach: true
- CreationTimestamp: Fri, 19 Aug 2022 19:45:19 +0000
- Taints: node.kubernetes.io/not-ready:NoExecute
- node.kubernetes.io/not-ready:NoSchedule
- Unschedulable: false
- Lease:
- HolderIdentity: cp
- AcquireTime: <unset>
- RenewTime: Fri, 19 Aug 2022 22:04:04 +0000
- Conditions:
- Type Status LastHeartbeatTime LastTransitionTime Reason Message
- ---- ------ ----------------- ------------------ ------ -------
- MemoryPressure False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
- DiskPressure False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
- PIDPressure False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
- Ready False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
- Addresses:
- InternalIP: 10.2.0.5
- Hostname: cp
- Capacity:
- cpu: 2
- ephemeral-storage: 20134592Ki
- hugepages-1Gi: 0
- hugepages-2Mi: 0
- memory: 8137476Ki
- pods: 110
- Allocatable:
- cpu: 2
- ephemeral-storage: 18556039957
- hugepages-1Gi: 0
- hugepages-2Mi: 0
- memory: 8035076Ki
- pods: 110
- System Info:
- Machine ID: fff78146cb1e357890a1028fae17828d
- System UUID: fff78146-cb1e-3578-90a1-028fae17828d
- Boot ID: a461b440-a502-452d-a6d6-6de511f8f630
- Kernel Version: 5.15.0-1016-gcp
- OS Image: Ubuntu 20.04.4 LTS
- Operating System: linux
- Architecture: amd64
- Container Runtime Version: containerd://1.6.7
- Kubelet Version: v1.24.1
- Kube-Proxy Version: v1.24.1
- PodCIDR: 192.168.0.0/24
- PodCIDRs: 192.168.0.0/24
- Non-terminated Pods: (5 in total)
- Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
- --------- ---- ------------ ---------- --------------- ------------- ---
- kube-system etcd-cp 100m (5%) 0 (0%) 100Mi (1%) 0 (0%) 138m
- kube-system kube-apiserver-cp 250m (12%) 0 (0%) 0 (0%) 0 (0%) 138m
- kube-system kube-controller-manager-cp 200m (10%) 0 (0%) 0 (0%) 0 (0%) 138m
- kube-system kube-proxy-mrj6b 0 (0%) 0 (0%) 0 (0%) 0 (0%) 138m
- kube-system kube-scheduler-cp 100m (5%) 0 (0%) 0 (0%) 0 (0%) 138m
- Allocated resources:
- (Total limits may be over 100 percent, i.e., overcommitted.)
- Resource Requests Limits
- -------- -------- ------
- cpu 650m (32%) 0 (0%)
- memory 100Mi (1%) 0 (0%)
- ephemeral-storage 0 (0%) 0 (0%)
- hugepages-1Gi 0 (0%) 0 (0%)
- hugepages-2Mi 0 (0%) 0 (0%)
- Events:
- Type Reason Age From Message
- ---- ------ ---- ---- -------
- Normal Starting 42m kube-proxy
- Warning InvalidDiskCapacity 43m kubelet invalid capacity 0 on image filesystem
- Normal Starting 43m kubelet Starting kubelet.
- Normal NodeAllocatableEnforced 43m kubelet Updated Node Allocatable limit across pods
- Normal NodeHasSufficientMemory 43m (x8 over 43m) kubelet Node cp status is now: NodeHasSufficientMemory
- Normal NodeHasNoDiskPressure 43m (x7 over 43m) kubelet Node cp status is now: NodeHasNoDiskPressure
- Normal NodeHasSufficientPID 43m (x7 over 43m) kubelet Node cp status is now: NodeHasSufficientPID
- Normal RegisteredNode 42m node-controller Node cp event: Registered Node cp in Controller
Where the following error appears to be the main culprit.
- container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Any tips to get the cni plugin initialized? I have validated that
- SELinux is not installed with
sestatus Command 'sestatus' not found
- swap is off with
swapoff -a
invalid capacity 0 on image filesystem
seemed like it might be a culprit as well, but I don't see anything alarming in the following
- df -h
- Filesystem Size Used Avail Use% Mounted on
- /dev/root 20G 3.9G 16G 21% /
- devtmpfs 3.9G 0 3.9G 0% /dev
- tmpfs 3.9G 0 3.9G 0% /dev/shm
- tmpfs 795M 1.6M 794M 1% /run
- tmpfs 5.0M 0 5.0M 0% /run/lock
- tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
- /dev/loop0 56M 56M 0 100% /snap/core18/2538
- /dev/loop1 62M 62M 0 100% /snap/core20/1587
- /dev/loop2 304M 304M 0 100% /snap/google-cloud-cli/58
- /dev/loop3 47M 47M 0 100% /snap/snapd/16292
- /dev/loop4 68M 68M 0 100% /snap/lxd/22753
- /dev/sda15 105M 5.2M 100M 5% /boot/efi
- shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc8a9699f213ac53638eb5afa78b3d4f26925227b7d690b61e4ea0f212c044c4/shm
- shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1252fad3b6c04d24951354fca3aba5c4970bed78b835bcf15b1675891d16aea5/shm
- shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/8fa97bec0fce31245a2946ef5b605d4611a82e4867cef26ab96024e6be7fe377/shm
- shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a6a9d66d452157caf8b3cc7b4ac1b7c95d895f890eb0b4ae1d4cf475aa9d7421/shm
- shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c7fe176d5f0091b8c88446c05e9c8e50b393f33db35057d3879e7eb17037cd83/shm
- /dev/loop5 62M 62M 0 100% /snap/core20/1611
- /dev/loop6 304M 304M 0 100% /snap/google-cloud-cli/60
- tmpfs 795M 0 795M 0% /run/user/1001
Comments
-
The culprit was AppArmor running by default on the Ubuntu 20.04 image. As the course mentions, AppArmor needs to be disabled.
Create a new VM, but this time, prior to running k8scp.sh, try a
- sudo systemctl stop apparmor
- sudo systemctl disable apparmor
and then retry running k8scp.sh using their command
Wait ~30 seconds prior to checking the state as the kubelet starts up
0 -
Doesn't work for me
0 -
Sorry, I deployed v1.25.1, my issue was related to "SystemdCgroup", I solved with this:
https://github.com/containerd/containerd/issues/60090
Categories
- All Categories
- 138 LFX Mentorship
- 138 LFX Mentorship: Linux Kernel
- 815 Linux Foundation IT Professional Programs
- 366 Cloud Engineer IT Professional Program
- 184 Advanced Cloud Engineer IT Professional Program
- 83 DevOps Engineer IT Professional Program
- 151 Cloud Native Developer IT Professional Program
- 142 Express Training Courses & Microlearning
- 142 Express Courses - Discussion Forum
- Microlearning - Discussion Forum
- 6.6K Training Courses
- 48 LFC110 Class Forum - Discontinued
- 72 LFC131 Class Forum
- 49 LFD102 Class Forum
- 234 LFD103 Class Forum
- 21 LFD110 Class Forum
- 44 LFD121 Class Forum
- LFD123 Class Forum
- LFD125 Class Forum
- 18 LFD133 Class Forum
- 8 LFD134 Class Forum
- 18 LFD137 Class Forum
- 72 LFD201 Class Forum
- 5 LFD210 Class Forum
- 5 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 2 LFD233 Class Forum
- 4 LFD237 Class Forum
- 24 LFD254 Class Forum
- 719 LFD259 Class Forum
- 111 LFD272 Class Forum - Discontinued
- 4 LFD272-JP クラス フォーラム
- 13 LFD273 Class Forum
- 235 LFS101 Class Forum
- 2 LFS111 Class Forum
- 3 LFS112 Class Forum
- 3 LFS116 Class Forum
- 7 LFS118 Class Forum
- LFS120 Class Forum
- 9 LFS142 Class Forum
- 8 LFS144 Class Forum
- 4 LFS145 Class Forum
- 4 LFS146 Class Forum
- 16 LFS148 Class Forum
- 15 LFS151 Class Forum
- 5 LFS157 Class Forum
- 65 LFS158 Class Forum
- LFS158-JP クラス フォーラム
- 11 LFS162 Class Forum
- 2 LFS166 Class Forum
- 6 LFS167 Class Forum
- 3 LFS170 Class Forum
- 2 LFS171 Class Forum
- 3 LFS178 Class Forum
- 3 LFS180 Class Forum
- 2 LFS182 Class Forum
- 5 LFS183 Class Forum
- 33 LFS200 Class Forum
- 737 LFS201 Class Forum - Discontinued
- 3 LFS201-JP クラス フォーラム - Discontinued
- 20 LFS203 Class Forum
- 135 LFS207 Class Forum
- 2 LFS207-DE-Klassenforum
- 2 LFS207-JP クラス フォーラム
- 302 LFS211 Class Forum
- 56 LFS216 Class Forum
- 53 LFS241 Class Forum
- 50 LFS242 Class Forum
- 38 LFS243 Class Forum
- 16 LFS244 Class Forum
- 5 LFS245 Class Forum
- LFS246 Class Forum
- LFS248 Class Forum
- 86 LFS250 Class Forum
- 2 LFS250-JP クラス フォーラム
- 1 LFS251 Class Forum
- 156 LFS253 Class Forum
- 1 LFS254 Class Forum
- 2 LFS255 Class Forum
- 12 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.3K LFS258 Class Forum
- 11 LFS258-JP クラス フォーラム
- 135 LFS260 Class Forum
- 161 LFS261 Class Forum
- 43 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 24 LFS267 Class Forum
- 25 LFS268 Class Forum
- 37 LFS269 Class Forum
- 7 LFS270 Class Forum
- 202 LFS272 Class Forum - Discontinued
- 2 LFS272-JP クラス フォーラム
- 4 LFS147 Class Forum
- 2 LFS274 Class Forum
- 4 LFS281 Class Forum
- 16 LFW111 Class Forum
- 262 LFW211 Class Forum
- 185 LFW212 Class Forum
- 15 SKF100 Class Forum
- 1 SKF200 Class Forum
- 2 SKF201 Class Forum
- 797 Hardware
- 199 Drivers
- 68 I/O Devices
- 37 Monitors
- 104 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 85 Storage
- 761 Linux Distributions
- 82 Debian
- 67 Fedora
- 17 Linux Mint
- 13 Mageia
- 23 openSUSE
- 149 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 355 Ubuntu
- 470 Linux System Administration
- 39 Cloud Computing
- 71 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 102 System Management
- 47 Web Management
- 69 Mobile Computing
- 18 Android
- 38 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 379 Off Topic
- 115 Introductions
- 177 Small Talk
- 26 Study Material
- 808 Programming and Development
- 304 Kernel Development
- 486 Software Development
- 1.8K Software
- 263 Applications
- 183 Command Line
- 3 Compiling/Installing
- 988 Games
- 317 Installation
- 103 All In Program
- 103 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)