Welcome to the Linux Foundation Forum!

Lab 2.2 cp is showing NotReady due to cni plugin not initialized

Posts: 2
edited August 2022 in LFD259 Class Forum

When checking the status of the cp and worker nodes

  1. kubectl get node
  2. NAME STATUS ROLES AGE VERSION
  3. cp NotReady control-plane 135m v1.24.1
  4. worker Ready <none> 50m v1.24.1

and found cp is NotReady. I then tried a

  1. kubectl describe node cp
  2. Name: cp
  3. Roles: control-plane
  4. Labels: beta.kubernetes.io/arch=amd64
  5. beta.kubernetes.io/os=linux
  6. kubernetes.io/arch=amd64
  7. kubernetes.io/hostname=cp
  8. kubernetes.io/os=linux
  9. node-role.kubernetes.io/control-plane=
  10. node.kubernetes.io/exclude-from-external-load-balancers=
  11. Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
  12. node.alpha.kubernetes.io/ttl: 0
  13. volumes.kubernetes.io/controller-managed-attach-detach: true
  14. CreationTimestamp: Fri, 19 Aug 2022 19:45:19 +0000
  15. Taints: node.kubernetes.io/not-ready:NoExecute
  16. node.kubernetes.io/not-ready:NoSchedule
  17. Unschedulable: false
  18. Lease:
  19. HolderIdentity: cp
  20. AcquireTime: <unset>
  21. RenewTime: Fri, 19 Aug 2022 22:04:04 +0000
  22. Conditions:
  23. Type Status LastHeartbeatTime LastTransitionTime Reason Message
  24. ---- ------ ----------------- ------------------ ------ -------
  25. MemoryPressure False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
  26. DiskPressure False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
  27. PIDPressure False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
  28. Ready False Fri, 19 Aug 2022 22:02:02 +0000 Fri, 19 Aug 2022 19:45:12 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  29. Addresses:
  30. InternalIP: 10.2.0.5
  31. Hostname: cp
  32. Capacity:
  33. cpu: 2
  34. ephemeral-storage: 20134592Ki
  35. hugepages-1Gi: 0
  36. hugepages-2Mi: 0
  37. memory: 8137476Ki
  38. pods: 110
  39. Allocatable:
  40. cpu: 2
  41. ephemeral-storage: 18556039957
  42. hugepages-1Gi: 0
  43. hugepages-2Mi: 0
  44. memory: 8035076Ki
  45. pods: 110
  46. System Info:
  47. Machine ID: fff78146cb1e357890a1028fae17828d
  48. System UUID: fff78146-cb1e-3578-90a1-028fae17828d
  49. Boot ID: a461b440-a502-452d-a6d6-6de511f8f630
  50. Kernel Version: 5.15.0-1016-gcp
  51. OS Image: Ubuntu 20.04.4 LTS
  52. Operating System: linux
  53. Architecture: amd64
  54. Container Runtime Version: containerd://1.6.7
  55. Kubelet Version: v1.24.1
  56. Kube-Proxy Version: v1.24.1
  57. PodCIDR: 192.168.0.0/24
  58. PodCIDRs: 192.168.0.0/24
  59. Non-terminated Pods: (5 in total)
  60. Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
  61. --------- ---- ------------ ---------- --------------- ------------- ---
  62. kube-system etcd-cp 100m (5%) 0 (0%) 100Mi (1%) 0 (0%) 138m
  63. kube-system kube-apiserver-cp 250m (12%) 0 (0%) 0 (0%) 0 (0%) 138m
  64. kube-system kube-controller-manager-cp 200m (10%) 0 (0%) 0 (0%) 0 (0%) 138m
  65. kube-system kube-proxy-mrj6b 0 (0%) 0 (0%) 0 (0%) 0 (0%) 138m
  66. kube-system kube-scheduler-cp 100m (5%) 0 (0%) 0 (0%) 0 (0%) 138m
  67. Allocated resources:
  68. (Total limits may be over 100 percent, i.e., overcommitted.)
  69. Resource Requests Limits
  70. -------- -------- ------
  71. cpu 650m (32%) 0 (0%)
  72. memory 100Mi (1%) 0 (0%)
  73. ephemeral-storage 0 (0%) 0 (0%)
  74. hugepages-1Gi 0 (0%) 0 (0%)
  75. hugepages-2Mi 0 (0%) 0 (0%)
  76. Events:
  77. Type Reason Age From Message
  78. ---- ------ ---- ---- -------
  79. Normal Starting 42m kube-proxy
  80. Warning InvalidDiskCapacity 43m kubelet invalid capacity 0 on image filesystem
  81. Normal Starting 43m kubelet Starting kubelet.
  82. Normal NodeAllocatableEnforced 43m kubelet Updated Node Allocatable limit across pods
  83. Normal NodeHasSufficientMemory 43m (x8 over 43m) kubelet Node cp status is now: NodeHasSufficientMemory
  84. Normal NodeHasNoDiskPressure 43m (x7 over 43m) kubelet Node cp status is now: NodeHasNoDiskPressure
  85. Normal NodeHasSufficientPID 43m (x7 over 43m) kubelet Node cp status is now: NodeHasSufficientPID
  86. Normal RegisteredNode 42m node-controller Node cp event: Registered Node cp in Controller

Where the following error appears to be the main culprit.

  1. container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Any tips to get the cni plugin initialized? I have validated that

  • SELinux is not installed with sestatus Command 'sestatus' not found
  • swap is off with swapoff -a

invalid capacity 0 on image filesystem seemed like it might be a culprit as well, but I don't see anything alarming in the following

  1. df -h
  2. Filesystem Size Used Avail Use% Mounted on
  3. /dev/root 20G 3.9G 16G 21% /
  4. devtmpfs 3.9G 0 3.9G 0% /dev
  5. tmpfs 3.9G 0 3.9G 0% /dev/shm
  6. tmpfs 795M 1.6M 794M 1% /run
  7. tmpfs 5.0M 0 5.0M 0% /run/lock
  8. tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
  9. /dev/loop0 56M 56M 0 100% /snap/core18/2538
  10. /dev/loop1 62M 62M 0 100% /snap/core20/1587
  11. /dev/loop2 304M 304M 0 100% /snap/google-cloud-cli/58
  12. /dev/loop3 47M 47M 0 100% /snap/snapd/16292
  13. /dev/loop4 68M 68M 0 100% /snap/lxd/22753
  14. /dev/sda15 105M 5.2M 100M 5% /boot/efi
  15. shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc8a9699f213ac53638eb5afa78b3d4f26925227b7d690b61e4ea0f212c044c4/shm
  16. shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1252fad3b6c04d24951354fca3aba5c4970bed78b835bcf15b1675891d16aea5/shm
  17. shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/8fa97bec0fce31245a2946ef5b605d4611a82e4867cef26ab96024e6be7fe377/shm
  18. shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a6a9d66d452157caf8b3cc7b4ac1b7c95d895f890eb0b4ae1d4cf475aa9d7421/shm
  19. shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c7fe176d5f0091b8c88446c05e9c8e50b393f33db35057d3879e7eb17037cd83/shm
  20. /dev/loop5 62M 62M 0 100% /snap/core20/1611
  21. /dev/loop6 304M 304M 0 100% /snap/google-cloud-cli/60
  22. tmpfs 795M 0 795M 0% /run/user/1001

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 2
    edited August 2022

    The culprit was AppArmor running by default on the Ubuntu 20.04 image. As the course mentions, AppArmor needs to be disabled.

    Create a new VM, but this time, prior to running k8scp.sh, try a

    1. sudo systemctl stop apparmor
    2. sudo systemctl disable apparmor

    and then retry running k8scp.sh using their command

    Wait ~30 seconds prior to checking the state as the kubelet starts up

  • Doesn't work for me :(

  • Sorry, I deployed v1.25.1, my issue was related to "SystemdCgroup", I solved with this:
    https://github.com/containerd/containerd/issues/6009

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training