Welcome to the Linux Foundation Forum!

Lab 2.2 cp is showing NotReady due to cni plugin not initialized

rneureuther1
rneureuther1 Posts: 2
edited August 2022 in LFD259 Class Forum

When checking the status of the cp and worker nodes

kubectl get node
NAME     STATUS     ROLES           AGE    VERSION
cp       NotReady   control-plane   135m   v1.24.1
worker   Ready      <none>          50m    v1.24.1

and found cp is NotReady. I then tried a

kubectl describe node cp
Name:               cp
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=cp
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 19 Aug 2022 19:45:19 +0000
Taints:             node.kubernetes.io/not-ready:NoExecute
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  cp
  AcquireTime:     <unset>
  RenewTime:       Fri, 19 Aug 2022 22:04:04 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Addresses:
  InternalIP:  10.2.0.5
  Hostname:    cp
Capacity:
  cpu:                2
  ephemeral-storage:  20134592Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8137476Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  18556039957
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8035076Ki
  pods:               110
System Info:
  Machine ID:                 fff78146cb1e357890a1028fae17828d
  System UUID:                fff78146-cb1e-3578-90a1-028fae17828d
  Boot ID:                    a461b440-a502-452d-a6d6-6de511f8f630
  Kernel Version:             5.15.0-1016-gcp
  OS Image:                   Ubuntu 20.04.4 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.7
  Kubelet Version:            v1.24.1
  Kube-Proxy Version:         v1.24.1
PodCIDR:                      192.168.0.0/24
PodCIDRs:                     192.168.0.0/24
Non-terminated Pods:          (5 in total)
  Namespace                   Name                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                          ------------  ----------  ---------------  -------------  ---
  kube-system                 etcd-cp                       100m (5%)     0 (0%)      100Mi (1%)       0 (0%)         138m
  kube-system                 kube-apiserver-cp             250m (12%)    0 (0%)      0 (0%)           0 (0%)         138m
  kube-system                 kube-controller-manager-cp    200m (10%)    0 (0%)      0 (0%)           0 (0%)         138m
  kube-system                 kube-proxy-mrj6b              0 (0%)        0 (0%)      0 (0%)           0 (0%)         138m
  kube-system                 kube-scheduler-cp             100m (5%)     0 (0%)      0 (0%)           0 (0%)         138m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                650m (32%)  0 (0%)
  memory             100Mi (1%)  0 (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                   Age                From             Message
  ----     ------                   ----               ----             -------
  Normal   Starting                 42m                kube-proxy       
  Warning  InvalidDiskCapacity      43m                kubelet          invalid capacity 0 on image filesystem
  Normal   Starting                 43m                kubelet          Starting kubelet.
  Normal   NodeAllocatableEnforced  43m                kubelet          Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  43m (x8 over 43m)  kubelet          Node cp status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    43m (x7 over 43m)  kubelet          Node cp status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     43m (x7 over 43m)  kubelet          Node cp status is now: NodeHasSufficientPID
  Normal   RegisteredNode           42m                node-controller  Node cp event: Registered Node cp in Controller

Where the following error appears to be the main culprit.

container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Any tips to get the cni plugin initialized? I have validated that

  • SELinux is not installed with sestatus Command 'sestatus' not found
  • swap is off with swapoff -a

invalid capacity 0 on image filesystem seemed like it might be a culprit as well, but I don't see anything alarming in the following

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        20G  3.9G   16G  21% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           795M  1.6M  794M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0       56M   56M     0 100% /snap/core18/2538
/dev/loop1       62M   62M     0 100% /snap/core20/1587
/dev/loop2      304M  304M     0 100% /snap/google-cloud-cli/58
/dev/loop3       47M   47M     0 100% /snap/snapd/16292
/dev/loop4       68M   68M     0 100% /snap/lxd/22753
/dev/sda15      105M  5.2M  100M   5% /boot/efi
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc8a9699f213ac53638eb5afa78b3d4f26925227b7d690b61e4ea0f212c044c4/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1252fad3b6c04d24951354fca3aba5c4970bed78b835bcf15b1675891d16aea5/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/8fa97bec0fce31245a2946ef5b605d4611a82e4867cef26ab96024e6be7fe377/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a6a9d66d452157caf8b3cc7b4ac1b7c95d895f890eb0b4ae1d4cf475aa9d7421/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c7fe176d5f0091b8c88446c05e9c8e50b393f33db35057d3879e7eb17037cd83/shm
/dev/loop5       62M   62M     0 100% /snap/core20/1611
/dev/loop6      304M  304M     0 100% /snap/google-cloud-cli/60
tmpfs           795M     0  795M   0% /run/user/1001

Comments

  • rneureuther1
    rneureuther1 Posts: 2
    edited August 2022

    The culprit was AppArmor running by default on the Ubuntu 20.04 image. As the course mentions, AppArmor needs to be disabled.

    Create a new VM, but this time, prior to running k8scp.sh, try a

    sudo systemctl stop apparmor
sudo systemctl disable apparmor

    and then retry running k8scp.sh using their command

    Wait ~30 seconds prior to checking the state as the kubelet starts up

  • ibanez
    ibanez Posts: 4

    Doesn't work for me :(

  • ibanez
    ibanez Posts: 4

    Sorry, I deployed v1.25.1, my issue was related to "SystemdCgroup", I solved with this:
    https://github.com/containerd/containerd/issues/6009

Categories

Upcoming Training