Lab 2.2 cp is showing NotReady due to cni plugin not initialized

rneureuther1

When checking the status of the cp and worker nodes

kubectl get node
NAME     STATUS     ROLES           AGE    VERSION
cp       NotReady   control-plane   135m   v1.24.1
worker   Ready      <none>          50m    v1.24.1

and found cp is NotReady. I then tried a

kubectl describe node cp
Name:               cp
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=cp
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 19 Aug 2022 19:45:19 +0000
Taints:             node.kubernetes.io/not-ready:NoExecute
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  cp
  AcquireTime:     <unset>
  RenewTime:       Fri, 19 Aug 2022 22:04:04 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 19 Aug 2022 22:02:02 +0000   Fri, 19 Aug 2022 19:45:12 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Addresses:
  InternalIP:  10.2.0.5
  Hostname:    cp
Capacity:
  cpu:                2
  ephemeral-storage:  20134592Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8137476Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  18556039957
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8035076Ki
  pods:               110
System Info:
  Machine ID:                 fff78146cb1e357890a1028fae17828d
  System UUID:                fff78146-cb1e-3578-90a1-028fae17828d
  Boot ID:                    a461b440-a502-452d-a6d6-6de511f8f630
  Kernel Version:             5.15.0-1016-gcp
  OS Image:                   Ubuntu 20.04.4 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.7
  Kubelet Version:            v1.24.1
  Kube-Proxy Version:         v1.24.1
PodCIDR:                      192.168.0.0/24
PodCIDRs:                     192.168.0.0/24
Non-terminated Pods:          (5 in total)
  Namespace                   Name                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                          ------------  ----------  ---------------  -------------  ---
  kube-system                 etcd-cp                       100m (5%)     0 (0%)      100Mi (1%)       0 (0%)         138m
  kube-system                 kube-apiserver-cp             250m (12%)    0 (0%)      0 (0%)           0 (0%)         138m
  kube-system                 kube-controller-manager-cp    200m (10%)    0 (0%)      0 (0%)           0 (0%)         138m
  kube-system                 kube-proxy-mrj6b              0 (0%)        0 (0%)      0 (0%)           0 (0%)         138m
  kube-system                 kube-scheduler-cp             100m (5%)     0 (0%)      0 (0%)           0 (0%)         138m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                650m (32%)  0 (0%)
  memory             100Mi (1%)  0 (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                   Age                From             Message
  ----     ------                   ----               ----             -------
  Normal   Starting                 42m                kube-proxy       
  Warning  InvalidDiskCapacity      43m                kubelet          invalid capacity 0 on image filesystem
  Normal   Starting                 43m                kubelet          Starting kubelet.
  Normal   NodeAllocatableEnforced  43m                kubelet          Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  43m (x8 over 43m)  kubelet          Node cp status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    43m (x7 over 43m)  kubelet          Node cp status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     43m (x7 over 43m)  kubelet          Node cp status is now: NodeHasSufficientPID
  Normal   RegisteredNode           42m                node-controller  Node cp event: Registered Node cp in Controller

Where the following error appears to be the main culprit.

container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Any tips to get the cni plugin initialized? I have validated that

• SELinux is not installed with sestatus Command 'sestatus' not found
• swap is off with swapoff -a

invalid capacity 0 on image filesystem seemed like it might be a culprit as well, but I don't see anything alarming in the following

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        20G  3.9G   16G  21% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           795M  1.6M  794M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0       56M   56M     0 100% /snap/core18/2538
/dev/loop1       62M   62M     0 100% /snap/core20/1587
/dev/loop2      304M  304M     0 100% /snap/google-cloud-cli/58
/dev/loop3       47M   47M     0 100% /snap/snapd/16292
/dev/loop4       68M   68M     0 100% /snap/lxd/22753
/dev/sda15      105M  5.2M  100M   5% /boot/efi
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc8a9699f213ac53638eb5afa78b3d4f26925227b7d690b61e4ea0f212c044c4/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1252fad3b6c04d24951354fca3aba5c4970bed78b835bcf15b1675891d16aea5/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/8fa97bec0fce31245a2946ef5b605d4611a82e4867cef26ab96024e6be7fe377/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a6a9d66d452157caf8b3cc7b4ac1b7c95d895f890eb0b4ae1d4cf475aa9d7421/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c7fe176d5f0091b8c88446c05e9c8e50b393f33db35057d3879e7eb17037cd83/shm
/dev/loop5       62M   62M     0 100% /snap/core20/1611
/dev/loop6      304M  304M     0 100% /snap/google-cloud-cli/60
tmpfs           795M     0  795M   0% /run/user/1001

rneureuther1

The culprit was AppArmor running by default on the Ubuntu 20.04 image. As the course mentions, AppArmor needs to be disabled.

Create a new VM, but this time, prior to running k8scp.sh, try a

sudo systemctl stop apparmor
sudo systemctl disable apparmor

and then retry running k8scp.sh using their command

Wait ~30 seconds prior to checking the state as the kubelet starts up

ibanez

Doesn't work for me

ibanez

Sorry, I deployed v1.25.1, my issue was related to "SystemdCgroup", I solved with this:
https://github.com/containerd/containerd/issues/6009