LFD259 - Updated to v1.22.1 (10.14.2021)

The line for /etc/containers/storage.conf needs to be restored. The absence of those lines ends up with the "worker" node failing to start up critical containers.

diff --git a/SOLUTIONS/s_02/k8sSecond.sh b/SOLUTIONS/s_02/k8sSecond.sh
index 44060c2..adc691f 100755
--- a/SOLUTIONS/s_02/k8sSecond.sh
+++ b/SOLUTIONS/s_02/k8sSecond.sh
@@ -49,6 +49,9 @@ sudo apt-get update
 # Install cri-o
 sudo apt-get install -y cri-o cri-o-runc podman buildah

+# A bug fix to get past a cri-o update
+sudo sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
+
 sleep 3

 sudo systemctl daemon-reload

kube-system pod/calico-node-djwk2 0/1 Init:0/3 0 4m23s
kube-system pod/kube-proxy-dplzb 0/1 ContainerCreating 0 4m23s

$ kubectl describe -n kube-system pod/calico-node-djwk2

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               5m59s                default-scheduler  Successfully assigned kube-system/calico-node-djwk2 to worker
  Warning  FailedCreatePodSandBox  5m56s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0 in pod sandbox k8s_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0(9ba0a861831c3a59c3382e17221fc83a63c2240fe3d6e12f1e9ca9355915637e): error creating overlay mount to /var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/diff,workdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/work": invalid argument

$ kubectl describe -n kube-system pod/kube-proxy-dplzb

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               7m33s                  default-scheduler  Successfully assigned kube-system/kube-proxy-dplzb to worker
  Warning  FailedCreatePodSandBox  7m30s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0 in pod sandbox k8s_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0(9975181c1522f44afbb0b34af0f6bbadfc90e3cefff42cda4cc434fbb93c235b): error creating overlay mount to /var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/diff,workdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/work": invalid argument

TL;DR: these are Ubuntu 18.04 VMs, running on a KVM hypervisor

Sure. These are Ubuntu 18.04 VMs, based off of upstream provided base cloud image. They are running on a Ubuntu 20.04 hypervisor.

https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.img

Here is the virt-install script (for the 'worker' VM; the 'cp' script is the same except for the --name argument):

export OS_QCOW2=/var/lib/libvirt/images/worker.qcow2
scp /home/ubuntu/_images/ubuntu-18.04-server-cloudimg-amd64.img root@colo4:$OS_QCOW2
ssh root@colo4 qemu-img resize $OS_QCOW2 20G

virt-install \
  --connect qemu+ssh://root@colo4/system \
  --import \
  --virt-type kvm \
  --name worker \
  --ram 8192 \
  --cpu host-model \
  --vcpus 2 \
  --network bridge=br0,model=virtio \
  --disk path=$OS_QCOW2,format=qcow2 \
  --os-type=linux \
  --os-variant ubuntu18.04 \
  --autostart \
  --autoconsole text \
  --cloud-init user-data=./user-data,meta-data=./meta-data

Post virt-install, this is running the following scripts from the 10-13 tarball:

• LFD259/SOLUTIONS/s_02/k8scp.sh
• LFD259/SOLUTIONS/s_02/k8sSecond.sh

The problem arises after the 'kubeadm join ...' command. I am reproducing this right now. The absence of the 'sed' line results in:

$ kubectl get pod -n kube-system
NAME                                       READY   STATUS              RESTARTS   AGE
calico-kube-controllers-75f8f6cc59-wj5rf   1/1     Running             0          12m
calico-node-4knm9                          1/1     Running             0          12m
calico-node-5w7cd                          0/1     Init:0/3            0          9m27s
coredns-78fcd69978-26rtv                   1/1     Running             0          12m
coredns-78fcd69978-84hhb                   1/1     Running             0          12m
etcd-cp                                    1/1     Running             0          12m
kube-apiserver-cp                          1/1     Running             0          12m
kube-controller-manager-cp                 1/1     Running             0          12m
kube-proxy-62qw2                           0/1     ContainerCreating   0          9m27s
kube-proxy-zbq87                           1/1     Running             0          12m
kube-scheduler-cp                          1/1     Running             0          12m

@kingphil said:
The line for /etc/containers/storage.conf needs to be restored. The absence of those lines ends up with the "worker" node failing to start up critical containers.

diff --git a/SOLUTIONS/s_02/k8sSecond.sh b/SOLUTIONS/s_02/k8sSecond.sh
index 44060c2..adc691f 100755
--- a/SOLUTIONS/s_02/k8sSecond.sh
+++ b/SOLUTIONS/s_02/k8sSecond.sh
@@ -49,6 +49,9 @@ sudo apt-get update
 # Install cri-o
 sudo apt-get install -y cri-o cri-o-runc podman buildah
 
+# A bug fix to get past a cri-o update
+sudo sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
+
 sleep 3
 
 sudo systemctl daemon-reload

kube-system pod/calico-node-djwk2 0/1 Init:0/3 0 4m23s
kube-system pod/kube-proxy-dplzb 0/1 ContainerCreating 0 4m23s

$ kubectl describe -n kube-system pod/calico-node-djwk2

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               5m59s                default-scheduler  Successfully assigned kube-system/calico-node-djwk2 to worker
  Warning  FailedCreatePodSandBox  5m56s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0 in pod sandbox k8s_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0(9ba0a861831c3a59c3382e17221fc83a63c2240fe3d6e12f1e9ca9355915637e): error creating overlay mount to /var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/diff,workdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/work": invalid argument

$ kubectl describe -n kube-system pod/kube-proxy-dplzb

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               7m33s                  default-scheduler  Successfully assigned kube-system/kube-proxy-dplzb to worker
  Warning  FailedCreatePodSandBox  7m30s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0 in pod sandbox k8s_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0(9975181c1522f44afbb0b34af0f6bbadfc90e3cefff42cda4cc434fbb93c235b): error creating overlay mount to /var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/diff,workdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/work": invalid argument

I agree with this. I am facing the same issue.
I am using Virtualbox on Windows 11 Host with Ubuntu 18.04 Guest.

The Linux Foundation is fully on board with the "inclusive naming initiative" (https://inclusivenaming.org/ and has systematically implemented its guidelines every where it can in courses. Unfortunately, there are quite a few upstream projects that still use "master" and it is impossible to eliminate its use completely until they do, at which point the courses automatically will not be using these terms. It takes time, and even though CNCF is an initiator of the inclusive naming project, quite a few CNCF projects still use "master".

@kingphil said:
The line for /etc/containers/storage.conf needs to be restored. The absence of those lines ends up with the "worker" node failing to start up critical containers.

diff --git a/SOLUTIONS/s_02/k8sSecond.sh b/SOLUTIONS/s_02/k8sSecond.sh
index 44060c2..adc691f 100755
--- a/SOLUTIONS/s_02/k8sSecond.sh
+++ b/SOLUTIONS/s_02/k8sSecond.sh
@@ -49,6 +49,9 @@ sudo apt-get update
 # Install cri-o
 sudo apt-get install -y cri-o cri-o-runc podman buildah
 
+# A bug fix to get past a cri-o update
+sudo sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
+
 sleep 3
 
 sudo systemctl daemon-reload

kube-system pod/calico-node-djwk2 0/1 Init:0/3 0 4m23s
kube-system pod/kube-proxy-dplzb 0/1 ContainerCreating 0 4m23s

$ kubectl describe -n kube-system pod/calico-node-djwk2

```

Confirming this post! Guys, please fix it in the "k8sSecond.sh", I have wasted more than an hour to figure out why the steps are not working on VMware fusion.

I believe I am experiencing this same issue still. @serewicz