LFD259 - Updated to v1.22.1 (10.14.2021)

fcioanca

Hi,

LFD259 has been updated to Kubernetes v1.22.1. There is new content and labs to match the updated CKAD exam domains and competencies. To ensure you have access to the latest materials, please clear your cache.

Regards,
Flavia
The Linux Foundation Training Team

kingphil

The line for /etc/containers/storage.conf needs to be restored. The absence of those lines ends up with the "worker" node failing to start up critical containers.

diff --git a/SOLUTIONS/s_02/k8sSecond.sh b/SOLUTIONS/s_02/k8sSecond.sh
index 44060c2..adc691f 100755
--- a/SOLUTIONS/s_02/k8sSecond.sh
+++ b/SOLUTIONS/s_02/k8sSecond.sh
@@ -49,6 +49,9 @@ sudo apt-get update
 # Install cri-o
 sudo apt-get install -y cri-o cri-o-runc podman buildah

+# A bug fix to get past a cri-o update
+sudo sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
+
 sleep 3

 sudo systemctl daemon-reload

kube-system pod/calico-node-djwk2 0/1 Init:0/3 0 4m23s
kube-system pod/kube-proxy-dplzb 0/1 ContainerCreating 0 4m23s

$ kubectl describe -n kube-system pod/calico-node-djwk2

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               5m59s                default-scheduler  Successfully assigned kube-system/calico-node-djwk2 to worker
  Warning  FailedCreatePodSandBox  5m56s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0 in pod sandbox k8s_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0(9ba0a861831c3a59c3382e17221fc83a63c2240fe3d6e12f1e9ca9355915637e): error creating overlay mount to /var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/diff,workdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/work": invalid argument

$ kubectl describe -n kube-system pod/kube-proxy-dplzb

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               7m33s                  default-scheduler  Successfully assigned kube-system/kube-proxy-dplzb to worker
  Warning  FailedCreatePodSandBox  7m30s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0 in pod sandbox k8s_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0(9975181c1522f44afbb0b34af0f6bbadfc90e3cefff42cda4cc434fbb93c235b): error creating overlay mount to /var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/diff,workdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/work": invalid argument

serewicz

Hello,

Could you tell me about your environment. I am not seeing this error, but would like to understand where it could be found. Are you using VirtualBox or a cloud provider? What version of the OS are you using, etc?

Regards,

kingphil

TL;DR: these are Ubuntu 18.04 VMs, running on a KVM hypervisor

Sure. These are Ubuntu 18.04 VMs, based off of upstream provided base cloud image. They are running on a Ubuntu 20.04 hypervisor.

https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.img

Here is the virt-install script (for the 'worker' VM; the 'cp' script is the same except for the --name argument):

export OS_QCOW2=/var/lib/libvirt/images/worker.qcow2
scp /home/ubuntu/_images/ubuntu-18.04-server-cloudimg-amd64.img root@colo4:$OS_QCOW2
ssh root@colo4 qemu-img resize $OS_QCOW2 20G

virt-install \
  --connect qemu+ssh://root@colo4/system \
  --import \
  --virt-type kvm \
  --name worker \
  --ram 8192 \
  --cpu host-model \
  --vcpus 2 \
  --network bridge=br0,model=virtio \
  --disk path=$OS_QCOW2,format=qcow2 \
  --os-type=linux \
  --os-variant ubuntu18.04 \
  --autostart \
  --autoconsole text \
  --cloud-init user-data=./user-data,meta-data=./meta-data

Post virt-install, this is running the following scripts from the 10-13 tarball:

• LFD259/SOLUTIONS/s_02/k8scp.sh
• LFD259/SOLUTIONS/s_02/k8sSecond.sh

The problem arises after the 'kubeadm join ...' command. I am reproducing this right now. The absence of the 'sed' line results in:

$ kubectl get pod -n kube-system
NAME                                       READY   STATUS              RESTARTS   AGE
calico-kube-controllers-75f8f6cc59-wj5rf   1/1     Running             0          12m
calico-node-4knm9                          1/1     Running             0          12m
calico-node-5w7cd                          0/1     Init:0/3            0          9m27s
coredns-78fcd69978-26rtv                   1/1     Running             0          12m
coredns-78fcd69978-84hhb                   1/1     Running             0          12m
etcd-cp                                    1/1     Running             0          12m
kube-apiserver-cp                          1/1     Running             0          12m
kube-controller-manager-cp                 1/1     Running             0          12m
kube-proxy-62qw2                           0/1     ContainerCreating   0          9m27s
kube-proxy-zbq87                           1/1     Running             0          12m
kube-scheduler-cp                          1/1     Running             0          12m

kingphil

Starting from scratch again, and restoring the 3 lines as shown in the diff in my original response, the problem goes away; the two "stuck" pods are created after a minute or so.

k get pod -n kube-system -w

calico-node-d99gh 1/1 Running 0 66s

Let me know if more is needed from me on this, I'm happy to help.

supirman

@kingphil said:
The line for /etc/containers/storage.conf needs to be restored. The absence of those lines ends up with the "worker" node failing to start up critical containers.

diff --git a/SOLUTIONS/s_02/k8sSecond.sh b/SOLUTIONS/s_02/k8sSecond.sh
index 44060c2..adc691f 100755
--- a/SOLUTIONS/s_02/k8sSecond.sh
+++ b/SOLUTIONS/s_02/k8sSecond.sh
@@ -49,6 +49,9 @@ sudo apt-get update
 # Install cri-o
 sudo apt-get install -y cri-o cri-o-runc podman buildah
 
+# A bug fix to get past a cri-o update
+sudo sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
+
 sleep 3
 
 sudo systemctl daemon-reload

kube-system pod/calico-node-djwk2 0/1 Init:0/3 0 4m23s
kube-system pod/kube-proxy-dplzb 0/1 ContainerCreating 0 4m23s

$ kubectl describe -n kube-system pod/calico-node-djwk2

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               5m59s                default-scheduler  Successfully assigned kube-system/calico-node-djwk2 to worker
  Warning  FailedCreatePodSandBox  5m56s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0 in pod sandbox k8s_calico-node-djwk2_kube-system_7e9aa728-8fe9-4a29-bedf-40988dff8741_0(9ba0a861831c3a59c3382e17221fc83a63c2240fe3d6e12f1e9ca9355915637e): error creating overlay mount to /var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/diff,workdir=/var/lib/containers/storage/overlay/0ddf29c0abf8761cf2aa6c28c1f4022e3064636ecb9ea95c7b04bbe8f0bcd2e0/work": invalid argument

$ kubectl describe -n kube-system pod/kube-proxy-dplzb

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               7m33s                  default-scheduler  Successfully assigned kube-system/kube-proxy-dplzb to worker
  Warning  FailedCreatePodSandBox  7m30s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0 in pod sandbox k8s_kube-proxy-dplzb_kube-system_7fc66509-09a3-46e2-b118-39a7fe9c3139_0(9975181c1522f44afbb0b34af0f6bbadfc90e3cefff42cda4cc434fbb93c235b): error creating overlay mount to /var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LAOCFIHBLQ5ZHEGKPTYC6N36OP,upperdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/diff,workdir=/var/lib/containers/storage/overlay/85b1b5186e0d6506db20d5a31b9ddbdea0627ff19e0c6a411d30f30a2d82b3de/work": invalid argument

I agree with this. I am facing the same issue.
I am using Virtualbox on Windows 11 Host with Ubuntu 18.04 Guest.

tiffanyfay

How come the course still uses the "master" term?

coop

The Linux Foundation is fully on board with the "inclusive naming initiative" (https://inclusivenaming.org/ and has systematically implemented its guidelines every where it can in courses. Unfortunately, there are quite a few upstream projects that still use "master" and it is impossible to eliminate its use completely until they do, at which point the courses automatically will not be using these terms. It takes time, and even though CNCF is an initiator of the inclusive naming project, quite a few CNCF projects still use "master".

bharathsridhar8

Faced this issue in Lab 2.3 I am running the nodes on GCP, ubuntu 18.04LTS.

vardanm

@kingphil said:
The line for /etc/containers/storage.conf needs to be restored. The absence of those lines ends up with the "worker" node failing to start up critical containers.

diff --git a/SOLUTIONS/s_02/k8sSecond.sh b/SOLUTIONS/s_02/k8sSecond.sh
index 44060c2..adc691f 100755
--- a/SOLUTIONS/s_02/k8sSecond.sh
+++ b/SOLUTIONS/s_02/k8sSecond.sh
@@ -49,6 +49,9 @@ sudo apt-get update
 # Install cri-o
 sudo apt-get install -y cri-o cri-o-runc podman buildah
 
+# A bug fix to get past a cri-o update
+sudo sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
+
 sleep 3
 
 sudo systemctl daemon-reload

kube-system pod/calico-node-djwk2 0/1 Init:0/3 0 4m23s
kube-system pod/kube-proxy-dplzb 0/1 ContainerCreating 0 4m23s

$ kubectl describe -n kube-system pod/calico-node-djwk2

```

Confirming this post! Guys, please fix it in the "k8sSecond.sh", I have wasted more than an hour to figure out why the steps are not working on VMware fusion.

chrispokorni

Hi @vardanm,

The step works successfully in some tested cloud settings - AWS and GCP, while it seems to misbehave during local installations.

Regards,
-Chris

t.white.ucf

I believe I am experiencing this same issue still. @serewicz

chrispokorni

Hi @t.white.ucf,

Are you following the latest release of the lab guide? The "metacopy" reference is no longer part of any of the installation scripts. The course content release referenced by this discussion is close to two years old.

Regards,
-Chris