Lab 3.2 Reboot error, cannot to kube-apiserver, retried from scratch and getting same error

Hi @kenneth.kon001,

With Ubuntu 24.04 being the recommended guest OS for the lab environment (per Exercise 2.1: Overview and Preliminaries), I don't see the benefit of using an earlier release such as 20.04.

According to Step 11 of Lab 3.2, the repository service may take a few minutes to become available after the reboot, as all Kubernetes resources need to initialize and the repository components needs to launch as well. If not available after a few minutes, Step 11 also recommends checking .bashrc for the export repo=10.97.40.62:5000 instruction. If the instruction is there, you can simply source .bashrc to set and export the environment variable:

source $HOME/.bashrc

If this does not resolve your issue, check /etc/containerd/config.toml for the following entries:

[plugins."io.containerd.cri.v1.images".registry.mirrors."*"]
endpoint = ["http://10.97.40.62:5000"]

If your entries are different or missing, please update accordingly, then restart the containerd service:

sudo systemctl daemon-reload
sudo systemctl restart containerd

If attempting to provision a new lab environment and bootstrap a new Kubernetes cluster, I recommend downloading once again the SOLUTIONS tarball to ensure it includes the latest relevant configuration for the containerd runtime.

Regards,
-Chris

Hi @kenneth.kon001 I had the exact same issue today and I'm still having issues. After the reboot in Lab 3.2, Kubernetes wasn't working anymore. The issue was that the kubelet service wasn't able to start.
Running sudo journalctl -u kubelet -f showed an error like this:

validate CRI v1 runtime API for endpoint "unix:///var/run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService

After hours of debugging I found out that current version 2.2.0 of containerd potentially has a bug:

• https://github.com/containerd/containerd/issues/12636
• https://github.com/containerd/containerd/issues/12612

So I removed containerd, installed version 2.1.5 and re-created the containerd config.

I ran something like the following:

sudo systemctl stop containerd
sudo apt remove -y containerd.io containerd
wget https://github.com/containerd/containerd/releases/download/v2.1.5/containerd-2.1.5-linux-amd64.tar.gz
sudo tar -C /usr/local -xzf containerd-2.1.5-linux-amd64.tar.gz

sudo wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service -O /etc/systemd/system/containerd.service

sudo systemctl daemon-reload
sudo systemctl enable containerd
sudo systemctl start containerd

sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

And then I manually added the following in the correct place nested under [plugins.'io.containerd.cri.v1.images'.registry] in the /etc/containerd/config.toml.

[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = ''

  [plugins.'io.containerd.cri.v1.images'.registry.mirrors.'*']
    endpoint = ['http://10.97.40.62:5000']

And then restart containerd:
sudo systemctl restart containerd

After doing this for both control plane and worker it first seemed to have worked but now when I check the logs for kubelet on worker or control-plane I see this:

Dec 16 14:49:25 control-plane.us-central1-a.c.core-trees-481307-n8.internal kubelet[4875]: E1216 14:49:25.079840    4875 reconstruct.go:189] "Failed to get Node status to reconstruct device paths" err="nodes \"control-plane.us-central1-a.c.core-trees-481307-n8.internal\" is forbidden: User \"system:node:control-plane\" cannot get resource \"nodes\" in API group \"\" at the cluster scope: node 'control-plane' cannot read 'control-plane.us-central1-a.c.core-trees-481307-n8.internal', only its own Node object"

And both nodes show as "NotReady"

kubectl get nodes
NAME            STATUS     ROLES           AGE   VERSION
control-plane   NotReady   control-plane   25h   v1.34.1
worker          NotReady   <none>          25h   v1.34.1

I tried restarting everything multiple times without any success.

Hi @pcarr3,

As I investigate this behavior, please share details about your setup. What type of infrastructure hosts your lab VMs? What cloud or local hypervisor, what are the sizes of your VMs (CPU, RAM, disk), what guest OS, inbound firewall rule allowing all traffic?

Regards,
-Chris

Hi @kenneth.kon001, @pcarr3, @haeniya,

Thank you for all the details you provided from your systems.
Also appreciating your patience while I investigated the issue.

Since containerd v2 was released several configuration options have changed. We caught some, fixed them in the lab guide and SOLUTIONS scripts, but now additional changes are required. Until our course maintainer will merge these changes into the course and publish a new release, please use the solution below to fix the containerd behavior after the reboot.

After the completion of the local-repo-setup.sh script, perform the following config steps.

1. Backup your current containerd config file:

sudo cp /etc/containerd/config.toml /etc/containerd/config.toml-backup

2. Reset the containerd config file:

sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -e 's/SystemdCgroup = false/SystemdCgroup = true/g' -i /etc/containerd/config.toml

3. Edit the new containerd config file. Use the LINE number references to locate existing entries. LINE 54 needs to be modified. LINEs 56 57 58 are new entries. Ensure proper indentation for the new entries. For reference LINE 38 is not indented, LINE 39 indented 2 spaces, LINE 53 indented 4 spaces, LINE 54 indented 6 spaces, LINE 56 indented 6 spaces, LINE 57 indented 8 spaces, LINE 58 indented 10 spaces.

sudo vim /etc/containerd/config.toml

<file content truncated>
...
[plugins]                                         # LINE 38
  [plugins.'io.containerd.cri.v1.images']         # LINE 39
    ...

    [plugins.'io.containerd.cri.v1.images'.registry]    # LINE 53
      config_path = ''                                  # LINE 54 - REMOVE VALUE OF CONFIG PATH

      [plugins.'io.containerd.cri.v1.images'.registry.mirrors]           # LINE 56 - ADD THIS LINE
        [plugins.'io.containerd.cri.v1.images'.registry.mirrors.'*']     # LINE 57 - ADD THIS LINE
          endpoint = ['http://10.97.40.62:5000']                         # LINE 58 - ADD THIS LINE

    [plugins.'io.containerd.cri.v1.images'.image_decryption]
      key_model = 'node'
...

4. Reload daemon and restart containerd service:

sudo systemctl daemon-reload
sudo systemctl restart containerd

These steps need to be executed on both nodes. Start with the cp node, then complete them on the worker node. From here on, the upcoming steps from the lab guide should work, even after a new reboot.

Otherwise, the recommendation to closely follow the lab guide, still stands.

Regards,
-Chris