Welcome to the Linux Foundation Forum!

kubeadm init error: CRI v1 runtime API is not implemented

Hi, I'll write this post to help anyone encountering my same issue, given that I could not find the solution in past forum discussions.

I'm following the installation PDF for the class, exactly with all required versions: Ubuntu 20.04 (on a Google Cloud VM), kubeadm 1.24.1, etc.
But when executing the kubeadm init command I got the following error:

[init] Using Kubernetes version: v1.24.1
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2023-01-19T15:05:35Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

By searching the web it appears this is an issue with the old containerd provided by Ubuntu 20.
I solved it with the following steps (as root):
1. Set up the Docker repository as described in https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository
2. Remove the old containerd:apt remove containerd
3. Update repository data and install the new containerd: apt update, apt install containerd.io
4. Remove the installed default config file: rm /etc/containerd/config.toml
5. Restart containerd: systemctl restart containerd

The kubeadm init command worked fine afterwards.

Comments

  • Thanks for this! After three days of searching, you solved my problem in five minutes!

  • Thank you. It worked for me.
    On an Ubuntu 22.04.1 LTS, I have a warning [WARNING SystemVerification]: missing optional cgroups: blkio.
    Then I had to fix https://forum.linuxfoundation.org/discussion/862809/lab-3-1-kubeadm-init-error-creating-kube-proxy-service-account

  • Thank you mdevitis, Your solution worked perfectly.

  • Thank you sir, you are my HEROOOOOOOOOOO!!

  • rom1c
    rom1c Posts: 2

    Hello, I had the same problem (ubuntu focal 20.04). The first install when I started the class was OK in the end of 2022. I wanted to reinstall on a second system to test my ansible script and it ending with the CRI runtime error. Using the container.io package from Docker repo and removing the default config.toml file solve the problem.
    I'll verify on jammy (22.04) to check.

  • rom1c
    rom1c Posts: 2

    Same error on 22.04 with the containerd package from Ubuntu repository.
    There is also an error for blkio cgroups.
    Installing the containerd.io package from Docker repo and remove the default config file solve the problem.

  • farazoman
    farazoman Posts: 6
    edited January 2023

    Using 22.04 Ubuntu, and following the instructions above (i.e the first / top answer) it worked but with a slight addtion of a step. The repo for finding container.d wasn't added to my system so I had to add it as well. The instructions here helpd https://superuser.com/questions/1528265/package-containerd-io-is-not-available-when-installing-docker-on-ubuntu-19-10

    sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
    
  • I followed the the instructions to the end but my Calico Node was not ready! Whats the solution to this?

  • chrispokorni
    chrispokorni Posts: 2,155

    Hi @marviflame,

    Calico node pods are typically impacted by incorrect network configuration at the infrastructure layer. What type of infrastructure are you using to provision the hosts of the cluster, and what OS?

    Regards,
    -Chris

  • @chrispokorni I used the DigitalOcean Ubuntu 20.04 LTS Linux Server

  • @chrispokorni I had to use weave net pod network

  • chrispokorni
    chrispokorni Posts: 2,155

    Hi @marviflame,

    Alternate infrastructure than what is recommended in the lab guide may require additional configuration to accommodate the provider's network stack implementation.

    In addition, the introductory chapter of the course includes two video guides for provisioning the lab environment on AWS and GCP infrastructure, however, the networking requirements for any VPC, subnets, and firewall rules should be similar to DO.

    Regards,
    -Chris

  • Thanks a lot for this quick solution. You saved my day - I am doing self-cert and your to the point solutions worked perfectly. Keep up the good work!

  • Fresh deployment of Ubuntu 22.04 and install of containerd and I ran into this. Of all the fixes I tried this is the one that worked. Thanks for posting.

  • Hi,

    I followed the same steps:-
    Step1:

    apt remove containerd
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Package 'containerd' is not installed, so not removed
    0 upgraded, 0 newly installed, 0 to remove and 30 not upgraded.

    Step2:
    apt update
    Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
    Hit:2 http://us.archive.ubuntu.com/ubuntu focal InRelease
    Hit:3 https://download.docker.com/linux/ubuntu focal InRelease
    Get:4 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
    Get:5 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
    Get:6 http://us.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]
    Get:7 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,380 kB]
    Get:8 http://us.archive.ubuntu.com/ubuntu focal-updates/main Translation-en [409 kB]
    Get:9 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.3 kB]
    Get:10 http://us.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,607 kB]
    Get:11 http://us.archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [226 kB]
    Get:12 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,024 kB]
    Get:13 http://us.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [237 kB]
    Get:14 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.6 kB]
    Get:15 http://us.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,993 kB]
    Get:16 http://us.archive.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [12.2 kB]
    Get:17 http://us.archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [795 kB]
    Get:18 http://us.archive.ubuntu.com/ubuntu focal-security/universe Translation-en [154 kB]
    Get:19 http://us.archive.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [17.0 kB]
    Fetched 9,230 kB in 17s (546 kB/s)
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    30 packages can be upgraded. Run 'apt list --upgradable' to see them.

    Step3:
    apt install containerd.io
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    containerd.io is already the newest version (1.6.16-1).
    0 upgraded, 0 newly installed, 0 to remove and 30 not upgraded.

    Step4:
    rm /etc/containerd/config.toml
    rm: cannot remove '/etc/containerd/config.toml': No such file or directory

    Step5:
    systemctl status containerd
    ● containerd.service - containerd container runtime
    Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
    Active: active (running) since Mon 2023-02-13 23:24:56 UTC; 7s ago
    Docs: https://containerd.io
    Process: 574824 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Main PID: 574835 (containerd)
    Tasks: 23
    Memory: 22.6M
    CGroup: /system.slice/containerd.service
    └─574835 /usr/local/bin/containerd

    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.045913643Z" level=info msg="Start subscribing containerd event"
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046104300Z" level=info msg="Start recovering state"
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046041430Z" level=info msg=serving... address=/run/containerd/containe>
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046219362Z" level=info msg=serving... address=/run/containerd/containe>
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046280356Z" level=info msg="containerd successfully booted in 0.038047>
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046227827Z" level=info msg="Start event monitor"
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046488975Z" level=info msg="Start snapshots syncer"
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046531682Z" level=info msg="Start cni network conf syncer for default"
    Feb 13 23:24:56 usf-linux1 containerd[574835]: time="2023-02-13T23:24:56.046552645Z" level=info msg="Start streaming server"
    Feb 13 23:24:56 usf-linux1 systemd[1]: Started containerd container runtime.

    Final Step:
    sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket=/var/run/cri-dockerd.sock
    W0213 23:25:41.313871 574999 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
    [init] Using Kubernetes version: v1.26.1
    [preflight] Running pre-flight checks
    error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR CRI]: container runtime is not running: output: time="2023-02-13T23:25:41Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/cri-dockerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
    , error: exit status 1
    [preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
    To see the stack trace of this error execute with --v=5 or higher

    Any help appreciable.

  • Hi @manasfirst,

    I would recommend following the installation steps in the sequence presented in the latest release of the lab guide. Step 13 of Lab exercise 3.1 provides the commands that were missed earlier - that is to create the config.toml file, and restart containerd.

    In addition, in step 22 the initialization of the control plane is done with the help of a configuration manifest where the control plane endpoint is the k8scp alias set at step 21, and the Kubernetes version is set to 1.24.1. In doing so, the cluster is prepared for high availability discussed in Chapter 16, and the cluster upgrade from 1.24 to 1.25 will work as expected in a later chapter.

    Deviations from the installation steps will produce inconsistent results, and will make troubleshooting much more difficult.

    Regards,
    -Chris

  • I followed the same steps:-

    Well, they do not seem to be exactly the same steps.
    Your command output shows that you are starting from a different situation (containerd.io is already installed, the toml file is not present...), and you installed Kubernetes 1.26.1 instead of 1.24.1 as suggested by the 3.1 LAB. You are also using different arguments in the kubeadm init command. Indeed also the error is different: yours mentions the endpoint unix:///var/run/cri-dockerd.sock while in my case it was unix:///var/run/containerd/containerd.sock

    I think you should restart the installation from scratch, following the steps in the lab.

  • I had the same problem on Ubuntu 22.04 LTS. I think an easier solution is to edit the config for containerd because it has disabled the CRI plugin by default.

    Like this, or with your favorite editor: sudo nano /etc/containerd/config.toml and comment out the line that says disabled_plugins = ["cri"].

    Gotta restart containerd, but after that the CRI is enabled and there won't be any more complaints about not working when doing kubeadm init

  • @omar9000 if I remember correctly, when I experienced the issue with Ubuntu 20 I described in the initial post, the standard Ubuntu packages did not install any /etc/containerd/config.toml file.
    So the situation you describe with Ubuntu 22 seems to be different.

    By the way, commenting out the line in the config.toml file or completely deleting the file should have the same effect, because as far as I could see the disabled_plugins line was the only enabled option.

  • This was immensely helpful, thank you!

  • In my case an Ubuntu 20 and 22 have no 10-containerd-net.conflist so I used a basic one and then installed the runsc binary and made a slight change to add the runtime on the containerd config

    Create /etc/cni/net.d/10-containerd-net.conflist

    cat << EOF | sudo tee /etc/cni/net.d/10-containerd-net.conflist
    {
     "cniVersion": "1.0.0",
     "name": "containerd-net",
     "plugins": [
       {
         "type": "bridge",
         "bridge": "cni0",
         "isGateway": true,
         "ipMasq": true,
         "promiscMode": true,
         "ipam": {
           "type": "host-local",
           "ranges": [
             [{
               "subnet": "192.168.0.0/16"
             }]
           ],
           "routes": [
             { "dst": "0.0.0.0/0" },
             { "dst": "::/0" }
           ]
         }
       },
       {
         "type": "portmap",
         "capabilities": {"portMappings": true},
         "externalSetMarkChain": "KUBE-MARK-MASQ"
       }
     ]
    }
    EOF
    sudo swapoff -a
    sudo systemctl restart kubelet
    

    Install runsc

    The runsc installation is mentioned at the end of the readme file, but without it, containerd won't start. These are the instructions mentioned in gVisor documentation.

    (
      set -e
      ARCH=$(uname -m)
      URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}
      wget ${URL}/runsc ${URL}/runsc.sha512 \
        ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512
      sha512sum -c runsc.sha512 \
        -c containerd-shim-runsc-v1.sha512
      rm -f *.sha512
      chmod a+rx runsc containerd-shim-runsc-v1
      sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
    )
    

    Update containerd config to add the runsc runtime

    sudo sed -i 's/shim_debug = false$/shim_debug = true/' /etc/containerd/config.toml
    sudo sed -i '/shim_debug = true$/a \\n  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]\n    runtime_type = "io.containerd.runsc.v1"\n' /etc/containerd/config.toml
    sudo systemctl restart containerd
    

    Now create the cluster

    sudo kubeadm init  --pod-network-cidr 192.168.0.0/16 | tee /var/log/kubeinit.log
    
  • @manasfirst

    Here is what I did, note that "containerd config default | tee /etc/containerd/config.toml" creates a new config.toml, then edited it to set systemCgroup=true. Also note the crictl config set commands.

    mkdir -p /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    apt-get update && apt-get install containerd.io -y
    containerd config default | tee /etc/containerd/config.toml
    sed -e 's/SystemdCgroup = false/SystemdCgroup = true/g' -i /etc/containerd/config.toml
    crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock --set image-endpoint=unix:///run/containerd/containerd.sock
    systemctl restart containerd

  • ceromo82
    ceromo82 Posts: 1

    Worked like a charm for me! Thanks a lot bud!

  • staresky
    staresky Posts: 2
    edited June 2023

    Thank you!!
    In may case, I just remove /etc/containerd/config.toml and systemctl restart containerd and It's solved.

    Error messages

    # kubeadm init
    [init] Using Kubernetes version: v1.27.2
    [preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
    error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2023-06-04T19:05:15+09:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
    , error: exit status 1
    [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
    To see the stack trace of this error execute with --v=5 or higher
    

    Env

    OS: Almalinux 9.2 
    Kubernetes v1.27.2
    containerd containerd.io 1.6.21 3dce8eb055cbb6872793272b4f20ed16117344f8
    Docker version 24.0.2, build cb74dfc
    
  • staresky
    staresky Posts: 2
    edited June 2023

    @staresky

    In may case, I just remove /etc/containerd/config.toml and systemctl restart containerd and It's solved.

    I found a new solution
    Make comment out disabled_plugins = ["cri"] in /etc/containerd/config.toml instead of rm /etc/containerd/config.toml

  • Makrand
    Makrand Posts: 1

    Arlgiht. I am on 1.23 on prem k8s. This cluster was setup using containerd. The only reason config.toml was created here because the cluster was failing to initiate using kubeadm when I set it up.

    Here is how my toml looks

    root@stage-mast1:~# cat /etc/containerd/config.toml
    disabled_plugins = []
    imports = []
    oom_score = 0
    plugin_dir = ""
    required_plugins = []
    root = "/var/lib/containerd"
    state = "/run/containerd"
    version = 2

    Here comes the issue.
    I am trying to upgrade to 1.24. Facing same error as everyone else here.

    [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.24.15: output: time="2023-06-18T23:18:53Z" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
    , error: exit status 1
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-controller-manager:v1.24.15: output: time="2023-06-18T23:18:54Z" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
    , error: exit status 1
    

    This is HA cluster with 3 master and 8 workers. All nodes are ubuntu 20 LTS.

    Containerd version is as below. Same is CRI here

    >

    root@stage-mast1:~# ctr version
    Client:
    Version: 1.5.9-0ubuntu1~20.04.6
    Revision:
    Go version: go1.13.8

    Server:
    Version: 1.5.9-0ubuntu1~20.04.5
    Revision:
    UUID: 580cee63-8afd-4437-9447-e44b0501167a
    WARNING: version mismatch

    Should I try the solution here?

  • chrispokorni
    chrispokorni Posts: 2,155

    Hi @Makrand,

    The training material has been updated several times since Kubernetes 1.23 and 1.24. With that in mind I would recommend inspecting the official Kubernetes documentation for any specific instructions to upgrade the cluster from 1.23 to 1.24. For some earlier versions the upgrade instructions are very specific, and may not be supported by the more recent upgrade instructions.

    https://v1-24.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

    Regards,
    -Chris

  • Thankyou so much @mdevitis

  • xavierzip
    xavierzip Posts: 2
    edited February 18
    Feb 18 10:49:05 lfs-cp-1 containerd[4335]: time="2024-02-18T10:49:05.968032466Z" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
    Feb 18 10:49:05 lfs-cp-1 containerd[4335]: time="2024-02-18T10:49:05.968353080Z" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `systemd_cgroup` only works for runtime io.containerd.runtime.v1.linux"
    

    In my case, removing systemd_cgroup line from /etc/containerd/config.toml solved the issue.

Categories

Upcoming Training