Welcome to the Linux Foundation Forum!

LFD259 - course updated to v1.25.1 (11.8.2022)

Hello,

The course has been updated to Kubernetes v1.25.1. The majority of the updates are in labs, along with some typo corrections and small changes in lectures.

To ensure you have access to the latest updates, please clear your cache.

Regards,
Flavia

Comments

  • lf1d
    lf1d Posts: 13

    Thank you! Can we be provided with a changelog?

  • fcioanca
    fcioanca Posts: 1,886

    Hi @lf1d

    We recommend that you redo the labs using v1.25.1 version.

    Regards,
    Flavia

  • lf1d
    lf1d Posts: 13

    There are incorrect steps for lab 3.2. On page 21, step 4, that is not what the /etc/containers/registries.conf file looks like. Additionally, for step 5, the crio service is not installed. The k8scp.sh script from Lab 2.2 installs containerd.

    Let me know if I should create a separate post to address these issues

  • At lab 3.2 On Page 24, step 16 when trying to create the deployment using the got the error ErrImagePull

    student@cp:~$ kubectl get pods
    NAME READY STATUS RESTARTS AGE
    nginx-8686455c8b-4kz85 1/1 Running 2 (96m ago) 29h
    registry-7c4c89bb5c-pvt2k 1/1 Running 2 (96m ago) 29h
    try1-67d8f68fbc-6rlcc 0/1 ImagePullBackOff 0 8s
    try1-67d8f68fbc-fxqfj 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-mkqx2 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-nv257 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-rpbwj 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-szzlw 0/1 ImagePullBackOff 0 6s

  • Hi @dsmartins,

    Did the earlier curl command curl $repo/v2/_catalog work successfully from both nodes, and produced the expected output?

    Are the registry.conf and config.toml files on both nodes populated with the correct information (service IP, port, plugin entry, ...)?

    Regards,
    -Chris

  • lf1d
    lf1d Posts: 13

    @chrispokorni, as I mentioned above, the steps for lab 3.2 are incomplete and inaccurate. For example, the instructions for updating config.toml have been completely removed.

  • lf1d
    lf1d Posts: 13

    Lab 3.2
    On page 24, step 17, it says "On the second node" instead of "On the worker node".

    Lab 3.3
    Page 27, step 9: "This time we will add a Sidecar container to the pod running a simple application which will respond to port 8080." -- I noticed the word simple is wrapped in codeblocks, but it should not be in this context.

    Page 28, step 13: "In the next minute or so the Sidecar container in each pod, which was not running, will change status to Running." -- The Sidecar containers, aka goproxy, were running. It was the simpleapp container that was not running.

  • @chrispokorni yes the command work fine and shows in the cp and worker has the same result - same ip/port for the registry. I'm able to pull images at both using the sudo podman. As @lf1d said there isn't instructions for config.toml. What sould be the configuration needed at config.toml ?

  • lf1d
    lf1d Posts: 13

    @dsmartins, while in the course, go to:
    Menu -> Resources -> Files.

    You should see an option to View/Download LFD259 Lab Exercises 1.24.1 (7.25.2022). This version has the correct steps for Lab 3.2.

  • Hi I followed the latest Lab exercises but 3.2 still has problems. It tells you to verify and reboot. I verified via cURL that simpleapp is in the repo both from the cp and worker nodes. But when I try to use "kubectl create deployment try1 --image=$repo/simpleapp" I get the error ErrImagePull/ImagePullBackOff. But when I try to pull the image from podman in the cp aswell as the worker node it works... So I think something is keeping kubernetes from connecting to the local repo?

  • @taushifhab apparently this error we're all experiencing is caused by recent upstream changes :/

  • mehdi.brahmia
    mehdi.brahmia Posts: 2
    edited December 2022

    I faced the same problem, I had to engage the brain for once.

    To help others, here is how you troubleshoot and solve

    First describe the pod that was created as part of the deployment creation
    you will see
    " Failed to pull image "10.4.0.4:5000/simpleapp": rpc error: code = Unknown desc = failed to pull and unpack image "10.4.0.4:5000/simpleapp:latest": failed to resolve reference "10.4.0.4:5000/simpleapp:latest": failed to do request: Head "https://10.4.0.4:5000/v2/simpleapp/manifests/latest": http: server gave HTTP response to HTTPS client"

    Clearly your private registry cannot handle HTTPS (as expected) but your kubelet container runtime (containerd) is expecting it to be HTTPS by default

    To prove that : curl http://:5000/simpleapp should work however run the same command as https should fail

    the fix: change the containerd config to force it to accept unsecure connections
    sudo vim /etc/containerd/config.toml

    edit what you have with your own ip

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."10.4.0.4:5000".tls]
         insecure_skip_verify = true
    
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
       [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.4.0.4:5000"]
         endpoint = ["http://10.4.0.4:5000"]
    

    Final note check the typo on the "plugins" - it was singular on the original config file

    Edit: restart containerd to pick up the updated config : sudo systemctl restart containred

    Now you can redeploy your simpleapp deployment

    Hope it helps

  • tstaffordsmith
    tstaffordsmith Posts: 31
    edited December 2022

    @mehdi.brahmia thank you! For me, editing the typo from "plugin" to "plugins" on the second line of the registry.mirrors sections worked for me. There was no need for me to do more than that.

    @taushifhab @lf1d @dsmartins @dec698 @sebastianvaldez01 thought you guys might find this useful if you didn't have the issue resolved yet

  • For me, it still does not worked. Tried lot of different steps but with no win.
    Here is my config.toml

    disabled_plugins = []
    imports = ["/etc/containerd/config.toml"]
    oom_score = 0
    plugin_dir = ""
    required_plugins = []
    root = "/var/lib/containerd"
    state = "/run/containerd"
    temp = ""
    version = 2
    
    [cgroup]
      path = ""
    
    [debug]
      address = ""
      format = ""
      gid = 0
      level = ""
      uid = 0
    
    [grpc]
      address = "/run/containerd/containerd.sock"
      gid = 0
      max_recv_message_size = 16777216
      max_send_message_size = 16777216
      tcp_address = ""
      tcp_tls_ca = ""
      tcp_tls_cert = ""
      tcp_tls_key = ""
      uid = 0
    
    [metrics]
      address = ""
      grpc_histogram = false
    
    [plugins]
    
      [plugins."io.containerd.gc.v1.scheduler"]
        deletion_threshold = 0
        mutation_threshold = 100
        pause_threshold = 0.02
        schedule_delay = "0s"
        startup_delay = "100ms"
    
      [plugins."io.containerd.grpc.v1.cri"]
        device_ownership_from_security_context = false
        disable_apparmor = false
        disable_cgroup = false
        disable_hugetlb_controller = true
        disable_proc_mount = false
        disable_tcp_service = true
        enable_selinux = false
        enable_tls_streaming = false
        enable_unprivileged_icmp = false
        enable_unprivileged_ports = false
        ignore_image_defined_volumes = false
        max_concurrent_downloads = 3
        max_container_log_line_size = 16384
        netns_mounts_under_state_dir = false
        restrict_oom_score_adj = false
        sandbox_image = "registry.k8s.io/pause:3.6"
        selinux_category_range = 1024
        stats_collect_period = 10
        stream_idle_timeout = "4h0m0s"
        stream_server_address = "127.0.0.1"
        stream_server_port = "0"
        systemd_cgroup = false
        tolerate_missing_hugetlb_controller = true
        unset_seccomp_profile = ""
    
        [plugins."io.containerd.grpc.v1.cri".cni]
          bin_dir = "/opt/cni/bin"
          conf_dir = "/etc/cni/net.d"
          conf_template = ""
          ip_pref = ""
          max_conf_num = 1
    
        [plugins."io.containerd.grpc.v1.cri".containerd]
          default_runtime_name = "runc"
          disable_snapshot_annotations = true
          discard_unpacked_layers = false
          ignore_rdt_not_enabled_errors = false
          no_pivot = false
          snapshotter = "overlayfs"
    
          [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
            base_runtime_spec = ""
            cni_conf_dir = ""
            cni_max_conf_num = 0
            container_annotations = []
            pod_annotations = []
            privileged_without_host_devices = false
            runtime_engine = ""
            runtime_path = ""
            runtime_root = ""
            runtime_type = ""
    
            [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
    
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
    
            [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
              base_runtime_spec = ""
              cni_conf_dir = ""
              cni_max_conf_num = 0
              container_annotations = []
              pod_annotations = []
              privileged_without_host_devices = false
              runtime_engine = ""
              runtime_path = ""
              runtime_root = ""
              runtime_type = "io.containerd.runc.v2"
    
              [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
                BinaryName = ""
                CriuImagePath = ""
                CriuPath = ""
                CriuWorkPath = ""
                IoGid = 0
                IoUid = 0
                NoNewKeyring = false
                NoPivotRoot = false
                Root = ""
                ShimCgroup = ""
                SystemdCgroup = false
    
          [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
            base_runtime_spec = ""
            cni_conf_dir = ""
            cni_max_conf_num = 0
            container_annotations = []
            pod_annotations = []
            privileged_without_host_devices = false
            runtime_engine = ""
            runtime_path = ""
            runtime_root = ""
            runtime_type = ""
    
            [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
    
        [plugins."io.containerd.grpc.v1.cri".image_decryption]
          key_model = "node"
    
        [plugins."io.containerd.grpc.v1.cri".registry]
          config_path = ""
    
          [plugins."io.containerd.grpc.v1.cri".registry.auths]
    
          [plugins."io.containerd.grpc.v1.cri".registry.configs]
    
            [plugins."io.containerd.grpc.v1.cri".registry.configs."10.111.73.196:5000"]
    
              [plugins."io.containerd.grpc.v1.cri".registry.configs."10.111.73.196:5000".tls]
                ca_file = ""
                cert_file = ""
                insecure_skip_verify = true
                key_file = ""
    
          [plugins."io.containerd.grpc.v1.cri".registry.headers]
    
          [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    
            [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.111.73.196:5000"]
              endpoint = ["http://10.111.73.196:5000"]
    
        [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
          tls_cert_file = ""
          tls_key_file = ""
    
      [plugins."io.containerd.internal.v1.opt"]
        path = "/opt/containerd"
    
      [plugins."io.containerd.internal.v1.restart"]
        interval = "10s"
    
      [plugins."io.containerd.internal.v1.tracing"]
        sampling_ratio = 1.0
        service_name = "containerd"
    
      [plugins."io.containerd.metadata.v1.bolt"]
        content_sharing_policy = "shared"
    
      [plugins."io.containerd.monitor.v1.cgroups"]
        no_prometheus = false
    
      [plugins."io.containerd.runtime.v1.linux"]
        no_shim = false
        runtime = "runc"
        runtime_root = ""
        shim = "containerd-shim"
        shim_debug = false
    
      [plugins."io.containerd.runtime.v2.task"]
        platforms = ["linux/amd64"]
        sched_core = false
    
      [plugins."io.containerd.service.v1.diff-service"]
        default = ["walking"]
    
      [plugins."io.containerd.service.v1.tasks-service"]
        rdt_config_file = ""
    
      [plugins."io.containerd.snapshotter.v1.aufs"]
        root_path = ""
    
      [plugins."io.containerd.snapshotter.v1.btrfs"]
        root_path = ""
    
      [plugins."io.containerd.snapshotter.v1.devmapper"]
        async_remove = false
        base_image_size = ""
        discard_blocks = false
        fs_options = ""
        fs_type = ""
        pool_name = ""
        root_path = ""
    
      [plugins."io.containerd.snapshotter.v1.native"]
        root_path = ""
    
      [plugins."io.containerd.snapshotter.v1.overlayfs"]
        root_path = ""
        upperdir_label = false
    
      [plugins."io.containerd.snapshotter.v1.zfs"]
        root_path = ""
    
      [plugins."io.containerd.tracing.processor.v1.otlp"]
        endpoint = ""
        insecure = false
        protocol = ""
    
    [proxy_plugins]
    
    [stream_processors]
    
      [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
        accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
        args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
        env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
        path = "ctd-decoder"
        returns = "application/vnd.oci.image.layer.v1.tar"
    
      [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
        accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
        args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
        env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
        path = "ctd-decoder"
        returns = "application/vnd.oci.image.layer.v1.tar+gzip"
    
    [timeouts]
      "io.containerd.timeout.bolt.open" = "0s"
      "io.containerd.timeout.shim.cleanup" = "5s"
      "io.containerd.timeout.shim.load" = "5s"
      "io.containerd.timeout.shim.shutdown" = "3s"
      "io.containerd.timeout.task.state" = "2s"
    
    [ttrpc]
      address = ""
      gid = 0
      uid = 0
    

    Here is my registry.conf

    [[registry]]
    location = "10.111.73.196:5000"
    insecure = true
    
    
  • I also followed this and moved the config inside certs.d as explained here ->
    https://github.com/containerd/containerd/blob/main/docs/hosts.md

  • Here is the most recent version(part) of my config

          [plugins."io.containerd.grpc.v1.cri".registry.configs]
            [plugins."io.containerd.grpc.v1.cri".registry.configs."my-local-reg.io".tls]
              insecure_skip_verify = true
    
          [plugins."io.containerd.grpc.v1.cri".registry.headers]
    
          [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
            [plugins."io.containerd.grpc.v1.cri".registry.mirrors."my-local-reg.io"]
              endpoint = ["http://10.111.73.196:5000"]
    
        [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
          tls_cert_file = ""
          tls_key_file = ""
    
  • Hi @ashishsantikari,

    I'd recommend ensuring you have the latest release of the lab guide. This discussion thread is on the 11-08-2022 course release, while the latest course release is 11-23-2022.

    As suggested above, step 4 of lab exercise 3.2 has a slight typo, a missing "s".

    See the recommended correction in another discussion:

    https://forum.linuxfoundation.org/discussion/862689/3-2-9-failed-to-push-tagged-image-to-local-registry-server-gave-http-response-to-https-client#latest

    Regards,
    -Chris

  • @chrispokorni said:
    Hi @ashishsantikari,

    I'd recommend ensuring you have the latest release of the lab guide. This discussion thread is on the 11-08-2022 course release, while the latest course release is 11-23-2022.

    As suggested above, step 4 of lab exercise 3.2 has a slight typo, a missing "s".

    See the recommended correction in another discussion:

    https://forum.linuxfoundation.org/discussion/862689/3-2-9-failed-to-push-tagged-image-to-local-registry-server-gave-http-response-to-https-client#latest

    Regards,
    -Chris

    Please see the post above which has the configuration changes. Looks like I followed the docs and guide properly.

  • Hi @ashishsantikari,

    Any intermediary changes may impact the expected results.

    The missing "s" typo from config.toml in step 4, and the registry.conf file content are most often causing issues with this lab exercise. The equivalent tasks from step 12 are as important. The two VM reboots from step 13 may resolve any additional dependency issues.

    If none of this works, you may be facing other issues at VM/node level, or cloud VPC/hypervisor networking level.
    What type of infrastructure are you using for these labs? What is your OS? What firewall rule(s) do you have in place to manage traffic to/from your VMs?

    Regards,
    -Chris

  • @chrispokorni I am using GCP. OS is Ubuntu 20.04 on both nodes. No firewall enabled on both the nodes. I have equal config setup on both the nodes and I have rebooted the VM's multiple times.

  • chrispokorni
    chrispokorni Posts: 2,153

    Hi @ashishsantikari,

    For the GCE VM instances did you follow the video from the introductory chapter?

    Regards,
    -Chris

  • @chrispokorni said:
    Hi @ashishsantikari,

    For the GCE VM instances did you follow the video from the introductory chapter?

    Regards,
    -Chris

    @chrispokorni yes I followed the exact instructions.

  • chrispokorni
    chrispokorni Posts: 2,153

    Hi @ashishsantikari,

    I just went through two new clusters provisioned per the video instructions and followed all lab guide instructions to bootstrap Kubernetes and configure the local repository per the latest course release 11-23-2022, and outside of the "plugins" typo I did not have to correct anything else for both labs 2 and 3 to work successfully.

    Is there anything in the lab that you may be doing differently than what is suggested by the lab guide, or maybe the video?

    Regards,
    -Chris

Categories

Upcoming Training