LFD259 - course updated to v1.25.1 (11.8.2022)

fcioanca

Hello,

The course has been updated to Kubernetes v1.25.1. The majority of the updates are in labs, along with some typo corrections and small changes in lectures.

To ensure you have access to the latest updates, please clear your cache.

Regards,
Flavia

lf1d

Thank you! Can we be provided with a changelog?

fcioanca

Hi @lf1d

We recommend that you redo the labs using v1.25.1 version.

Regards,
Flavia

lf1d

There are incorrect steps for lab 3.2. On page 21, step 4, that is not what the /etc/containers/registries.conf file looks like. Additionally, for step 5, the crio service is not installed. The k8scp.sh script from Lab 2.2 installs containerd.

Let me know if I should create a separate post to address these issues

dsmartins

At lab 3.2 On Page 24, step 16 when trying to create the deployment using the got the error ErrImagePull

student@cp:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-8686455c8b-4kz85 1/1 Running 2 (96m ago) 29h
registry-7c4c89bb5c-pvt2k 1/1 Running 2 (96m ago) 29h
try1-67d8f68fbc-6rlcc 0/1 ImagePullBackOff 0 8s
try1-67d8f68fbc-fxqfj 0/1 ImagePullBackOff 0 6s
try1-67d8f68fbc-mkqx2 0/1 ImagePullBackOff 0 6s
try1-67d8f68fbc-nv257 0/1 ImagePullBackOff 0 6s
try1-67d8f68fbc-rpbwj 0/1 ImagePullBackOff 0 6s
try1-67d8f68fbc-szzlw 0/1 ImagePullBackOff 0 6s

chrispokorni

Hi @dsmartins,

Did the earlier curl command curl $repo/v2/_catalog work successfully from both nodes, and produced the expected output?

Are the registry.conf and config.toml files on both nodes populated with the correct information (service IP, port, plugin entry, ...)?

Regards,
-Chris

lf1d

@chrispokorni, as I mentioned above, the steps for lab 3.2 are incomplete and inaccurate. For example, the instructions for updating config.toml have been completely removed.

lf1d

Lab 3.2
On page 24, step 17, it says "On the second node" instead of "On the worker node".

Lab 3.3
Page 27, step 9: "This time we will add a Sidecar container to the pod running a simple application which will respond to port 8080." -- I noticed the word simple is wrapped in codeblocks, but it should not be in this context.

Page 28, step 13: "In the next minute or so the Sidecar container in each pod, which was not running, will change status to Running." -- The Sidecar containers, aka goproxy, were running. It was the simpleapp container that was not running.

dsmartins

@chrispokorni yes the command work fine and shows in the cp and worker has the same result - same ip/port for the registry. I'm able to pull images at both using the sudo podman. As @lf1d said there isn't instructions for config.toml. What sould be the configuration needed at config.toml ?

lf1d

@dsmartins, while in the course, go to:
Menu -> Resources -> Files.

You should see an option to View/Download LFD259 Lab Exercises 1.24.1 (7.25.2022). This version has the correct steps for Lab 3.2.

taushifhab

Hi I followed the latest Lab exercises but 3.2 still has problems. It tells you to verify and reboot. I verified via cURL that simpleapp is in the repo both from the cp and worker nodes. But when I try to use "kubectl create deployment try1 --image=$repo/simpleapp" I get the error ErrImagePull/ImagePullBackOff. But when I try to pull the image from podman in the cp aswell as the worker node it works... So I think something is keeping kubernetes from connecting to the local repo?

tstaffordsmith

@taushifhab apparently this error we're all experiencing is caused by recent upstream changes

mehdi.brahmia

I faced the same problem, I had to engage the brain for once.

To help others, here is how you troubleshoot and solve

First describe the pod that was created as part of the deployment creation
you will see
" Failed to pull image "10.4.0.4:5000/simpleapp": rpc error: code = Unknown desc = failed to pull and unpack image "10.4.0.4:5000/simpleapp:latest": failed to resolve reference "10.4.0.4:5000/simpleapp:latest": failed to do request: Head "https://10.4.0.4:5000/v2/simpleapp/manifests/latest": http: server gave HTTP response to HTTPS client"

Clearly your private registry cannot handle HTTPS (as expected) but your kubelet container runtime (containerd) is expecting it to be HTTPS by default

To prove that : curl http://:5000/simpleapp should work however run the same command as https should fail

the fix: change the containerd config to force it to accept unsecure connections
sudo vim /etc/containerd/config.toml

edit what you have with your own ip

  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."10.4.0.4:5000".tls]
     insecure_skip_verify = true

  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
   [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.4.0.4:5000"]
     endpoint = ["http://10.4.0.4:5000"]

Final note check the typo on the "plugins" - it was singular on the original config file

Edit: restart containerd to pick up the updated config : sudo systemctl restart containred

Now you can redeploy your simpleapp deployment

Hope it helps

tstaffordsmith

@mehdi.brahmia thank you! For me, editing the typo from "plugin" to "plugins" on the second line of the registry.mirrors sections worked for me. There was no need for me to do more than that.

@taushifhab @lf1d @dsmartins @dec698 @sebastianvaldez01 thought you guys might find this useful if you didn't have the issue resolved yet

ashishsantikari

For me, it still does not worked. Tried lot of different steps but with no win.
Here is my config.toml

disabled_plugins = []
imports = ["/etc/containerd/config.toml"]
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.k8s.io/pause:3.6"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

        [plugins."io.containerd.grpc.v1.cri".registry.configs."10.111.73.196:5000"]

          [plugins."io.containerd.grpc.v1.cri".registry.configs."10.111.73.196:5000".tls]
            ca_file = ""
            cert_file = ""
            insecure_skip_verify = true
            key_file = ""

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.111.73.196:5000"]
          endpoint = ["http://10.111.73.196:5000"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.internal.v1.tracing"]
    sampling_ratio = 1.0
    service_name = "containerd"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
    sched_core = false

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.service.v1.tasks-service"]
    rdt_config_file = ""

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    discard_blocks = false
    fs_options = ""
    fs_type = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""
    upperdir_label = false

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

  [plugins."io.containerd.tracing.processor.v1.otlp"]
    endpoint = ""
    insecure = false
    protocol = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.bolt.open" = "0s"
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

Here is my registry.conf

[[registry]]
location = "10.111.73.196:5000"
insecure = true

ashishsantikari

I also followed this and moved the config inside certs.d as explained here ->
https://github.com/containerd/containerd/blob/main/docs/hosts.md

ashishsantikari

Here is the most recent version(part) of my config

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."my-local-reg.io".tls]
          insecure_skip_verify = true

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."my-local-reg.io"]
          endpoint = ["http://10.111.73.196:5000"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

chrispokorni

Hi @ashishsantikari,

I'd recommend ensuring you have the latest release of the lab guide. This discussion thread is on the 11-08-2022 course release, while the latest course release is 11-23-2022.

As suggested above, step 4 of lab exercise 3.2 has a slight typo, a missing "s".

See the recommended correction in another discussion:

https://forum.linuxfoundation.org/discussion/862689/3-2-9-failed-to-push-tagged-image-to-local-registry-server-gave-http-response-to-https-client#latest

Regards,
-Chris

ashishsantikari

@chrispokorni said:
Hi @ashishsantikari,

I'd recommend ensuring you have the latest release of the lab guide. This discussion thread is on the 11-08-2022 course release, while the latest course release is 11-23-2022.

As suggested above, step 4 of lab exercise 3.2 has a slight typo, a missing "s".

See the recommended correction in another discussion:

https://forum.linuxfoundation.org/discussion/862689/3-2-9-failed-to-push-tagged-image-to-local-registry-server-gave-http-response-to-https-client#latest

Regards,
-Chris

Please see the post above which has the configuration changes. Looks like I followed the docs and guide properly.

chrispokorni

Hi @ashishsantikari,

Any intermediary changes may impact the expected results.

The missing "s" typo from config.toml in step 4, and the registry.conf file content are most often causing issues with this lab exercise. The equivalent tasks from step 12 are as important. The two VM reboots from step 13 may resolve any additional dependency issues.

If none of this works, you may be facing other issues at VM/node level, or cloud VPC/hypervisor networking level.
What type of infrastructure are you using for these labs? What is your OS? What firewall rule(s) do you have in place to manage traffic to/from your VMs?

Regards,
-Chris

ashishsantikari

@chrispokorni I am using GCP. OS is Ubuntu 20.04 on both nodes. No firewall enabled on both the nodes. I have equal config setup on both the nodes and I have rebooted the VM's multiple times.

chrispokorni

Hi @ashishsantikari,

For the GCE VM instances did you follow the video from the introductory chapter?

Regards,
-Chris

ashishsantikari

@chrispokorni said:
Hi @ashishsantikari,

For the GCE VM instances did you follow the video from the introductory chapter?

Regards,
-Chris

@chrispokorni yes I followed the exact instructions.

chrispokorni

Hi @ashishsantikari,

I just went through two new clusters provisioned per the video instructions and followed all lab guide instructions to bootstrap Kubernetes and configure the local repository per the latest course release 11-23-2022, and outside of the "plugins" typo I did not have to correct anything else for both labs 2 and 3 to work successfully.

Is there anything in the lab that you may be doing differently than what is suggested by the lab guide, or maybe the video?

Regards,
-Chris