Welcome to the Linux Foundation Forum!

LFD259 - course updated to v1.25.1 (11.8.2022)

Posts: 2,295
in LFD259 Class Forum

Hello,

The course has been updated to Kubernetes v1.25.1. The majority of the updates are in labs, along with some typo corrections and small changes in lectures.

To ensure you have access to the latest updates, please clear your cache.

Regards,
Flavia

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 13

    Thank you! Can we be provided with a changelog?

  • Posts: 2,295

    Hi @lf1d

    We recommend that you redo the labs using v1.25.1 version.

    Regards,
    Flavia

  • Posts: 13

    There are incorrect steps for lab 3.2. On page 21, step 4, that is not what the /etc/containers/registries.conf file looks like. Additionally, for step 5, the crio service is not installed. The k8scp.sh script from Lab 2.2 installs containerd.

    Let me know if I should create a separate post to address these issues

  • Posts: 2

    At lab 3.2 On Page 24, step 16 when trying to create the deployment using the got the error ErrImagePull

    student@cp:~$ kubectl get pods
    NAME READY STATUS RESTARTS AGE
    nginx-8686455c8b-4kz85 1/1 Running 2 (96m ago) 29h
    registry-7c4c89bb5c-pvt2k 1/1 Running 2 (96m ago) 29h
    try1-67d8f68fbc-6rlcc 0/1 ImagePullBackOff 0 8s
    try1-67d8f68fbc-fxqfj 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-mkqx2 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-nv257 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-rpbwj 0/1 ImagePullBackOff 0 6s
    try1-67d8f68fbc-szzlw 0/1 ImagePullBackOff 0 6s

  • Posts: 2,451

    Hi @dsmartins,

    Did the earlier curl command curl $repo/v2/_catalog work successfully from both nodes, and produced the expected output?

    Are the registry.conf and config.toml files on both nodes populated with the correct information (service IP, port, plugin entry, ...)?

    Regards,
    -Chris

  • Posts: 13

    @chrispokorni, as I mentioned above, the steps for lab 3.2 are incomplete and inaccurate. For example, the instructions for updating config.toml have been completely removed.

  • Posts: 13

    Lab 3.2
    On page 24, step 17, it says "On the second node" instead of "On the worker node".

    Lab 3.3
    Page 27, step 9: "This time we will add a Sidecar container to the pod running a simple application which will respond to port 8080." -- I noticed the word simple is wrapped in codeblocks, but it should not be in this context.

    Page 28, step 13: "In the next minute or so the Sidecar container in each pod, which was not running, will change status to Running." -- The Sidecar containers, aka goproxy, were running. It was the simpleapp container that was not running.

  • Posts: 2

    @chrispokorni yes the command work fine and shows in the cp and worker has the same result - same ip/port for the registry. I'm able to pull images at both using the sudo podman. As @lf1d said there isn't instructions for config.toml. What sould be the configuration needed at config.toml ?

  • Posts: 13

    @dsmartins, while in the course, go to:
    Menu -> Resources -> Files.

    You should see an option to View/Download LFD259 Lab Exercises 1.24.1 (7.25.2022). This version has the correct steps for Lab 3.2.

  • Posts: 1

    Hi I followed the latest Lab exercises but 3.2 still has problems. It tells you to verify and reboot. I verified via cURL that simpleapp is in the repo both from the cp and worker nodes. But when I try to use "kubectl create deployment try1 --image=$repo/simpleapp" I get the error ErrImagePull/ImagePullBackOff. But when I try to pull the image from podman in the cp aswell as the worker node it works... So I think something is keeping kubernetes from connecting to the local repo?

  • Posts: 31

    @taushifhab apparently this error we're all experiencing is caused by recent upstream changes :/

  • Posts: 2
    edited December 2022

    I faced the same problem, I had to engage the brain for once.

    To help others, here is how you troubleshoot and solve

    First describe the pod that was created as part of the deployment creation
    you will see
    " Failed to pull image "10.4.0.4:5000/simpleapp": rpc error: code = Unknown desc = failed to pull and unpack image "10.4.0.4:5000/simpleapp:latest": failed to resolve reference "10.4.0.4:5000/simpleapp:latest": failed to do request: Head "https://10.4.0.4:5000/v2/simpleapp/manifests/latest": http: server gave HTTP response to HTTPS client"

    Clearly your private registry cannot handle HTTPS (as expected) but your kubelet container runtime (containerd) is expecting it to be HTTPS by default

    To prove that : curl http://:5000/simpleapp should work however run the same command as https should fail

    the fix: change the containerd config to force it to accept unsecure connections
    sudo vim /etc/containerd/config.toml

    edit what you have with your own ip

    1.   [plugins."io.containerd.grpc.v1.cri".registry.configs]
    2.     [plugins."io.containerd.grpc.v1.cri".registry.configs."10.4.0.4:5000".tls]
    3.      insecure_skip_verify = true
    4.  
    5.   [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    6.    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.4.0.4:5000"]
    7.      endpoint = ["http://10.4.0.4:5000"]

    Final note check the typo on the "plugins" - it was singular on the original config file

    Edit: restart containerd to pick up the updated config : sudo systemctl restart containred

    Now you can redeploy your simpleapp deployment

    Hope it helps

  • Posts: 31
    edited December 2022

    @mehdi.brahmia thank you! For me, editing the typo from "plugin" to "plugins" on the second line of the registry.mirrors sections worked for me. There was no need for me to do more than that.

    @taushifhab @lf1d @dsmartins @dec698 @sebastianvaldez01 thought you guys might find this useful if you didn't have the issue resolved yet

  • Posts: 13

    For me, it still does not worked. Tried lot of different steps but with no win.
    Here is my config.toml

    1. disabled_plugins = []
    2. imports = ["/etc/containerd/config.toml"]
    3. oom_score = 0
    4. plugin_dir = ""
    5. required_plugins = []
    6. root = "/var/lib/containerd"
    7. state = "/run/containerd"
    8. temp = ""
    9. version = 2
    10.  
    11. [cgroup]
    12.   path = ""
    13.  
    14. [debug]
    15.   address = ""
    16.   format = ""
    17.   gid = 0
    18.   level = ""
    19.   uid = 0
    20.  
    21. [grpc]
    22.   address = "/run/containerd/containerd.sock"
    23.   gid = 0
    24.   max_recv_message_size = 16777216
    25.   max_send_message_size = 16777216
    26.   tcp_address = ""
    27.   tcp_tls_ca = ""
    28.   tcp_tls_cert = ""
    29.   tcp_tls_key = ""
    30.   uid = 0
    31.  
    32. [metrics]
    33.   address = ""
    34.   grpc_histogram = false
    35.  
    36. [plugins]
    37.  
    38.   [plugins."io.containerd.gc.v1.scheduler"]
    39.     deletion_threshold = 0
    40.     mutation_threshold = 100
    41.     pause_threshold = 0.02
    42.     schedule_delay = "0s"
    43.     startup_delay = "100ms"
    44.  
    45.   [plugins."io.containerd.grpc.v1.cri"]
    46.     device_ownership_from_security_context = false
    47.     disable_apparmor = false
    48.     disable_cgroup = false
    49.     disable_hugetlb_controller = true
    50.     disable_proc_mount = false
    51.     disable_tcp_service = true
    52.     enable_selinux = false
    53.     enable_tls_streaming = false
    54.     enable_unprivileged_icmp = false
    55.     enable_unprivileged_ports = false
    56.     ignore_image_defined_volumes = false
    57.     max_concurrent_downloads = 3
    58.     max_container_log_line_size = 16384
    59.     netns_mounts_under_state_dir = false
    60.     restrict_oom_score_adj = false
    61.     sandbox_image = "registry.k8s.io/pause:3.6"
    62.     selinux_category_range = 1024
    63.     stats_collect_period = 10
    64.     stream_idle_timeout = "4h0m0s"
    65.     stream_server_address = "127.0.0.1"
    66.     stream_server_port = "0"
    67.     systemd_cgroup = false
    68.     tolerate_missing_hugetlb_controller = true
    69.     unset_seccomp_profile = ""
    70.  
    71.     [plugins."io.containerd.grpc.v1.cri".cni]
    72.       bin_dir = "/opt/cni/bin"
    73.       conf_dir = "/etc/cni/net.d"
    74.       conf_template = ""
    75.       ip_pref = ""
    76.       max_conf_num = 1
    77.  
    78.     [plugins."io.containerd.grpc.v1.cri".containerd]
    79.       default_runtime_name = "runc"
    80.       disable_snapshot_annotations = true
    81.       discard_unpacked_layers = false
    82.       ignore_rdt_not_enabled_errors = false
    83.       no_pivot = false
    84.       snapshotter = "overlayfs"
    85.  
    86.       [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
    87.         base_runtime_spec = ""
    88.         cni_conf_dir = ""
    89.         cni_max_conf_num = 0
    90.         container_annotations = []
    91.         pod_annotations = []
    92.         privileged_without_host_devices = false
    93.         runtime_engine = ""
    94.         runtime_path = ""
    95.         runtime_root = ""
    96.         runtime_type = ""
    97.  
    98.         [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
    99.  
    100.       [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
    101.  
    102.         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    103.           base_runtime_spec = ""
    104.           cni_conf_dir = ""
    105.           cni_max_conf_num = 0
    106.           container_annotations = []
    107.           pod_annotations = []
    108.           privileged_without_host_devices = false
    109.           runtime_engine = ""
    110.           runtime_path = ""
    111.           runtime_root = ""
    112.           runtime_type = "io.containerd.runc.v2"
    113.  
    114.           [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    115.             BinaryName = ""
    116.             CriuImagePath = ""
    117.             CriuPath = ""
    118.             CriuWorkPath = ""
    119.             IoGid = 0
    120.             IoUid = 0
    121.             NoNewKeyring = false
    122.             NoPivotRoot = false
    123.             Root = ""
    124.             ShimCgroup = ""
    125.             SystemdCgroup = false
    126.  
    127.       [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
    128.         base_runtime_spec = ""
    129.         cni_conf_dir = ""
    130.         cni_max_conf_num = 0
    131.         container_annotations = []
    132.         pod_annotations = []
    133.         privileged_without_host_devices = false
    134.         runtime_engine = ""
    135.         runtime_path = ""
    136.         runtime_root = ""
    137.         runtime_type = ""
    138.  
    139.         [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
    140.  
    141.     [plugins."io.containerd.grpc.v1.cri".image_decryption]
    142.       key_model = "node"
    143.  
    144.     [plugins."io.containerd.grpc.v1.cri".registry]
    145.       config_path = ""
    146.  
    147.       [plugins."io.containerd.grpc.v1.cri".registry.auths]
    148.  
    149.       [plugins."io.containerd.grpc.v1.cri".registry.configs]
    150.  
    151.         [plugins."io.containerd.grpc.v1.cri".registry.configs."10.111.73.196:5000"]
    152.  
    153.           [plugins."io.containerd.grpc.v1.cri".registry.configs."10.111.73.196:5000".tls]
    154.             ca_file = ""
    155.             cert_file = ""
    156.             insecure_skip_verify = true
    157.             key_file = ""
    158.  
    159.       [plugins."io.containerd.grpc.v1.cri".registry.headers]
    160.  
    161.       [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    162.  
    163.         [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.111.73.196:5000"]
    164.           endpoint = ["http://10.111.73.196:5000"]
    165.  
    166.     [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
    167.       tls_cert_file = ""
    168.       tls_key_file = ""
    169.  
    170.   [plugins."io.containerd.internal.v1.opt"]
    171.     path = "/opt/containerd"
    172.  
    173.   [plugins."io.containerd.internal.v1.restart"]
    174.     interval = "10s"
    175.  
    176.   [plugins."io.containerd.internal.v1.tracing"]
    177.     sampling_ratio = 1.0
    178.     service_name = "containerd"
    179.  
    180.   [plugins."io.containerd.metadata.v1.bolt"]
    181.     content_sharing_policy = "shared"
    182.  
    183.   [plugins."io.containerd.monitor.v1.cgroups"]
    184.     no_prometheus = false
    185.  
    186.   [plugins."io.containerd.runtime.v1.linux"]
    187.     no_shim = false
    188.     runtime = "runc"
    189.     runtime_root = ""
    190.     shim = "containerd-shim"
    191.     shim_debug = false
    192.  
    193.   [plugins."io.containerd.runtime.v2.task"]
    194.     platforms = ["linux/amd64"]
    195.     sched_core = false
    196.  
    197.   [plugins."io.containerd.service.v1.diff-service"]
    198.     default = ["walking"]
    199.  
    200.   [plugins."io.containerd.service.v1.tasks-service"]
    201.     rdt_config_file = ""
    202.  
    203.   [plugins."io.containerd.snapshotter.v1.aufs"]
    204.     root_path = ""
    205.  
    206.   [plugins."io.containerd.snapshotter.v1.btrfs"]
    207.     root_path = ""
    208.  
    209.   [plugins."io.containerd.snapshotter.v1.devmapper"]
    210.     async_remove = false
    211.     base_image_size = ""
    212.     discard_blocks = false
    213.     fs_options = ""
    214.     fs_type = ""
    215.     pool_name = ""
    216.     root_path = ""
    217.  
    218.   [plugins."io.containerd.snapshotter.v1.native"]
    219.     root_path = ""
    220.  
    221.   [plugins."io.containerd.snapshotter.v1.overlayfs"]
    222.     root_path = ""
    223.     upperdir_label = false
    224.  
    225.   [plugins."io.containerd.snapshotter.v1.zfs"]
    226.     root_path = ""
    227.  
    228.   [plugins."io.containerd.tracing.processor.v1.otlp"]
    229.     endpoint = ""
    230.     insecure = false
    231.     protocol = ""
    232.  
    233. [proxy_plugins]
    234.  
    235. [stream_processors]
    236.  
    237.   [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    238.     accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    239.     args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    240.     env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    241.     path = "ctd-decoder"
    242.     returns = "application/vnd.oci.image.layer.v1.tar"
    243.  
    244.   [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    245.     accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    246.     args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    247.     env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    248.     path = "ctd-decoder"
    249.     returns = "application/vnd.oci.image.layer.v1.tar+gzip"
    250.  
    251. [timeouts]
    252.   "io.containerd.timeout.bolt.open" = "0s"
    253.   "io.containerd.timeout.shim.cleanup" = "5s"
    254.   "io.containerd.timeout.shim.load" = "5s"
    255.   "io.containerd.timeout.shim.shutdown" = "3s"
    256.   "io.containerd.timeout.task.state" = "2s"
    257.  
    258. [ttrpc]
    259.   address = ""
    260.   gid = 0
    261.   uid = 0

    Here is my registry.conf

    1. [[registry]]
    2. location = "10.111.73.196:5000"
    3. insecure = true
    4.  
  • Posts: 13

    I also followed this and moved the config inside certs.d as explained here ->
    https://github.com/containerd/containerd/blob/main/docs/hosts.md

  • Posts: 13

    Here is the most recent version(part) of my config

    1.       [plugins."io.containerd.grpc.v1.cri".registry.configs]
    2.         [plugins."io.containerd.grpc.v1.cri".registry.configs."my-local-reg.io".tls]
    3.           insecure_skip_verify = true
    4.  
    5.       [plugins."io.containerd.grpc.v1.cri".registry.headers]
    6.  
    7.       [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    8.         [plugins."io.containerd.grpc.v1.cri".registry.mirrors."my-local-reg.io"]
    9.           endpoint = ["http://10.111.73.196:5000"]
    10.  
    11.     [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
    12.       tls_cert_file = ""
    13.       tls_key_file = ""
  • Posts: 2,451

    Hi @ashishsantikari,

    I'd recommend ensuring you have the latest release of the lab guide. This discussion thread is on the 11-08-2022 course release, while the latest course release is 11-23-2022.

    As suggested above, step 4 of lab exercise 3.2 has a slight typo, a missing "s".

    See the recommended correction in another discussion:

    https://forum.linuxfoundation.org/discussion/862689/3-2-9-failed-to-push-tagged-image-to-local-registry-server-gave-http-response-to-https-client#latest

    Regards,
    -Chris

  • Posts: 13

    @chrispokorni said:
    Hi @ashishsantikari,

    I'd recommend ensuring you have the latest release of the lab guide. This discussion thread is on the 11-08-2022 course release, while the latest course release is 11-23-2022.

    As suggested above, step 4 of lab exercise 3.2 has a slight typo, a missing "s".

    See the recommended correction in another discussion:

    https://forum.linuxfoundation.org/discussion/862689/3-2-9-failed-to-push-tagged-image-to-local-registry-server-gave-http-response-to-https-client#latest

    Regards,
    -Chris

    Please see the post above which has the configuration changes. Looks like I followed the docs and guide properly.

  • Posts: 2,451

    Hi @ashishsantikari,

    Any intermediary changes may impact the expected results.

    The missing "s" typo from config.toml in step 4, and the registry.conf file content are most often causing issues with this lab exercise. The equivalent tasks from step 12 are as important. The two VM reboots from step 13 may resolve any additional dependency issues.

    If none of this works, you may be facing other issues at VM/node level, or cloud VPC/hypervisor networking level.
    What type of infrastructure are you using for these labs? What is your OS? What firewall rule(s) do you have in place to manage traffic to/from your VMs?

    Regards,
    -Chris

  • Posts: 13

    @chrispokorni I am using GCP. OS is Ubuntu 20.04 on both nodes. No firewall enabled on both the nodes. I have equal config setup on both the nodes and I have rebooted the VM's multiple times.

  • Posts: 2,451

    Hi @ashishsantikari,

    For the GCE VM instances did you follow the video from the introductory chapter?

    Regards,
    -Chris

  • Posts: 13

    @chrispokorni said:
    Hi @ashishsantikari,

    For the GCE VM instances did you follow the video from the introductory chapter?

    Regards,
    -Chris

    @chrispokorni yes I followed the exact instructions.

  • Posts: 2,451

    Hi @ashishsantikari,

    I just went through two new clusters provisioned per the video instructions and followed all lab guide instructions to bootstrap Kubernetes and configure the local repository per the latest course release 11-23-2022, and outside of the "plugins" typo I did not have to correct anything else for both labs 2 and 3 to work successfully.

    Is there anything in the lab that you may be doing differently than what is suggested by the lab guide, or maybe the video?

    Regards,
    -Chris

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Quick Links

Categories

Upcoming Training