Welcome to the Linux Foundation Forum!

Lab 4.1, exec'ing into etcd container fails due to lack of shell

josefassad
josefassad Posts: 6
edited December 10 in LFS258 Class Forum

Way at the top of the lab, there's this:
student@cp:˜$ kubectl -n kube-system exec -it etcd-<Tab> -- sh
On my end, I'm getting this complaint back:

username@control-plane:~$ kubectl exec -it --namespace kube-system etcd-cp -- sh 
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "<REDACTED_ID>": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Of course there's no such error if I try to call -- etcdctl instead of -- sh. I figure at some point that container went toolless? Is that right?

Answers

  • ucd4evr
    ucd4evr Posts: 1

    Did you try /bin/sh ?

  • josefassad
    josefassad Posts: 6

    Yes. Same error. I'll note though, I'm upgrading 1.34.2 to 1.34.3. So a bit newer than the versions specified in the lab.

    I also checked for other ultra-common tools such as ls, even echo * gives the same error.

    I mean I've worked around it, I'm just wondering if it's PEBKAC or if it's the course materials.

  • chrispokorni
    chrispokorni Posts: 2,551

    Hi @josefassad,

    I just verified on both Kubernetes v1.33.1 (etcd 3.5.21) and v1.34.1 (etcd 3.6.4) the command behaves as presented in the lab guide.
    -- sh allowed me to execute pwd, cd, and even echo * inside the etcd container of the etcd-cp pod.
    -- sh -c 'echo *' also worked.
    -- sh -c 'etcdctl' and -- etcdctl also worked.

    I recommend checking the official etcd.io site for any known issues or bugs and suggested workarounds or fixes.

    Regards,
    -Chris

  • josefassad
    josefassad Posts: 6

    Hi Chris,

    Thanks for checking! What an odd thing.

    At any rate I have an effective workaround so I'm not too bothered. I appreciate the time you took to try to reproduce.

  • josefassad
    josefassad Posts: 6
    edited December 10

    Hang on, let me be a bit more persistent. Looking at the etcd source I see from the Dockerfile the image is built on Distroless and contains little else than etcd, etcdctl, and etcdutl.

    On the one hand that explains why I can only run those in a kubectl exec. On the other hand, now I'm wondering why @chrispokorni 's image has more things in it.

    I'm still thinking PEBKAC but I'm intrigued.

  • chriswood123
    chriswood123 Posts: 1

    I have the same issue.

    @josefassad what did you do as a workaround? I'm thinking of creating an etcdctl deployment for running these commands, or maybe just installing it directly on the control plane host.

  • josefassad
    josefassad Posts: 6

    @chriswood123 my workaround is a lot simpler, I just call etcdctl directly. So if the lab says something like

    kubectl -n kube-system exec -it etcd-cp -- sh \
-c "ETCDCTL_API=3 \
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \ 
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt \ 
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key \
etcdctl endpoint health"

    I'm calling

    kubectl exec -ti -n kube-system etcd-cp -- etcdctl \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health

    I figured this out when I looked at the current etcd image's Dockerfile. They're using distroless and copying the etcd binaries over. So I just configured the etcdctl invocation using flags instead of environment variables.

    Hope this works for you!

  • chrispokorni
    chrispokorni Posts: 2,551

    Hi @josefassad and @chriswood123,

    Would you mind providing the etcd version that you are reporting about? This would help the course maintainer determine if these are isolated incidents or perhaps more generic across multiple etcd releases, requiring a permanent solution within the lab exercises.

    You can find the version with either one of the two commands below:

    kubectl -n kube-system exec etcd-cp -- etcdctl version
kubectl -n kube-system describe pod etcd-cp | grep Image:

    As I posted earlier, when I tried to reproduce the issue, all commands worked as presented in the lab guide. I attempted to reproduce the issue on two Kubernetes releases: v1.33.1 (etcd 3.5.21) and v1.34.1 (etcd 3.6.4).

    Regards,
    -Chris

  • josefassad
    josefassad Posts: 6
    edited 7:37PM

    Hi @chrispokorni . You bet. I'll actually grab version strings from kubeadm too since that's the fella that got me this version of etcd.

    josef@control-plane:~/tmp/echo-server$ kubectl exec -n kube-system etcd-cp -- etcdctl version | cowsay
 _________________________________________
< etcdctl version: 3.6.5 API version: 3.6 >
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
josef@control-plane:~/tmp/echo-server$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"34", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.34.3", GitCommit:"df11db1c0f08fab3c0baee1e5ce6efbf816af7f1", GitTreeState:"clean", BuildDate:"2025-12-09T15:05:15Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}

    So digging around a bit, here's what I think is going on. I was naughty and used k8s 1.34 when in fact Lab 3.1 said to use 1.33. Looking at the k8s changelog for 1.34, I note the following:

    Updated etcd version to v3.6.0. (#131501, @joshjms) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing]

    Looking at the corresponding etcd changelog, you can see they went distroless (Dockerfile for reference). A whole four years ago!

    It's distroless, so there's no busybox or any other kind of userland in there. Hence no bash or sh. So as far as hypotheses go, this one only has one weakness: that you tested against a 1.34 line and found a shell in the etcd container.

Categories

Upcoming Training