Welcome to the Linux Foundation Forum!

Lab 4.1, exec'ing into etcd container fails due to lack of shell

josefassad
josefassad Posts: 5
edited December 10 in LFS258 Class Forum

Way at the top of the lab, there's this:
student@cp:˜$ kubectl -n kube-system exec -it etcd-<Tab> -- sh
On my end, I'm getting this complaint back:

username@control-plane:~$ kubectl exec -it --namespace kube-system etcd-cp -- sh 
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "<REDACTED_ID>": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Of course there's no such error if I try to call -- etcdctl instead of -- sh. I figure at some point that container went toolless? Is that right?

Answers

  • ucd4evr
    ucd4evr Posts: 1

    Did you try /bin/sh ?

  • josefassad
    josefassad Posts: 5

    Yes. Same error. I'll note though, I'm upgrading 1.34.2 to 1.34.3. So a bit newer than the versions specified in the lab.

    I also checked for other ultra-common tools such as ls, even echo * gives the same error.

    I mean I've worked around it, I'm just wondering if it's PEBKAC or if it's the course materials.

  • chrispokorni
    chrispokorni Posts: 2,550

    Hi @josefassad,

    I just verified on both Kubernetes v1.33.1 (etcd 3.5.21) and v1.34.1 (etcd 3.6.4) the command behaves as presented in the lab guide.
    -- sh allowed me to execute pwd, cd, and even echo * inside the etcd container of the etcd-cp pod.
    -- sh -c 'echo *' also worked.
    -- sh -c 'etcdctl' and -- etcdctl also worked.

    I recommend checking the official etcd.io site for any known issues or bugs and suggested workarounds or fixes.

    Regards,
    -Chris

  • josefassad
    josefassad Posts: 5

    Hi Chris,

    Thanks for checking! What an odd thing.

    At any rate I have an effective workaround so I'm not too bothered. I appreciate the time you took to try to reproduce.

  • josefassad
    josefassad Posts: 5
    edited December 10

    Hang on, let me be a bit more persistent. Looking at the etcd source I see from the Dockerfile the image is built on Distroless and contains little else than etcd, etcdctl, and etcdutl.

    On the one hand that explains why I can only run those in a kubectl exec. On the other hand, now I'm wondering why @chrispokorni 's image has more things in it.

    I'm still thinking PEBKAC but I'm intrigued.

  • chriswood123
    chriswood123 Posts: 1

    I have the same issue.

    @josefassad what did you do as a workaround? I'm thinking of creating an etcdctl deployment for running these commands, or maybe just installing it directly on the control plane host.

  • josefassad
    josefassad Posts: 5

    @chriswood123 my workaround is a lot simpler, I just call etcdctl directly. So if the lab says something like

    kubectl -n kube-system exec -it etcd-cp -- sh \
-c "ETCDCTL_API=3 \
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \ 
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt \ 
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key \
etcdctl endpoint health"

    I'm calling

    kubectl exec -ti -n kube-system etcd-cp -- etcdctl \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health

    I figured this out when I looked at the current etcd image's Dockerfile. They're using distroless and copying the etcd binaries over. So I just configured the etcdctl invocation using flags instead of environment variables.

    Hope this works for you!

Categories

Upcoming Training