Lab 4.1, exec'ing into etcd container fails due to lack of shell

josefassad

Way at the top of the lab, there's this:
student@cp:˜$ kubectl -n kube-system exec -it etcd-<Tab> -- sh
On my end, I'm getting this complaint back:

username@control-plane:~$ kubectl exec -it --namespace kube-system etcd-cp -- sh 
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "<REDACTED_ID>": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Of course there's no such error if I try to call -- etcdctl instead of -- sh. I figure at some point that container went toolless? Is that right?

ucd4evr

Did you try /bin/sh ?

josefassad

Yes. Same error. I'll note though, I'm upgrading 1.34.2 to 1.34.3. So a bit newer than the versions specified in the lab.

I also checked for other ultra-common tools such as ls, even echo * gives the same error.

I mean I've worked around it, I'm just wondering if it's PEBKAC or if it's the course materials.

chrispokorni

Hi @josefassad,

I just verified on both Kubernetes v1.33.1 (etcd 3.5.21) and v1.34.1 (etcd 3.6.4) the command behaves as presented in the lab guide.
-- sh allowed me to execute pwd, cd, and even echo * inside the etcd container of the etcd-cp pod.
-- sh -c 'echo *' also worked.
-- sh -c 'etcdctl' and -- etcdctl also worked.

I recommend checking the official etcd.io site for any known issues or bugs and suggested workarounds or fixes.

Regards,
-Chris

josefassad

Hi Chris,

Thanks for checking! What an odd thing.

At any rate I have an effective workaround so I'm not too bothered. I appreciate the time you took to try to reproduce.

josefassad

Hang on, let me be a bit more persistent. Looking at the etcd source I see from the Dockerfile the image is built on Distroless and contains little else than etcd, etcdctl, and etcdutl.

On the one hand that explains why I can only run those in a kubectl exec. On the other hand, now I'm wondering why @chrispokorni 's image has more things in it.

I'm still thinking PEBKAC but I'm intrigued.