Lab 4.1, exec'ing into etcd container fails due to lack of shell

I have the same issue.

@josefassad what did you do as a workaround? I'm thinking of creating an etcdctl deployment for running these commands, or maybe just installing it directly on the control plane host.

Hi @josefassad and @chriswood123,

Would you mind providing the etcd version that you are reporting about? This would help the course maintainer determine if these are isolated incidents or perhaps more generic across multiple etcd releases, requiring a permanent solution within the lab exercises.

You can find the version with either one of the two commands below:

kubectl -n kube-system exec etcd-cp -- etcdctl version
kubectl -n kube-system describe pod etcd-cp | grep Image:

As I posted earlier, when I tried to reproduce the issue, all commands worked as presented in the lab guide. I attempted to reproduce the issue on two Kubernetes releases: v1.33.1 (etcd 3.5.21) and v1.34.1 (etcd 3.6.4).

Regards,
-Chris

Hi @josefassad,

Thank you for the detailed output. When etcd went distroless I remember we experienced similar issues, but eventually some CLI tools/utilities have been added back into the image and the expected behaviors resumed (for the purpose of our lab exercises). Therefore -- sh -c 'some-command' worked without any issues.

I will monitor github for reported etcd issues to determine if this is temporary or a permanent reduction of the image footprint. Until we know more, the workaround you recommended above can be successfully used for Lab exercise 4.1 and can be tailored for other scenarios as well.

Regards,
-Chris

Hi @chriswood123 @josefassad,

I took a deeper dive into the etcd image inconsistencies, and I uncovered the following (so far):

KUBERNETES      ETCD
RELEASE         IMAGE       sh SUPPORT
-----------------------------------------------
1.33.1          3.5.21          YES
1.33.2          3.5.21          YES
1.33.3          3.5.21          YES
1.33.4          3.5.21          YES
1.33.5          3.5.21          YES
1.33.6          3.5.24          NO
1.33.7          3.5.24          NO
1.34.1          3.6.4           YES
1.34.2          3.6.5           NO
1.34.3          3.6.5           NO

This helps us to plan ahead for possible changes to the lab guide to account for missing sh shell.

For the time being, the .1 Kubernetes patch releases support the instructions as they are presented in the lab guide where etcdctl tool runs in the container's shell:

kubectl -n kube-system exec -it etcd-cp -- sh -c \
  "ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
  ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt \
  ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key \
  etcdctl endpoint health"

However, if deviating from the lab guide by installing patch releases with etcd images missing sh shell support, the instructions need to be adapted to call etcdctl directly:

kubectl -n kube-system exec etcd-cp -- \
  etcdctl endpoint health \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Regards,
-Chris

I just want to add a "This affects me too" here. My deviation from the labguide is running kubeadm 1.33.7 and so on, pluss using Debian Trixie as distro.

kubectl -n kube-system exec etcd-k8smn01 -- /usr/local/bin/etcd --version
etcd Version: 3.5.24
Git SHA: e72f3c2
Go Version: go1.24.9
Go OS/Arch: linux/amd64

Thanks! I initially thought that this is a thing with arm64 image. But even googling took me here.