Welcome to the Linux Foundation Forum!

5.7.4 - etcd

I think it'd be handy showing in the lab how to create the etcd-cp that is used here, presuming it's a copy of etcd that has been created for debugging?

If I run the command as it is shown I get:
Error from server (NotFound): pods "etcd-cp" not found

I'm pretty sure I've seen at some point how to do this, but I cannot remember it.

If I substitute for the etcd master I'm running, I get an error regarding the CA Cert:

kubectl -n kube-system exec -it etcd-k8smaster -- sh -c "ECTCDCTL_API=3
etcdctl --endpoints= get /registry/secrets/default/first"
{"level":"warn","ts":"2021-10-15T06:26:54.179Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bd8bf03-173b-48ff-9305-0a65281c1900/","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\""}
Error: context deadline exceeded
command terminated with exit code 1


  • chrispokorni
    chrispokorni Posts: 1,964

    Hi @dnx,

    The name of the etcd pod includes the hostname of the control-plane node etcd-hostname. In the example, the hostname is cp.


  • dnx
    dnx Posts: 32

    Thanks Chris, I wondered that after I posted it, cp threw me off the track haha, guess that's named after you :D Any hint on the error I received?

  • chrispokorni
    chrispokorni Posts: 1,964

    Hi @dnx,

    The error may be caused by unexpected key and cert files. Did you try running the command with the key and cert shown in the lab guide?

    The multi-line command itself may not be parsed correctly by the command line client especially when copy/pasted from PDF; converting it into a single-line command may also help.

    This is assuming that your etcd pod is named etcd-k8smaster where k8smaster is your control-plane node hostname and not just an alias.

    -CP :wink:

  • dnx
    dnx Posts: 32

    Here is the output I based that on:

    $ k get po --all-namespaces |grep etcd
    kube-system   etcd-k8smaster                             1/1     Running   0          2d11h
    $ sudo grep etcd /etc/kubernetes/manifests/kube-apiserver.yaml
        - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
        - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
        - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
        - --etcd-servers=

    I did manage to get it working:

    $ kubectl -n kube-system exec -it etcd-k8smaster -- sh -c "ECTCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints= get /registry/secrets/default/first"

    I got the cert details from the following. I'm not sure how /etc/kubernetes/pki/apiserver-etcd-client.crt becomes /etc/kubernetes/pki/etcd/server.crt:

    $ k describe po etcd-k8smaster -n kube-system |grep pki
          /etc/kubernetes/pki/etcd from etcd-certs (rw)
        Path:          /etc/kubernetes/pki/etcd


Upcoming Training