5.7.4 - etcd

dnx

I think it'd be handy showing in the lab how to create the etcd-cp that is used here, presuming it's a copy of etcd that has been created for debugging?

If I run the command as it is shown I get:
Error from server (NotFound): pods "etcd-cp" not found

I'm pretty sure I've seen at some point how to do this, but I cannot remember it.

If I substitute for the etcd master I'm running, I get an error regarding the CA Cert:

kubectl -n kube-system exec -it etcd-k8smaster -- sh -c "ECTCDCTL_API=3
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt
ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key
etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"

{"level":"warn","ts":"2021-10-15T06:26:54.179Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bd8bf03-173b-48ff-9305-0a65281c1900/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\""}
Error: context deadline exceeded
command terminated with exit code 1

chrispokorni

Hi @dnx,

The name of the etcd pod includes the hostname of the control-plane node etcd-hostname. In the example, the hostname is cp.

Regards,
-Chris

dnx

Thanks Chris, I wondered that after I posted it, cp threw me off the track haha, guess that's named after you Any hint on the error I received?

chrispokorni

Hi @dnx,

The error may be caused by unexpected key and cert files. Did you try running the command with the key and cert shown in the lab guide?

The multi-line command itself may not be parsed correctly by the command line client especially when copy/pasted from PDF; converting it into a single-line command may also help.

This is assuming that your etcd pod is named etcd-k8smaster where k8smaster is your control-plane node hostname and not just an alias.

Regards,
-CP

dnx

Here is the output I based that on:

$ k get po --all-namespaces |grep etcd
kube-system   etcd-k8smaster                             1/1     Running   0          2d11h

$ sudo grep etcd /etc/kubernetes/manifests/kube-apiserver.yaml
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379

I did manage to get it working:

$ kubectl -n kube-system exec -it etcd-k8smaster -- sh -c "ECTCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"
/registry/secrets/default/first
k8s


v1Secret

firstdefault"*$c02f5f24-a877-4b6a-ab03-8d5ba8fc36de2ȶzb
kubectl-createUpdatevȶFieldsV1:0
.{"f:data":{".":{},"f:somekey":{}},"f:type":{}}
somekeyfindmeOpaque"

I got the cert details from the following. I'm not sure how /etc/kubernetes/pki/apiserver-etcd-client.crt becomes /etc/kubernetes/pki/etcd/server.crt:

$ k describe po etcd-k8smaster -n kube-system |grep pki
      --cert-file=/etc/kubernetes/pki/etcd/server.crt
      --key-file=/etc/kubernetes/pki/etcd/server.key
      --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
      --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
      --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
      --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
      /etc/kubernetes/pki/etcd from etcd-certs (rw)
    Path:          /etc/kubernetes/pki/etcd