Welcome to the Linux Foundation Forum!

5.7.4 - etcd

I think it'd be handy showing in the lab how to create the etcd-cp that is used here, presuming it's a copy of etcd that has been created for debugging?

If I run the command as it is shown I get:
Error from server (NotFound): pods "etcd-cp" not found

I'm pretty sure I've seen at some point how to do this, but I cannot remember it.

If I substitute for the etcd master I'm running, I get an error regarding the CA Cert:

kubectl -n kube-system exec -it etcd-k8smaster -- sh -c "ECTCDCTL_API=3
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt
ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key
etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"
{"level":"warn","ts":"2021-10-15T06:26:54.179Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bd8bf03-173b-48ff-9305-0a65281c1900/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\""}
Error: context deadline exceeded
command terminated with exit code 1

Comments

  • chrispokorni
    chrispokorni Posts: 2,155

    Hi @dnx,

    The name of the etcd pod includes the hostname of the control-plane node etcd-hostname. In the example, the hostname is cp.

    Regards,
    -Chris

  • dnx
    dnx Posts: 32

    Thanks Chris, I wondered that after I posted it, cp threw me off the track haha, guess that's named after you :D Any hint on the error I received?

  • chrispokorni
    chrispokorni Posts: 2,155

    Hi @dnx,

    The error may be caused by unexpected key and cert files. Did you try running the command with the key and cert shown in the lab guide?

    The multi-line command itself may not be parsed correctly by the command line client especially when copy/pasted from PDF; converting it into a single-line command may also help.

    This is assuming that your etcd pod is named etcd-k8smaster where k8smaster is your control-plane node hostname and not just an alias.

    Regards,
    -CP :wink:

  • dnx
    dnx Posts: 32

    Here is the output I based that on:

    $ k get po --all-namespaces |grep etcd
    kube-system   etcd-k8smaster                             1/1     Running   0          2d11h
    
    $ sudo grep etcd /etc/kubernetes/manifests/kube-apiserver.yaml
        - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
        - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
        - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
        - --etcd-servers=https://127.0.0.1:2379
    

    I did manage to get it working:

    $ kubectl -n kube-system exec -it etcd-k8smaster -- sh -c "ECTCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"
    /registry/secrets/default/first
    k8s
    
    
    v1Secret
    
    firstdefault"*$c02f5f24-a877-4b6a-ab03-8d5ba8fc36de2ȶzb
    kubectl-createUpdatevȶFieldsV1:0
    .{"f:data":{".":{},"f:somekey":{}},"f:type":{}}
    somekeyfindmeOpaque"
    

    I got the cert details from the following. I'm not sure how /etc/kubernetes/pki/apiserver-etcd-client.crt becomes /etc/kubernetes/pki/etcd/server.crt:

    $ k describe po etcd-k8smaster -n kube-system |grep pki
          --cert-file=/etc/kubernetes/pki/etcd/server.crt
          --key-file=/etc/kubernetes/pki/etcd/server.key
          --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
          --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
          --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
          --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
          /etc/kubernetes/pki/etcd from etcd-certs (rw)
        Path:          /etc/kubernetes/pki/etcd
    

Categories

Upcoming Training