Lab 4.1 - sh: find: command not found

danielclough

Lab 4.1 step 2 (b):

In order to use TLS, find the three files that need to be passed with the etcdctl command. Change into the directory and view available files.
# find . -name etcd

My shell returns sh: find: command not found

Also, none of the ls commands work for me.

Again returning, sh: ls: command not found

I can create a snapshot by entering the ETCDCTL commands.

rilindo

I ran into this as well. I am not entirely sure if it is because the etcd container is updated to a very minimal image, but I found a workaround:

1. On the control panel / master server, I looked for the etcd container:

sudo docker ps -a | grep etcd
efb1d0cf67a5        0369cf4303ff           "etcd --advertise-cl…"   About an hour ago   Up About an hour                                   k8s_etcd_etcd-cp01_kube-system_c14cf7d1dfa698068ffc2f808ebe757d_5
eb00f4b04d24        k8s.gcr.io/pause:3.2   "/pause"                 About an hour ago   Up About an hour                                   k8s_POD_etcd-cp01_kube-system_c14cf7d1dfa698068ffc2f808ebe757d_5
f0cc0d56dfbf        0369cf4303ff           "etcd --advertise-cl…"   32 hours ago        Exited (0) 32 hours ago                            k8s_etcd_etcd-cp01_kube-system_c14cf7d1dfa698068ffc2f808ebe757d_4
4ed688ab3e53        k8s.gcr.io/pause:3.2   "/pause"                 32 hours ago        Exited (0) 32 hours ago                            k8s_POD_etcd-cp01_kube-system_c14cf7d1dfa698068ffc2f808ebe757d_4

2. Once I found the container, I ran docker inspec on the container and look at the parameters passed to the containers that reference the etc directory, like so:

rilindo@cp01:~$ sudo docker inspect efb1d0cf67a5 | grep etc
        "Path": "etcd",
            "--cert-file=/etc/kubernetes/pki/etcd/server.crt",
            "--data-dir=/var/lib/etcd",
            "--key-file=/etc/kubernetes/pki/etcd/server.key",
            "--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt",
            "--peer-key-file=/etc/kubernetes/pki/etcd/peer.key",
            "--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt",
            "--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"
        "HostsPath": "/var/lib/kubelet/pods/c14cf7d1dfa698068ffc2f808ebe757d/etc-hosts",
        "Name": "/k8s_etcd_etcd-cp01_kube-system_c14cf7d1dfa698068ffc2f808ebe757d_5",
                "/var/lib/etcd:/var/lib/etcd",
                "/etc/kubernetes/pki/etcd:/etc/kubernetes/pki/etcd",
                "/var/lib/kubelet/pods/c14cf7d1dfa698068ffc2f808ebe757d/etc-hosts:/etc/hosts",
                "/var/lib/kubelet/pods/c14cf7d1dfa698068ffc2f808ebe757d/containers/etcd/7aecfad3:/dev/termination-log"
                "Source": "/var/lib/etcd",
                "Destination": "/var/lib/etcd",
                "Source": "/etc/kubernetes/pki/etcd",
                "Destination": "/etc/kubernetes/pki/etcd",
                "Source": "/var/lib/kubelet/pods/c14cf7d1dfa698068ffc2f808ebe757d/etc-hosts",
                "Destination": "/etc/hosts",
                "Source": "/var/lib/kubelet/pods/c14cf7d1dfa698068ffc2f808ebe757d/containers/etcd/7aecfad3",
                "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
                "etcd",
                "--cert-file=/etc/kubernetes/pki/etcd/server.crt",
                "--data-dir=/var/lib/etcd",
                "--key-file=/etc/kubernetes/pki/etcd/server.key",
                "--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt",
                "--peer-key-file=/etc/kubernetes/pki/etcd/peer.key",
                "--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt",
                "--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"
                "io.kubernetes.container.logpath": "/var/log/pods/kube-system_etcd-cp01_c14cf7d1dfa698068ffc2f808ebe757d/etcd/5.log",
                "io.kubernetes.container.name": "etcd",
                "io.kubernetes.pod.name": "etcd-cp01",

3. Now I got the paths, I can simply log in to the etcd pod and using the paths I found in my grep output, backup the etcd database.

chrispokorni

Hi @danielclough,

It seems the etcd container image has been modified and commands that recently worked have been removed from the container's environment. Bummer...

@rilindo, alternatively you can display the content of the /etc/kubernetes/manifests/etcd.yaml manifest, or kubectl describe the etcd pod.

Regards,
-Chris

rilindo

@chrispokorni That is even easier. I would go with that approach, then.

serewicz

Hello,

Indeed, it looks like the newly updated image only has two commands, cp and sh. Makes things a bit more difficult to work with, for sure. It no longer even has commands such as ls, nor a way of installing new software.

I'm sure it was a great thought for those who assume no one else would need to work inside the image. Probably a "security feature" of some sort.

Thanks for the heads up.

Regards,

johnley

Then how can I delete the file inside the container? When I tried to restore the etcd backup, I got the following error "Error: data-dir "default.etcd" exists"; I found default.etcd and snapshot.db under /tmp, I would like to remove those 2 files and retry toe etcd restore, how can I remove those 2 files inside the etcd container?

Thanks

chrispokorni

Hi @johnley,

Due to continuous changes in Kubernetes and its components, luckily, the latest image versions of the etcd container image supports again most commands that allow us to manage artifacts inside the running container. That means that now we can run again commands such as cd, ls, cp, mv, rm, rmdir, touch, grep, cat, echo, which, ip in an interactive terminal on the etcd container.

Regards,
-Chris