kubernetes installed by kind can't pull container images

dzhigalin

I've installed k8s with kind using the instructions from the course. When I started with the first lab with consul, I found that my kubelet running inside consul-control-plane container fails to pull images of the emojivoto applications:

vagrant@k8s:~/lab3.2$ kubectl get po -n emojivoto
NAME                        READY   STATUS             RESTARTS   AGE
emoji-65df4d68f7-9nhqf      0/1     ImagePullBackOff   0          19m
vote-bot-7c59767698-dk5pc   0/1     ImagePullBackOff   0          29m
voting-768f496cd8-8xlrd     0/1     ErrImagePull       0          29m
web-545f869fc4-6z5g8        0/1     ImagePullBackOff   0          29m

in order to exclude any problems with name resolving I've changed nameserver in /etc/resolve.cof to google dns and I can see that name resolution works from within the consul-control-plane container.

vagrant@k8s:~/lab3.2$ docker exec -it consul-control-plane bash
root@consul-control-plane:/# cat /etc/resolv.conf
nameserver 8.8.8.8
root@consul-control-plane:/# curl -v  google.com
*   Trying 172.217.23.142:80...
* TCP_NODELAY set
* Connected to google.com (172.217.23.142) port 80 (#0)


  GET / HTTP/1.1

But at the same time in journal for kubelet that is running inside that container I still see that it fails to pull the image

root@consul-control-plane:/# journalctl -fu kubelet.service
-- Logs begin at Tue 2021-01-26 20:44:41 UTC. --
Jan 26 21:24:51 consul-control-plane kubelet[4620]: E0126 21:24:51.745867    4620 pod_workers.go:191] Error syncing pod 5896010f-a7b1-41d4-92ff-4460b862a146 ("voting-768f496cd8-8xlrd_emojivoto(5896010f-a7b1-41d4-92ff-4460b862a146)"), skipping: failed to "StartContainer" for "voting-svc" with ImagePullBackOff: "Back-off pulling image \"docker.l5d.io/buoyantio/emojivoto-voting-svc:v11\""
Jan 26 21:24:53 consul-control-plane kubelet[4620]: E0126 21:24:53.745825    4620 pod_workers.go:191] Error syncing pod 40b8f63d-c56c-4916-a07a-5c6ef05f3b1c ("vote-bot-7c59767698-dk5pc_emojivoto(40b8f63d-c56c-4916-a07a-5c6ef05f3b1c)"), skipping: failed to "StartContainer" for "vote-bot" with ImagePullBackOff: "Back-off pulling image \"docker.l5d.io/buoyantio/emojivoto-web:v11\""
Jan 26 21:25:01 consul-control-plane kubelet[4620]: E0126 21:25:01.746200    4620 pod_workers.go:191] Error syncing pod 053f9fe8-ab56-4653-a23d-1584c4a1a0f0 ("web-545f869fc4-6z5g8_emojivoto(053f9fe8-ab56-4653-a23d-1584c4a1a0f0)"), skipping: failed to "StartContainer" for "web-svc" with ImagePullBackOff: "Back-off pulling image \"docker.l5d.io/buoyantio/emojivoto-web:v11\""

I tried to restart it, but it didn't help. Any ideas how I can troubleshoot this further?

dzhigalin

One addition. When I changed registry from docker.l5d.io to just docker hub (default one) it worked:

vagrant@k8s:~/lab3.2$ kubectl get po -n emojivoto
NAME                       READY   STATUS    RESTARTS   AGE
emoji-5fcc588b77-m7kdd     1/1     Running   0          44s
vote-bot-658cb85b6-xj4zz   1/1     Running   0          44s
voting-5858f494d-cx2dp     1/1     Running   0          44s
web-7ddc6b67d6-zcxps       1/1     Running   0          43s