k8s unable to pull image from the local unsecured registry

mark.kharitonov

In Lab 3.2. (Configure a Local Repository) we spin up a local unsecured registry from which k8s would pull the simple app image. However, I am unable to make it work.

So, before creating the deployment everything seems to be in order:

student@master:~$ curl 10.97.82.186:5000/v2/_catalog
{"repositories":["simpleapp"]}
student@master:~$ k get deploy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
nginx      1/1     1            1           118m
registry   1/1     1            1           118m
student@master:~$ k get pod
NAME                       READY   STATUS    RESTARTS      AGE
nginx-6488f757bc-cf4q4     1/1     Running   1 (51m ago)   118m
registry-d4cf9fd7d-qj6tn   1/1     Running   1 (51m ago)   118m
student@master:~$ sudo podman images
REPOSITORY                   TAG         IMAGE ID      CREATED      SIZE
localhost/simpleapp          latest      bb19ffc6050a  2 hours ago  943 MB
10.97.82.186:5000/simpleapp  latest      bb19ffc6050a  2 hours ago  943 MB
docker.io/library/python     3           e285995a3494  8 days ago   943 MB
10.97.82.186:5000/tagtest    latest      9c6f07244728  6 weeks ago  5.83 MB
student@master:~$ echo $repo
10.97.82.186:5000
student@master:~$

Let us create the deployment as per the lab instructions:

student@master:~$ k create deployment try1 --image=$repo/simpleapp
deployment.apps/try1 created
student@master:~$ k describe pod try1-5f97db4fb8-j9csw |grep Failed
  Warning  Failed     11s                kubelet            Failed to pull image "10.97.82.186:5000/simpleapp": rpc error: code = Unknown desc = failed to pull and unpack image "10.97.82.186:5000/simpleapp:latest": failed to resolve reference "10.97.82.186:5000/simpleapp:latest": failed to do request: Head https://10.97.82.186:5000/v2/simpleapp/manifests/latest: http: server gave HTTP response to HTTPS client
  Warning  Failed     11s                kubelet            Error: ErrImagePull
  Warning  Failed     10s (x2 over 11s)  kubelet            Error: ImagePullBackOff
student@master:~$

What I find suspicious is the url https://10.97.82.186:5000/v2/simpleapp/manifests/latest - no way https is going to work here.

How do we fix it?

P.S.

Also posted the question here - https://stackoverflow.com/questions/73807830/k8s-unable-to-pull-image-from-the-local-unsecured-registry

chrispokorni

Hi @mark.kharitonov,

On which node is the try1 pod scheduled? Typically (but not always) at this step it gets scheduled on the worker/second node.

Can you validate that registry.conf and config.toml files respectively are identically configured between the two nodes of your cluster? In addition, the runtime restart and eventually the node reboot are also successful? The necessary commands are presented in Lab 3.2 steps 12 and 13.

Regards,
-Chris

mark.kharitonov

As per the lab instructions I modified the relevant configuration files both on the master and the worker nodes.

So, on the master:

student@master:~$ cat /etc/containers/registries.conf.d/registry.conf
[[registry]]
location = "10.97.82.186:5000"
insecure = true
student@master:~$ diff -U3 /etc/containerd/config.toml /etc/containerd/config.toml.orig
--- /etc/containerd/config.toml 2022-09-21 21:22:37.032171446 +0000
+++ /etc/containerd/config.toml.orig    2022-09-22 03:35:37.032007211 +0000
@@ -152,9 +152,6 @@

       [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

-      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
-        endpoint = ["10.97.82.186:5000"]
-
     [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
       tls_cert_file = ""
       tls_key_file = ""
student@master:~$

Now on the worker:

student@worker:~$ cat /etc/containers/registries.conf.d/registry.conf
[[registry]]
location = "10.97.82.186:5000"
insecure = true
student@worker:~$ diff -U3 /etc/containerd/config.toml /etc/containerd/config.toml.orig
--- /etc/containerd/config.toml 2022-09-21 22:07:27.199770673 +0000
+++ /etc/containerd/config.toml.orig    2022-09-22 15:26:21.280537739 +0000
@@ -136,9 +136,6 @@

       [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

-      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
-        endpoint = ["10.97.82.186:5000"]
-
     [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
       tls_cert_file = ""
       tls_key_file = ""
student@worker:~$

Both machines for stopped for the night. After booting them the only thing I had to repeat is push the simpleapp image to the local repository again. I need to check it, seems like the storage was not persisted in between reboots, but this is a different issue. Anyway, all seems working fine:

On the master:

student@master:~$ curl $repo/v2/_catalog
{"repositories":["simpleapp"]}
student@master:~$

On the worker:

student@worker:~$ curl $repo/v2/_catalog
{"repositories":["simpleapp"]}
student@worker:~$

Is there anything else I can check to help you to help me?

mark.kharitonov

My SO question was answered and the answer is to change endpoint = ["10.97.82.186:5000"] to endpoint = ["http://10.97.82.186:5000"]

The lab should be updated.

mretfaster

For anyone else stumbling upon this thread as I am,
The LFD directions I have do show the http:// prefix as suggested above, in section 3.2!

mark.kharitonov

It does indeed. I do not know how I could miss it.

taepyongyang

This does not appear at all in the V2022-11-23 labs, but it was necessary to proceed. Appreciate the help here.

d95pari

I ran into the same problem but eventually got it to work.

1.
The lab has the following line:

[plugin."io.containerd.grpc.v1.cri".registry.mirrors."*"] #<-- Add these two lines

I changed "plugin" to "plugins" as mentioned here and in the downloadable archive.

2.
I also needed to add an image tag when creating the deployment.

kubectl create deployment try1 --image=$repo/simpleapp:latest