Welcome to the Linux Foundation Forum!

k8s unable to pull image from the local unsecured registry

mark.kharitonov Posts: 17
edited September 2022 in LFD259 Class Forum

In Lab 3.2. (Configure a Local Repository) we spin up a local unsecured registry from which k8s would pull the simple app image. However, I am unable to make it work.

So, before creating the deployment everything seems to be in order:

student@master:~$ curl
student@master:~$ k get deploy
nginx      1/1     1            1           118m
registry   1/1     1            1           118m
student@master:~$ k get pod
NAME                       READY   STATUS    RESTARTS      AGE
nginx-6488f757bc-cf4q4     1/1     Running   1 (51m ago)   118m
registry-d4cf9fd7d-qj6tn   1/1     Running   1 (51m ago)   118m
student@master:~$ sudo podman images
REPOSITORY                   TAG         IMAGE ID      CREATED      SIZE
localhost/simpleapp          latest      bb19ffc6050a  2 hours ago  943 MB  latest      bb19ffc6050a  2 hours ago  943 MB
docker.io/library/python     3           e285995a3494  8 days ago   943 MB    latest      9c6f07244728  6 weeks ago  5.83 MB
student@master:~$ echo $repo

Let us create the deployment as per the lab instructions:

student@master:~$ k create deployment try1 --image=$repo/simpleapp
deployment.apps/try1 created
student@master:~$ k describe pod try1-5f97db4fb8-j9csw |grep Failed
  Warning  Failed     11s                kubelet            Failed to pull image "": rpc error: code = Unknown desc = failed to pull and unpack image "": failed to resolve reference "": failed to do request: Head http: server gave HTTP response to HTTPS client
  Warning  Failed     11s                kubelet            Error: ErrImagePull
  Warning  Failed     10s (x2 over 11s)  kubelet            Error: ImagePullBackOff

What I find suspicious is the url - no way https is going to work here.

How do we fix it?


Also posted the question here - https://stackoverflow.com/questions/73807830/k8s-unable-to-pull-image-from-the-local-unsecured-registry


  • Hi @mark.kharitonov,

    On which node is the try1 pod scheduled? Typically (but not always) at this step it gets scheduled on the worker/second node.

    Can you validate that registry.conf and config.toml files respectively are identically configured between the two nodes of your cluster? In addition, the runtime restart and eventually the node reboot are also successful? The necessary commands are presented in Lab 3.2 steps 12 and 13.


  • As per the lab instructions I modified the relevant configuration files both on the master and the worker nodes.

    So, on the master:

    student@master:~$ cat /etc/containers/registries.conf.d/registry.conf
    location = ""
    insecure = true
    student@master:~$ diff -U3 /etc/containerd/config.toml /etc/containerd/config.toml.orig
    --- /etc/containerd/config.toml 2022-09-21 21:22:37.032171446 +0000
    +++ /etc/containerd/config.toml.orig    2022-09-22 03:35:37.032007211 +0000
    @@ -152,9 +152,6 @@
    -      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
    -        endpoint = [""]
           tls_cert_file = ""
           tls_key_file = ""

    Now on the worker:

    student@worker:~$ cat /etc/containers/registries.conf.d/registry.conf
    location = ""
    insecure = true
    student@worker:~$ diff -U3 /etc/containerd/config.toml /etc/containerd/config.toml.orig
    --- /etc/containerd/config.toml 2022-09-21 22:07:27.199770673 +0000
    +++ /etc/containerd/config.toml.orig    2022-09-22 15:26:21.280537739 +0000
    @@ -136,9 +136,6 @@
    -      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
    -        endpoint = [""]
           tls_cert_file = ""
           tls_key_file = ""

    Both machines for stopped for the night. After booting them the only thing I had to repeat is push the simpleapp image to the local repository again. I need to check it, seems like the storage was not persisted in between reboots, but this is a different issue. Anyway, all seems working fine:

    On the master:

    student@master:~$ curl $repo/v2/_catalog

    On the worker:

    student@worker:~$ curl $repo/v2/_catalog

    Is there anything else I can check to help you to help me?

  • mark.kharitonov
    mark.kharitonov Posts: 17
    edited September 2022

    My SO question was answered and the answer is to change endpoint = [""] to endpoint = [""]

    The lab should be updated.

  • For anyone else stumbling upon this thread as I am,
    The LFD directions I have do show the http:// prefix as suggested above, in section 3.2!

  • It does indeed. I do not know how I could miss it.

  • This does not appear at all in the V2022-11-23 labs, but it was necessary to proceed. Appreciate the help here.

  • I ran into the same problem but eventually got it to work.

    The lab has the following line:

    [plugin."io.containerd.grpc.v1.cri".registry.mirrors."*"] #<-- Add these two lines

    I changed "plugin" to "plugins" as mentioned here and in the downloadable archive.

    I also needed to add an image tag when creating the deployment.

    kubectl create deployment try1 --image=$repo/simpleapp:latest


Upcoming Training