Welcome to the Linux Foundation Forum!

k8s unable to pull image from the local unsecured registry

Posts: 17
edited September 2022 in LFD259 Class Forum

In Lab 3.2. (Configure a Local Repository) we spin up a local unsecured registry from which k8s would pull the simple app image. However, I am unable to make it work.

So, before creating the deployment everything seems to be in order:

  1. student@master:~$ curl 10.97.82.186:5000/v2/_catalog
  2. {"repositories":["simpleapp"]}
  3. student@master:~$ k get deploy
  4. NAME READY UP-TO-DATE AVAILABLE AGE
  5. nginx 1/1 1 1 118m
  6. registry 1/1 1 1 118m
  7. student@master:~$ k get pod
  8. NAME READY STATUS RESTARTS AGE
  9. nginx-6488f757bc-cf4q4 1/1 Running 1 (51m ago) 118m
  10. registry-d4cf9fd7d-qj6tn 1/1 Running 1 (51m ago) 118m
  11. student@master:~$ sudo podman images
  12. REPOSITORY TAG IMAGE ID CREATED SIZE
  13. localhost/simpleapp latest bb19ffc6050a 2 hours ago 943 MB
  14. 10.97.82.186:5000/simpleapp latest bb19ffc6050a 2 hours ago 943 MB
  15. docker.io/library/python 3 e285995a3494 8 days ago 943 MB
  16. 10.97.82.186:5000/tagtest latest 9c6f07244728 6 weeks ago 5.83 MB
  17. student@master:~$ echo $repo
  18. 10.97.82.186:5000
  19. student@master:~$

Let us create the deployment as per the lab instructions:

  1. student@master:~$ k create deployment try1 --image=$repo/simpleapp
  2. deployment.apps/try1 created
  3. student@master:~$ k describe pod try1-5f97db4fb8-j9csw |grep Failed
  4. Warning Failed 11s kubelet Failed to pull image "10.97.82.186:5000/simpleapp": rpc error: code = Unknown desc = failed to pull and unpack image "10.97.82.186:5000/simpleapp:latest": failed to resolve reference "10.97.82.186:5000/simpleapp:latest": failed to do request: Head https://10.97.82.186:5000/v2/simpleapp/manifests/latest: http: server gave HTTP response to HTTPS client
  5. Warning Failed 11s kubelet Error: ErrImagePull
  6. Warning Failed 10s (x2 over 11s) kubelet Error: ImagePullBackOff
  7. student@master:~$

What I find suspicious is the url https://10.97.82.186:5000/v2/simpleapp/manifests/latest - no way https is going to work here.

How do we fix it?

P.S.

Also posted the question here - https://stackoverflow.com/questions/73807830/k8s-unable-to-pull-image-from-the-local-unsecured-registry

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Answers

  • Hi @mark.kharitonov,

    On which node is the try1 pod scheduled? Typically (but not always) at this step it gets scheduled on the worker/second node.

    Can you validate that registry.conf and config.toml files respectively are identically configured between the two nodes of your cluster? In addition, the runtime restart and eventually the node reboot are also successful? The necessary commands are presented in Lab 3.2 steps 12 and 13.

    Regards,
    -Chris

  • As per the lab instructions I modified the relevant configuration files both on the master and the worker nodes.

    So, on the master:

    1. student@master:~$ cat /etc/containers/registries.conf.d/registry.conf
    2. [[registry]]
    3. location = "10.97.82.186:5000"
    4. insecure = true
    5. student@master:~$ diff -U3 /etc/containerd/config.toml /etc/containerd/config.toml.orig
    6. --- /etc/containerd/config.toml 2022-09-21 21:22:37.032171446 +0000
    7. +++ /etc/containerd/config.toml.orig 2022-09-22 03:35:37.032007211 +0000
    8. @@ -152,9 +152,6 @@
    9.  
    10. [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    11.  
    12. - [plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
    13. - endpoint = ["10.97.82.186:5000"]
    14. -
    15. [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
    16. tls_cert_file = ""
    17. tls_key_file = ""
    18. student@master:~$

    Now on the worker:

    1. student@worker:~$ cat /etc/containers/registries.conf.d/registry.conf
    2. [[registry]]
    3. location = "10.97.82.186:5000"
    4. insecure = true
    5. student@worker:~$ diff -U3 /etc/containerd/config.toml /etc/containerd/config.toml.orig
    6. --- /etc/containerd/config.toml 2022-09-21 22:07:27.199770673 +0000
    7. +++ /etc/containerd/config.toml.orig 2022-09-22 15:26:21.280537739 +0000
    8. @@ -136,9 +136,6 @@
    9.  
    10. [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    11.  
    12. - [plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
    13. - endpoint = ["10.97.82.186:5000"]
    14. -
    15. [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
    16. tls_cert_file = ""
    17. tls_key_file = ""
    18. student@worker:~$

    Both machines for stopped for the night. After booting them the only thing I had to repeat is push the simpleapp image to the local repository again. I need to check it, seems like the storage was not persisted in between reboots, but this is a different issue. Anyway, all seems working fine:

    On the master:

    1. student@master:~$ curl $repo/v2/_catalog
    2. {"repositories":["simpleapp"]}
    3. student@master:~$

    On the worker:

    1. student@worker:~$ curl $repo/v2/_catalog
    2. {"repositories":["simpleapp"]}
    3. student@worker:~$

    Is there anything else I can check to help you to help me?

  • Posts: 17
    edited September 2022

    My SO question was answered and the answer is to change endpoint = ["10.97.82.186:5000"] to endpoint = ["http://10.97.82.186:5000"]

    The lab should be updated.

  • For anyone else stumbling upon this thread as I am,
    The LFD directions I have do show the http:// prefix as suggested above, in section 3.2!

  • It does indeed. I do not know how I could miss it.

  • This does not appear at all in the V2022-11-23 labs, but it was necessary to proceed. Appreciate the help here.

  • I ran into the same problem but eventually got it to work.

    1.
    The lab has the following line:

    [plugin."io.containerd.grpc.v1.cri".registry.mirrors."*"] #<-- Add these two lines

    I changed "plugin" to "plugins" as mentioned here and in the downloadable archive.

    2.
    I also needed to add an image tag when creating the deployment.

    kubectl create deployment try1 --image=$repo/simpleapp:latest

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training