Welcome to the Linux Foundation Forum!

LFD256 Lab 3.2 using V 2022-11-23

edited December 2022 in LFD259 Class Forum

Looking for advise.

My control plane and worker node are configured:

  • has podman installed
  • $repo env = the cluster IP from kubectl get svc | grep registry ( w/ :5000)
  • curl $repo/v2/_catalog on both control node and worker node output the simpleapp in the registry
  • /etc/containerd/config.toml & /etc/containers/registries.conf.d/registry.conf
    on both control node and worker node are configured with this cluster IP

on control node I ran:

sudo podman build -t simpleapp .
sudo podman tag simpleapp $repo/simpleapp
sudo podman push $repo/simpleapp

on worker node I ran as a debug step sudo podman pull $repo/simpleapp - and this was successful!

The wild thing is that on my worker node i successfully pull using the command
sudo podman pull $repo/simpleapp

My issue is when running:
kubectl create deployment try1 --image=$repo/simpleapp

my output is:

  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  15m                 default-scheduler  Successfully assigned default/try1-6568894f-6p9vp to ip-172-31-4-176
  Warning  Failed     14m (x6 over 15m)   kubelet            Error: ImagePullBackOff
  Normal   Pulling    14m (x4 over 15m)   kubelet            Pulling image ""
  Warning  Failed     14m (x4 over 15m)   kubelet            Failed to pull image "": rpc error: code = Unknown desc = failed to pull and unpack image "": failed to resolve reference "": failed to do request: Head "": http: server gave HTTP response to HTTPS client
  Warning  Failed     14m (x4 over 15m)   kubelet            Error: ErrImagePull
  Normal   BackOff    27s (x65 over 15m)  kubelet            Back-off pulling image ""


  • @sebastianvaldez01 the same thing happened to me and I've been trying to resolve it for the past few days

  • Hi @sebastianvaldez01, @tstaffordsmith,

    The error you are experiencing is caused by recent upstream changes. We will post an update once they get resolved or with a possible workaround.


  • Is the workaround available yet?

  • lf1d
    lf1d Posts: 13
    edited December 2022

    @chrispokorni, upstream changes with what provider? I have been able to successfully create the registry, push an image, and pull from it without encountering an error.

    [email protected]:~$ kubectl get nodes
    NAME     STATUS   ROLES           AGE     VERSION
    cp       Ready    control-plane   2m43s   v1.25.1
    worker   Ready    <none>          2m12s   v1.25.1
    [email protected]:~$ kubectl create deployment try1 --image=$repo/simpleapp
    deployment.apps/try1 created
    [email protected]:~$ kubectl scale deployment try1 --replicas=6
    deployment.apps/try1 scaled
    [email protected]:~$ kubectl get pods -o wide
    NAME                        READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
    nginx-8686455c8b-x82wr      1/1     Running   0          4m15s   cp       <none>           <none>
    registry-7c4c89bb5c-dg79b   1/1     Running   0          4m15s   cp       <none>           <none>
    try1-849fdb77f8-2wwk6       1/1     Running   0          40s   cp       <none>           <none>
    try1-849fdb77f8-5c2rv       1/1     Running   0          40s   worker   <none>           <none>
    try1-849fdb77f8-nnpq5       1/1     Running   0          62s   worker   <none>           <none>
    try1-849fdb77f8-t2qq7       1/1     Running   0          40s   worker   <none>           <none>
    try1-849fdb77f8-vb2tx       1/1     Running   0          40s   cp       <none>           <none>
    try1-849fdb77f8-z2qs4       1/1     Running   0          40s   worker   <none>           <none>
  • I think you are facing the same issue as I did. I provided a fix in a response to this thread : https://forum.linuxfoundation.org/discussion/862401/lfd259-course-updated-to-v1-25-1-11-8-2022#latest


Upcoming Training