LFD256 Lab 3.2 using V 2022-11-23

sebastianvaldez01

Looking for advise.

My control plane and worker node are configured:

• has podman installed
• $repo env = the cluster IP from kubectl get svc | grep registry ( w/ :5000)
• curl $repo/v2/_catalog on both control node and worker node output the simpleapp in the registry
• /etc/containerd/config.toml & /etc/containers/registries.conf.d/registry.conf
  on both control node and worker node are configured with this cluster IP

on control node I ran:

sudo podman build -t simpleapp .
sudo podman tag simpleapp $repo/simpleapp
sudo podman push $repo/simpleapp

on worker node I ran as a debug step sudo podman pull $repo/simpleapp - and this was successful!

The wild thing is that on my worker node i successfully pull using the command
sudo podman pull $repo/simpleapp

My issue is when running:
kubectl create deployment try1 --image=$repo/simpleapp

my output is:

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  15m                 default-scheduler  Successfully assigned default/try1-6568894f-6p9vp to ip-172-31-4-176
  Warning  Failed     14m (x6 over 15m)   kubelet            Error: ImagePullBackOff
  Normal   Pulling    14m (x4 over 15m)   kubelet            Pulling image "10.102.69.34:5000/simpleapp"
  Warning  Failed     14m (x4 over 15m)   kubelet            Failed to pull image "10.102.69.34:5000/simpleapp": rpc error: code = Unknown desc = failed to pull and unpack image "10.102.69.34:5000/simpleapp:latest": failed to resolve reference "10.102.69.34:5000/simpleapp:latest": failed to do request: Head "https://10.102.69.34:5000/v2/simpleapp/manifests/latest": http: server gave HTTP response to HTTPS client
  Warning  Failed     14m (x4 over 15m)   kubelet            Error: ErrImagePull
  Normal   BackOff    27s (x65 over 15m)  kubelet            Back-off pulling image "10.102.69.34:5000/simpleapp"

tstaffordsmith

@sebastianvaldez01 the same thing happened to me and I've been trying to resolve it for the past few days

chrispokorni

Hi @sebastianvaldez01, @tstaffordsmith,

The error you are experiencing is caused by recent upstream changes. We will post an update once they get resolved or with a possible workaround.

Regards,
-Chris

dec698

Is the workaround available yet?

lf1d

@chrispokorni, upstream changes with what provider? I have been able to successfully create the registry, push an image, and pull from it without encountering an error.

ubuntu@cp:~$ kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
cp       Ready    control-plane   2m43s   v1.25.1
worker   Ready    <none>          2m12s   v1.25.1

ubuntu@cp:~$ kubectl create deployment try1 --image=$repo/simpleapp
deployment.apps/try1 created

ubuntu@cp:~$ kubectl scale deployment try1 --replicas=6
deployment.apps/try1 scaled

ubuntu@cp:~$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
nginx-8686455c8b-x82wr      1/1     Running   0          4m15s   192.168.242.65   cp       <none>           <none>
registry-7c4c89bb5c-dg79b   1/1     Running   0          4m15s   192.168.242.67   cp       <none>           <none>
try1-849fdb77f8-2wwk6       1/1     Running   0          40s     192.168.242.70   cp       <none>           <none>
try1-849fdb77f8-5c2rv       1/1     Running   0          40s     192.168.171.66   worker   <none>           <none>
try1-849fdb77f8-nnpq5       1/1     Running   0          62s     192.168.171.65   worker   <none>           <none>
try1-849fdb77f8-t2qq7       1/1     Running   0          40s     192.168.171.67   worker   <none>           <none>
try1-849fdb77f8-vb2tx       1/1     Running   0          40s     192.168.242.71   cp       <none>           <none>
try1-849fdb77f8-z2qs4       1/1     Running   0          40s     192.168.171.68   worker   <none>           <none>

mehdi.brahmia

I think you are facing the same issue as I did. I provided a fix in a response to this thread : https://forum.linuxfoundation.org/discussion/862401/lfd259-course-updated-to-v1-25-1-11-8-2022#latest