LAB 3.2: Replicas of try1 pod running correctly on cp but showing as 'ImagePullBackOff' on worker

gabbybamps

I have followed all the steps in Lab3.2 up to the point of deployment of try1 using the image from the local registry. After creating the deployment and replicas, those assigned on the cp run perfectly but those assigned on the worker node show as 'ImagePullBackOff'. I have made all the necessary changes in the registry.conf and config.toml files both on the cp and the worker.
Attached is the output of 'kubectl get pod -o wide'.
Any help will be appreciated. Thanks!

gabbybamps

This is the events from describe:

chrispokorni

Hi @gabbybamps,

I would recommend correcting the typo that causes the error, as it is indicated by the events output:

the invalid character " " in host name "http:// 10.108.149.137:5000"

That is an extra empty space between "//" and "10", which should not be there.
Your correct host name should be http://10.108.149.137:5000

Regards,
-Chris

gabbybamps

@chrispokorni so yes you are right, I was having an extra space in the hostname of the config.toml file of the worker. All works now, thanks!

tymg

I ran into the exact issue

The solution was in the worker node, the /etc/containerd/config.toml file had a typo:

The course says:

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."*"] #<-- Add these two lines
endpoint = ["http://10.97.40.62:5000"]

It should be

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"] #<-- Add these two lines
endpoint = ["http://10.97.40.62:5000"]

The second plugins is missing an ‘s’

See https://github.com/containerd/containerd/issues/4938#issuecomment-851480730