lab.3.1 Deploying a New Application. pod image pull error.

raghava.cheruku

i am following the lab guide exactly, and up this commnad.

#docker-compose up  output on my  test box is correct.

but when I run, pod status errimagepull. I understand the error that its not able to pull the image from local docker image repo. but its trying docker.com to pull the iamge but its failing.

 

admin1@k8master:/localdocker$ kubectl create -f localregistry.yaml

service/nginx created

service/registry created

deployment.extensions/nginx created

persistentvolumeclaim/nginx-claim0 created

deployment.extensions/registry created

persistentvolumeclaim/registry-claim0 created

admin1@k8master:/localdocker$ kubectl get pods,svc,pvc,pv,deploy

NAME                            READY     STATUS         RESTARTS   AGE

pod/nginx-6b58d9cdfd-dhjqd      0/1       ErrImagePull   0          2s

pod/registry-795c6c8b8f-64nj5   0/1       ErrImagePull   0          2s

 

 

can someone explain whats I need to do fix this ?

chrispokorni

Hi, 

There are a few intermediary steps between the ones mentioned above. Did all those intermediary steps produce expected outputs?

Are your VM instances on Google Cloud? If so, did you create the instances inside a VPC? 

I have seen similar errors when the VM instances were not created inside a VPC. It seems to be a strange behavior of GCP in this case, and I was able to move forward once I created a VPC and the VM instances inside the VPC.

Regards, 

-Chris

raghava.cheruku

Hi Chris,

There are a few intermediary steps between the ones mentioned above. Did all those intermediary steps produce expected outputs?

yes. steps all outputs are as given in the lab guide, but one doubt, Lab guide tasks about "localdocker​ registry:" but howt to check that my local docker registry is corretly configured and kubernates pointing to local docker registry ?

extract from labguide

"Go to the configuration file for a ​localdocker​ registry: student@ckad-1:~$ cd /localdocker/ student@ckad-1:~/localdocker$ ls data  docker-compose.yaml  nginx 

Convert the Docker file into a single YAML file for use with Kubernetes. Not all objects convert exactly from Docker to kompose; you will get errors about the mount syntax for the new volumes. They can be safely ignored.  "

 

------------

I have my local desktop with 2 VMs with OracleVirtual machines created with Suse linux 15.4 version. ontop i have installed kubernates.

 

admin1@k8master:~$ kubectl describe pods

Name:           nginx-6b58d9cdfd-dhjqd

Namespace:      default

Node:           k8second/192.168.0.45

Start Time:     Mon, 30 Jul 2018 07:17:06 +0200

Labels:         io.kompose.service=nginx

                pod-template-hash=2614857898

Annotations:    <none>

Status:         Pending

IP:             192.168.27.215

Controlled By:  ReplicaSet/nginx-6b58d9cdfd

Containers:

  nginx:

    Container ID:

    Image:          nginx:1.12

    Image ID:

    Port:           443/TCP

    Host Port:      0/TCP

    State:          Waiting

      Reason:       ImagePullBackOff

    Ready:          False

    Restart Count:  0

    Environment:    <none>

    Mounts:

      /etc/nginx/conf.d from nginx-claim0 (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9nj9b (ro)

Conditions:

  Type           Status

  Initialized    True

  Ready          False

  PodScheduled   True

Volumes:

  nginx-claim0:

    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName:  nginx-claim0

    ReadOnly:   false

  default-token-9nj9b:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  default-token-9nj9b

    Optional:    false

QoS Class:       BestEffort

Node-Selectors:  <none>

Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s

                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:

  Type     Reason                 Age                From               Message

  ----     ------                 ----               ----               -------

  Normal   SuccessfulMountVolume  2d                 kubelet, k8second  MountVolume.SetUp succeeded for volume "task-pv-volume"

  Normal   SuccessfulMountVolume  2d                 kubelet, k8second  MountVolume.SetUp succeeded for volume "default-token-9nj9b"

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "nginx:1.12": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:48485->[::1]                         :53: read: connection refused

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "nginx:1.12": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:56983->[::1]                         :53: read: connection refused

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "nginx:1.12": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:45017->[::1]                         :53: read: connection refused

  Warning  Failed                 2d (x4 over 2d)    kubelet, k8second  Error: ErrImagePull

  Normal   Pulling                2d (x4 over 2d)    kubelet, k8second  pulling image "nginx:1.12"

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "nginx:1.12": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:44154->[::1]                         :53: read: connection refused

  Normal   BackOff                2d (x5 over 2d)    kubelet, k8second  Back-off pulling image "nginx:1.12"

  Warning  Failed                 2d (x126 over 2d)  kubelet, k8second  Error: ImagePullBackOff

  Normal   Scheduled              34m                default-scheduler  Successfully assigned nginx-6b58d9cdfd-dhjqd to k8second

Name:           registry-795c6c8b8f-64nj5

Namespace:      default

Node:           k8second/192.168.0.45

Start Time:     Mon, 30 Jul 2018 07:17:06 +0200

Labels:         io.kompose.service=registry

                pod-template-hash=3517274649

Annotations:    <none>

Status:         Pending

IP:             192.168.27.216

Controlled By:  ReplicaSet/registry-795c6c8b8f

Containers:

  registry:

    Container ID:

    Image:          registry:2

    Image ID:

    Port:           5000/TCP

    Host Port:      0/TCP

    State:          Waiting

      Reason:       ImagePullBackOff

    Ready:          False

    Restart Count:  0

    Environment:

      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY:  /data

    Mounts:

      /data from registry-claim0 (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9nj9b (ro)

Conditions:

  Type           Status

  Initialized    True

  Ready          False

  PodScheduled   True

Volumes:

  registry-claim0:

    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName:  registry-claim0

    ReadOnly:   false

  default-token-9nj9b:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  default-token-9nj9b

    Optional:    false

QoS Class:       BestEffort

Node-Selectors:  <none>

Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s

                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:

  Type     Reason                 Age                From               Message

  ----     ------                 ----               ----               -------

  Normal   SuccessfulMountVolume  2d                 kubelet, k8second  MountVolume.SetUp succeeded for volume "registryvm"

  Normal   SuccessfulMountVolume  2d                 kubelet, k8second  MountVolume.SetUp succeeded for volume "default-token-9nj9b"

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "registry:2": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:42661->[::1]                         :53: read: connection refused

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "registry:2": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:33018->[::1]                         :53: read: connection refused

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "registry:2": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:51314->[::1]                         :53: read: connection refused

  Normal   BackOff                2d (x5 over 2d)    kubelet, k8second  Back-off pulling image "registry:2"

  Warning  Failed                 2d (x4 over 2d)    kubelet, k8second  Error: ErrImagePull

  Normal   Pulling                2d (x4 over 2d)    kubelet, k8second  pulling image "registry:2"

  Warning  Failed                 2d                 kubelet, k8second  Failed to pull image "registry:2": rpc error: code = Unknown desc = Er                         ror response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:44726->[::1]                         :53: read: connection refused

  Warning  Failed                 2d (x127 over 2d)  kubelet, k8second  Error: ImagePullBackOff

  Normal   Scheduled              34m                default-scheduler  Successfully assigned registry-795c6c8b8f-64nj5 to k8second

admin1@k8master:~$

 

chrispokorni

Hi, thanks for the detailed output. It seems there is a connection issue between k8second and k8master. Traffic to certain ports may be blocked by either the firewall from Suse and/or a networking setting between the VMs. For the purpose of the labs I remember making sure that my firewall was disabled on master and nodes, and my VMs were able to communicate with each other and the internet.

Regards, 

-Chris

raghava.cheruku

Thanks Chris. you are write. my VM's DNS entry into /etc/resolv.conf was not persistance which cuased the issue.

fixing the dns resolved the issue.