Lab 3.2. Deploy simpleapp + error on workernode

Fafa0803

Hi,

I am stuck on lab 3.2. from the task 29. to task 35.

What I have done so far:

• I have pushed the simpleapp to the local registry.
• I created the daemon.json file with the line { "insecure-registries":["IP-Adress:5000"] }
• I restarted the local registry

My Problem is:
1) I can not pull the image from the worker node
2) I can not deploy the whole application (6 pods / 6 replicas) from the master node. 3 Pods (I assume the pods for the worker node) are not starting and I get an error message for these pods.

Error Messages for:
1) sudo docker pull 10.103.255.115:5000/tagtest
Using default tag: latest
Error response from daemon: Get https://10.103.255.115:5000/v2/: http: server gave HTTP response to HTTPS client

2) Please have a look at the screenshot:

I assumed that there is something wrong with the worker node, but as you can see, the worker node is configured in the K8s cluster.

I am using the aws cloud and configured the two virtual machines as described in the video.
I have no clue how to fix this problem.

Could you help please?
Thank you very much.

chrispokorni

Hi @Fafa0803,

In step 30, what image are you pulling from the private registry? From your notes, you are attempting to pull the tagtest image. According to step 30, you should be pulling the simpleapp image.

Also, what image are you using for the try1 deployment in step 31?

As an overall verification, try to run step 21 from both nodes, your ckad1 and ckad2 (assuming that 10.103.255.115 is the ClusterIP of your registry service) :

curl http://10.103.255.115:5000/v2/

It may have been successful from ckad1, but I would like to see the output it produces when attempted from ckad2.

Regards,
-Chris

serewicz

Hello,

As well as what Chris suggested, if you run kubectl get pods -o wide, and find that all the failed pods are on the worker, this usually indicates the steps to setup the repository on the worker was skipped or incorrectly done. Make sure the /etc/docker/daemon.json file on the worker is the same as the master, and remember you have two services to restart.

Regards,

Fafa0803

Hi,

Thank you for your quick response .

1) I am sorry. This was my mistake. It works with the simpleapp.

2) For step 31 I used simpleapp - its saved in my command line. But I tried a second deployment with the name try2 and it does also not work:

The command: curl http://10.103.255.115:5000/v2/ is producing the following results:

Masternode:

Workernode:

Thank you for help.

Best regards,
Fabian

Fafa0803

I just saw the comment from serewicz. Thank you. I will try this also.

Update:
I checked the daemon.json on the master and worker node. Both were identically and i also restarted the service.
I still can not scale the application to 6 nodes. I still have the errors for the 3 pods.

Best regards,
Fabian

Fafa0803

Good morning,

to further test the issue, i created a new deployment file, which i found on the kubernetes documentation: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ and i changed the replicas to 6.

When I created a new deployment using this file, all pods are deployed to the master and worker node.

chrispokorni

Hi @Fafa0803,

The nginx deployment does not help with your original issue reported above, because it does not make use of the private registry we are setting up as part of Lab 3.

Would you be able to provide the output to the command requested by @serewicz earlier?

In addition, picking the name of a troubled pod (one with ImagePullBackOff or ErrImagePull), can you show the Events (displayed at the bottom of the output) from the command:

kubectl describe pod <troubled-pod-name>

Regards,
-Chris

Fafa0803

Now I am really confused, because after starting my machines today, i can not get the pods running.

I get this error message:

So I started again from step 29. On the step 30, i get the following error message on the worker node:

When i continue with stept 31 and 32 on master and look at my pods, I get this:

I would have thought, that I can get at least the pods on master running.
When I describe one pod, than i get this:

Is the "simpleapp" no longer on the repository?
How can that be?

Now I am totaly lost.

serewicz

Hello,

Many things can happen when you reboot a node, such as a change in IP addresses due to DHCP. As everything is an API call, this can be problematic. That the nginx pod continues to run would indicate that the issue is only with the repository configuration, not the Kubernetes cluster itself.

First, let us see the condition of the node , infrastructure pods, and services. Please run
kubectl get node
kubectl -n kube-system get pod
kubectl get svc --all-namespaces

Then show the contents, on both nodes, of /etc/docker/daemon.json.

Finally what are the errors when you run kubectl describe pod try2-some-long-name where you use one of your pod's actual name instead. Near the end of the output you should see several errors, probably having do with not finding the repository. Please paste all of them.

Hopefully from this output we'll know where to look next.

Regards,

Fafa0803

Hello @serewicz ,

thank you so much for your help.

Please find the output for the following commands below:

Kubectl get node

kubectl -n kube-system get pod

kubectl get svc --all-namespaces

/etc/docker/daemon.json

Master:

Worker:

kubectl describe pod try2-some-long-name

Again, thank you very much for your help .

Best regards,
Fabian

chrispokorni

Hi Fabian,

As you can see from your output above, the daemon.json file on your worker node is incomplete. Edit the file to add the missing port number, then restart the service.

Regards,
-Chris

Fafa0803

Hi chris,

thank you for your help.

I fixed this issue:

I also restarted the try2 deployment:

Pod description:

The registry ist still running:

How can I check, if a docker image is still in the local registry?
When I want to pull the simpleapp, i get the following error message:

Thank you for your help.

Fafa0803

Hi,

I checked the docker repository again and found several images with the name simpleapp
One with a wrong port number.

This command works also:

Should I delete a some docker images and push a new image to the repository?

Thank you very much for your helpf.

Best regards,
Fabian

chrispokorni

Hi Fabian,

You can remove any unwanted image from the registry, if needed.

Revisit the steps on the master node to create the image and push it to the private registry. Then continue with the verification steps for the worker node, to ensure you can pull the new image from there as well.

Moving forward please double-check your work, because such typos can cause serious issues with your cluster.

Regards,
-Chris

Fafa0803

Hi @chrispokorni , @serewicz ,

thank you very much for your help.
I repeated the labs 3.1 and 3.2. and now it all works.
I am not sure where the problem was, but nevertheless i learned a lot.

Thank you very much for your help and patience.

Best regards,
Fabian

chrispokorni

Glad it all works!

Keep in mind that slight typos and omissions, introduced early on, may not immediately cause errors. They may reveal issues in later exercises, making it so much more difficult to troubleshoot and fix (if possible).

Regards,
-Chris

Fafa0803

Hello,

I have problems with the local Docker registry, again.

When i wanted to continue with the Lab 3.3, on September 29, i had the same problems than before (as described above).
My deployment files could not pull the image from the local docker registry.
So i learned all about the probes from the official documentation website.

But now, in Lab 5.1., we are using the simpleapp and the local docker registry again and I have the same problems.

Can i find the same image on the official docker registry, so i can circumvent the local docker registry?
I want to continue my labs .

Best regards,
Fabian

Fafa0803

I set the imagePullPolicy to Never. Now this works .

chrispokorni

@Fafa0803, that implies that you will manually pull the image from the private registry to the local cache of the ckad-2 node, whenever updating the container image in the registry.