ImagePullBackoff

Hello all.
I need some help on this problem.
So, I'm using Ubuntu 16.04 and have one master and one worker.
My master node ip is 192.168.0.35
My worker node ip is 192.168.0.36
Both have static bridge enp0s3 connection, gateway 192.168.0.1, dns 8.8.8.8.
I changed my coredns pod configMap before.
There was an error which ImagePullBackoff status when i tried to pull image from docker to make a pod, but this only happened on my master node. On my worker node, all worked fine, no error and got running status. But this matter will be solved when i restart the master node, may it one times or more until all pods were running. After restart one times or more, i got running status on all pods.
Anybody know why did this happen? Is there a problem with my network configurations?
Comments
-
Hello,
If you can share the error and the command you ran to generate it, it will help with the troubleshooting.
Changing the network configuration after the cluster has been initialized is tricky at best. I note you are using an IP in the 192.168.0.0 range, which is the default range used by Calico to provide IP addresses to the pods. This can cause network issues.
To troubleshoot I would look at the IP of the master node when this problem occurs. Then use docker run to see if I can pull and run the container outside of Kubernetes. This will narrow down where to look for the issue.
I think a better solution is to rebuild the cluster and either change the range of IPs of the nodes, or edit the calico.yaml file and the kubeadm init command to use a range without overlap. If you reference the lab exercise you'll note there are steps exactly about this issue and where to find the settings in the calico.yaml file and then the kubeadm init command to avoid the issue of having overlapping IPs.
Regards,
0 -
Several master node restarts forced your pods to receive new IP addresses, probably until there was no more overlap in IPs with your nodes' IPs. In multi-node clusters, on the first node, the calico network plugin will assign IPs to pods from the 192.168.0.X subnet, on the second node the plugin will assign IPs from 192.168.1.X, on a potential third node from 192.168.2.X, ... - provided your calico was started with its default configuration. Keeping this in mind, I would see how IPs of pods running on the master node and the IPs of your nodes would overlap, considering both ranges are on 192.168.0.X, causing DNS confusion.
Regards,
-Chris0 -
I will make it more details.
Master node IP 192.168.0.35/24, hostname master
Worker node IP 192.168.0.36/24, hostname ubuntu
Both are static IPs, bridge enp0s3, gateway 192.168.0.1, dns 8.8.8.8
Calico is using the default, which is 192.168.0.0/16, i did not change anything
Also, master node and worker node both has same docker0 bridge IP which is 172.17.0.1/16
Actually, i'm wondering is it okay if both of my nodes have same docker0 with same IP?This is my cluster info
This is my /etc/hosts on master
This is my /etc/hosts on worker
When i ran docker run on my worker, it worked fine. But when i ran docker run on my master, i got this error
The command i ran before i got the error are
- kubectl apply -n sock-shop -f complete-demo.yaml
- kubectl get pods -n sock-shop -o wide
This is the pod that got the error
1. payment-7d4d4bf9hb4-hgbrx
2. queue-master-6b5b5c7658-rr5gbDo you reference about Exercise 3.1 from number 8 until number 10?
Am i having Network Issues because of how i configured my network?0 -
Hi @neirkate,
Thank you for all the details provided. There seem to be several issues related to a misconfigured cluster. As mentioned earlier the default pods IP range 192.168.0.0/16, used by calico and the kubeadm init command, overlap the node IPs configured by your hypervisor. To fix this issue the suggestion was to rebuild the cluster and use different IP ranges: either change the IPs assigned to your nodes by the hypervisor or change the calico.yaml file and issue kubeadm init command with a different IP range for pods.
There also seems to be some naming confusion between the ubuntu node and worker node. It seems that pods are scheduled on a node named ubuntu when they should be running on the worker.
I agree with the suggestion of starting clean from 2 new nodes, and re-install all the components and re-create the cluster, by making sure there is no overlapping between nodes' IP range and pods' IP range. Both nodes should have the same type of networking interface, with promiscuous mode set to allow-all traffic.Good luck,
-Chris0 -
@chrispokorni
Sorry for the late reply.
I just knew it that i had configured my cluster wrongly, especially the network part.
Okay, i will do that. Seems like that is the best solution for this problem.
Thank you for your help.Have a nice day.
0
Categories
- All Categories
- 51 LFX Mentorship
- 104 LFX Mentorship: Linux Kernel
- 576 Linux Foundation IT Professional Programs
- 304 Cloud Engineer IT Professional Program
- 125 Advanced Cloud Engineer IT Professional Program
- 53 DevOps Engineer IT Professional Program
- 61 Cloud Native Developer IT Professional Program
- 5 Express Training Courses
- 5 Express Courses - Discussion Forum
- 2K Training Courses
- 19 LFC110 Class Forum
- 7 LFC131 Class Forum
- 27 LFD102 Class Forum
- 157 LFD103 Class Forum
- 20 LFD121 Class Forum
- 1 LFD137 Class Forum
- 61 LFD201 Class Forum
- 1 LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 611 LFD259 Class Forum
- 105 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFD273 Class Forum
- 2 LFS145 Class Forum
- 24 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 11 LFS203 Class Forum
- 75 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 36 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 140 LFS253 Class Forum
- LFS254 Class Forum
- 1.1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 93 LFS260 Class Forum
- 132 LFS261 Class Forum
- 33 LFS262 Class Forum
- 80 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 18 LFS267 Class Forum
- 17 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS274 Class Forum
- LFS281 Class Forum
- 235 LFW211 Class Forum
- 172 LFW212 Class Forum
- 7 SKF100 Class Forum
- SKF200 Class Forum
- 903 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 116 Multimedia
- 209 Networking
- 101 Printers & Scanners
- 85 Storage
- 763 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 142 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 357 Ubuntu
- 479 Linux System Administration
- 41 Cloud Computing
- 70 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 108 System Management
- 49 Web Management
- 68 Mobile Computing
- 23 Android
- 30 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 537 Off Topic
- 131 Introductions
- 217 Small Talk
- 21 Study Material
- 826 Programming and Development
- 278 Kernel Development
- 514 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 61 All In Program
- 61 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)