calico-node pod: CrashLoopBackOff, coredns pod: ContainerCreating in vagrant

Hello,
I'm trying to follow LAB 3_[1-5] but got the following error:
I have deleted two coredns pods as recommended in LAB_3.3 but still have this state of new spawned pods.
Some logs:
kubectl -n kube-system describe pods coredns-5c98db65d4-dt499
gives
... Warning FailedCreatePodSandBox 7m8s (x4 over 7m11s) kubelet, master-node (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bb3a35937e92bf16d7d2d565ecef5c7259e2a7a391141df19f21c4cd6cc08172" network for pod "coredns-5c98db65d4-dt499": NetworkPlugin cni failed to set up pod "coredns-5c98db65d4-dt499_kube-system" network: no podCidr for node master-node
kubectl -n kube-system describe pods calico-node-w62gg
gives:
... Warning Unhealthy 10m (x7 over 11m) kubelet, worker-node Readiness probe failed: Threshold time for bird readiness check: 30s calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp 127.0.0.1:9099: connect: connection refused ...
- promisc mode on master and worker is enabled, e.g. master node:
$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 02:32:7c:93:db:e1 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global enp0s3 valid_lft forever preferred_lft forever inet6 fe80::32:7cff:fe93:dbe1/64 scope link valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:97:49:fb brd ff:ff:ff:ff:ff:ff inet 10.0.0.10/24 brd 10.0.0.255 scope global enp0s8 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:fe97:49fb/64 scope link valid_lft forever preferred_lft forever 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:0f:c8:1b:f4 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 5: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1 link/ipip 0.0.0.0 brd 0.0.0.0
I have few questions:
- What is the reason of this problem?
- How can I fix it?
Comments
-
Here is my two vagrant files for master node and for worker node.
master:
# -*- mode: ruby -*- # vi: set ft=ruby : $configureMasterNodeBox = <<-SCRIPT #LFS258 apt-get update && apt-get upgrade -y apt-get install -y docker.io cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - apt-get update apt-get install -y kubeadm=1.15.1-00 kubelet=1.15.1-00 kubectl=1.15.1-00 wget https://tinyurl.com/yb4xturm -O rbac-kdd.yaml wget https://tinyurl.com/y8lvqc9g -O calico.yaml IP_ADDR=`ifconfig enp0s8 | grep Mask | awk '{print $2}'| cut -f2 -d:` echo "$IP_ADDR k8smaster" >> /etc/hosts cat <<EOF >kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: 1.15.1 #<-- Use the word stable for newest version controlPlaneEndpoint: "k8smaster:6443" #<-- Use the node alias not the IP networking: podSubnet: 192.168.0.0/16 EOF kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.out # Save output for future review #copying credentials to regular user - vagrant sudo --user=vagrant mkdir -p /home/vagrant/.kube cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config SCRIPT $kubectl = <<-SCRIPT mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config sudo cp /root/rbac-kdd.yaml . kubectl apply -f rbac-kdd.yaml sudo cp /root/calico.yaml . kubectl apply -f calico.yaml source <(kubectl completion bash) echo "source <(kubectl completion bash)" >> ~/.bashrc sudo kubeadm config print init-defaults SCRIPT Vagrant.configure("2") do |config| config.vm.box = "ubuntu/xenial64" config.vm.hostname = "master-node" config.vm.network :private_network, ip: "10.0.0.10" config.vm.provision "shell", inline: $configureMasterNodeBox config.vm.provision "shell", inline: $kubectl, privileged: false end
worker:
# -*- mode: ruby -*- # vi: set ft=ruby : $configureWorkerNodeBox = <<-SCRIPT apt-get update && apt-get upgrade -y apt-get install -y docker.io cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - apt-get update apt-get install -y kubeadm=1.15.1-00 kubelet=1.15.1-00 kubectl=1.15.1-00 SCRIPT Vagrant.configure("2") do |config| config.vm.box = "ubuntu/xenial64" config.vm.hostname = "worker-node" config.vm.network :private_network, ip: "10.0.0.11" config.vm.provision "shell", inline: $configureWorkerNodeBox end
for worker node I run by hands:
# vim /etc/hosts 10.0.0.10 k8smaster kubeadm join ...
Inspired by https://github.com/ecomm-integration-ballerina/kubernetes-cluster/blob/master/Vagrantfile which actually works.
Just wondering what did I missed?
0 -
Seems I found the problem place.
OK if I use the following provisioning command:
IP_ADDR=`ifconfig enp0s8 | grep Mask | awk '{print $2}'| cut -f2 -d:` HOST_NAME=$(hostname -s) kubeadm init --apiserver-advertise-address=$IP_ADDR --apiserver-cert-extra-sans=$IP_ADDR --node-name $HOST_NAME --pod-network-cidr=172.16.0.0/16
NOT OK if I use the following provisioning command:
cat <<EOF >kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: 1.15.1 #<-- Use the word stable for newest version controlPlaneEndpoint: "k8smaster:6443" #<-- Use the node alias not the IP networking: podSubnet: 192.168.0.0/16 EOF kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.out # Save output for future review
Still not sure why it happening. Could someone explain in details?
1 -
Hi @b10s ,
From your first output, it seems that the networking between your nodes is not configured properly.
When provisioning local nodes it is recommended to enable promiscuous mode with allow-all-traffic (all sources, all destinations, all ports, all protocols) in order to allow all Kubernetes agents to talk to each other.
On the Ubuntu nodes once provisioned, also check any firewalls which may block some traffic.If the tutorial you follow works, then you can always just replace the configuration options with the ones from the lab exercise in this course, and see what happens then.
If all else fails, you can simply spin-up two Ubuntu 16.04 LTS VMs with VirtualBox, configure them with promiscuous mode to allow-all-traffic and continue from there.
Good luck!
-Chris0 -
@chrispokorni thank you for checking.
If by
not configured properly
you meantpromiscuous mode
then in my first output you can seePROMISC
.It is at the very beginning, the out of
ip a
.Or have I misunderstood you? Can you explain, why do you think I didn't configure promiscuous mode ?
0 -
When configuring the promiscuous mode you have 3 available options: allow all, allow VMs, deny. Is yous configured to allow all?
0 -
@chrispokorni I hope it is allow all since VirtualBox does it as a bridge
0
Categories
- All Categories
- 51 LFX Mentorship
- 104 LFX Mentorship: Linux Kernel
- 576 Linux Foundation IT Professional Programs
- 304 Cloud Engineer IT Professional Program
- 125 Advanced Cloud Engineer IT Professional Program
- 53 DevOps Engineer IT Professional Program
- 61 Cloud Native Developer IT Professional Program
- 5 Express Training Courses
- 5 Express Courses - Discussion Forum
- 2.1K Training Courses
- 19 LFC110 Class Forum
- 7 LFC131 Class Forum
- 27 LFD102 Class Forum
- 158 LFD103 Class Forum
- 21 LFD121 Class Forum
- 1 LFD137 Class Forum
- 61 LFD201 Class Forum
- 1 LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 612 LFD259 Class Forum
- 105 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFD273 Class Forum
- 2 LFS145 Class Forum
- 25 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 11 LFS203 Class Forum
- 76 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 37 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 141 LFS253 Class Forum
- LFS254 Class Forum
- 1.1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 93 LFS260 Class Forum
- 132 LFS261 Class Forum
- 33 LFS262 Class Forum
- 80 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 18 LFS267 Class Forum
- 18 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS274 Class Forum
- LFS281 Class Forum
- 236 LFW211 Class Forum
- 172 LFW212 Class Forum
- 7 SKF100 Class Forum
- SKF200 Class Forum
- 903 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 116 Multimedia
- 209 Networking
- 101 Printers & Scanners
- 85 Storage
- 763 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 142 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 357 Ubuntu
- 479 Linux System Administration
- 41 Cloud Computing
- 70 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 108 System Management
- 49 Web Management
- 68 Mobile Computing
- 23 Android
- 30 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 538 Off Topic
- 131 Introductions
- 217 Small Talk
- 22 Study Material
- 826 Programming and Development
- 278 Kernel Development
- 514 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 61 All In Program
- 61 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)