Welcome to the Linux Foundation Forum!

calico-node pod: CrashLoopBackOff, coredns pod: ContainerCreating in vagrant

Hello,
I'm trying to follow LAB 3_[1-5] but got the following error:

error screen shoot

I have deleted two coredns pods as recommended in LAB_3.3 but still have this state of new spawned pods.

Some logs:

  1. kubectl -n kube-system describe pods coredns-5c98db65d4-dt499 gives
  1. ...
  2. Warning FailedCreatePodSandBox 7m8s (x4 over 7m11s) kubelet, master-node (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bb3a35937e92bf16d7d2d565ecef5c7259e2a7a391141df19f21c4cd6cc08172" network for pod "coredns-5c98db65d4-dt499": NetworkPlugin cni failed to set up pod "coredns-5c98db65d4-dt499_kube-system" network: no podCidr for node master-node
  1. kubectl -n kube-system describe pods calico-node-w62gggives:
  1. ...
  2. Warning Unhealthy 10m (x7 over 11m) kubelet, worker-node Readiness probe failed: Threshold time for bird readiness check: 30s
  3. calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp 127.0.0.1:9099: connect: connection refused
  4. ...
  1. promisc mode on master and worker is enabled, e.g. master node:
  1. $ ip a
  2. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
  3. link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  4. inet 127.0.0.1/8 scope host lo
  5. valid_lft forever preferred_lft forever
  6. inet6 ::1/128 scope host
  7. valid_lft forever preferred_lft forever
  8. 2: enp0s3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
  9. link/ether 02:32:7c:93:db:e1 brd ff:ff:ff:ff:ff:ff
  10. inet 10.0.2.15/24 brd 10.0.2.255 scope global enp0s3
  11. valid_lft forever preferred_lft forever
  12. inet6 fe80::32:7cff:fe93:dbe1/64 scope link
  13. valid_lft forever preferred_lft forever
  14. 3: enp0s8: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
  15. link/ether 08:00:27:97:49:fb brd ff:ff:ff:ff:ff:ff
  16. inet 10.0.0.10/24 brd 10.0.0.255 scope global enp0s8
  17. valid_lft forever preferred_lft forever
  18. inet6 fe80::a00:27ff:fe97:49fb/64 scope link
  19. valid_lft forever preferred_lft forever
  20. 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
  21. link/ether 02:42:0f:c8:1b:f4 brd ff:ff:ff:ff:ff:ff
  22. inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
  23. valid_lft forever preferred_lft forever
  24. 5: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1
  25. link/ipip 0.0.0.0 brd 0.0.0.0
  26.  

I have few questions:

  1. What is the reason of this problem?
  2. How can I fix it?

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 45
    edited September 2019

    Here is my two vagrant files for master node and for worker node.

    master:

    1. # -*- mode: ruby -*-
    2. # vi: set ft=ruby :
    3.  
    4. $configureMasterNodeBox = <<-SCRIPT
    5. #LFS258
    6. apt-get update && apt-get upgrade -y
    7. apt-get install -y docker.io
    8.  
    9. cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
    10. deb http://apt.kubernetes.io/ kubernetes-xenial main
    11. EOF
    12.  
    13. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
    14. apt-get update
    15. apt-get install -y kubeadm=1.15.1-00 kubelet=1.15.1-00 kubectl=1.15.1-00
    16. wget https://tinyurl.com/yb4xturm -O rbac-kdd.yaml
    17. wget https://tinyurl.com/y8lvqc9g -O calico.yaml
    18. IP_ADDR=`ifconfig enp0s8 | grep Mask | awk '{print $2}'| cut -f2 -d:`
    19. echo "$IP_ADDR k8smaster" >> /etc/hosts
    20.  
    21. cat <<EOF >kubeadm-config.yaml
    22. apiVersion: kubeadm.k8s.io/v1beta2
    23. kind: ClusterConfiguration
    24. kubernetesVersion: 1.15.1 #<-- Use the word stable for newest version
    25. controlPlaneEndpoint: "k8smaster:6443" #<-- Use the node alias not the IP
    26. networking:
    27. podSubnet: 192.168.0.0/16
    28. EOF
    29.  
    30. kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.out # Save output for future review
    31. #copying credentials to regular user - vagrant
    32. sudo --user=vagrant mkdir -p /home/vagrant/.kube
    33. cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
    34. chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config
    35. SCRIPT
    36.  
    37.  
    38. $kubectl = <<-SCRIPT
    39. mkdir -p $HOME/.kube
    40. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    41. sudo chown $(id -u):$(id -g) $HOME/.kube/config
    42. sudo cp /root/rbac-kdd.yaml .
    43. kubectl apply -f rbac-kdd.yaml
    44. sudo cp /root/calico.yaml .
    45. kubectl apply -f calico.yaml
    46. source <(kubectl completion bash)
    47. echo "source <(kubectl completion bash)" >> ~/.bashrc
    48. sudo kubeadm config print init-defaults
    49. SCRIPT
    50.  
    51. Vagrant.configure("2") do |config|
    52. config.vm.box = "ubuntu/xenial64"
    53. config.vm.hostname = "master-node"
    54. config.vm.network :private_network, ip: "10.0.0.10"
    55. config.vm.provision "shell", inline: $configureMasterNodeBox
    56. config.vm.provision "shell", inline: $kubectl, privileged: false
    57. end

    worker:

    1. # -*- mode: ruby -*-
    2. # vi: set ft=ruby :
    3.  
    4. $configureWorkerNodeBox = <<-SCRIPT
    5. apt-get update && apt-get upgrade -y
    6. apt-get install -y docker.io
    7.  
    8. cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
    9. deb http://apt.kubernetes.io/ kubernetes-xenial main
    10. EOF
    11.  
    12. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
    13. apt-get update
    14. apt-get install -y kubeadm=1.15.1-00 kubelet=1.15.1-00 kubectl=1.15.1-00
    15. SCRIPT
    16.  
    17. Vagrant.configure("2") do |config|
    18. config.vm.box = "ubuntu/xenial64"
    19. config.vm.hostname = "worker-node"
    20. config.vm.network :private_network, ip: "10.0.0.11"
    21. config.vm.provision "shell", inline: $configureWorkerNodeBox
    22. end

    for worker node I run by hands:

    1. # vim /etc/hosts
    2. 10.0.0.10 k8smaster
    3.  
    4. kubeadm join ...

    Inspired by https://github.com/ecomm-integration-ballerina/kubernetes-cluster/blob/master/Vagrantfile which actually works.

    Just wondering what did I missed?

  • Posts: 45
    edited September 2019

    Seems I found the problem place.

    OK if I use the following provisioning command:

    1. IP_ADDR=`ifconfig enp0s8 | grep Mask | awk '{print $2}'| cut -f2 -d:`
    2. HOST_NAME=$(hostname -s)
    3. kubeadm init --apiserver-advertise-address=$IP_ADDR --apiserver-cert-extra-sans=$IP_ADDR --node-name $HOST_NAME --pod-network-cidr=172.16.0.0/16

    NOT OK if I use the following provisioning command:

    1. cat <<EOF >kubeadm-config.yaml
    2. apiVersion: kubeadm.k8s.io/v1beta2
    3. kind: ClusterConfiguration
    4. kubernetesVersion: 1.15.1 #<-- Use the word stable for newest version
    5. controlPlaneEndpoint: "k8smaster:6443" #<-- Use the node alias not the IP
    6. networking:
    7. podSubnet: 192.168.0.0/16
    8. EOF
    9.  
    10. kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.out # Save output for future review
    11.  

    Still not sure why it happening. Could someone explain in details?

  • Hi @b10s ,

    From your first output, it seems that the networking between your nodes is not configured properly.
    When provisioning local nodes it is recommended to enable promiscuous mode with allow-all-traffic (all sources, all destinations, all ports, all protocols) in order to allow all Kubernetes agents to talk to each other.
    On the Ubuntu nodes once provisioned, also check any firewalls which may block some traffic.

    If the tutorial you follow works, then you can always just replace the configuration options with the ones from the lab exercise in this course, and see what happens then.

    If all else fails, you can simply spin-up two Ubuntu 16.04 LTS VMs with VirtualBox, configure them with promiscuous mode to allow-all-traffic and continue from there.

    Good luck!
    -Chris

  • Posts: 45
    edited September 2019

    @chrispokorni thank you for checking.

    If by not configured properly you meant promiscuous mode then in my first output you can see PROMISC.

    It is at the very beginning, the out of ip a.

    Or have I misunderstood you? Can you explain, why do you think I didn't configure promiscuous mode ?

  • When configuring the promiscuous mode you have 3 available options: allow all, allow VMs, deny. Is yous configured to allow all?

  • Posts: 45

    @chrispokorni I hope it is allow all since VirtualBox does it as a bridge

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training