Welcome to the Linux Foundation Forum!

Lab 3.2 kubeadm join request refused

hmchen47
hmchen47 Posts: 3
edited March 2018 in LFS258 Class Forum

Hello all,

I am following the instruction to do Lab 3.2 Grow the Cluster.  I use my own laptop to setup the environment.  Here are teh spec of my environment.

Host OS: Windows 10

VM Hypervisor: VirtualBox 

Guest OS: Ubuntu 17.10

I setup two VMs.  The master node is the one I regular used for my projects.  The second one is dedicated for the project.

At step 5 of the Lab instruction, I cannot pass the settings.

Here I dump the info I capture for this lab.



Master node:

$ hmchen@Prj:ch03(k8s03)$ ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:c1:e5:84 brd ff:ff:ff:ff:ff:ff

    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3

       valid_lft 69118sec preferred_lft 69118sec

    inet6 fe80::a9ce:981a:d8f:529c/64 scope link 

       valid_lft forever preferred_lft forever

3: br-2a405b0dbcdc: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 

    link/ether 02:42:b3:b9:b1:a1 brd ff:ff:ff:ff:ff:ff

    inet 172.19.0.1/16 scope global br-2a405b0dbcdc

       valid_lft forever preferred_lft forever

4: br-4bb1837b90b1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 

    link/ether 02:42:1f:16:7c:e3 brd ff:ff:ff:ff:ff:ff

    inet 172.18.0.1/16 scope global br-4bb1837b90b1

       valid_lft forever preferred_lft forever

5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 

    link/ether 02:42:b2:41:94:1b brd ff:ff:ff:ff:ff:ff

    inet 172.17.0.1/16 scope global docker0

       valid_lft forever preferred_lft forever

6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 

    link/ether f2:79:48:46:d8:60 brd ff:ff:ff:ff:ff:ff

    inet 10.244.0.0/32 scope global flannel.1

       valid_lft forever preferred_lft forever

    inet6 fe80::f079:48ff:fe46:d860/64 scope link 

       valid_lft forever preferred_lft forever

7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000

    link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff

    inet 10.244.0.1/24 scope global cni0

       valid_lft forever preferred_lft forever

    inet6 fe80::cc83:4fff:feda:d3a1/64 scope link 

       valid_lft forever preferred_lft forever

8: veth156a7dc9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 

    link/ether b2:69:f7:04:6a:20 brd ff:ff:ff:ff:ff:ff link-netnsid 0

    inet6 fe80::b069:f7ff:fe04:6a20/64 scope link 

       valid_lft forever preferred_lft forever



$ sudo kubeadm token list

[sudo] password for hmchen: 

TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS

44d148.2a750f5fe1f90105   23h       2018-03-28T12:09:11-07:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

ba3edb.1b9344bf6ce19d91   29m       2018-03-27T13:17:26-07:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token



$ openssl x509  -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa \

                -pubin -outform der 2>/dev/null | openssl dgst \

                -sha256 -hex | sed 's/^.* //'

34e555e3f502b0b0aa0786ea841fa8b62daa66569863d62adc34f8e72dc3da41

 

Second Node:

$ ping 10.0.2.15 ==> master node reachable



# kubeadm join --ignore-preflight-errors=cri   --token 44d148.2a750f5fe1f90105 10.0.2.15:6443 --discovery-token-ca-cert-hash sha256:34e555e3f502b0b0aa0786ea841fa8b62daa66569863d62adc34f8e72dc3da41

[preflight] Running pre-flight checks.

[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'

[WARNING FileExisting-crictl]: crictl not found in system path

[discovery] Trying to connect to API Server "10.0.2.15:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://10.0.2.15:6443"

[discovery] Failed to request cluster info, will try again: [Get https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.0.2.15:6443: getsockopt: connection refused]

[discovery] Failed to request cluster info, will try again: [Get https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.0.2.15:6443: getsockopt: connection refused]

 

I have tried other interfaces but none working even with ping.

Comments

  • chrispokorni
    chrispokorni Posts: 2,346
    edited March 2018

    Hi, while I have not tried yet doing the labs on local VMs, I can only think of a networking issue between the 2 nodes:

    1 - is your master listening and accepting traffic on port 6443? The reason I asked is because you mentioned you have used this node for other projects as well.

    2 -  can you run successfully a telnet or nc from the second node with the master IP and port 6443?

    Good luck!

    -Chris

  • serewicz
    serewicz Posts: 1,000
    edited March 2018

    Hello,

    In addition to the errors Chris mentioned, I noticed it said:


    [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'

    Did you install docker on both nodes already? Perhaps return to the lab guide and ensure all necessary steps have been completed on both nodes. The master node must be up and in a Ready state. Then try to join. If you have continued issues, use netcat (nc <server> 6443) to see if the port is reachable. As Chris mentioned it may be a firewall either on the nodes or the host.

    Regards,

     

  • hmchen47
    hmchen47 Posts: 3

    serewicz

    I did solve the issue mention above after running the suggest command 'systemctl enable docker.service' in the later run.  Currently I am working on the port issue as suggested.

    Regards,

    Fred

  • serewicz
    serewicz Posts: 1,000
    edited March 2018

    Hi Fred,

    Another tool to consider is wireshark.  https://www.wireshark.org/docs/wsug_html_chunked/ChUseMainWindowSection.html

    By capturing all the packets on the master node, you can filter for inboud traffic from the second node. If the packet is being accepted I would think it has something to do with the state of the master, or the TLS/SSL key in use.

    Regards,

  • hmchen47
    hmchen47 Posts: 3
    edited March 2018

    Chris, serewicz,

    My issue most likely about the VM network setting and routing issue.  I install the VMs with NAT.  I will try  bridgeing or other options.  Once I have the result I will post the consequence here.

    Regards,

    Fred

  • chrispokorni
    chrispokorni Posts: 2,346
    edited March 2018

    Hi Fred, although a good learning experience, it is unfortunate that you have to go through all this trouble to setup your environment in order to complete the labs. There was another post in this forum about a similar issue (Ubuntu VMs on a local windows workstation), which did not get resolved. The successful alternative, however, was to complete these labs in the cloud, either on GCP or AWS. They offer 1 free year, and I would think that Azure has a similar offer. I have completed most labs by now on GCP and I only ran into a minor firewall issue, which I resolved with a few simple commands.

    Good luck!

    -Chris

  • chrispokorni
    chrispokorni Posts: 2,346
    edited April 2018

    Hi Fred, @hmchen47

    I was able to run thru lab 3 by setting up Bridged networking adapter on the master and minion VMs, and I posted some of the steps and my findings in the topic below:

    https://www.linux.com/forums/lfs258-class-forum/lab-3233-master-node-network-problem

    Regards, 

    -Chris

  • ingernet
    ingernet Posts: 1

    I thought it was really interesting that, although I created the worker as essentially a clone of the master by choosing "create similar" in GCP per the instructions, installing docker on it didn't enable it. Here were the instructions on Lab 3.2:

    IF you chose Docker on the master:
    root@worker: ̃# apt-get install -y docker.io

    ...And here's what I did:

    root@lfs258-worker:~# apt-get install -y docker.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libnuma1
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  bridge-utils cgroupfs-mount containerd pigz runc ubuntu-fan
Suggested packages:
  ifupdown aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
  bridge-utils cgroupfs-mount containerd docker.io pigz runc ubuntu-fan
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 63.7 MB of archives.
After this operation, 319 MB of additional disk space will be used.
Get:1 http://us-west1.gce.archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:2 http://us-west1.gce.archive.ubuntu.com/ubuntu bionic/main amd64 bridge-utils amd64 1.5-15ubuntu1 [30.1 kB]
Get:3 http://us-west1.gce.archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6320 B]
Get:4 http://us-west1.gce.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 runc amd64 1.0.0~rc10-0ubuntu1~18.04.2 [2000 kB]
Get:5 http://us-west1.gce.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 containerd amd64 1.3.3-0ubuntu1~18.04.2 [21.7 MB]
Get:6 http://us-west1.gce.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 docker.io amd64 19.03.6-0ubuntu1~18.04.2 [39.9 MB]
Get:7 http://us-west1.gce.archive.ubuntu.com/ubuntu bionic/main amd64 ubuntu-fan all 0.12.10 [34.7 kB]
Fetched 63.7 MB in 1s (56.7 MB/s)      
Preconfiguring packages ...
Selecting previously unselected package pigz.
(Reading database ... 65498 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package bridge-utils.
Preparing to unpack .../1-bridge-utils_1.5-15ubuntu1_amd64.deb ...
Unpacking bridge-utils (1.5-15ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../2-cgroupfs-mount_1.4_all.deb ...
Unpacking cgroupfs-mount (1.4) ...
Selecting previously unselected package runc.
Preparing to unpack .../3-runc_1.0.0~rc10-0ubuntu1~18.04.2_amd64.deb ...
Unpacking runc (1.0.0~rc10-0ubuntu1~18.04.2) ...
Selecting previously unselected package containerd.
Preparing to unpack .../4-containerd_1.3.3-0ubuntu1~18.04.2_amd64.deb ...
Unpacking containerd (1.3.3-0ubuntu1~18.04.2) ...
Selecting previously unselected package docker.io.
Preparing to unpack .../5-docker.io_19.03.6-0ubuntu1~18.04.2_amd64.deb ...
Unpacking docker.io (19.03.6-0ubuntu1~18.04.2) ...
Selecting previously unselected package ubuntu-fan.
Preparing to unpack .../6-ubuntu-fan_0.12.10_all.deb ...
Unpacking ubuntu-fan (0.12.10) ...
Setting up runc (1.0.0~rc10-0ubuntu1~18.04.2) ...
Setting up cgroupfs-mount (1.4) ...
Setting up containerd (1.3.3-0ubuntu1~18.04.2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up bridge-utils (1.5-15ubuntu1) ...
Setting up ubuntu-fan (0.12.10) ...
Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service → /lib/systemd/system/ubuntu-fan.service.
Setting up pigz (2.4-1) ...
Setting up docker.io (19.03.6-0ubuntu1~18.04.2) ...
Adding group `docker' (GID 116) ...
Done.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
docker.service is a disabled or a static unit, not starting it.
Processing triggers for systemd (237-3ubuntu10.43) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...

    Of note is the tiny notice near the end, "docker.service is a disabled or a static unit, not starting it." Odd that this would happen on the worker but not the control plane.

    So I enabled it:

    root@lfs258-worker:~# systemctl enable docker.service
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.

    That seemed to work.

Categories

Upcoming Training