Lab 16.2: unable to join second master node

Hi Chris,

thank you for your reply and your help. I was busy and couldn't reply earlier.
Regarding your questiions:

• Is this used consistently in your cluster, in all the /etc/hosts files, kubeadm init command parameters, and kubeadm join command?
  Yes i did use that consistently, while working through the Labs.
  k8s-master has the gcloud internal ip 10.156.0.5.
  k8s-master-2 has the gcloud internal ip 10.156.0.12.
  k8s-proxy has the gcloud internal ip 10.156.0.10.
  In all /etc/hosts/ files k8s-master is mapped to 10.156.0.10.

• kubeadm init
  I don't have my kubeadm-config.yaml from Lab 3 anymore, so i cannot answer your question regarding kubeadm init.
  All i know is, that joining worker nodes wasn't a problem.
  The kube adm init command i use to join the new node to the existing master shows also that connectivity is not an issue:
  It's too long to paste it here, but i've attached the output.

It's possible to curl on the api through the k8s-proxy.

sutdent@k8s-master-2:~$ curl k8s-master:6443 -v
* Rebuilt URL to: k8s-master:6443/
*   Trying 10.156.0.10...
* Connected to k8s-master (10.156.0.10) port 6443 (#0)


  GET / HTTP/1.1

• haproxy config
  haproxy.cfg seems correct to me; i also see traffic in the stats increasing, whle curl'ing on the api.

• IP change
  The internal IPs from the VMs didn't change.

Any ideas? Thanks again!
Peter

Hi Chris,

thank you for your reply and suggestions. I was busy again and couldn't continue here, until now.
My cluster VMs were powered off, but after poweron cluster is running again.

So the instructions tells me to use the student user, but with sudo. So basically your're right.
I tried to join the second master again. Generated token, SSL hash & master certificate on the running master to join the new master node. This time i used root. It fails again with the same Error:

root@k8s-master-2:~# kubeadm join k8s-master:6443 --token 1uwx2s.fs12j9vumu59utl9 --discovery-token-ca-cert-hash sha256:4df75f2
61e68bf47c0117bde46899745956883a592f7f996843698f5d1053876 --control-plane --certificate-key 2a5cca83e6a6d4022652ab372dc137d46d2
3adb39a70614ec957ea99f21ce8ab
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". P
lease follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
error execution phase preflight:
One or more conditions for hosting a new control plane instance is not satisfied.

unable to add a new control plane instance a cluster that doesn't have a stable controlPlaneEndpoint address

Please ensure that:  
* The cluster has a stable controlPlaneEndpoint address.
* The certificates that must be shared among control plane instances are provided.


To see the stack trace of this error execute with --v=5 or higher

Now what's interesting is, that i tried to join the node as a "worker node" instead, by omitting the --control-plane argument.
And that worked!

root@k8s-master-2:~# kubeadm join k8s-master:6443 --token 1uwx2s.fs12j9vumu59utl9 --discovery-token-ca-cert-hash sha256:4df75f261e68bf47c0117bde46899745956883a592f7f996843698f5d1053876  --certificate-key 2a5cca83e6a6d4022652ab372dc137d46d23adb39a70614ec957ea99f21ce8ab
W0620 08:40:36.959233   20255 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Now i have another worker node in the cluster

student@k8s-master:~$ k get nodes
NAME           STATUS   ROLES    AGE    VERSION
k8s-master     Ready    master   147d   v1.17.2
k8s-master-2   Ready    <none>   18m    v1.17.2
k8s-worker     Ready    <none>   147d   v1.17.2

I scaled an exisiting deployment and the new pods are also running on the new node.

I don't know what to try next, but i'll try again with a new blank VM.

Thank's again for the feedback!
Best
Peter