Unable to add Node - Lab 3.2

rthothadri

kubeadm join --token 5e0890.01221d0246c8ea8e 10.128.0.4:6443 --discovery-token-ca-cert-hash \

sha256:e3b0c44298fc1c149.......e4649b934ca495991b7852b855

The above command is retruning the following error:

[discovery] Trying to connect to API Server "10.128.0.4:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://10.128.0.4:6443"

[discovery] Failed to connect to API Server "10.128.0.4:6443": cluster CA found in cluster-info configmap is invalid: public key sha256:9b263f52d90b62458a6a6c6.......02ddc34bf26e1ac not pinned

I couldn't find any information about how to resolve this error.

I'm using GCE VM Instances.

serewicz

Hello,

I would begin by first checking that server IP address matches. Were you using the join statement from the output of kubeadm init?  If it's been over 24 hours you may need to generate a new certificate on the master server.   You may have a firewall inbetween the nodes. 

If the IP address is accurate and it has been less than 24 hours since you initialied the master, use the nc command to check port 6443 like this: nc 10.10.128.0.4 6443 

If you get a prompt back, then the worker node cannot reach the server and I would look for a firewall issue. Next I would check that adding the networking policy file worked and all the Pods are running and in good shape with kubectl show pods --all-namespaces -o wide

Please report back if this works or not. If you get errors along the way please share the output so we can further troubleshoot the issue.

Regards,

chrispokorni

Hi, 

I notice you are providing one public key in your kubeadm join command:  e3b0c4429... and there is a different key in the error output: 9b263f52d90b...

The key in the error output is the expexted key you should have in the kubeadm join command. Try to join with the 9b263f52d90b... key. Basically if you provide the wrong key in the kubeadm join, the error spits out the expected key to join the cluster, which is not something very secure if you asked me...

I just reproduced the error by providing a different key than the one expected...

Good luck!

-Chris

rthothadri

Node1a - Master:

oracle@k8-node1a:~$ sudo kubeadm token create

5ed1ee.fc9580d58d001c0e

oracle@k8-node1a:~$ sudo kubeadm token list

TOKEN                     TTL       EXPIRES                USAGES                   DESCRIPTION   EXTRA GROUPS

5ed1ee.fc9580d58d001c0e   23h       2018-03-21T00:51:28Z   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

oracle@k8-node1a:~$

oracle@k8-node1a:~$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der> /dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

writing RSA key

oracle@k8-node1a:~$

Node2a:

root@k8-node2a:~# kubeadm join --token 5ed1ee.fc9580d58d001c0e 10.128.0.4:6443 --discovery-token-ca-cert-hash \

> sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

[preflight] Running pre-flight checks.

        [WARNING FileExisting-crictl]: crictl not found in system path

[discovery] Trying to connect to API Server "10.128.0.4:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://10.128.0.4:6443"

[discovery] Failed to connect to API Server "10.128.0.4:6443": cluster CA found in cluster-info configmap is invalid: public key sha256:9b263f52d90b62458a6a6c6d5d415e9110fc6dcb9bf8392f102ddc34bf26e1ac not pinned

[discovery] Trying to connect to API Server "10.128.0.4:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://10.128.0.4:6443"

[discovery] Failed to connect to API Server "10.128.0.4:6443": cluster CA found in cluster-info configmap is invalid: public key sha256:9b263f52d90b62458a6a6c6d5d415e9110fc6dcb9bf8392f102ddc34bf26e1ac not pinned

root@k8-node2a:~#

root@k8-node2a:~# kubeadm join --token 5ed1ee.fc9580d58d001c0e 10.128.0.4:6443 --discovery-token-ca-cert-hash \

> sha256:9b263f52d90b62458a6a6c6d5d415e9110fc6dcb9bf8392f102ddc34bf26e1ac

[preflight] Running pre-flight checks.

        [WARNING FileExisting-crictl]: crictl not found in system path

[discovery] Trying to connect to API Server "10.128.0.4:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://10.128.0.4:6443"

[discovery] Requesting info from "https://10.128.0.4:6443" again to validate TLS against the pinned public key

[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.128.0.4:6443"

[discovery] Successfully established connection with API Server "10.128.0.4:6443"

This node has joined the cluster:

* Certificate signing request was sent to master and a response

  was received.

* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

root@k8-node2a:~#

Hi Chris,

Thanks for your suggestion. I was able to use the key from the error output to join successfully.

I'm not sure why openssl command from Master Node is not returning the key that I should use for kubeadm join.

I've provided the output from Node1a and Node2a.

Regards,

Ram

 

chrispokorni

Hi Ram, 

There are a few characters missing from your 'openssl', my guess is due to copy/paste. 

From https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/  and from the course/labs manual, the complete 'openssl' is the following: 

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

Compare it with yours, try both and see what the difference is... I assume this one will generate the expected hash.

Good luck!

-Chris

rthothadri

Thanks Chris! Yes, it was missing "2" after der. I got the correct public key now!

oracle@k8-node1a:~$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

9b263f52d90b62458a6a6c6d5d415e9110fc6dcb9bf8392f102ddc34bf26e1ac

oracle@k8-node1a:~$

 

chrispokorni

I am glad it worked. With these long commands and lots of piping it is easy to miss a few characters when copy/pasting, and copying from the pdf lab manual is not always seamless either. 

What helps me in these cases is pasting the content into a text editor, double-checking for accuracy and only then copy from the editor and paste into the terminal. 

mians71

I had a similar issue, but mine was a different typo:

sudo openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | open rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

Notice the second command was open instead of openssl. Redirecting stderr hid that it was failing