adding a worker node to cluster giving error

santoshsub123

Hello instructors

i added a worker node to make my cluster from 1 master node 1 worker node to 1 master node 2 worker nodes

i recently added 2nd worker node where kubeadm join is failing. please advise what needs to be done.

chrispokorni

Hi @santoshsub123,

There are several reasons why a kubeadm join command may fail. These reasons may be included in the output produced by the command, or they may be deduced from warning and error messages displayed.

Please provide the output of the failed join command, to help us troubleshoot the issue.

Regards,
-Chris

santoshsub123

sudo kubeadm join 172.31.35.59:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:72461e0dace6a289d3dd15ef5cc4a6fca76036fcc992093e844e386579a37557 --v=5
I0823 09:32:58.630290 7362 join.go:395] [preflight] found NodeName empty; using OS hostname as NodeName
I0823 09:32:58.630600 7362 initconfiguration.go:115] detected and using CRI socket: /var/run/crio/crio.sock
[preflight] Running pre-flight checks
I0823 09:32:58.630681 7362 preflight.go:90] [preflight] Running general checks
I0823 09:32:58.630730 7362 checks.go:250] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0823 09:32:58.630784 7362 checks.go:287] validating the existence of file /etc/kubernetes/kubelet.conf
I0823 09:32:58.630797 7362 checks.go:287] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0823 09:32:58.630807 7362 checks.go:103] validating the container runtime
I0823 09:32:58.642908 7362 checks.go:377] validating the presence of executable crictl
I0823 09:32:58.643568 7362 checks.go:336] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0823 09:32:58.643890 7362 checks.go:336] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0823 09:32:58.644106 7362 checks.go:654] validating whether swap is enabled or not
I0823 09:32:58.644570 7362 checks.go:377] validating the presence of executable conntrack
I0823 09:32:58.644785 7362 checks.go:377] validating the presence of executable ip
I0823 09:32:58.645002 7362 checks.go:377] validating the presence of executable iptables
I0823 09:32:58.645215 7362 checks.go:377] validating the presence of executable mount
I0823 09:32:58.645428 7362 checks.go:377] validating the presence of executable nsenter
I0823 09:32:58.645642 7362 checks.go:377] validating the presence of executable ebtables
I0823 09:32:58.645836 7362 checks.go:377] validating the presence of executable ethtool
I0823 09:32:58.645900 7362 checks.go:377] validating the presence of executable socat
I0823 09:32:58.646086 7362 checks.go:377] validating the presence of executable tc
I0823 09:32:58.646568 7362 checks.go:377] validating the presence of executable touch
I0823 09:32:58.646608 7362 checks.go:525] running all checks
I0823 09:32:58.662878 7362 checks.go:408] checking whether the given node name is valid and reachable using net.LookupHost
I0823 09:32:58.663608 7362 checks.go:623] validating kubelet version
I0823 09:32:58.768408 7362 checks.go:129] validating if the "kubelet" service is enabled and active
I0823 09:32:58.784302 7362 checks.go:202] validating availability of port 10250
I0823 09:32:58.784503 7362 checks.go:287] validating the existence of file /etc/kubernetes/pki/ca.crt
I0823 09:32:58.784524 7362 checks.go:437] validating if the connectivity type is via proxy or direct
I0823 09:32:58.784567 7362 join.go:465] [preflight] Discovering cluster-info
I0823 09:32:58.784597 7362 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "172.31.35.59:6443"
I0823 09:32:58.795930 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again

santoshsub123

main.main
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
couldn't validate the identity of the API Server

santoshsub123

main.main
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
error execution phase preflight

main.main
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371

chrispokorni

Hi @santoshsub123,

How did you generate the token abcdef.0123456789abcdef and the discovery token hash 72461e0dace6a289d3dd15ef5cc4a6fca76036fcc992093e844e386579a37557 prior to running the join command?

Regards,
-Chris

santoshsub123

sudo kubeadm join 172.31.35.59:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:72461e0dace6a289d3dd15ef5cc4a6fca76036fcc992093e844e386579a37557 --v=5

I0823 09:32:58.784524 7362 checks.go:437] validating if the connectivity type is via proxy or direct
I0823 09:32:58.784567 7362 join.go:465] [preflight] Discovering cluster-info
I0823 09:32:58.784597 7362 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "172.31.35.59:6443"

I0823 09:34:24.465385 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:34:30.175613 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:34:35.606899 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:34:41.052016 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:34:47.074828 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:34:52.408849 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:34:57.722085 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:03.268472 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:09.198803 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:15.498788 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:20.947894 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:26.398686 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:32.535410 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:38.466388 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:44.778274 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:50.832257 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:35:56.626164 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:01.677345 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:06.924151 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:12.842566 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:19.310145 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:24.434379 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:30.331614 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:35.429836 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:41.475208 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:46.940728 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:52.205340 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:36:58.030695 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:03.855385 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:09.277700 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:14.922959 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:20.724888 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:26.109930 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:31.541856 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:37.734133 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:43.894861 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:50.220400 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0823 09:37:55.673063 7362 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
could not find a JWS signature in the cluster-info ConfigMap for token ID "abcdef"

MassimilianoGullusci

Hi @chrispokorni and @serewicz ,

I've the same issue on worker node after a successful run of k8sSecond.sh.
In my case the discovery token hash was generated by control-plane node.

I've preserved the CP installation output as suggested in your guide.
Attached CP cluster-info and worker join command ouput too.

Please help me to understand where I'm wrong.

I've stopped to go on with this course cause I've become father on May, but I want to solve all start/preparation issues and go on.

Thanks a lot.

Massimiliano

chrispokorni

Hi @MassimilianoGullusci,

From the timestamps in your outputs you may have an expired bootstrap token. Try running the following commands in the order provided below, to cleanup the worker, generate a new token/hash combination on the cp node, and finally join the worker with the cp node:

1 - On worker node: $ sudo kubeadm reset

2 - On cp node: $ sudo kubeadm token create --print-join-command

3 - On worker node: $ sudo kubeadm join ... <as generated by the command above>

Regards,
-Chris

MassimilianoGullusci

Hi @chrispokorni,

even if I had a great hope on your suggestion success, it fails creating a new token.

Output of failed commends in attachment.

I share with you my ".kube/config" file got from CP node.

Is there something wrong?
It was generated automatically.

Which "condition" it expects during token generation?

Thanks.

Massimiliano

MassimilianoGullusci

Hi @chrispokorni,

to avoid possible errors due to out of time joining of my worker node, I've executed again all steps from the start.

Now I can see CP and WORKER in the cluster.

I'm very happy.

But I have a question:

What is missing to fix errors risen by execution of your suggested commands?

Thanks

Massimiliano

MassimilianoGullusci

Sorry. Found it.
Thanks a lot.