Lab 3.2 new token not having a JWS signature in cluster-info ConfigMap

Chup4Chups

Hi,

I started few days ago the LFS258 training and I am stuck on lab exercise 3.2.
I created the 2 VMs, master and worker, with VirtualBox (host-only + NAT) and set static IPs

From 3.1, I created the cluster with the kubeadm-config.yaml below:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.56.100
  bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.18.1
controlPlaneEndpoint: "k8smaster:6443"
networking:
  podSubnet: 192.168.0.0/16

I successfully applied calico config to the cluster as well.
You can find the kubeadm-init.out in attachment.

From 3.2, I created a new token from the master

student@master:~$ sudo kubeadm --v=5 token create --print-join-command
I0510 08:57:08.914031 27949 token.go:121] [token] validating mixed arguments
I0510 08:57:08.914123 27949 token.go:130] [token] getting Clientsets from kubeconfig file
I0510 08:57:08.914149 27949 cmdutil.go:79] Using kubeconfig file: /home/student/.kube/config
I0510 08:57:08.919229 27949 token.go:243] [token] loading configurations
I0510 08:57:08.920355 27949 interface.go:400] Looking for default routes with IPv4 addresses
I0510 08:57:08.921316 27949 interface.go:405] Default route transits interface "enp0s3"
I0510 08:57:08.922168 27949 interface.go:208] Interface enp0s3 is up
I0510 08:57:08.923092 27949 interface.go:256] Interface "enp0s3" has 2 addresses :[192.168.56.100/24 fe80::a00:27ff:fef8:907f/64].
I0510 08:57:08.923849 27949 interface.go:223] Checking addr 192.168.56.100/24.
I0510 08:57:08.924634 27949 interface.go:230] IP found 192.168.56.100
I0510 08:57:08.925321 27949 interface.go:262] Found valid IPv4 address 192.168.56.100 for interface "enp0s3".
I0510 08:57:08.925966 27949 interface.go:411] Found active IP 192.168.56.100
W0510 08:57:08.926707 27949 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
I0510 08:57:08.927506 27949 token.go:255] [token] creating token
kubeadm join k8smaster:6443 --token 8j77yo.vrv09tpb6wqpf2p9 --discovery-token-ca-cert-hash sha256:d21ca296b30091b304dfa03fb3b600e32eb67cef13c3f51badd835e25dfad1ba

student@master:~$ kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
8j77yo.vrv09tpb6wqpf2p9 23h 2020-05-11T08:57:08Z authentication,signing system:bootstrappers:kubeadm:default-node-token

and when I use it to join the cluster from the worker, I get the following issue:

root@worker:/home/student# kubeadm join --v=5 k8smaster:6443 --token 8j77yo.vrv09tpb6wqpf2p9 --discovery-token-ca-cert-hash sha256:d21ca296b30091b304dfa03fb3b600e32eb67cef13c3f51badd835e25dfad1ba
W0510 08:59:12.753901 1121 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
I0510 08:59:12.754736 1121 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName
I0510 08:59:12.755402 1121 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock
[preflight] Running pre-flight checks
I0510 08:59:12.756629 1121 preflight.go:90] [preflight] Running general checks
I0510 08:59:12.757284 1121 checks.go:249] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0510 08:59:12.758224 1121 checks.go:286] validating the existence of file /etc/kubernetes/kubelet.conf
I0510 08:59:12.758814 1121 checks.go:286] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0510 08:59:12.759434 1121 checks.go:102] validating the container runtime
I0510 08:59:12.854290 1121 checks.go:128] validating if the service is enabled and active
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0510 08:59:13.009580 1121 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0510 08:59:13.009770 1121 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0510 08:59:13.009860 1121 checks.go:649] validating whether swap is enabled or not
I0510 08:59:13.009893 1121 checks.go:376] validating the presence of executable conntrack
I0510 08:59:13.009923 1121 checks.go:376] validating the presence of executable ip
I0510 08:59:13.009945 1121 checks.go:376] validating the presence of executable iptables
I0510 08:59:13.009963 1121 checks.go:376] validating the presence of executable mount
I0510 08:59:13.009980 1121 checks.go:376] validating the presence of executable nsenter
I0510 08:59:13.009995 1121 checks.go:376] validating the presence of executable ebtables
I0510 08:59:13.010011 1121 checks.go:376] validating the presence of executable ethtool
I0510 08:59:13.010029 1121 checks.go:376] validating the presence of executable socat
I0510 08:59:13.010044 1121 checks.go:376] validating the presence of executable tc
I0510 08:59:13.010062 1121 checks.go:376] validating the presence of executable touch
I0510 08:59:13.010093 1121 checks.go:520] running all checks
I0510 08:59:13.139333 1121 checks.go:406] checking whether the given node name is reachable using net.LookupHost
I0510 08:59:13.141594 1121 checks.go:618] validating kubelet version
I0510 08:59:13.237892 1121 checks.go:128] validating if the service is enabled and active
I0510 08:59:13.262185 1121 checks.go:201] validating availability of port 10250
I0510 08:59:13.262628 1121 checks.go:286] validating the existence of file /etc/kubernetes/pki/ca.crt
I0510 08:59:13.262712 1121 checks.go:432] validating if the connectivity type is via proxy or direct
I0510 08:59:13.262745 1121 join.go:441] [preflight] Discovering cluster-info
I0510 08:59:13.262774 1121 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "k8smaster:6443"
I0510 08:59:13.277069 1121 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "8j77yo", will try again
I0510 08:59:13.283618 1121 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "8j77yo", will try again

looking at journalctl on the master, I can only see these errors in loop from the master:

May 10 09:24:56 master kubelet[26506]: E0510 09:24:56.925984 26506 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-master_kube-system(5ee8922c9ad7b5a8f3f59abd72df0a78)": Timeout: request did not complete within requested timeout 34s
May 10 09:24:59 master kubelet[26506]: E0510 09:24:59.158408 26506 controller.go:136] failed to ensure node lease exists, will retry in 7s, error: proto: Lease: illegal tag -633754067 (wire type 29289705834)
May 10 09:25:06 master kubelet[26506]: E0510 09:25:06.167262 26506 controller.go:136] failed to ensure node lease exists, will retry in 7s, error: proto: Lease: illegal tag -633754067 (wire type 29289705834)
May 10 09:25:06 master kubelet[26506]: E0510 09:25:06.252399 26506 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: proto: VolumeMount: wiretype end group for non-group

from the worker

May 10 09:26:42 worker systemd[1]: Configuration file /lib/systemd/system/kubelet.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
May 10 09:26:42 worker systemd[1]: Configuration file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.

Any help and advice to troubleshoot further would be appreciated