Hostname mismatch causing kubelet issue after restarting

beedle

Hello,
I just want to share an issue I had at step 3.2.11, after restarting the nodes. I was unable to reach the registry address. After a bit of troubleshooting, I noticed that both my nodes where "NotReady". When checking their status and that the kubelet service kept restarting:

 01:21:42 student@cp ~ →  journalctl -xeu kubelet --no-pager | tail -50
  Feb 05 13:22:30 cp.<redacted>.internal kubelet[40281]: E0205 13:22:30.110490   40281 reconstruct.go:189] "Failed to get Node status to
   reconstruct device paths" err="nodes \"cp.us-central1-a.c.<redacted>.internal\" not found"

The issue was that the hostname is the full FQDN cp.us-central1-a.c..internal, while my node in Kubernetes is registered as just cp. After the reboot, kubelet is using the FQDN and can't find itself in the cluster.

Solution: Add hostname-override to kubelet

On the control plane node, run:

sudo vi /var/lib/kubelet/kubeadm-flags.env

Change this line:

KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.k8s.io/pause:3.10.1"

To this:

KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.k8s.io/pause:3.10.1 --hostname-override=cp"

Then restart kubelet:

sudo systemctl restart kubelet

Finally my nodes were "Ready" when running:
kubectl get nodes

Not sure why this happened, my nodes have the proper hostname in GCE.