Recover cluster. How can I restart kube-apiserver?

ilmx

I am trying to post a question but it says "I am blocked"... What can I do?

ilmx

(it is likely because I am pasting some commands output that are flagged as "suspicious")

ilmx

I will create my question removing some of the command output:

I had a computer crash while the two VMs of the cluster were running. After rebooting, the cluster is not accessible via kubelet:

kubectl get nodes
couldn't get current server API group list: Get "https://k8scp:6443/api?timeout=32s": dial tcp 192.168.138.131:6443: connect: connection refused

Noting is listening in the port:
nc: connect to 192.168.138.131 port 6443 (tcp) failed: Connection refused

Kubelet is running:
sudo service kubelet status
Active: active (running) since Wed 2024-08-14 08:43:13 UTC; 2min 52s ago
...

I tried "sudo swapoff -a" and others...

How can I recover the cluster?
I thought of "reset" or even "init", but I would need to rejoin all nodes, I suppose. This is something that could happen in real life...

chrispokorni

Hi @ilmx,

After rebooting the VMs, do they preserve their original IP addresses? If the IPs change, the cluster's identity changes, and the access credentials of kubectl are invalidated.

If IPs are preserved, verify the kubelet and containerd services. They need to be active and running.

sudo systemctl status kubelet
sudo systemctl status containerd

If you do decide to reset your cluster, then all nodes need to be reset. First the control plane reset and init, then the .kube/config file re-generated. The worker needs to be reset and join with the newly generated join command from the current init output. Also, when listing nodes with
kubectl get nodes -o wide
ensure only current nodes are listed. If any of the old node entries are shown, remove them from the cluster with
kubectl delete node node-name.

Last but not least, I would recommend the VMs to have IP addresses that are NOT from the 192.168.0.0/16 subnet.

Regards,
-Chris

ilmx

Thanks Chris for the information and the suggestion