Welcome to the Linux Foundation Forum!

Recover cluster. How can I restart kube-apiserver?

ilmx
ilmx Posts: 18
in LFS258 Class Forum

I am trying to post a question but it says "I am blocked"... What can I do?

Answers

  • ilmx
    ilmx Posts: 18

    (it is likely because I am pasting some commands output that are flagged as "suspicious")

  • ilmx
    ilmx Posts: 18

    I will create my question removing some of the command output:

    I had a computer crash while the two VMs of the cluster were running. After rebooting, the cluster is not accessible via kubelet:

    kubectl get nodes
    couldn't get current server API group list: Get "https://k8scp:6443/api?timeout=32s": dial tcp 192.168.138.131:6443: connect: connection refused

    Noting is listening in the port:
    nc: connect to 192.168.138.131 port 6443 (tcp) failed: Connection refused

    Kubelet is running:
    sudo service kubelet status
    Active: active (running) since Wed 2024-08-14 08:43:13 UTC; 2min 52s ago
    ...

    I tried "sudo swapoff -a" and others...

    How can I recover the cluster?
    I thought of "reset" or even "init", but I would need to rejoin all nodes, I suppose. This is something that could happen in real life...

  • chrispokorni
    chrispokorni Posts: 2,606

    Hi @ilmx,

    After rebooting the VMs, do they preserve their original IP addresses? If the IPs change, the cluster's identity changes, and the access credentials of kubectl are invalidated.

    If IPs are preserved, verify the kubelet and containerd services. They need to be active and running.

    sudo systemctl status kubelet
    sudo systemctl status containerd

    If you do decide to reset your cluster, then all nodes need to be reset. First the control plane reset and init, then the .kube/config file re-generated. The worker needs to be reset and join with the newly generated join command from the current init output. Also, when listing nodes with
    kubectl get nodes -o wide
    ensure only current nodes are listed. If any of the old node entries are shown, remove them from the cluster with
    kubectl delete node node-name.

    Last but not least, I would recommend the VMs to have IP addresses that are NOT from the 192.168.0.0/16 subnet.

    Regards,
    -Chris

  • ilmx
    ilmx Posts: 18

    Thanks Chris for the information and the suggestion

Categories

Upcoming Training