Lab 2.2 - Port 6443 stops responding intermittently on the control plane

I am facing the same issue, it is not like it does not work, sometimes it does, but 95% of the time it is down. I have been wasting time Googling the issue, everywhere they talk of swapfile, but I do not even have swap space on my EC2 instance.

Hi @scottengle,

This is a strange behavior indeed. If it is due to GCP issues, there isn't much we can do about it. However, at times it may be related to how the cluster nodes VMs have been provisioned, the VPC, VPC firewall, OS distribution and version, VM CPU, memory and disk space, etc...
From your description it seems that the OS version 20.04 LTS should not be the issue, as it is the recommended version by the lab guide.
In order to eliminate any possible networking issues did you happen to follow the demo video from the introductory chapter on how to provision the GCE instances? It offers important tips on GCP GCE provisioning, and on VPC and firewall configuration.

If you experience slowness in your cluster again, try to run (when possible) the following command and provide its output in the forum to help troubleshoot the issue:

kubectl get pods --all-namespaces -o wide
OR
kubectl get po -A -owide

Regards,
-Chris

Hi @chrispokorni,I also join this thread.

I'm laso facing problems when creating the first pod example (basicpod/nginx).

 kubectl get pod
NAME       READY   STATUS              RESTARTS   AGE
basicpod   0/1     ContainerCreating   0          113s

kubectl describe pod basicpod
 
Warning  FailedCreatePodSandBox  43s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/0080880d559fb50ea77e6ea23dcb0369f1382f38aed54f13482afc7d2af46609/log.json: no such file or directory): fork/exec /usr/local/bin/runc: exec format error: unknown
  Warning  FailedCreatePodSandBox  4s (x3 over 28s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/fbc9a5468aca6d6c0c3bc9bd162354d4adaeb81ba0ef95581c1da837031bc770/log.json: no such file or directory): fork/exec /usr/local/bin/runc: exec format error: unknown

I have also tried with the image: arm64v8/nginx.

Could it be some kind of incompatibility between the latest version of nginx with version 20.04?

Or could it be related to problems I've had launching k8scp.sh and containerd?

https://forum.linuxfoundation.org/discussion/comment/35215#Comment_35215

To install manually I used

sudo apt install containerd

my node and containers status

ubuntu@ip-172-31-47-37:/usr/local/bin$ kubectl get pod -n kube-system namespace
Error from server (NotFound): pods "namespace" not found
ubuntu@ip-172-31-47-37:/usr/local/bin$ kubectl get pod -n kube-system
NAME                                       READY   STATUS              RESTARTS      AGE
calico-kube-controllers-5b97f5d8cf-sfwfb   1/1     Running             2 (34m ago)   27h
calico-node-5h77g                          0/1     Init:0/3            0             3h17m
calico-node-9vz4r                          1/1     Running             2 (34m ago)   27h
coredns-6d4b75cb6d-b5tf6                   1/1     Running             2 (34m ago)   27h
coredns-6d4b75cb6d-wknrz                   1/1     Running             2 (34m ago)   27h
etcd-ip-172-31-47-37                       1/1     Running             2 (34m ago)   27h
kube-apiserver-ip-172-31-47-37             1/1     Running             2 (34m ago)   27h
kube-controller-manager-ip-172-31-47-37    1/1     Running             2 (34m ago)   27h
kube-proxy-8wpqj                           1/1     Running             2 (34m ago)   27h
kube-proxy-dk9p6                           0/1     ContainerCreating   0             3h17m
kube-scheduler-ip-172-31-47-37             1/1     Running             2 (34m ago)   27h
ubuntu@ip-172-31-47-37:/usr/local/bin$ kubectl get node
NAME               STATUS   ROLES           AGE     VERSION
ip-172-31-41-155   Ready    <none>          3h18m   v1.24.1
ip-172-31-47-37    Ready    control-plane   27h     v1.24.1

BR
Alberto

Hi @chrispokorni,

kubectl get pods -A -o wide
NAMESPACE     NAME                                       READY   STATUS              RESTARTS       AGE    IP               NODE               NOMINATED NODE   READINESS GATES
default       basicpod                                   0/1     ContainerCreating   0              7h2m   <none>           ip-172-31-41-155   <none>           <none>
kube-system   calico-kube-controllers-5b97f5d8cf-sfwfb   1/1     Running             3 (118s ago)   33h    192.168.196.10   ip-172-31-47-37    <none>           <none>
kube-system   calico-node-5h77g                          0/1     Init:0/3            0              10h    172.31.41.155    ip-172-31-41-155   <none>           <none>
kube-system   calico-node-9vz4r                          1/1     Running             3 (118s ago)   33h    172.31.47.37     ip-172-31-47-37    <none>           <none>
kube-system   coredns-6d4b75cb6d-b5tf6                   1/1     Running             3 (118s ago)   33h    192.168.196.12   ip-172-31-47-37    <none>           <none>
kube-system   coredns-6d4b75cb6d-wknrz                   1/1     Running             3 (118s ago)   33h    192.168.196.11   ip-172-31-47-37    <none>           <none>
kube-system   etcd-ip-172-31-47-37                       1/1     Running             3 (118s ago)   33h    172.31.47.37     ip-172-31-47-37    <none>           <none>
kube-system   kube-apiserver-ip-172-31-47-37             1/1     Running             3 (118s ago)   33h    172.31.47.37     ip-172-31-47-37    <none>           <none>
kube-system   kube-controller-manager-ip-172-31-47-37    1/1     Running             3 (118s ago)   33h    172.31.47.37     ip-172-31-47-37    <none>           <none>
kube-system   kube-proxy-8wpqj                           1/1     Running             3 (118s ago)   33h    172.31.47.37     ip-172-31-47-37    <none>           <none>
kube-system   kube-proxy-dk9p6                           0/1     ContainerCreating   0              10h    172.31.41.155    ip-172-31-41-155   <none>           <none>
kube-system   kube-scheduler-ip-172-31-47-37             1/1     Running             3 (118s ago)   33h    172.31.47.37     ip-172-31-47-37    <none>           <none>

kubectl get nodes -o wide
NAME               STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
ip-172-31-41-155   Ready    <none>          10h   v1.24.1   172.31.41.155   <none>        Ubuntu 20.04.5 LTS   5.15.0-1017-aws   containerd://1.5.9
ip-172-31-47-37    Ready    control-plane   33h   v1.24.1   172.31.47.37    <none>        Ubuntu 20.04.5 LTS   5.15.0-1017-aws   containerd://1.5.9

Thanks for helping.

BR
Alberto