LFS258 Lab 12.3 - Kubernetes Dashboard Inaccessible

TheFutonEng

After following the instructions in lab 12.3, everything worked until arriving at the "Configure the dashboard section."

The dashboard objects were successfully created:

[user@cka-master01]$ kubectl create -f https://bit.ly/2OFQRMy namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created [user@cka-master01]$

The service for the dashboard was successfully modified to use a NodePort:

[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 19m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 19m [user@cka-master01]$

And the clusterrolebinding was created:

[user@cka-master01]$ kubectl create clusterrolebinding dashaccess --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard clusterrolebinding.rbac.authorization.k8s.io/dashaccess created [user@cka-master01]$

The issue appears to be that the pod isn't responding to any requests. Attempting to hit the dashboard through the service address fails:

[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 33m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 33m [user@cka-master01]$ date ; telnet 10.110.107.206 31497 ; date Thu Jan 14 12:13:00 UTC 2021 Trying 10.110.107.206... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:15:11 UTC 2021 [user@cka-master01]$

As does trying to go direct to the pod:

[user@cka-master01]$ kubectl -n kubernetes-dashboard describe po kubernetes-dashboard-864f6467f8-g6qkp | egrep -i '(^IP:|Port:)' IP: 10.1.198.32 Port: 8443/TCP Host Port: 0/TCP [user@cka-master01]$ date ; telnet 10.1.198.32 8443 ; date Thu Jan 14 12:18:27 UTC 2021 Trying 10.1.198.32... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:20:38 UTC 2021 [user@cka-master01]$

The logs of the pod don't really seem to indicate a problem:

[user@cka-master01]$ kubectl -n kubernetes-dashboard logs kubernetes-dashboard-864f6467f8-g6qkp 2021/01/14 12:08:24 Using namespace: kubernetes-dashboard 2021/01/14 12:08:24 Using in-cluster config to connect to apiserver 2021/01/14 12:08:24 Starting overwatch 2021/01/14 12:08:24 Using secret token for csrf signing 2021/01/14 12:08:24 Initializing csrf token from kubernetes-dashboard-csrf secret 2021/01/14 12:08:24 Successful initial request to the apiserver, version: v1.19.0 2021/01/14 12:08:24 Generating JWE encryption key 2021/01/14 12:08:24 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting 2021/01/14 12:08:24 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard 2021/01/14 12:08:24 Initializing JWE encryption key from synchronized object 2021/01/14 12:08:24 Creating in-cluster Sidecar client 2021/01/14 12:08:24 Auto-generating certificates 2021/01/14 12:08:24 Successfully created certificates 2021/01/14 12:08:24 Serving securely on HTTPS port: 8443 2021/01/14 12:08:24 Successful request to sidecar [user@cka-master01]$

I've tried deleting the dashboard pod as that seems to be step one in a lot of troubleshooting scenarios but that also did not work. Any help would be greatly appreciated, thanks!

serewicz

Hello,

When trying to understand the logs output you have posted, it is jumbled and difficult to read. Perhaps paste as a quote or code in order to preserve the formatting.

First off, what are you using to run the labs GCE, AWS, Digital Ocean, VirtualBox, etc?

Have you ensured that ALL traffic is allowed to the nodes?

Are any of your pods showing other than all containers running and in a ready status?

What version of the software did you deploy?

It appears as if you are using the NodePort 31497. Instead of telnet to the port what happens if you use nc, or curl?

Also, if using a provider such as GCE, use the PUBLIC iP address from outside the lab environment to view the dashboard. For example, if my PUBLIC IP to the GCE node was 35.101.11.22 I would use a browser on my local machine and go to 35.101.11.22:31497, not 10.110.107.206:31497 from inside the node due to how software defined networks and routing work.

Regards,

chrispokorni

Hi @TheFutonEng,

Also be aware that accessing the Dashboard through the Google Chrome browser may not be successful, due to some recent upgrades in how the browser handles insecure connections.

I would recommend using the Firefox browser as an alternative, because that is what seemed to work when Chrome failed to display the Dashboard.

Regards,
-Chris

TheFutonEng

Thanks for the responses @serewicz and @chrispokorni.

I'm running my cluster out of my house on a combination of desktop machines and VMs, all running Ubuntu 20.04.

I haven't put any network policies in place or modified iptables rules so I think all traffic is allowed. I've only made changes that the labs outlined.

There is a single calico pod that isn't running:

[rmengert@cka-master01]$ kubectl get pods --all-namespaces 
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                seconddb-mariadb-client                      1/1     Running   0          3d
kube-system            calico-kube-controllers-76d4774d89-t2dm9     1/1     Running   0          21d
kube-system            calico-node-9824l                            1/1     Running   0          25d
kube-system            calico-node-vqbqj                            1/1     Running   4          25d
kube-system            calico-node-zl9zc                            0/1     Running   2          25d
<-omitted->

Restarting that pod seems to have restored connectivity and the dashboard is now accessible. And not via Chrome as @chrispokorni pointed out. Thank you both for your support!

-Futon

mikerossiter

Hi. I had the same problem until I used the browser "Midori" instead of Chrome (this is in Ubuntu not Windows). I clicked the little padlock up in the address bar and selected "Trust this website" and then I could enter the token code to access the dashboard.