Welcome to the Linux Foundation Forum!

LFS258 Lab 12.3 - Kubernetes Dashboard Inaccessible

TheFutonEng
TheFutonEng Posts: 7
in LFS258 Class Forum

After following the instructions in lab 12.3, everything worked until arriving at the "Configure the dashboard section."

The dashboard objects were successfully created:

[user@cka-master01]$ kubectl create -f https://bit.ly/2OFQRMy namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created [user@cka-master01]$

The service for the dashboard was successfully modified to use a NodePort:

[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 19m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 19m [user@cka-master01]$

And the clusterrolebinding was created:

[user@cka-master01]$ kubectl create clusterrolebinding dashaccess --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard clusterrolebinding.rbac.authorization.k8s.io/dashaccess created [user@cka-master01]$

The issue appears to be that the pod isn't responding to any requests. Attempting to hit the dashboard through the service address fails:

[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 33m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 33m [user@cka-master01]$ date ; telnet 10.110.107.206 31497 ; date Thu Jan 14 12:13:00 UTC 2021 Trying 10.110.107.206... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:15:11 UTC 2021 [user@cka-master01]$

As does trying to go direct to the pod:

[user@cka-master01]$ kubectl -n kubernetes-dashboard describe po kubernetes-dashboard-864f6467f8-g6qkp | egrep -i '(^IP:|Port:)' IP: 10.1.198.32 Port: 8443/TCP Host Port: 0/TCP [user@cka-master01]$ date ; telnet 10.1.198.32 8443 ; date Thu Jan 14 12:18:27 UTC 2021 Trying 10.1.198.32... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:20:38 UTC 2021 [user@cka-master01]$

The logs of the pod don't really seem to indicate a problem:

[user@cka-master01]$ kubectl -n kubernetes-dashboard logs kubernetes-dashboard-864f6467f8-g6qkp 2021/01/14 12:08:24 Using namespace: kubernetes-dashboard 2021/01/14 12:08:24 Using in-cluster config to connect to apiserver 2021/01/14 12:08:24 Starting overwatch 2021/01/14 12:08:24 Using secret token for csrf signing 2021/01/14 12:08:24 Initializing csrf token from kubernetes-dashboard-csrf secret 2021/01/14 12:08:24 Successful initial request to the apiserver, version: v1.19.0 2021/01/14 12:08:24 Generating JWE encryption key 2021/01/14 12:08:24 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting 2021/01/14 12:08:24 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard 2021/01/14 12:08:24 Initializing JWE encryption key from synchronized object 2021/01/14 12:08:24 Creating in-cluster Sidecar client 2021/01/14 12:08:24 Auto-generating certificates 2021/01/14 12:08:24 Successfully created certificates 2021/01/14 12:08:24 Serving securely on HTTPS port: 8443 2021/01/14 12:08:24 Successful request to sidecar [user@cka-master01]$

I've tried deleting the dashboard pod as that seems to be step one in a lot of troubleshooting scenarios but that also did not work. Any help would be greatly appreciated, thanks!

Comments

  • chrispokorni
    chrispokorni Posts: 2,606

    Hi @TheFutonEng,

    Also be aware that accessing the Dashboard through the Google Chrome browser may not be successful, due to some recent upgrades in how the browser handles insecure connections.

    I would recommend using the Firefox browser as an alternative, because that is what seemed to work when Chrome failed to display the Dashboard.

    Regards,
    -Chris

  • TheFutonEng
    TheFutonEng Posts: 7

    Thanks for the responses @serewicz and @chrispokorni.

    I'm running my cluster out of my house on a combination of desktop machines and VMs, all running Ubuntu 20.04.

    I haven't put any network policies in place or modified iptables rules so I think all traffic is allowed. I've only made changes that the labs outlined.

    There is a single calico pod that isn't running:

    [rmengert@cka-master01]$ kubectl get pods --all-namespaces 
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                seconddb-mariadb-client                      1/1     Running   0          3d
kube-system            calico-kube-controllers-76d4774d89-t2dm9     1/1     Running   0          21d
kube-system            calico-node-9824l                            1/1     Running   0          25d
kube-system            calico-node-vqbqj                            1/1     Running   4          25d
kube-system            calico-node-zl9zc                            0/1     Running   2          25d
<-omitted->

    Restarting that pod seems to have restored connectivity and the dashboard is now accessible. And not via Chrome as @chrispokorni pointed out. Thank you both for your support!

    -Futon

  • mikerossiter
    mikerossiter Posts: 21

    Hi. I had the same problem until I used the browser "Midori" instead of Chrome (this is in Ubuntu not Windows). I clicked the little padlock up in the address bar and selected "Trust this website" and then I could enter the token code to access the dashboard.

Categories

Upcoming Training