LFS258 Lab 12.3 - Kubernetes Dashboard Inaccessible

After following the instructions in lab 12.3, everything worked until arriving at the "Configure the dashboard section."
The dashboard objects were successfully created:
[[email protected]]$ kubectl create -f https://bit.ly/2OFQRMy namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created [[email protected]]$
The service for the dashboard was successfully modified to use a NodePort:
[[email protected]]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 19m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 19m [[email protected]]$
And the clusterrolebinding was created:
[[email protected]]$ kubectl create clusterrolebinding dashaccess --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard clusterrolebinding.rbac.authorization.k8s.io/dashaccess created [[email protected]]$
The issue appears to be that the pod isn't responding to any requests. Attempting to hit the dashboard through the service address fails:
[[email protected]]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 33m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 33m [[email protected]]$ date ; telnet 10.110.107.206 31497 ; date Thu Jan 14 12:13:00 UTC 2021 Trying 10.110.107.206... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:15:11 UTC 2021 [[email protected]]$
As does trying to go direct to the pod:
[[email protected]]$ kubectl -n kubernetes-dashboard describe po kubernetes-dashboard-864f6467f8-g6qkp | egrep -i '(^IP:|Port:)' IP: 10.1.198.32 Port: 8443/TCP Host Port: 0/TCP [[email protected]]$ date ; telnet 10.1.198.32 8443 ; date Thu Jan 14 12:18:27 UTC 2021 Trying 10.1.198.32... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:20:38 UTC 2021 [[email protected]]$
The logs of the pod don't really seem to indicate a problem:
[[email protected]]$ kubectl -n kubernetes-dashboard logs kubernetes-dashboard-864f6467f8-g6qkp 2021/01/14 12:08:24 Using namespace: kubernetes-dashboard 2021/01/14 12:08:24 Using in-cluster config to connect to apiserver 2021/01/14 12:08:24 Starting overwatch 2021/01/14 12:08:24 Using secret token for csrf signing 2021/01/14 12:08:24 Initializing csrf token from kubernetes-dashboard-csrf secret 2021/01/14 12:08:24 Successful initial request to the apiserver, version: v1.19.0 2021/01/14 12:08:24 Generating JWE encryption key 2021/01/14 12:08:24 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting 2021/01/14 12:08:24 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard 2021/01/14 12:08:24 Initializing JWE encryption key from synchronized object 2021/01/14 12:08:24 Creating in-cluster Sidecar client 2021/01/14 12:08:24 Auto-generating certificates 2021/01/14 12:08:24 Successfully created certificates 2021/01/14 12:08:24 Serving securely on HTTPS port: 8443 2021/01/14 12:08:24 Successful request to sidecar [[email protected]]$
I've tried deleting the dashboard pod as that seems to be step one in a lot of troubleshooting scenarios but that also did not work. Any help would be greatly appreciated, thanks!
Comments
-
Hello,
When trying to understand the logs output you have posted, it is jumbled and difficult to read. Perhaps paste as a quote or code in order to preserve the formatting.
First off, what are you using to run the labs GCE, AWS, Digital Ocean, VirtualBox, etc?
Have you ensured that ALL traffic is allowed to the nodes?
Are any of your pods showing other than all containers running and in a ready status?
What version of the software did you deploy?
It appears as if you are using the NodePort 31497. Instead of telnet to the port what happens if you use nc, or curl?
Also, if using a provider such as GCE, use the PUBLIC iP address from outside the lab environment to view the dashboard. For example, if my PUBLIC IP to the GCE node was 35.101.11.22 I would use a browser on my local machine and go to 35.101.11.22:31497, not 10.110.107.206:31497 from inside the node due to how software defined networks and routing work.
Regards,
1 -
Hi @TheFutonEng,
Also be aware that accessing the Dashboard through the Google Chrome browser may not be successful, due to some recent upgrades in how the browser handles insecure connections.
I would recommend using the Firefox browser as an alternative, because that is what seemed to work when Chrome failed to display the Dashboard.
Regards,
-Chris1 -
Thanks for the responses @serewicz and @chrispokorni.
I'm running my cluster out of my house on a combination of desktop machines and VMs, all running Ubuntu 20.04.
I haven't put any network policies in place or modified iptables rules so I think all traffic is allowed. I've only made changes that the labs outlined.
There is a single calico pod that isn't running:
[[email protected]]$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default seconddb-mariadb-client 1/1 Running 0 3d kube-system calico-kube-controllers-76d4774d89-t2dm9 1/1 Running 0 21d kube-system calico-node-9824l 1/1 Running 0 25d kube-system calico-node-vqbqj 1/1 Running 4 25d kube-system calico-node-zl9zc 0/1 Running 2 25d <-omitted->
Restarting that pod seems to have restored connectivity and the dashboard is now accessible. And not via Chrome as @chrispokorni pointed out. Thank you both for your support!
-Futon
0 -
Hi. I had the same problem until I used the browser "Midori" instead of Chrome (this is in Ubuntu not Windows). I clicked the little padlock up in the address bar and selected "Trust this website" and then I could enter the token code to access the dashboard.
0
Categories
- 9.9K All Categories
- 29 LFX Mentorship
- 82 LFX Mentorship: Linux Kernel
- 473 Linux Foundation Boot Camps
- 268 Cloud Engineer Boot Camp
- 96 Advanced Cloud Engineer Boot Camp
- 44 DevOps Engineer Boot Camp
- 32 Cloud Native Developer Boot Camp
- 1 Express Training Courses
- 1 Express Courses - Discussion Forum
- 1.6K Training Courses
- 18 LFC110 Class Forum
- 4 LFC131 Class Forum
- 19 LFD102 Class Forum
- 133 LFD103 Class Forum
- 9 LFD121 Class Forum
- 60 LFD201 Class Forum
- LFD210 Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- 23 LFD254 Class Forum
- 545 LFD259 Class Forum
- 100 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFS145 Class Forum
- 20 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 1 LFS203 Class Forum
- 37 LFS207 Class Forum
- 296 LFS211 Class Forum
- 53 LFS216 Class Forum
- 45 LFS241 Class Forum
- 40 LFS242 Class Forum
- 33 LFS243 Class Forum
- 10 LFS244 Class Forum
- 27 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- 131 LFS253 Class Forum
- 967 LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 86 LFS260 Class Forum
- 124 LFS261 Class Forum
- 29 LFS262 Class Forum
- 78 LFS263 Class Forum
- 15 LFS264 Class Forum
- 10 LFS266 Class Forum
- 17 LFS267 Class Forum
- 16 LFS268 Class Forum
- 14 LFS269 Class Forum
- 195 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 207 LFW211 Class Forum
- 148 LFW212 Class Forum
- 892 Hardware
- 213 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 206 Networking
- 100 Printers & Scanners
- 85 Storage
- 747 Linux Distributions
- 88 Debian
- 64 Fedora
- 13 Linux Mint
- 13 Mageia
- 24 openSUSE
- 133 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 354 Ubuntu
- 469 Linux System Administration
- 38 Cloud Computing
- 68 Command Line/Scripting
- Github systems admin projects
- 93 Linux Security
- 77 Network Management
- 107 System Management
- 48 Web Management
- 62 Mobile Computing
- 22 Android
- 26 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 524 Off Topic
- 127 Introductions
- 210 Small Talk
- 19 Study Material
- 783 Programming and Development
- 257 Kernel Development
- 492 Software Development
- 919 Software
- 255 Applications
- 181 Command Line
- 2 Compiling/Installing
- 76 Games
- 316 Installation
- 46 All In Program
- 46 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)