LFS258 Lab 12.3 - Kubernetes Dashboard Inaccessible

After following the instructions in lab 12.3, everything worked until arriving at the "Configure the dashboard section."
The dashboard objects were successfully created:
[user@cka-master01]$ kubectl create -f https://bit.ly/2OFQRMy namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created [user@cka-master01]$
The service for the dashboard was successfully modified to use a NodePort:
[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 19m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 19m [user@cka-master01]$
And the clusterrolebinding was created:
[user@cka-master01]$ kubectl create clusterrolebinding dashaccess --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard clusterrolebinding.rbac.authorization.k8s.io/dashaccess created [user@cka-master01]$
The issue appears to be that the pod isn't responding to any requests. Attempting to hit the dashboard through the service address fails:
[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 33m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 33m [user@cka-master01]$ date ; telnet 10.110.107.206 31497 ; date Thu Jan 14 12:13:00 UTC 2021 Trying 10.110.107.206... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:15:11 UTC 2021 [user@cka-master01]$
As does trying to go direct to the pod:
[user@cka-master01]$ kubectl -n kubernetes-dashboard describe po kubernetes-dashboard-864f6467f8-g6qkp | egrep -i '(^IP:|Port:)' IP: 10.1.198.32 Port: 8443/TCP Host Port: 0/TCP [user@cka-master01]$ date ; telnet 10.1.198.32 8443 ; date Thu Jan 14 12:18:27 UTC 2021 Trying 10.1.198.32... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:20:38 UTC 2021 [user@cka-master01]$
The logs of the pod don't really seem to indicate a problem:
[user@cka-master01]$ kubectl -n kubernetes-dashboard logs kubernetes-dashboard-864f6467f8-g6qkp 2021/01/14 12:08:24 Using namespace: kubernetes-dashboard 2021/01/14 12:08:24 Using in-cluster config to connect to apiserver 2021/01/14 12:08:24 Starting overwatch 2021/01/14 12:08:24 Using secret token for csrf signing 2021/01/14 12:08:24 Initializing csrf token from kubernetes-dashboard-csrf secret 2021/01/14 12:08:24 Successful initial request to the apiserver, version: v1.19.0 2021/01/14 12:08:24 Generating JWE encryption key 2021/01/14 12:08:24 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting 2021/01/14 12:08:24 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard 2021/01/14 12:08:24 Initializing JWE encryption key from synchronized object 2021/01/14 12:08:24 Creating in-cluster Sidecar client 2021/01/14 12:08:24 Auto-generating certificates 2021/01/14 12:08:24 Successfully created certificates 2021/01/14 12:08:24 Serving securely on HTTPS port: 8443 2021/01/14 12:08:24 Successful request to sidecar [user@cka-master01]$
I've tried deleting the dashboard pod as that seems to be step one in a lot of troubleshooting scenarios but that also did not work. Any help would be greatly appreciated, thanks!
Comments
-
Hello,
When trying to understand the logs output you have posted, it is jumbled and difficult to read. Perhaps paste as a quote or code in order to preserve the formatting.
First off, what are you using to run the labs GCE, AWS, Digital Ocean, VirtualBox, etc?
Have you ensured that ALL traffic is allowed to the nodes?
Are any of your pods showing other than all containers running and in a ready status?
What version of the software did you deploy?
It appears as if you are using the NodePort 31497. Instead of telnet to the port what happens if you use nc, or curl?
Also, if using a provider such as GCE, use the PUBLIC iP address from outside the lab environment to view the dashboard. For example, if my PUBLIC IP to the GCE node was 35.101.11.22 I would use a browser on my local machine and go to 35.101.11.22:31497, not 10.110.107.206:31497 from inside the node due to how software defined networks and routing work.
Regards,
1 -
Hi @TheFutonEng,
Also be aware that accessing the Dashboard through the Google Chrome browser may not be successful, due to some recent upgrades in how the browser handles insecure connections.
I would recommend using the Firefox browser as an alternative, because that is what seemed to work when Chrome failed to display the Dashboard.
Regards,
-Chris1 -
Thanks for the responses @serewicz and @chrispokorni.
I'm running my cluster out of my house on a combination of desktop machines and VMs, all running Ubuntu 20.04.
I haven't put any network policies in place or modified iptables rules so I think all traffic is allowed. I've only made changes that the labs outlined.
There is a single calico pod that isn't running:
[rmengert@cka-master01]$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default seconddb-mariadb-client 1/1 Running 0 3d kube-system calico-kube-controllers-76d4774d89-t2dm9 1/1 Running 0 21d kube-system calico-node-9824l 1/1 Running 0 25d kube-system calico-node-vqbqj 1/1 Running 4 25d kube-system calico-node-zl9zc 0/1 Running 2 25d <-omitted->
Restarting that pod seems to have restored connectivity and the dashboard is now accessible. And not via Chrome as @chrispokorni pointed out. Thank you both for your support!
-Futon
0 -
Hi. I had the same problem until I used the browser "Midori" instead of Chrome (this is in Ubuntu not Windows). I clicked the little padlock up in the address bar and selected "Trust this website" and then I could enter the token code to access the dashboard.
0
Categories
- All Categories
- 49 LFX Mentorship
- 102 LFX Mentorship: Linux Kernel
- 552 Linux Foundation Boot Camps
- 296 Cloud Engineer Boot Camp
- 118 Advanced Cloud Engineer Boot Camp
- 52 DevOps Engineer Boot Camp
- 53 Cloud Native Developer Boot Camp
- 4 Express Training Courses
- 4 Express Courses - Discussion Forum
- 1.9K Training Courses
- 18 LFC110 Class Forum
- 6 LFC131 Class Forum
- 25 LFD102 Class Forum
- 150 LFD103 Class Forum
- 17 LFD121 Class Forum
- LFD137 Class Forum
- 61 LFD201 Class Forum
- LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 597 LFD259 Class Forum
- 102 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- LFD273 Class Forum
- 2 LFS145 Class Forum
- 24 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 3 LFS203 Class Forum
- 69 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 33 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 140 LFS253 Class Forum
- 1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 92 LFS260 Class Forum
- 129 LFS261 Class Forum
- 32 LFS262 Class Forum
- 79 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 17 LFS267 Class Forum
- 17 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS281 Class Forum
- 221 LFW211 Class Forum
- 167 LFW212 Class Forum
- SKF100 Class Forum
- 901 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 208 Networking
- 101 Printers & Scanners
- 85 Storage
- 761 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 141 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 356 Ubuntu
- 477 Linux System Administration
- 41 Cloud Computing
- 69 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 77 Network Management
- 108 System Management
- 49 Web Management
- 66 Mobile Computing
- 23 Android
- 29 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 536 Off Topic
- 131 Introductions
- 216 Small Talk
- 21 Study Material
- 816 Programming and Development
- 275 Kernel Development
- 507 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 59 All In Program
- 59 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)