LFS258 Lab 12.3 - Kubernetes Dashboard Inaccessible
After following the instructions in lab 12.3, everything worked until arriving at the "Configure the dashboard section."
The dashboard objects were successfully created:
[user@cka-master01]$ kubectl create -f https://bit.ly/2OFQRMy namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created [user@cka-master01]$
The service for the dashboard was successfully modified to use a NodePort:
[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 19m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 19m [user@cka-master01]$
And the clusterrolebinding was created:
[user@cka-master01]$ kubectl create clusterrolebinding dashaccess --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard clusterrolebinding.rbac.authorization.k8s.io/dashaccess created [user@cka-master01]$
The issue appears to be that the pod isn't responding to any requests. Attempting to hit the dashboard through the service address fails:
[user@cka-master01]$ kubectl -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.107.26.79 <none> 8000/TCP 33m kubernetes-dashboard NodePort 10.110.107.206 <none> 443:31497/TCP 33m [user@cka-master01]$ date ; telnet 10.110.107.206 31497 ; date Thu Jan 14 12:13:00 UTC 2021 Trying 10.110.107.206... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:15:11 UTC 2021 [user@cka-master01]$
As does trying to go direct to the pod:
[user@cka-master01]$ kubectl -n kubernetes-dashboard describe po kubernetes-dashboard-864f6467f8-g6qkp | egrep -i '(^IP:|Port:)' IP: 10.1.198.32 Port: 8443/TCP Host Port: 0/TCP [user@cka-master01]$ date ; telnet 10.1.198.32 8443 ; date Thu Jan 14 12:18:27 UTC 2021 Trying 10.1.198.32... telnet: Unable to connect to remote host: Connection timed out Thu Jan 14 12:20:38 UTC 2021 [user@cka-master01]$
The logs of the pod don't really seem to indicate a problem:
[user@cka-master01]$ kubectl -n kubernetes-dashboard logs kubernetes-dashboard-864f6467f8-g6qkp 2021/01/14 12:08:24 Using namespace: kubernetes-dashboard 2021/01/14 12:08:24 Using in-cluster config to connect to apiserver 2021/01/14 12:08:24 Starting overwatch 2021/01/14 12:08:24 Using secret token for csrf signing 2021/01/14 12:08:24 Initializing csrf token from kubernetes-dashboard-csrf secret 2021/01/14 12:08:24 Successful initial request to the apiserver, version: v1.19.0 2021/01/14 12:08:24 Generating JWE encryption key 2021/01/14 12:08:24 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting 2021/01/14 12:08:24 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard 2021/01/14 12:08:24 Initializing JWE encryption key from synchronized object 2021/01/14 12:08:24 Creating in-cluster Sidecar client 2021/01/14 12:08:24 Auto-generating certificates 2021/01/14 12:08:24 Successfully created certificates 2021/01/14 12:08:24 Serving securely on HTTPS port: 8443 2021/01/14 12:08:24 Successful request to sidecar [user@cka-master01]$
I've tried deleting the dashboard pod as that seems to be step one in a lot of troubleshooting scenarios but that also did not work. Any help would be greatly appreciated, thanks!
Comments
-
Hello,
When trying to understand the logs output you have posted, it is jumbled and difficult to read. Perhaps paste as a quote or code in order to preserve the formatting.
First off, what are you using to run the labs GCE, AWS, Digital Ocean, VirtualBox, etc?
Have you ensured that ALL traffic is allowed to the nodes?
Are any of your pods showing other than all containers running and in a ready status?
What version of the software did you deploy?
It appears as if you are using the NodePort 31497. Instead of telnet to the port what happens if you use nc, or curl?
Also, if using a provider such as GCE, use the PUBLIC iP address from outside the lab environment to view the dashboard. For example, if my PUBLIC IP to the GCE node was 35.101.11.22 I would use a browser on my local machine and go to 35.101.11.22:31497, not 10.110.107.206:31497 from inside the node due to how software defined networks and routing work.
Regards,
1 -
Hi @TheFutonEng,
Also be aware that accessing the Dashboard through the Google Chrome browser may not be successful, due to some recent upgrades in how the browser handles insecure connections.
I would recommend using the Firefox browser as an alternative, because that is what seemed to work when Chrome failed to display the Dashboard.
Regards,
-Chris1 -
Thanks for the responses @serewicz and @chrispokorni.
I'm running my cluster out of my house on a combination of desktop machines and VMs, all running Ubuntu 20.04.
I haven't put any network policies in place or modified iptables rules so I think all traffic is allowed. I've only made changes that the labs outlined.
There is a single calico pod that isn't running:
[rmengert@cka-master01]$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default seconddb-mariadb-client 1/1 Running 0 3d kube-system calico-kube-controllers-76d4774d89-t2dm9 1/1 Running 0 21d kube-system calico-node-9824l 1/1 Running 0 25d kube-system calico-node-vqbqj 1/1 Running 4 25d kube-system calico-node-zl9zc 0/1 Running 2 25d <-omitted->
Restarting that pod seems to have restored connectivity and the dashboard is now accessible. And not via Chrome as @chrispokorni pointed out. Thank you both for your support!
-Futon
0 -
Hi. I had the same problem until I used the browser "Midori" instead of Chrome (this is in Ubuntu not Windows). I clicked the little padlock up in the address bar and selected "Trust this website" and then I could enter the token code to access the dashboard.
0
Categories
- All Categories
- 208 LFX Mentorship
- 208 LFX Mentorship: Linux Kernel
- 738 Linux Foundation IT Professional Programs
- 340 Cloud Engineer IT Professional Program
- 167 Advanced Cloud Engineer IT Professional Program
- 67 DevOps Engineer IT Professional Program
- 133 Cloud Native Developer IT Professional Program
- 123 Express Training Courses
- 123 Express Courses - Discussion Forum
- 6K Training Courses
- 40 LFC110 Class Forum - Discontinued
- 67 LFC131 Class Forum
- 40 LFD102 Class Forum
- 223 LFD103 Class Forum
- 18 LFD110 Class Forum
- 35 LFD121 Class Forum
- 18 LFD133 Class Forum
- 7 LFD134 Class Forum
- 18 LFD137 Class Forum
- 71 LFD201 Class Forum
- 4 LFD210 Class Forum
- 3 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 2 LFD233 Class Forum
- 4 LFD237 Class Forum
- 24 LFD254 Class Forum
- 691 LFD259 Class Forum
- 111 LFD272 Class Forum
- 4 LFD272-JP クラス フォーラム
- 11 LFD273 Class Forum
- 118 LFS101 Class Forum
- 1 LFS111 Class Forum
- 3 LFS112 Class Forum
- 2 LFS116 Class Forum
- 4 LFS118 Class Forum
- 4 LFS142 Class Forum
- 5 LFS144 Class Forum
- 4 LFS145 Class Forum
- 2 LFS146 Class Forum
- 3 LFS147 Class Forum
- 9 LFS151 Class Forum
- 2 LFS157 Class Forum
- 19 LFS158 Class Forum
- 6 LFS162 Class Forum
- 2 LFS166 Class Forum
- 4 LFS167 Class Forum
- 2 LFS170 Class Forum
- 2 LFS171 Class Forum
- 3 LFS178 Class Forum
- 3 LFS180 Class Forum
- 2 LFS182 Class Forum
- 5 LFS183 Class Forum
- 31 LFS200 Class Forum
- 737 LFS201 Class Forum - Discontinued
- 3 LFS201-JP クラス フォーラム
- 18 LFS203 Class Forum
- 120 LFS207 Class Forum
- 2 LFS207-DE-Klassenforum
- 1 LFS207-JP クラス フォーラム
- 302 LFS211 Class Forum
- 56 LFS216 Class Forum
- 51 LFS241 Class Forum
- 46 LFS242 Class Forum
- 38 LFS243 Class Forum
- 14 LFS244 Class Forum
- 2 LFS245 Class Forum
- 46 LFS250 Class Forum
- 2 LFS250-JP クラス フォーラム
- 1 LFS251 Class Forum
- 148 LFS253 Class Forum
- 1 LFS254 Class Forum
- 1 LFS255 Class Forum
- 7 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.2K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 117 LFS260 Class Forum
- 158 LFS261 Class Forum
- 42 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 24 LFS267 Class Forum
- 19 LFS268 Class Forum
- 30 LFS269 Class Forum
- 201 LFS272 Class Forum
- 2 LFS272-JP クラス フォーラム
- 1 LFS274 Class Forum
- 4 LFS281 Class Forum
- 9 LFW111 Class Forum
- 258 LFW211 Class Forum
- 181 LFW212 Class Forum
- 13 SKF100 Class Forum
- 1 SKF200 Class Forum
- 1 SKF201 Class Forum
- 791 Hardware
- 199 Drivers
- 68 I/O Devices
- 37 Monitors
- 98 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 85 Storage
- 755 Linux Distributions
- 82 Debian
- 67 Fedora
- 16 Linux Mint
- 13 Mageia
- 23 openSUSE
- 147 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 352 Ubuntu
- 465 Linux System Administration
- 39 Cloud Computing
- 71 Command Line/Scripting
- Github systems admin projects
- 91 Linux Security
- 78 Network Management
- 101 System Management
- 47 Web Management
- 56 Mobile Computing
- 17 Android
- 28 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 366 Off Topic
- 114 Introductions
- 171 Small Talk
- 20 Study Material
- 534 Programming and Development
- 293 Kernel Development
- 223 Software Development
- 1.2K Software
- 212 Applications
- 182 Command Line
- 3 Compiling/Installing
- 405 Games
- 312 Installation
- 79 All In Program
- 79 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)