LAB 12.3 Metrics pod failing currently unable to handle the request (get nodes.metrics.k8s.io)
For those with issues in this lab. When running kubectl top pod --all-namespaces I received an error. I found errors in the logs stating unable to handle the request (get nodes.metrics.k8s.io). I upgraded K8's to v1.16.4 and it has fixed the problem.
Comments
-
Hi @scubaman31,
Thanks for reporting this strange behavior.
Although I did not have to go thru the upgrade process (to v1.16.4) for the metrics-server to work, the only strange behavior I noticed was a slight delay of thekubectl top ...
command to produce an output with usage data when running it for the first time after the metrics-server deployment update.Regards,
-Chris0 -
Haven't been able to get metrics-server working on any version so far.
username@cka-master:~/metrics-server/deploy$ kubectl top node cka-master Error from server (NotFound): nodemetrics.metrics.k8s.io "cka-master" not found
I thought it was a problem with my configuration, so I reinstalled mutliple times and even tried different pod networks, different OSes (take systemd resolv.conf DNS out of the equation...) and it still won't work.
I'm on K8S 1.16.4.I get a whole lot of this:
E0109 10:00:38.777388 1 reststorage.go:135] unable to fetch node metrics for node "cka-master": no metrics known for node E0109 10:00:38.777413 1 reststorage.go:135] unable to fetch node metrics for node "cka-node01": no metrics known for node E0109 10:00:40.310346 1 reststorage.go:135] unable to fetch node metrics for node "cka-master": no metrics known for node E0109 10:00:40.310392 1 reststorage.go:135] unable to fetch node metrics for node "cka-node01": no metrics known for node E0109 10:00:45.215954 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-master: unable to fetch metrics from Kubelet cka-master (cka-master): Get https://cka-master:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-master on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01): Get https://cka-node01:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-node01 on 10.96.0.10:53: no such host] E0109 10:01:45.210111 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-master: unable to fetch metrics from Kubelet cka-master (cka-master): Get https://cka-master:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-master on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01): Get https://cka-node01:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-node01 on 10.96.0.10:53: no such host] E0109 10:02:45.211043 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01)
0 -
Hi @undine,
I see you mentioned v1.16.4 of Kubernetes. Any particular reason why you upgraded from the version suggested by the lab?
I deployed the metrics-server successfully on 2 separate clusters, v1.16.1 and v1.16.4. I did not run into any issues, although I tried really hard to break my cluster.
Since I have no visibility into your cluster's configuration, would you mind providing the output of the following commands?
kubectl version
kubectl get nodes
kubectl get pods --all-namespaces -o wide
cat kubeadm-config.yaml
Thanks,
-Chris0
Categories
- All Categories
- 217 LFX Mentorship
- 217 LFX Mentorship: Linux Kernel
- 791 Linux Foundation IT Professional Programs
- 353 Cloud Engineer IT Professional Program
- 178 Advanced Cloud Engineer IT Professional Program
- 82 DevOps Engineer IT Professional Program
- 147 Cloud Native Developer IT Professional Program
- 137 Express Training Courses
- 137 Express Courses - Discussion Forum
- 6.2K Training Courses
- 47 LFC110 Class Forum - Discontinued
- 71 LFC131 Class Forum
- 42 LFD102 Class Forum
- 226 LFD103 Class Forum
- 18 LFD110 Class Forum
- 38 LFD121 Class Forum
- 18 LFD133 Class Forum
- 7 LFD134 Class Forum
- 18 LFD137 Class Forum
- 71 LFD201 Class Forum
- 4 LFD210 Class Forum
- 5 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 2 LFD233 Class Forum
- 4 LFD237 Class Forum
- 24 LFD254 Class Forum
- 697 LFD259 Class Forum
- 111 LFD272 Class Forum
- 4 LFD272-JP クラス フォーラム
- 12 LFD273 Class Forum
- 148 LFS101 Class Forum
- 1 LFS111 Class Forum
- 3 LFS112 Class Forum
- 2 LFS116 Class Forum
- 4 LFS118 Class Forum
- LFS120 Class Forum
- 7 LFS142 Class Forum
- 5 LFS144 Class Forum
- 4 LFS145 Class Forum
- 2 LFS146 Class Forum
- 3 LFS147 Class Forum
- 1 LFS148 Class Forum
- 15 LFS151 Class Forum
- 2 LFS157 Class Forum
- 29 LFS158 Class Forum
- 7 LFS162 Class Forum
- 2 LFS166 Class Forum
- 4 LFS167 Class Forum
- 3 LFS170 Class Forum
- 2 LFS171 Class Forum
- 3 LFS178 Class Forum
- 3 LFS180 Class Forum
- 2 LFS182 Class Forum
- 5 LFS183 Class Forum
- 31 LFS200 Class Forum
- 737 LFS201 Class Forum - Discontinued
- 3 LFS201-JP クラス フォーラム
- 18 LFS203 Class Forum
- 134 LFS207 Class Forum
- 2 LFS207-DE-Klassenforum
- 1 LFS207-JP クラス フォーラム
- 302 LFS211 Class Forum
- 56 LFS216 Class Forum
- 52 LFS241 Class Forum
- 48 LFS242 Class Forum
- 38 LFS243 Class Forum
- 15 LFS244 Class Forum
- 2 LFS245 Class Forum
- LFS246 Class Forum
- 48 LFS250 Class Forum
- 2 LFS250-JP クラス フォーラム
- 1 LFS251 Class Forum
- 152 LFS253 Class Forum
- 1 LFS254 Class Forum
- 1 LFS255 Class Forum
- 7 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.2K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 118 LFS260 Class Forum
- 159 LFS261 Class Forum
- 42 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 24 LFS267 Class Forum
- 22 LFS268 Class Forum
- 30 LFS269 Class Forum
- LFS270 Class Forum
- 202 LFS272 Class Forum
- 2 LFS272-JP クラス フォーラム
- 1 LFS274 Class Forum
- 4 LFS281 Class Forum
- 9 LFW111 Class Forum
- 259 LFW211 Class Forum
- 181 LFW212 Class Forum
- 13 SKF100 Class Forum
- 1 SKF200 Class Forum
- 1 SKF201 Class Forum
- 795 Hardware
- 199 Drivers
- 68 I/O Devices
- 37 Monitors
- 102 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 85 Storage
- 758 Linux Distributions
- 82 Debian
- 67 Fedora
- 17 Linux Mint
- 13 Mageia
- 23 openSUSE
- 148 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 353 Ubuntu
- 468 Linux System Administration
- 39 Cloud Computing
- 71 Command Line/Scripting
- Github systems admin projects
- 93 Linux Security
- 78 Network Management
- 102 System Management
- 47 Web Management
- 63 Mobile Computing
- 18 Android
- 33 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 371 Off Topic
- 114 Introductions
- 174 Small Talk
- 22 Study Material
- 805 Programming and Development
- 303 Kernel Development
- 484 Software Development
- 1.8K Software
- 261 Applications
- 183 Command Line
- 3 Compiling/Installing
- 987 Games
- 317 Installation
- 97 All In Program
- 97 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)