LAB 12.3 Metrics pod failing currently unable to handle the request (get nodes.metrics.k8s.io)

scubaman31

For those with issues in this lab. When running kubectl top pod --all-namespaces I received an error. I found errors in the logs stating unable to handle the request (get nodes.metrics.k8s.io). I upgraded K8's to v1.16.4 and it has fixed the problem.

chrispokorni

Hi @scubaman31,

Thanks for reporting this strange behavior.
Although I did not have to go thru the upgrade process (to v1.16.4) for the metrics-server to work, the only strange behavior I noticed was a slight delay of the kubectl top ... command to produce an output with usage data when running it for the first time after the metrics-server deployment update.

Regards,
-Chris

undine

Haven't been able to get metrics-server working on any version so far.

username@cka-master:~/metrics-server/deploy$ kubectl top node cka-master
Error from server (NotFound): nodemetrics.metrics.k8s.io "cka-master" not found

I thought it was a problem with my configuration, so I reinstalled mutliple times and even tried different pod networks, different OSes (take systemd resolv.conf DNS out of the equation...) and it still won't work.
I'm on K8S 1.16.4.

I get a whole lot of this:

E0109 10:00:38.777388       1 reststorage.go:135] unable to fetch node metrics for node "cka-master": no metrics known for node
E0109 10:00:38.777413       1 reststorage.go:135] unable to fetch node metrics for node "cka-node01": no metrics known for node
E0109 10:00:40.310346       1 reststorage.go:135] unable to fetch node metrics for node "cka-master": no metrics known for node
E0109 10:00:40.310392       1 reststorage.go:135] unable to fetch node metrics for node "cka-node01": no metrics known for node
E0109 10:00:45.215954       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-master: unable to fetch metrics from Kubelet cka-master (cka-master): Get https://cka-master:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-master on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01): Get https://cka-node01:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-node01 on 10.96.0.10:53: no such host]
E0109 10:01:45.210111       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-master: unable to fetch metrics from Kubelet cka-master (cka-master): Get https://cka-master:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-master on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01): Get https://cka-node01:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-node01 on 10.96.0.10:53: no such host]
E0109 10:02:45.211043       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01)

chrispokorni

Hi @undine,

I see you mentioned v1.16.4 of Kubernetes. Any particular reason why you upgraded from the version suggested by the lab?

I deployed the metrics-server successfully on 2 separate clusters, v1.16.1 and v1.16.4. I did not run into any issues, although I tried really hard to break my cluster.

Since I have no visibility into your cluster's configuration, would you mind providing the output of the following commands?
kubectl version
kubectl get nodes
kubectl get pods --all-namespaces -o wide
cat kubeadm-config.yaml

Thanks,
-Chris