Welcome to the Linux Foundation Forum!

LAB 12.3 Metrics pod failing currently unable to handle the request (get nodes.metrics.k8s.io)

For those with issues in this lab. When running kubectl top pod --all-namespaces I received an error. I found errors in the logs stating unable to handle the request (get nodes.metrics.k8s.io). I upgraded K8's to v1.16.4 and it has fixed the problem.

Comments

  • chrispokorni
    chrispokorni Posts: 2,376

    Hi @scubaman31,

    Thanks for reporting this strange behavior.
    Although I did not have to go thru the upgrade process (to v1.16.4) for the metrics-server to work, the only strange behavior I noticed was a slight delay of the kubectl top ... command to produce an output with usage data when running it for the first time after the metrics-server deployment update.

    Regards,
    -Chris

  • undine
    undine Posts: 1

    Haven't been able to get metrics-server working on any version so far.

    username@cka-master:~/metrics-server/deploy$ kubectl top node cka-master
    Error from server (NotFound): nodemetrics.metrics.k8s.io "cka-master" not found
    

    I thought it was a problem with my configuration, so I reinstalled mutliple times and even tried different pod networks, different OSes (take systemd resolv.conf DNS out of the equation...) and it still won't work.
    I'm on K8S 1.16.4.

    I get a whole lot of this:

    E0109 10:00:38.777388       1 reststorage.go:135] unable to fetch node metrics for node "cka-master": no metrics known for node
    E0109 10:00:38.777413       1 reststorage.go:135] unable to fetch node metrics for node "cka-node01": no metrics known for node
    E0109 10:00:40.310346       1 reststorage.go:135] unable to fetch node metrics for node "cka-master": no metrics known for node
    E0109 10:00:40.310392       1 reststorage.go:135] unable to fetch node metrics for node "cka-node01": no metrics known for node
    E0109 10:00:45.215954       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-master: unable to fetch metrics from Kubelet cka-master (cka-master): Get https://cka-master:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-master on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01): Get https://cka-node01:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-node01 on 10.96.0.10:53: no such host]
    E0109 10:01:45.210111       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-master: unable to fetch metrics from Kubelet cka-master (cka-master): Get https://cka-master:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-master on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01): Get https://cka-node01:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup cka-node01 on 10.96.0.10:53: no such host]
    E0109 10:02:45.211043       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cka-node01: unable to fetch metrics from Kubelet cka-node01 (cka-node01)
    
  • chrispokorni
    chrispokorni Posts: 2,376

    Hi @undine,

    I see you mentioned v1.16.4 of Kubernetes. Any particular reason why you upgraded from the version suggested by the lab?

    I deployed the metrics-server successfully on 2 separate clusters, v1.16.1 and v1.16.4. I did not run into any issues, although I tried really hard to break my cluster.

    Since I have no visibility into your cluster's configuration, would you mind providing the output of the following commands?
    kubectl version
    kubectl get nodes
    kubectl get pods --all-namespaces -o wide
    cat kubeadm-config.yaml

    Thanks,
    -Chris

Categories

Upcoming Training