LAB 13.3: Adding tools for monitoring and metrics - Metrics API not available after 10+ minutes

@chrispokorni I've got the same problem. This is the kubectl -n kube-system describe deployment metrics-server :

Containers:
   metrics-server:
    Image:      k8s.gcr.io/metrics-server/metrics-server:v0.3.7
    Port:       4443/TCP
    Host Port:  0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-insecure-tls
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

Hi @chrispokorni I am facing the same issue, after reading the documentation I am not sure If I have to add an additional node, still not working for me even applying the new version.

I've added the configuration previous mentioned on this post, and still not working. This is the logs of one pod:

I0103 01:17:10.298497 1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0103 01:17:11.417949 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0103 01:17:11.418056 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0103 01:17:11.418124 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0103 01:17:11.418199 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0103 01:17:11.418247 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0103 01:17:11.418317 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0103 01:17:11.418679 1 secure_serving.go:267] Serving securely on [::]:4443
I0103 01:17:11.418808 1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0103 01:17:11.419381 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
W0103 01:17:11.419697 1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0103 01:17:11.519010 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0103 01:17:11.519024 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0103 01:17:11.519052 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
E0103 01:17:24.912919 1 scraper.go:140] "Failed to scrape node" err="Get \"https://worker:10250/metrics/resource\": context deadline exceeded" node="worker"
E0103 01:17:24.912964 1 scraper.go:140] "Failed to scrape node" err="Get \"https://cp:10250/metrics/resource\": context deadline exceeded" node="cp"
I0103 01:17:29.229201 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0103 01:17:39.233714 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
E0103 01:17:39.913491 1 scraper.go:140] "Failed to scrape node" err="Get \"https://cp:10250/metrics/resource\": context deadline exceeded" node="cp"
E0103 01:17:39.913492 1 scraper.go:140] "Failed to scrape node" err="Get \"https://worker:10250/metrics/resource\": context deadline exceeded" node="worker"
I0103 01:17:49.230115 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

Name: metrics-server
Namespace: kube-system
CreationTimestamp: Wed, 03 Jan 2024 01:08:57 +0000
Labels: k8s-app=metrics-server
Annotations: deployment.kubernetes.io/revision: 3
Selector: k8s-app=metrics-server
Replicas: 1 desired | 1 updated | 2 total | 0 available | 2 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 0 max unavailable, 25% max surge
Pod Template:
Labels: k8s-app=metrics-server
Service Account: metrics-server
Containers:
metrics-server:
Image: registry.k8s.io/metrics-server/metrics-server:v0.6.4
Port: 4443/TCP
Host Port: 0/TCP
Args:
--cert-dir=/tmp
--secure-port=4443
--kubelet-insecure-tls
--kubelet-preferred-address-types=Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP
--kubelet-use-node-status-port
--metric-resolution=15s
Requests:
cpu: 100m
memory: 200Mi
Liveness: http-get http://:http/livez delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/readyz delay=20s timeout=1s period=10s #success=1 #failure=3
Environment:
Mounts:
/tmp from tmp-dir (rw)
Volumes:
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
Priority Class Name: system-cluster-critical
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: metrics-server-fbb469ccc (0/0 replicas created), metrics-server-67865f7db4 (1/1 replicas created)
NewReplicaSet: metrics-server-b58456f69 (1/1 replicas created)
Events: