Lab 11.1. Service Mesh linkerd viz check

eboyko

Hi

When I try "linkerd viz check" I get an error:
container "linkerd-proxy" in pod "metrics-api-54888ffcc5-jkcxm" is not ready

How to resolve this?

Output of "kubectl -n linkerd-viz describe pods metrics-api-54888ffcc5-jkcxm":
Name: metrics-api-54888ffcc5-jkcxm
Namespace: linkerd-viz
Priority: 0
Service Account: metrics-api
Node: cp/10.22.22.114
Start Time: Thu, 11 Jan 2024 15:02:51 +0000
Labels: component=metrics-api
linkerd.io/extension=viz
pod-template-hash=54888ffcc5
Annotations: checksum/config: b73fb1bf343c4203fbab8ee108c5eba2e07d184177e204677dc83d4cad2cd12b
cluster-autoscaler.kubernetes.io/safe-to-evict: true
config.alpha.linkerd.io/proxy-wait-before-exit-seconds: 0
linkerd.io/created-by: linkerd/helm stable-2.14.8
linkerd.io/inject: enabled
Status: Running
SeccompProfile: RuntimeDefault
IP: 192.168.0.22
IPs:
IP: 192.168.0.22
Controlled By: ReplicaSet/metrics-api-54888ffcc5
Containers:
metrics-api:
Container ID: containerd://99208e15ec3f614f49febf02b1e277e7768767b24580f6b5c9bff280845a57cf
Image: cr.l5d.io/linkerd/metrics-api:stable-2.14.8
Image ID: cr.l5d.io/linkerd/metrics-api@sha256:dce7f2085d26d1d526d2c4c2a69c81eda6d89f1abf17404e7746acf5582a901c
Ports: 8085/TCP, 9995/TCP
Host Ports: 0/TCP, 0/TCP
SeccompProfile: RuntimeDefault
Args:
-controller-namespace=linkerd
-log-level=info
-log-format=plain
-cluster-domain=cluster.local
-prometheus-url=http://prometheus.linkerd-viz.svc.cluster.local:9090
-enable-pprof=false
State: Running
Started: Thu, 11 Jan 2024 15:02:53 +0000
Ready: True
Restart Count: 0
Liveness: http-get http://:9995/ping delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:9995/ready delay=0s timeout=1s period=10s #success=1 #failure=7
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n5lm9 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-n5lm9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

chrispokorni

Hi @eboyko,

In situations like this one, the actual Events (not displayed above) are the most helpful for troubleshooting purposes.

In addition, what is the state of your cluster overall? What infrastructure is hosting your cluster?
What are the outputs of the following commands:

kubectl get nodes -o wide
kubectl get pods -A -o wide

Regards,
-Chris

chrispokorni

Also, @eboyko,

How did you install the linkerd service mesh? Did you follow the installation sequence from the lab guide?

Regards,
-Chris

eboyko

Hi @chrispokorni,

I did follow the installation sequence in step 1 of lab guide.

I also tried to reinstall linkerd using:
linkerd viz uninstall | kubectl delete -f -
linkerd uninstall | kubectl delete -f -
and then repeating step 1 of lab guide.

My VM for CP:
VMware ESXi Hypervisor
4 CPUs
16 GB memory
200 GB HDD
Ubuntu 20.04.6 LTS
1 network adapter

My VM for Woker:
VMware ESXi Hypervisor
4 CPUs
8 GB memory
200 GB HDD
Ubuntu 20.04.6 LTS
1 network adapter

eboyko

Output of following commands in attached file (kubectl.txt):
kubectl get events --all-namespaces (during installation of linkerd and linkerd-viz)
kubectl get nodes -o wide
kubectl get pods -A -o wide

chrispokorni

Hi @eboyko,

Similar issues reported on github eventually got resolved by opening up the hypervisor firewall. In your situation it would be the ESXi firewall to allow all inbound traffic from all sources to all ports all protocols.

Regards,
-Chris

eboyko

Hi, @chrispokorni,

I managed to solve this issue by adding .cluster.local to no_proxy in /etc/kubernetes/manifests/kube-apiserver.yaml.

Thank you!