Lab 11.2 ingress controller - one ingress is restarting

bdkdavid

I have 3 node cp - 5 node dp - an haproxy

the node keeps restarting then goes to a CrashLoopBackOff

myingress-ingress-nginx-controller-2p98l 1/2 Running 5 (5s ago) 5m5s
myingress-ingress-nginx-controller-gzrjd 1/1 Running 0 82m
myingress-ingress-nginx-controller-vfw4m 1/1 Running 0 82m
myingress-ingress-nginx-controller-w9zr9 1/1 Running 0 82m
myingress-ingress-nginx-controller-zj6nk 1/1 Running 0 82m

kubectl get pods | grep myingress-ingress-nginx-controller-2p98l
myingress-ingress-nginx-controller-2p98l 1/2 CrashLoopBackOff 7 (2m1s ago) 14m

The tail of the describe pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned default/myingress-ingress-nginx-controller-2p98l to kworker03-test
Normal Created 15m kubelet Created container linkerd-init
Normal Started 15m kubelet Started container linkerd-init
Normal Pulled 15m kubelet Container image "cr.l5d.io/linkerd/proxy-init:v1.5.3" already present on machine
Normal Started 15m kubelet Started container linkerd-proxy
Normal Pulled 15m kubelet Container image "cr.l5d.io/linkerd/proxy:stable-2.11.3" already present on machine
Normal Created 15m kubelet Created container linkerd-proxy
Warning Unhealthy 14m (x5 over 15m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 502
Normal Killing 14m kubelet Container controller failed liveness probe, will be restarted
Warning FailedPreStopHook 14m kubelet Exec lifecycle hook ([/wait-shutdown]) for Container "controller" in Pod "myingress-ingress-nginx-controller-2p98l_default(888768f2-a660-4a9b-a293-6e4ba7912ab4)" failed - error: command '/wait-shutdown' exited with 137: , message: ""
Normal Pulled 14m (x2 over 15m) kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8" already present on machine
Normal Started 14m (x2 over 15m) kubelet Started container controller
Normal Created 14m (x2 over 15m) kubelet Created container controller
Warning Unhealthy 10m (x31 over 15m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 502
Warning BackOff 14s (x38 over 9m23s) kubelet Back-off restarting failed container

bdkdavid

here is the events

chrispokorni

Hi @bdkdavid,

Assuming you meant that one pod goes to CrashLoopBackOff... is this behavior observed immediately after the linkerd inject of the ingress? If linkerd is 2.11.3, what command was run to inject the ingress?

Regards,
-Chris

bdkdavid

Yes

The command:
kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress - | kubectl apply -f -

Also

it seems to be just on that node : all the pods on the other nodes seem to be fine
the myingress-ingress-nginx-controller pods seem to be working also

the node 3 has 1/2 where the onthers have only 1

I built all the nodes the same configs not sure why that is having a problem

Also I have some screen shots:
why is it only showing one functioning?

Here are some screen shots

chrispokorni

Hi @bdkdavid,

Please use the linkerd inject command as recommended here. The additional options will allow for all the ingress controller pods to be successfully injected and reach running state. This is a minor change that needs to be considered for linkerd 2.11, but it was not needed for the earlier version 2.10.

Regards,
-Chris

bdkdavid

do i need to uninstall and reinstall or can I just apply it?

bdkdavid

I uninstalled with helm and applied command:
kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress --skip-inbound-ports 443 --skip-outbound-ports 443 - | kubectl apply -f -

Works Now!

chrispokorni

Hi @bdkdavid,

The kubectl get ds ... | linkerd uninject ... | kubectl apply -f - was another option to reverse the ingress controller injection.

Regards,
-Chris