Welcome to the Linux Foundation Forum!

Lab 11.2 ingress controller - one ingress is restarting

bdkdavid
bdkdavid Posts: 32
edited July 2022 in LFS258 Class Forum

I have 3 node cp - 5 node dp - an haproxy

the node keeps restarting then goes to a CrashLoopBackOff

myingress-ingress-nginx-controller-2p98l 1/2 Running 5 (5s ago) 5m5s
myingress-ingress-nginx-controller-gzrjd 1/1 Running 0 82m
myingress-ingress-nginx-controller-vfw4m 1/1 Running 0 82m
myingress-ingress-nginx-controller-w9zr9 1/1 Running 0 82m
myingress-ingress-nginx-controller-zj6nk 1/1 Running 0 82m

kubectl get pods | grep myingress-ingress-nginx-controller-2p98l
myingress-ingress-nginx-controller-2p98l 1/2 CrashLoopBackOff 7 (2m1s ago) 14m

The tail of the describe pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned default/myingress-ingress-nginx-controller-2p98l to kworker03-test
Normal Created 15m kubelet Created container linkerd-init
Normal Started 15m kubelet Started container linkerd-init
Normal Pulled 15m kubelet Container image "cr.l5d.io/linkerd/proxy-init:v1.5.3" already present on machine
Normal Started 15m kubelet Started container linkerd-proxy
Normal Pulled 15m kubelet Container image "cr.l5d.io/linkerd/proxy:stable-2.11.3" already present on machine
Normal Created 15m kubelet Created container linkerd-proxy
Warning Unhealthy 14m (x5 over 15m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 502
Normal Killing 14m kubelet Container controller failed liveness probe, will be restarted
Warning FailedPreStopHook 14m kubelet Exec lifecycle hook ([/wait-shutdown]) for Container "controller" in Pod "myingress-ingress-nginx-controller-2p98l_default(888768f2-a660-4a9b-a293-6e4ba7912ab4)" failed - error: command '/wait-shutdown' exited with 137: , message: ""
Normal Pulled 14m (x2 over 15m) kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8" already present on machine
Normal Started 14m (x2 over 15m) kubelet Started container controller
Normal Created 14m (x2 over 15m) kubelet Created container controller
Warning Unhealthy 10m (x31 over 15m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 502
Warning BackOff 14s (x38 over 9m23s) kubelet Back-off restarting failed container

Comments

  • bdkdavid
    bdkdavid Posts: 32

    here is the events

  • chrispokorni
    chrispokorni Posts: 2,606

    Hi @bdkdavid,

    Assuming you meant that one pod goes to CrashLoopBackOff... is this behavior observed immediately after the linkerd inject of the ingress? If linkerd is 2.11.3, what command was run to inject the ingress?

    Regards,
    -Chris

  • bdkdavid
    bdkdavid Posts: 32

    Yes

    The command:
    kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress - | kubectl apply -f -

    Also

    it seems to be just on that node : all the pods on the other nodes seem to be fine
    the myingress-ingress-nginx-controller pods seem to be working also

    the node 3 has 1/2 where the onthers have only 1

    I built all the nodes the same configs not sure why that is having a problem

    Also I have some screen shots:
    why is it only showing one functioning?

    Here are some screen shots

  • chrispokorni
    chrispokorni Posts: 2,606

    Hi @bdkdavid,

    Please use the linkerd inject command as recommended here. The additional options will allow for all the ingress controller pods to be successfully injected and reach running state. This is a minor change that needs to be considered for linkerd 2.11, but it was not needed for the earlier version 2.10.

    Regards,
    -Chris

  • bdkdavid
    bdkdavid Posts: 32

    do i need to uninstall and reinstall or can I just apply it?

  • bdkdavid
    bdkdavid Posts: 32

    I uninstalled with helm and applied command:
    kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress --skip-inbound-ports 443 --skip-outbound-ports 443 - | kubectl apply -f -

    Works Now!

  • chrispokorni
    chrispokorni Posts: 2,606

    Hi @bdkdavid,

    The kubectl get ds ... | linkerd uninject ... | kubectl apply -f - was another option to reverse the ingress controller injection.

    Regards,
    -Chris

Categories

Upcoming Training