Welcome to the Linux Foundation Forum!

Connection closed error=ingress-mode routing is HTTP-only in lab 11.2 ingress

Options
theooze
theooze Posts: 1
in LFS258 Class Forum

At the end of lab 11.2 I follow the instruction to inject the linkerd into the ingress daemonset:
kubectl get ds myingress-ingress-nginx-controller -o yaml | \
linkerd inject --ingress - | kubectl apply -f -
_
(step 11).
this seems to succeed, but then i see a problem w/ my ingress pods:
_kubectl get pods
NAME READY STATUS RESTARTS AGE
myingress-ingress-nginx-controller-k98cq 1/1 Running 0 4d23h
myingress-ingress-nginx-controller-tld7h 1/2 CrashLoopBackOff 7 16m
web-one-8f86bc5f4-kxt4v 1/1 Running 0 94m
web-one-8f86bc5f4-q6w2k 1/1 Running 0 94m
web-two-8f86bc5f4-2tbxw 1/1 Running 0 94m
web-two-8f86bc5f4-972pt 1/1 Running 0 94m

looking at the logs in the linkerd-proxy container in the pod, i see these errors:
kubectl logs myingress-ingress-nginx-controller-tld7h linkerd-proxy

[ 0.056026s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: myingress-ingress-nginx.default.serviceaccount.identity.linkerd.cluster.local
[ 0.056865s] INFO ThreadId(01) outbound: linkerd_app_outbound: Outbound routing in ingress-mode
[ 3.909738s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 4.912101s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 5.914622s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 6.917707s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 7.920402s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 8.922144s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 9.924893s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 10.927779s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 11.930325s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 12.933975s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 13.935358s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 15.004374s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 16.006018s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 17.008912s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 17.847279s] INFO ThreadId(01) inbound:server{port=10254}:rescue{client.addr=192.168.0.113:57816}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)

I'm not sure what to do w/ this - is the linkerd-proxy for ingress trying to use https and i need to enable that in the ingress controller ?
i didn't see this in exercise 11.1, and there i was able to connect the deployments to linkerd w/o issue.

Comments

  • chrispokorni
    chrispokorni Posts: 2,165
    Options

    Hi @theooze,

    I was able to reproduce this issue and investigate a bit. It seems that on top of the errors reported above, the ingress 'controller' container of the myingress-ingress-nginx-... pod is failing the liveness probe.

    What I remember for sure is that we did not see this behavior with Linkerd v2.10, it only manifested with the latest Linkerd v2.11.

    The fact that in the meantime the NGINX ingress controller helm chart has been updated to a newer version of Nginx does not help either :wink:

    Unless it is just a temporary bug, the way to move forward is by trial and error to find the right combination of Nginx and Linkerd versions that would not break.

    Regards,
    -Chris

  • olmorupert
    olmorupert Posts: 14
    edited December 2021
    Options

    Confirmed you need linkerd v2.10 , that works with the latest ingress-nginx ( 4.0.13) . Wasting time here again.

  • giulianopz
    giulianopz Posts: 1
    Options

    Thanks, @olmorupert.
    I can confirm: I managed to make it work downgrading linkerd version in the setup sh file:
    LINKERD2_VERSION=${LINKERD2_VERSION:-stable-2.10.0}
    The latest ingress-nginx version in use is 4.0.18 in my case.

  • miguelgalindez
    miguelgalindez Posts: 1
    Options

    @giulianopz said:
    Thanks, @olmorupert.
    I can confirm: I managed to make it work downgrading linkerd version in the setup sh file:
    LINKERD2_VERSION=${LINKERD2_VERSION:-stable-2.10.0}
    The latest ingress-nginx version in use is 4.0.18 in my case.

    You can also set the env variable LINKERD2_VERSION with the proper version so that you don't need to edit the setup sh file. I mean something like this:

    • export LINKERD2_VERSION=stable-2.10.0

    Then you continue with the installation steps as normal:

    • curl -sL run.linkerd.io/install | sh
    • And so on...
  • chrispokorni
    chrispokorni Posts: 2,165
    Options

    Hi @miguelgalindez,

    The latest linkerd release can be instaled as well, however, Step 11 of Lab exercise 11.2 needs to be modified as follows:

    kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress --skip-inbound-ports 443 --skip-outbound-ports 443 - | kubectl apply -f -

    Regards,
    -Chris

  • punkoivan
    punkoivan Posts: 5
    Options

    Thanks @chrispokorni
    looks like started from linkerd version 2.11 they've added pods prob by default
    (not sure I've got their versioning though, it could be default https as well).

  • punkoivan
    punkoivan Posts: 5
    Options

    Btw, the entire lab with linkerd is a little bit confusing - it is not clear what's the role of linkerd.

    While nginx ingress controller (as well as any other) can work perfectly without any mesh.
    I assume that the main role is to provide easy termination SSL (without populating it to nginx and to application behind proxy) and cross-pod communication, but it is not obvious from the lab.

  • SevenD
    SevenD Posts: 3
    Options

    Hello @chrispokorni,

    Thank you! While troubleshooting can sometimes be a good learning exercise, I am a bit over it at this point of the course.

Categories

Upcoming Training