Connection closed error=ingress-mode routing is HTTP-only in lab 11.2 ingress

theooze

At the end of lab 11.2 I follow the instruction to inject the linkerd into the ingress daemonset:
kubectl get ds myingress-ingress-nginx-controller -o yaml | \
linkerd inject --ingress - | kubectl apply -f -
_
(step 11).
this seems to succeed, but then i see a problem w/ my ingress pods:
_kubectl get pods
NAME READY STATUS RESTARTS AGE
myingress-ingress-nginx-controller-k98cq 1/1 Running 0 4d23h
myingress-ingress-nginx-controller-tld7h 1/2 CrashLoopBackOff 7 16m
web-one-8f86bc5f4-kxt4v 1/1 Running 0 94m
web-one-8f86bc5f4-q6w2k 1/1 Running 0 94m
web-two-8f86bc5f4-2tbxw 1/1 Running 0 94m
web-two-8f86bc5f4-972pt 1/1 Running 0 94m

looking at the logs in the linkerd-proxy container in the pod, i see these errors:
kubectl logs myingress-ingress-nginx-controller-tld7h linkerd-proxy

[ 0.056026s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: myingress-ingress-nginx.default.serviceaccount.identity.linkerd.cluster.local
[ 0.056865s] INFO ThreadId(01) outbound: linkerd_app_outbound: Outbound routing in ingress-mode
[ 3.909738s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 4.912101s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 5.914622s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 6.917707s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 7.920402s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 8.922144s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 9.924893s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 10.927779s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 11.930325s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 12.933975s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 13.935358s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 15.004374s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 16.006018s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 17.008912s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 17.847279s] INFO ThreadId(01) inbound:server{port=10254}:rescue{client.addr=192.168.0.113:57816}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)

I'm not sure what to do w/ this - is the linkerd-proxy for ingress trying to use https and i need to enable that in the ingress controller ?
i didn't see this in exercise 11.1, and there i was able to connect the deployments to linkerd w/o issue.

chrispokorni

Hi @theooze,

I was able to reproduce this issue and investigate a bit. It seems that on top of the errors reported above, the ingress 'controller' container of the myingress-ingress-nginx-... pod is failing the liveness probe.

What I remember for sure is that we did not see this behavior with Linkerd v2.10, it only manifested with the latest Linkerd v2.11.

The fact that in the meantime the NGINX ingress controller helm chart has been updated to a newer version of Nginx does not help either

Unless it is just a temporary bug, the way to move forward is by trial and error to find the right combination of Nginx and Linkerd versions that would not break.

Regards,
-Chris

olmorupert

Confirmed you need linkerd v2.10 , that works with the latest ingress-nginx ( 4.0.13) . Wasting time here again.

giulianopz

Thanks, @olmorupert.
I can confirm: I managed to make it work downgrading linkerd version in the setup sh file:
LINKERD2_VERSION=${LINKERD2_VERSION:-stable-2.10.0}
The latest ingress-nginx version in use is 4.0.18 in my case.

miguelgalindez

@giulianopz said:
Thanks, @olmorupert.
I can confirm: I managed to make it work downgrading linkerd version in the setup sh file:
LINKERD2_VERSION=${LINKERD2_VERSION:-stable-2.10.0}
The latest ingress-nginx version in use is 4.0.18 in my case.

You can also set the env variable LINKERD2_VERSION with the proper version so that you don't need to edit the setup sh file. I mean something like this:

• export LINKERD2_VERSION=stable-2.10.0

Then you continue with the installation steps as normal:

• curl -sL run.linkerd.io/install | sh
• And so on...

chrispokorni

Hi @miguelgalindez,

The latest linkerd release can be instaled as well, however, Step 11 of Lab exercise 11.2 needs to be modified as follows:

kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress --skip-inbound-ports 443 --skip-outbound-ports 443 - | kubectl apply -f -

Regards,
-Chris

punkoivan

Thanks @chrispokorni
looks like started from linkerd version 2.11 they've added pods prob by default
(not sure I've got their versioning though, it could be default https as well).

punkoivan

Btw, the entire lab with linkerd is a little bit confusing - it is not clear what's the role of linkerd.

While nginx ingress controller (as well as any other) can work perfectly without any mesh.
I assume that the main role is to provide easy termination SSL (without populating it to nginx and to application behind proxy) and cross-pod communication, but it is not obvious from the lab.

SevenD

Hello @chrispokorni,

Thank you! While troubleshooting can sometimes be a good learning exercise, I am a bit over it at this point of the course.