11.2 Ingress Controller Lab Issue - Bug Found

melchior

Hi guys !

I hit what I believe is a documentation bug on LAB 11.2.
It was a good thing as I learned a lot more stuff troubleshooting (maybe this bug is there for that reason )
I followed all the steps an the Ingress was not working so I could not access the nginx by curling the ClusterIP of the service and by adding the Host header to curl like in the doc:

curl -H "Host: www.external.com" http://10.102.167.117

1 - Root cause:

The ingress.yaml file does not contain the default class for Ingress on the definition.

LOGS from Ingress Controller:

vxr@ubuntu-k8s-master:~$ kubectl logs myingress-ingress-nginx-controller-cqbt5 -v3
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.2
  Build:         2b8ed4511af75a7c41e52726b0644d600fc7961b
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

W0928 10:33:48.752897       7 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0928 10:33:48.753025       7 main.go:221] "Creating API client" host="https://10.96.0.1:443"
I0928 10:33:48.760687       7 main.go:265] "Running in Kubernetes cluster" major="1" minor="21" git="v1.21.1" state="clean" commit="5e58841cce77d4bc13713ad2b91fa0d961e69192" platform="linux/amd64"
I0928 10:33:49.212817       7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0928 10:33:49.230844       7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0928 10:33:49.256887       7 nginx.go:253] "Starting NGINX Ingress controller"
I0928 10:33:49.269543       7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"default", Name:"myingress-ingress-nginx-controller", UID:"11845428-96da-4338-a23f-99a916f3fc04", APIVersion:"v1", ResourceVersion:"3434117", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap default/myingress-ingress-nginx-controller
I0928 10:33:50.458770       7 nginx.go:295] "Starting NGINX process"
I0928 10:33:50.458840       7 leaderelection.go:243] attempting to acquire leader lease default/ingress-controller-leader...
I0928 10:33:50.459073       7 nginx.go:315] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0928 10:33:50.459341       7 controller.go:152] "Configuration changes detected, backend reload required"
I0928 10:33:50.464504       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-748d8ff6c7-stgsn"
I0928 10:33:50.518415       7 controller.go:169] "Backend successfully reloaded"
I0928 10:33:50.518462       7 controller.go:180] "Initial sync, sleeping for 1 second"
I0928 10:33:50.518568       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"myingress-ingress-nginx-controller-cqbt5", UID:"45b03c4c-26d3-4246-955b-396d21ec1dfe", APIVersion:"v1", ResourceVersion:"3434201", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0928 10:34:25.927925       7 leaderelection.go:253] successfully acquired lease default/ingress-controller-leader
I0928 10:34:25.928006       7 status.go:84] "New leader elected" identity="myingress-ingress-nginx-controller-cqbt5"
I0928 10:37:32.933703       7 main.go:101] "successfully validated configuration, accepting" ingress="ingress-test/default"
I0928 10:37:32.938381       7 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="default/ingress-test" error="ingress does not contain a valid IngressClass"
I0928 11:00:45.760557       7 store.go:336] "Ignoring ingress because of error while validating ingress class" ingress="default/ingress-test" error="ingress does not contain a valid IngressClass"
I0928 11:06:34.098285       7 store.go:435] "ignoring ingressclass as the spec.controller is not the same of this ingress" ingressclass="external-lb"
I0928 11:06:44.600291       7 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="default/ingress-test" error="ingress does not contain a valid IngressClass"
I0928 11:08:50.621894       7 store.go:336] "Ignoring ingress because of error while validating ingress class" ingress="default/ingress-test" error="ingress does not contain a valid IngressClass"
I0928 11:09:01.521031       7 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="default/ingress-test" error="no object matching key \"external-lb\" in local store"
I0928 11:12:46.121788       7 store.go:336] "Ignoring ingress because of error while validating ingress class" ingress="default/ingress-test" error="no object matching key \"external-lb\" in local store"
I0928 11:16:56.358722       7 store.go:452] "ignoring ingressclass as the spec.controller is not the same of this ingress" ingressclass="external-lb"
I0928 11:17:38.810403       7 store.go:365] "Found valid IngressClass" ingress="default/ingress-test" ingressclass="nginx"
I0928 11:17:38.810552       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-test", UID:"1e442104-5e27-4dc9-8e51-d9d4ab0bef95", APIVersion:"networking.k8s.io/v1", ResourceVersion:"3438329", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0928 11:17:38.810867       7 controller.go:152] "Configuration changes detected, backend reload required"
I0928 11:17:38.890003       7 controller.go:169] "Backend successfully reloaded"
I0928 11:17:38.890337       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"myingress-ingress-nginx-controller-cqbt5", UID:"45b03c4c-26d3-4246-955b-396d21ec1dfe", APIVersion:"v1", ResourceVersion:"3434201", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.89.0 - - [28/Sep/2021:11:18:27 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/7.58.0" 80 0.000 [default-web-one-80] [] 192.168.89.36:80 615 0.000 200 8b2e251094e3504862db879039d270e5
vxr@ubuntu-k8s-master:~$

If I list the default ingress classes:

vxr@ubuntu-k8s-master:~$ kubectl get ingressclasses.networking.k8s.io nginx
NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       95m
vxr@ubuntu-k8s-master:~$

2 - Resolution

Add the default class to the ingress.yaml file

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-test
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: www.external.com
    http:
      paths:
      - backend:
          service:
            name: web-one
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

3 - Verification

vxr@ubuntu-k8s-master:~$ curl -H "Host: www.external.com" http://10.102.167.117
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

chrispokorni

Hi @melchior,

Thank you for your post. The ingress class annotation has also been addressed in this forum post:

Lab 11.2 Ingress Controller - connection refused on public ip address

Regards,
-Chris

melchior

Thanks ! I did not realize this issue had been addressed on another post !

olmorupert

I also had to add

spec:
ingressClassName: nginx

olmorupert

Another bug found another freaking hours lost on this course.

Don't use the latest linkerd from the course.

Use specifically: https://github.com/linkerd/linkerd2/releases/tag/stable-2.10.0

works with ingress-nginx Chart 4.0.13

If you use the latest you won't be able to create the ingress controller daemonset with the linkerd injected settings. it will stay stuck at pending. with errors

[ 0.400754s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 1.402211s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 2.403196s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only
[ 3.404067s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=ingress-mode routing is HTTP-only

googling on the error lead to a link in this forum:

https://forum.linuxfoundation.org/discussion/860134/connection-closed-error-ingress-mode-routing-is-http-only-in-lab-11-2-ingress

so this is a duplicate. i hope to spare you some time. if you met above issue (missing annotation setting in course) as well, like me.

savoir

The whole LF Kubernets Fundamentals is essentially a garbage training full of faults and bugs. Unfortunately I found out that too late to get the refund.

hansbogert

Why is this not updated in the actual course material? Not a big issue for me, but my colleagues who have less K8s experience will lose a lot of time with this.

donaldsebleung

Just wanted to say that I encountered the same issue just now with Lab 11.2, which took me a while Googling and reading through the Ingress docs to realize that adding ingressClassName: nginx solved the issue. It would be great to see the lab material updated ASAP to resolve the issue, thanks!

P.S. Not related to the issue, but so far, I found the course educational and definitely a lot more in depth compared to the average online Kubernetes tutorial, also much easier to get started with than just reading the official docs. But it definitely does require a good understanding of Linux administration and networking to begin with, one which simply going through LFS101x and LFS158x does not suffice. I would suggest mentioning at the start of the course additional prerequisites such as LFS201 and possibly even LFS211, thanks!