Welcome to the Linux Foundation Forum!

Lab 6.2: error 404 when getting to the tester service via ingress

YuriiKantonistov
YuriiKantonistov Posts: 23
in LFS260 Class Forum

Hello,
hope you can guide me on how to troubleshoot and to fix this ingress issue below.

Ingress installation according to the lab steps went successfully, fixed ingress-controller service acount so it accesses ingressclass.
Now tester pod, ingress controller pod and tester service are up and running. Can get to the service directly, but error 404 if going in via ingress:

cp-$ curl http://192.168.122.70 -H 'Host: nginx.192.168.122.70.nip.io'
<html><body><h1> 404 Not Found </h1>
The requested URL was not found.
</body></html>

*** tester pod is up and running in pods network at ip 192.168.196.101:
NAME READY STATUS RESTARTS AGE IP
pod/tester-77f475f4f4-bptdm 1/1 Running 3 2d11h 192.168.196.101

*** tester service listening on port 80 at cluster ip 10.96.240.204:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/tester ClusterIP 10.96.240.204 80/TCP 35h app=tester

*** configured ingress rules are seen and to me looks correct:
cp-$ kubectl describe ingress.networking.k8s.io/tester
Name: tester
. . .
Rules:
Host Path Backends
---- ---- --------
nginx.192.168.122.70.nip.io / tester:80 (192.168.196.101:80)
Annotations:
Events:

*** ingress pod is running on a host ip 192.168.122.70 and listens on port 80:
NAME READY STATUS RESTARTS AGE IP
haproxy-ingress-s8fkh 1/1 Running 2 36h 192.168.122.70

Can see haproxy really running listens on port 80 on a host:

cp-# ss -4lpn|grep haproxy
. . .
tcp LISTEN 0 2000 0.0.0.0:80 0.0.0.0:* users:(("haproxy",pid=3586,fd=4))
tcp LISTEN 0 2000 0.0.0.0:443 0.0.0.0:* users:(("haproxy",pid=3586,fd=5))

P.S. I know I can try to re-create HAProxy-ingress controller by using Helm and official guide etc., but guess it worth to spend time on troubleshooting to get a better understanding on how it works internally.

Comments

  • chrispokorni
    chrispokorni Posts: 2,016

    Hi @YuriiKantonistov,

    What are the ranges of the Pod network and the host network? Is there any IP address overlap between the two?

    Regards,
    -Chris

  • YuriiKantonistov
    YuriiKantonistov Posts: 23

    Thank you for the quick response. Damn, really they overlap.

    VM host net is 192.168.122/24 and Pods network is the default one 192.168.0.0/16, just did not think about it when was preparing VMs.

    But is it really the root cause for 404 error?

    Because otherwise evertything works e.g. I can fetch test nginx web page if not using ingress proxy:

    • from cp master VM host(=192.168.122.70) by e.g. curl 192.168.196.101
    • from another pod(=192.168.196.95) by e.g. pod IP curl 192.168.196.101
    • from another pod(=192.168.196.95) by e.g. ClusterIP curl 10.96.240.204

    Why then HAProxy (=192.168.122.70) is the only one which cannot get to 192.168.196.101?

  • chrispokorni
    chrispokorni Posts: 2,016

    Hi @YuriiKantonistov,

    Traffic can be allowed/blocked either by an OS level firewall, a hypervisor level firewall, VPC firewall, Security Group, etc...

    The IP overlap however, is common cause for many routing issues within the cluster. The recommendation is to avoid any IP overlaps between Node IPs, Pod IPs, and Service IPs.

    Regards,
    -Chris

  • felixkrohn
    felixkrohn Posts: 5
    edited January 2022

    with this (current) version of haproxy ingress controller you need to annotate the ingress with the corresponding "ingress class", either on the fly after having created the Ingress object:
    kc annotate ingress tester kubernetes.io/ingress.class=haproxy
    or within the yaml:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:                                          # <== add this
    kubernetes.io/ingress.class: haproxy  # <== add this
  name: tester
[...]

    For debugging, you can check if the controller picked up you ingress:
    kc exec -n ingress-controller haproxy-ingress-7fk44 -t -i -- cat /etc/haproxy/haproxy.cfg
    adapt the pod's name accordingly, and watch out for a line containing (in this example) backend default_tester_80

    Also check the logs of the pod (kc logs -n ingress-controller --tail=10 haproxy-ingress-7fk44) to see if the controller is acting on creating/changing the Ingress object:

    I0120 21:25:27.455209       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"tester", UID:"7589344b-f7ca-47c9-ba4c-f441c7f23ea4", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1483239", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/tester
I0120 21:25:28.317963       8 controller.go:317] starting haproxy update id=9
I0120 21:25:28.359947       8 instance.go:340] haproxy successfully reloaded (embedded)
I0120 21:25:28.360269       8 controller.go:326] finish haproxy update id=9: parse_ingress=0.467661ms write_maps=0.912796ms write_config=2.915139ms reload_haproxy=37.616997ms total=41.912593ms

    HTH.
    Felix

  • felixkrohn
    felixkrohn Posts: 5

    For completeness' sake, you can also disable this behaviour of haproxy ingress controller, see https://haproxy-ingress.github.io/docs/configuration/command-line/#ingress-class:

    --ingress-class: defines the value of kubernetes.io/ingress.class annotation this controller should listen to. The default value is haproxy if not declared.
    --watch-ingress-without-class: defines if this controller should also listen to ingress resources that doesn’t declare neither the kubernetes.io/ingress.class annotation nor the .spec.ingressClassName field. The default since v0.12 is to ignore ingress without class annotation and class name.

    TL;DR: you can add the config option --watch-ingress-without-class in the DaemonSet.

    Felix

  • YuriiKantonistov
    YuriiKantonistov Posts: 23

    @felixkrohn said:
    with this (current) version of haproxy ingress controller you need to annotate the ingress with the corresponding "ingress class", either on the fly after having created the Ingress object:

    . . .

    annotations: # <== add this
    kubernetes.io/ingress.class: haproxy # <== add this
    name: tester
    [...]

    . . .

    This was the decisive move, it started to function immediately with the annotation.

    Thank you for the help and for the hints, now understand it much better.

  • crandon
    crandon Posts: 2

    @felixkrohn is completely right. I'd add one more possible solution:

    ---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: haproxy                                                         # <-- This can be custom
spec:
  controller: haproxy-ingress.github.io/controller        # <-- This is fixed as per https://haproxy-ingress.github.io/docs/configuration/keys/#class-matter
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tester
spec:  
  ingressClassName: haproxy                                  # <-- Should match the above IngressClass resource's name
  rules:
  - host: nginx.10.2.0.28.nip.io                                  #<-- Edit this to by YOUR IP
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: tester
            port:
              number: 80

Categories

Upcoming Training