Welcome to the Linux Foundation Forum!

Lab 11.2 Ingress Controller - connection refused on public ip address

I have installed Kubernetes on two Google cloud VM instances. Each node has a public IP address.

I have been through all the lessons and labs step by step and everything has worked correctly and matched the lab output until I got to lab 11.2 Ingress Controller. Everything is working except that the curl command to the public IP address from outside of the cluster is refused.

curl -H "Host: internal.org" [xxx.xxx.xxx.xxx]
curl: (7) Failed to connect to [xxx.xxx.xxx.xxx] port 80: Connection refused

Linkerd is running and accessible at that same IP address (on port 31500). I can ping the public IP address. I followed the early instructions for adding a firewall rule to allow traffic on all ports.

The NGINX ingress controller is installed, but shows a EXTERNAL-IP. I don't know if that's normal because the lab PDF never shows anything different. It doesn't seem correct, though.

default service/myingress-ingress-nginx-controller LoadBalancer 10.111.7.242 80:30499/TCP,443:32138/TCP 93m
default service/myingress-ingress-nginx-controller-admission ClusterIP 10.100.37.237 443/TCP 93m

Can anyone help me figure out what I've missed?

Comments

  • chrispokorni
    chrispokorni Posts: 1,170

    Hi @spongocoel,

    Do you have the output where the nginx ingress controller service shows the external IP assigned to it?

    When validating that the nginx ingress pods are running, do you see them distributed as node agents, where each node is running exactly one myingress pod?

    Is this behavior observed only when testing ingress to the internal service? What about the external service?

    Are you curling straight to the IP address, or do you have http:// prefixing it?

    When curling to a node IP address (private or public), keep in mind you are exposing the service as a NodePort, so the node IP alone will not work, you'd need to provide the high NodePort value with curl.

    Regards,
    -Chris

  • The ingress controller service does not show an external IP address. I just realized that earlier I wrote "shows a EXTERNAL-IP" when I meant to say "shows no EXTERNAL-IP":

    $ kubectl get service -A | grep myingress
    default myingress-ingress-nginx-controller LoadBalancer 10.111.7.242 80:30499/TCP,443:32138/TCP 24h
    default myingress-ingress-nginx-controller-admission ClusterIP 10.100.37.237 443/TCP 24h

    There is an nginx ingress pod running on both nodes:

    $ kubectl get pod -A -o=wide | grep myingress
    default myingress-ingress-nginx-controller-gz8cb 2/2 Running 2 24h 192.168.171.75 worker
    default myingress-ingress-nginx-controller-tpx4x 2/2 Running 2 24h 192.168.219.127 master

    Both of these calls from the master or worker node return the correct html response from the web server:

    curl -H "Host: internal.org" http://10.111.7.242
    curl -H "Host:www.external.com" http://10.111.7.242

    Curling to the public ip address of either node, with or without "http://" prefixed results in a "connection refused". I've tried it from the master node of the cluster as well as from my laptop.

    My understanding is that I am curling to the public ip address of the nodes in the cluster, and that the ingress service is exposed as a LoadBalancer type. I think the point is to allow access to the cluster on port 80 and via the ingress rules, route to the appropriate pod to respond to the request. The goal, as I understand it, is to not have to use a "weird" high numbered port to serve http requests from the cluster.

    I don't see, however, how the node's port 80 is routed to the cluster. I assumed that was part of the ingress controller install, or was part of the Linkerd install (I still don't understand exactly what that install was for).

    Thanks for your help. I really appreciate it.

    • Scott
  • I just realized the forum software is eating text within angle brackets. The service external ip address was being removed from the output. I'll try again, replacing angle brackets with square brackets:

    $ kubectl get service -A | grep myingress
    default myingress-ingress-nginx-controller LoadBalancer 10.111.7.242 [pending] 80:30499/TCP,443:32138/TCP 25h
    default myingress-ingress-nginx-controller-admission ClusterIP 10.100.37.237 [none] 443/TCP 25h

  • chrispokorni
    chrispokorni Posts: 1,170

    Hi @spongocoel,

    Thanks for clarifying.

    A LoadBalancer type service, without an actual external load balancer (which is expected in our scenario), will behave as a typical NodePort type service. When attempting to access such a service on public or private node IP addresses, the NodePort needs to be specified as well curl IPaddress:nodeport

    Regards,
    -Chris

  • When you say that an external load balancer is expected, is that something I missed in the coursework? Do I need to install something separately for that? Is it covered in the course?

    I'm using Google Cloud Platform VMs since that was suggested in the early course materials.

    The lab exercises were written and tested using Ubuntu instances running on Google Cloud Platform.

    Is that learning I need to do independently, outside of the course?

  • chrispokorni
    chrispokorni Posts: 1,170

    Hi @spongocoel,

    There is no load balancer included in the lab work, which is the reason why the External-IP field of the service remains in a pending state.

    Regards,
    -Chris

  • proliant
    proliant Posts: 5
    edited August 25

    I got a similar problem starting from today on lab 11.2

    curl -H "Host: www.external.com" x.x.x.x always return 404 Not Found error.
    curl -H "Host: internal.org" x.x.x.x also return the same error.

    For the past 2 weeks I didn't encounter this problem...

    The one I pulled this morning, by default, is ingress-nginx-4.0.1.tgz
    The problem is solved after pulling an older version: helm fetch --version 3.36.0 ingress/ingress-nginx

    FYI:
    https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx
    4.0.1 24 Aug, 2021
    3.36.0 22 Aug, 2021

    My lab is a KVM with 3 Ubuntu 18.04.5 LTS running on a LM 20.1 wtih 32GB RAM computer...

  • chrispokorni
    chrispokorni Posts: 1,170

    Hi @proliant,

    Thank you for your feedback. I have seen similar issues reported on and off with various versions of the ingress-nginx helm chart, that seem to be related to the configuration of the chart.

    Regards,
    -Chris

  • Hi,

    I ran into the same issue

    curl -H "Host: www.external.com" x.x.x.x always return 404 Not Found error.

    I found out that the newest version of the ingress-nginx controller (4.0.x) requires some additional infos in the yaml file of the Ingress resource!

    According to the lab documenation the yaml should look like this:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: ingress-test
      namespace: default
    spec:
      rules:
      - host: www.external.com
    (...)
    

    That it is not enough anymore!
    With 4.0.x you need to declare the "ingress.class" !!
    So you need to add the following annotation ;-)

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        kubernetes.io/ingress.class: nginx
      name: ingress-test
      namespace: default
    spec:
      rules:
    (...)
    

    annotations:
    kubernetes.io/ingress.class: nginx

    does the trick even with 4.0.x

    Hope this helps others and the author of the lab documentation incoperates this update.

    Cheers, Michael

  • serewicz
    serewicz Posts: 950

    Appreciate the feedback. Indeed there was a breaking change recently. We will update the material and get a new version out soon.

    Regards,

Categories

Upcoming Training