Lab 11.2 Ingress Controller - connection refused on public ip address

spongocoel

I have installed Kubernetes on two Google cloud VM instances. Each node has a public IP address.

I have been through all the lessons and labs step by step and everything has worked correctly and matched the lab output until I got to lab 11.2 Ingress Controller. Everything is working except that the curl command to the public IP address from outside of the cluster is refused.

curl -H "Host: internal.org" [xxx.xxx.xxx.xxx]
curl: (7) Failed to connect to [xxx.xxx.xxx.xxx] port 80: Connection refused

Linkerd is running and accessible at that same IP address (on port 31500). I can ping the public IP address. I followed the early instructions for adding a firewall rule to allow traffic on all ports.

The NGINX ingress controller is installed, but shows a EXTERNAL-IP. I don't know if that's normal because the lab PDF never shows anything different. It doesn't seem correct, though.

default service/myingress-ingress-nginx-controller LoadBalancer 10.111.7.242 80:30499/TCP,443:32138/TCP 93m
default service/myingress-ingress-nginx-controller-admission ClusterIP 10.100.37.237 443/TCP 93m

Can anyone help me figure out what I've missed?

chrispokorni

Hi @spongocoel,

Do you have the output where the nginx ingress controller service shows the external IP assigned to it?

When validating that the nginx ingress pods are running, do you see them distributed as node agents, where each node is running exactly one myingress pod?

Is this behavior observed only when testing ingress to the internal service? What about the external service?

Are you curling straight to the IP address, or do you have http:// prefixing it?

When curling to a node IP address (private or public), keep in mind you are exposing the service as a NodePort, so the node IP alone will not work, you'd need to provide the high NodePort value with curl.

Regards,
-Chris

spongocoel

The ingress controller service does not show an external IP address. I just realized that earlier I wrote "shows a EXTERNAL-IP" when I meant to say "shows no EXTERNAL-IP":

$ kubectl get service -A | grep myingress
default myingress-ingress-nginx-controller LoadBalancer 10.111.7.242 80:30499/TCP,443:32138/TCP 24h
default myingress-ingress-nginx-controller-admission ClusterIP 10.100.37.237 443/TCP 24h

There is an nginx ingress pod running on both nodes:

$ kubectl get pod -A -o=wide | grep myingress
default myingress-ingress-nginx-controller-gz8cb 2/2 Running 2 24h 192.168.171.75 worker
default myingress-ingress-nginx-controller-tpx4x 2/2 Running 2 24h 192.168.219.127 master

Both of these calls from the master or worker node return the correct html response from the web server:

curl -H "Host: internal.org" http://10.111.7.242
curl -H "Host:www.external.com" http://10.111.7.242

Curling to the public ip address of either node, with or without "http://" prefixed results in a "connection refused". I've tried it from the master node of the cluster as well as from my laptop.

My understanding is that I am curling to the public ip address of the nodes in the cluster, and that the ingress service is exposed as a LoadBalancer type. I think the point is to allow access to the cluster on port 80 and via the ingress rules, route to the appropriate pod to respond to the request. The goal, as I understand it, is to not have to use a "weird" high numbered port to serve http requests from the cluster.

I don't see, however, how the node's port 80 is routed to the cluster. I assumed that was part of the ingress controller install, or was part of the Linkerd install (I still don't understand exactly what that install was for).

Thanks for your help. I really appreciate it.

• Scott

spongocoel

I just realized the forum software is eating text within angle brackets. The service external ip address was being removed from the output. I'll try again, replacing angle brackets with square brackets:

$ kubectl get service -A | grep myingress
default myingress-ingress-nginx-controller LoadBalancer 10.111.7.242 [pending] 80:30499/TCP,443:32138/TCP 25h
default myingress-ingress-nginx-controller-admission ClusterIP 10.100.37.237 [none] 443/TCP 25h

chrispokorni

Hi @spongocoel,

Thanks for clarifying.

A LoadBalancer type service, without an actual external load balancer (which is expected in our scenario), will behave as a typical NodePort type service. When attempting to access such a service on public or private node IP addresses, the NodePort needs to be specified as well curl IPaddress:nodeport

Regards,
-Chris

spongocoel

When you say that an external load balancer is expected, is that something I missed in the coursework? Do I need to install something separately for that? Is it covered in the course?

I'm using Google Cloud Platform VMs since that was suggested in the early course materials.

The lab exercises were written and tested using Ubuntu instances running on Google Cloud Platform.

Is that learning I need to do independently, outside of the course?

chrispokorni

Hi @spongocoel,

There is no load balancer included in the lab work, which is the reason why the External-IP field of the service remains in a pending state.

Regards,
-Chris

proliant

I got a similar problem starting from today on lab 11.2

curl -H "Host: www.external.com" x.x.x.x always return 404 Not Found error.
curl -H "Host: internal.org" x.x.x.x also return the same error.

For the past 2 weeks I didn't encounter this problem...

The one I pulled this morning, by default, is ingress-nginx-4.0.1.tgz
The problem is solved after pulling an older version: helm fetch --version 3.36.0 ingress/ingress-nginx

FYI:
https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx
4.0.1 24 Aug, 2021
3.36.0 22 Aug, 2021

My lab is a KVM with 3 Ubuntu 18.04.5 LTS running on a LM 20.1 wtih 32GB RAM computer...

chrispokorni

Hi @proliant,

Thank you for your feedback. I have seen similar issues reported on and off with various versions of the ingress-nginx helm chart, that seem to be related to the configuration of the chart.

Regards,
-Chris

micahel71

Hi,

I ran into the same issue

curl -H "Host: www.external.com" x.x.x.x always return 404 Not Found error.

I found out that the newest version of the ingress-nginx controller (4.0.x) requires some additional infos in the yaml file of the Ingress resource!

According to the lab documenation the yaml should look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-test
  namespace: default
spec:
  rules:
  - host: www.external.com
(...)

That it is not enough anymore!
With 4.0.x you need to declare the "ingress.class" !!
So you need to add the following annotation ;-)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: ingress-test
  namespace: default
spec:
  rules:
(...)

annotations:
kubernetes.io/ingress.class: nginx

does the trick even with 4.0.x

Hope this helps others and the author of the lab documentation incoperates this update.

Cheers, Michael

serewicz

Appreciate the feedback. Indeed there was a breaking change recently. We will update the material and get a new version out soon.

Regards,

captixn

Hi,
The material doesn't seem to be updated. I followed the labs closely and was stuck for two hours until I realized the ingestStorageClass annotation/field was missing.

Just a kind reminder to put a note about IngestClass in the material.

Regards,

savoir

asdf

tfunke

Please update the material.. I don't understand why I had to get stuck here for an hour when this is known since last year September...

hansbogert

Putting it here as well, update this! Run some CI on your documentation. The material costs a comfy penny, surely you can put some CI in place? Devops mentality for your documentation as well please.