Lab 10.1. "Connection refused" after Ingress setup

Gim6626

Hi!

I'm following guide from "Lab 10.1. Advanced Service Exposure". All steps succeeded, but after adopting ingress.rule.yaml (I needed to change it a bit after format changes like serviceName -> service.name etc) and kubectl create -f ingress.rule.yaml I get following error while trying to test my setup:

$ curl -H "Host: www.example.com" http://k8smaster/
curl: (7) Failed to connect to k8smaster port 80: Connection refused

Seems like something is not setup correctly, but I'm new to k8s and have no idea how to investigate and solve it.

Could anybody help me?

Thanks in advance.

jcremp77

Hi @Gim6626 -

Not sure why you had to make that change. The file in my lab is like the lab guide. Change 'service.name' back to 'serviceName'; and check all the indentations. Should work. I just completed the lab.

chrispokorni

Hi @Gim6626,

On the machine where the curl command is run, is k8smaster configured as the alias to your control-plane/master node, or any other node where a traefik controller is running?

Regards,
-Chris

Gim6626

Thank you for trying to help me!

@jcremp77

Tried again to use original config, here is result:

$ kubectl create -f ingress.rule.yaml
error: error validating "ingress.rule.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false

And here is the link from where I got a fix - https://stackoverflow.com/questions/64125048/get-error-unknown-field-servicename-in-io-k8s-api-networking-v1-ingressbacken

@chrispokorni

Sure:

$ grep k8smaster /etc/hosts
192.168.56.104 k8smaster
$ ping k8smaster
PING k8smaster (192.168.56.104) 56(84) bytes of data.
64 bytes from k8smaster (192.168.56.104): icmp_seq=1 ttl=64 time=0.050 ms

chrispokorni

Hi @Gim6626,

The validation error is YAML related. Just recently the Ingress API resource matured from beta to stable support. This introduced a several changes into the YAML definition of the object. The first affected property is the API version, which changed from networking.k8s.io/v1beta1 to networking.k8s.io/v1. While the beta version is being deprecated (before it is no longer supported), the API will accept both versions: beta (v1beta1) and stable (v1).

The stable level of support also introduces changes in the format of the YAML manifest that defines the Ingress resource. During this transition phase, the Kubernetes API supports both Ingress resource API formats, but the API version declared at the top of the definition file and the included properties have to match the correct version.

From your output it seems that your nodes may be assigned IP addresses (192.168.56.104) from the same range as the default pod network range 192.168.0.0/16 managed y the Calico plugin. Is this your case? This causes networking, routing, and DNS issues in your cluster. The network IP addresses of your nodes should not overlap with the pods network and with the services Cluster IP subnet.

Regards,
-Chris

kaiwhata

Hi there,
I'm having the same issue for the same reason (i think!), but I'm not sure how to resolve.
When I run kubectl create -f ingress.rule.yaml using the course tarball ingress.rule.yaml i get:
error: error validating "ingress.rule.orig.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false
Which is due to the v1beta ->v1 switch. Switching networking.k8s.io/v1 to networking.k8s.io/v1beta1 resolves this issue for me.

The nginx (secondapp) is available to ClusterIPs ( curl -H "Host: www.example.com" http://192.168.226.65:80/ and curl -H "Host: www.example.com" http://10.97.198.188:80/ both return the service), and the service is available on my external IP (due to the NodePort using the high default port 30713). However, I cant seem to allow external access via port 80 - which leads me to think this is still an Ingress configuration issue.

For reference - my master node IP is 10.2.0.7 (and this is the alias set in /etc/hosts for k8smaster)
A couple of other things I have noticed:
1) kubectl get services shows no external IP set:
elf@master-1:~$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 109m
secondapp NodePort 10.97.198.188 80:30713/TCP 84m

2) kubectl get ingress shows no address for teh ingress:
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-test www.example.com 80 12m

So im reasonably certain there's some configuration issue with the ingress, even though it is being accepted - but it could also be the configuration of the backend service.

Any pointers/suggestions would be much appreciated!
Cheers

kaiwhata

As a follow up - kubectl describe ingress ended up showing that this was due to an error with the default-http-backend (which was why the backend wasn't pointing to realisic cluster IPs). Output:
kubectl describe ingress
Name: ingress-test
Namespace: default
Address:
Default backend: default-http-backend:80 ()
Rules:
Host Path Backends
---- ---- --------
www.example.com
/ secondapp:80 (192.168.226.65:80)
thirdpage.org
/ thirdpage:80 (192.168.226.66:80)
Annotations: kubernetes.io/ingress.class: traefik
Events:

chrispokorni

Hi @kaiwhata,

The Ingress resource just graduated to stable level of support, and the format of its yaml manifest slightly changed. If you take a look at the documentation on Ingress, you will find this section most up-to-date yaml format that can be used for this lab exercise.

If you pay close attention to the lab steps where the ingress is tested, curl is run against IP addresses of the Nodes (both public and private), not the Pod IPs, and not the Service ClusterIPs.

Regards,
-Chris

kaiwhata

Thanks Chris. For clarity on my part - my node IP in this instance is 10.2.0.7 (which is running on the master, the same as the setup given in the lab 10.1). Attempting to curl this (or the k8smaster alias) yields:
curl -H "Host: www.example.com" http://10.2.0.7/ curl: (7) Failed to connect to 10.2.0.7 port 80: Connection refused

My curling of the other IPs was simply to demonstrate that the service appears to be running correctly. For reference (and to help anyone else stuck with this issue) the two ingress.rule.yaml files that is used for the api/v1 and the apiv1beta are:

cat ingress.rule.v1betav1.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-test annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: www.example.com http: paths: - path: / backend: serviceName: secondapp servicePort: 80

cat ingress.rule.v1.yaml
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-test annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: "www.example.com" http: paths: - pathType: ImplementationSpecific path: "/" backend: service: name: secondapp port: number: 80
If you have any more debugging advice that would be much appreciated

kaiwhata

Hi Chris, thanks. I included the curl requests to the Pod and Service IPs previously just to show that the Pod and Service appear to be working. Curling the Node IP (which in this case, is that of my master node, 10.2.0.7) returns a 'port 80 connection refused' error.

In case it helps other students here are the two versions of the ingress.rule.yaml file that I used to bypass the validation error (one for v1 and one for v1betav1):

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-test
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: secondapp
          servicePort: 80

and

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-test
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: "www.example.com"
    http:
      paths:
      - pathType: ImplementationSpecific
        path: "/"
        backend:
          service:
            name: secondapp
            port:
              number: 80

Any further advice on how to troubleshoot this would be much appreciated.

kaiwhata

Ah ok. I think I understand now. From Kubernetes v1.18, alongside defining the ingress you must also define an ingress class (https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation). Here's my ingress class file -

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: traefik
spec:
  controller: example.com/traefik-ingress-controller
  parameters:
    apiGroup: k8s.example.com
    kind: IngressParameters
    name: traefik

No errors upon creation - but still doesnt appear to resolve the issue. I'll attempt to use a default IngressClass instead of traefik

chrispokorni

Hi @kaiwhata,

The API seems to support the annotation of the Ingress resource in Kubernetes v1.20, without producing any errors or validation warnings.

The curl header inserted with the -Htag is helpful when testing the Ingress, otherwise curling Pod IPs and Service ClusterIPs do not require the header tag.

After deploying the ingress resource, and the Traefik ingress controller, how many traefik-ingress-controller-... Pod instances do you have running in your cluster? Is there an instance running on your master node (IP 10.2.0.7)?

Regards,
-Chris

benrio

Hi @chrispokorni and all,

I am running into exactly the same issues than @kaiwhata: Connection refused when curling the k8smaster alias or the master node "public" IP to reach secondapp.

First, I confirm that the API accepts both versions v1 and v1beta1. Using v1beta1 will only lead to a warning, and the yaml output will be reformatted to the new standard. Therefore the current issue is not related to the version of the API.

To answer your question, as expected there is one traefik-ingress-controller-... pod running in the kube-system namespace, on the **worker **node. Should this instance run on the master node? If so, I'll need to untaint the master node to let it run there, but I don't believe this is the issue.

Still new to Kubernetes, so any ideas to troubleshoot this issue would be greatly appreciated.
Also, has anyone tested the nginx ingress controller so far, instead of the traefik one?

Thanks and Regards, Ben

benrio

Note that I updated the Traefik image from 1.7.13 to 1.7.24 as suggested in another thread. Still the same issue.

benrio

Also, note that there is no IP overlap between Node and Pods subnet, like I could read for others. Nodes having 10.1.3.0/24, and Pods having 192.168.0.0/16 CIDR.

chrispokorni

Hi @benrio,

The behavior of you ingress is expected based on the details you shared. With an ingress controller instance only running on the worker node, you'd need to test your ingress with the worker node's IP address. This may be the result of a missed step in lab 3, and if the taint still exists on your master node.

Regards,
-Chris

benrio

Master untainted, Ingress controller now running on the master node: Everything works now as expected.
Thank you very much!