How to troubleshoot Ingress tls?

Hello Fazlur,

Overview: pods, services, ingress, secret

student@master:~$ k get all,ing,secret
NAME           READY   STATUS    RESTARTS      AGE
pod/service1   1/1     Running   1 (48m ago)   14h
pod/service2   1/1     Running   1 (48m ago)   14h
 
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   22d
service/service1     ClusterIP   10.102.136.184   <none>        80/TCP    14h
service/service2     ClusterIP   10.96.42.137     <none>        80/TCP    14h
 
NAME                                       CLASS   HOSTS   ADDRESS    PORTS     AGE
ingress.networking.k8s.io/secure-ingress   nginx   *       10.3.0.8   80, 443   14h
 
NAME                    TYPE                DATA   AGE
secret/secure-ingress   kubernetes.io/tls   2      13h

The pods and services service1 and service2 has been created just with commands:

$ k run service1 --image=nginx
$ k run service2 --image=nginx
$ k expose pod service1 --port=80
$ k expose pod service2 --port=80

Ingress:

student@master:~$ k get ing secure-ingress -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/"},"creationTimestamp":"2025-01-15T16:53:24Z","generation":2,"name":"secure-ingress","namespace":"default","resourceVersion":"45424","uid":"9c3e0f93-4696-4556-be4b-2aabd09f5b06"},"spec":{"ingressClassName":"nginx","rules":[{"host":"secure-ingress.com","http":{"paths":[{"backend":{"service":{"name":"service1","port":{"number":80}}},"path":"/service1","pathType":"Prefix"},{"backend":{"service":{"name":"service2","port":{"number":80}}},"path":"/service2","pathType":"Prefix"}]}}]},"status":{"loadBalancer":{"ingress":[{"ip":"10.3.0.8"}]}}}
    nginx.ingress.kubernetes.io/rewrite-target: /
  creationTimestamp: "2025-01-15T16:53:24Z"
  generation: 6
  name: secure-ingress
  namespace: default
  resourceVersion: "53142"
  uid: 9c3e0f93-4696-4556-be4b-2aabd09f5b06
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: service1
            port:
              number: 80
        path: /service1
        pathType: Prefix
      - backend:
          service:
            name: service2
            port:
              number: 80
        path: /service2
        pathType: Prefix
  tls:
  - hosts:
    - secure-ingress.com
    secretName: secure-ingress
status:
  loadBalancer:
    ingress:
    - ip: 10.3.0.8

Secret: certificate

student@master:~$ k get secrets secure-ingress -o yaml | grep tls.crt | sed 's/  tls.crt: //g' | base64 -d |  openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            07:2f:4e:67:a7:01:32:9d:6f:3f:c3:e3:e7:50:a7:c1:1d:15:c1:8f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = secure-ingress.com
        Validity
            Not Before: Jan 15 17:32:29 2025 GMT
            Not After : Jan 15 17:32:29 2026 GMT
        Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = secure-ingress.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
... skipped ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                3D:0A:11:E6:BD:16:E1:F9:99:D0:1A:BD:15:CD:4F:F1:48:8F:0F:DA
            X509v3 Authority Key Identifier:
                keyid:3D:0A:11:E6:BD:16:E1:F9:99:D0:1A:BD:15:CD:4F:F1:48:8F:0F:DA
 
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
... skipped ...

Ingress Controller secure port: 31766

student@master:~$ k get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller   NodePort   10.109.18.140   <none>        80:32633/TCP,443:31766/TCP   14h

For testing I used following curl command from control plane (with output)

student@master:~$ curl -kv https://secure-ingress.com:31766/service1 --resolve secure-ingress.com:31766:$(curl ifconfig.io)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0    161      0 --:--:-- --:--:-- --:--:--   161
* Added secure-ingress.com:31766:35.208.133.226 to DNS cache
* Hostname secure-ingress.com was found in DNS cache
*   Trying 35.208.133.226:31766...
* TCP_NODELAY set
* Connected to secure-ingress.com (35.208.133.226) port 31766 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Jan 16 06:28:22 2025 GMT
*  expire date: Jan 16 06:28:22 2026 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5602c1acb0e0)
  GET /service1 HTTP/2
  Host: secure-ingress.com:31766
  user-agent: curl/7.68.0
  accept: */*
 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
  HTTP/2 200
  date: Thu, 16 Jan 2025 07:20:43 GMT
  content-type: text/html
  content-length: 615
  last-modified: Tue, 26 Nov 2024 15:55:00 GMT
  etag: "6745ef54-267"
  accept-ranges: bytes
  strict-transport-security: max-age=15724800; includeSubDomains
... reponse content is skipped ...
 Connection #0 to host secure-ingress.com left intact

Thank you and sorry for lengthy message,
Dmytro

Well, thank you very much everybody.

I specified host in a rule like this

. . .
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
. . .
    host: secure-ingress.com # ADDED THIS LINE
  tls:
  - hosts:
    - secure-ingress.com
    secretName: secure-ingress
. . .

and ingress started to use secure-ingress instead of Fake, as I wanted. It may seem that having host in a rule part is essential to make tls work, so I did another experiment and then removed it. I expected that ingress would start to use Fake as before, but it continued to use the right secret (i.e. secure-ingress) despite absent host in a rule.

If I understand correctly, in order to choose the right rule (and hence service to serve the request) ingress uses HTTP header Host. This works with both plain HTTP and HTTPS protocols. But ingress can't choose the right tls secret with this approach because it has make the right decision on TLS shandshake and there is no HTTP header at this moment. Therefore it uses SNI TLS extention (where client sends desired hostname in TLS negotiation's ClientHello message). So, these mechanisms are completely indepdendent and therefore technically there is no need to have host specified in a single rule. And perhaps that's why tls section is not inside rules in ingress resource. However, it looks like NGINX Ingress Controller has some issues when it tries to convert ingress resource into internal NGINX configuration when it has tls, but doesn't have host in rules.