Welcome to the Linux Foundation Forum!

How to troubleshoot Ingress tls?

Hi,

During preparation to exam I installed NGINX Ingress controller.

I followed these instructions:
https://github.com/killer-sh/cks-course-environment/blob/master/Resources.md#cluster-setup---secure-ingress

Also I created 2 pods (service1 and service2 with --image=nginx) and exposed them as ClusterIP services.

Then created Ingress (without tls), which looks exactly the same as here:
https://github.com/killer-sh/cks-course-environment/blob/master/course-content/cluster-setup/secure-ingress/secure-ingress-step1.yaml

I tested it and it worked well: routing to both services worked, http and https worked as well, but https worked with Fake certificate, which is ok for now.

Then I created key and self-signed certificate with CN=secure-ingress.com, created secret and configured it in Ingress.

  tls:
  - hosts:
      - secure-ingress.com
    secretName: secure-ingress

and tested it again with curl command and --resolve parameter.

The routing worked as before, but it still used Fake certificate, not the one I just generated. I doublechecked that ingress and its secret are in the same namespace, it's 'default' in my case.
Also, I looked into ingress-nginx-controller logs:

k logs -n ingress-nginx pod/ingress-nginx-controller-7858c949d8-rpf5g -f

I saw requests like

192.168.0.0 - - [15/Jan/2025:17:43:55 +0000] "GET /service1/ HTTP/2.0" 200 9 "-" "curl/7.68.0" 45 0.000 [default-service1-80] [] 192.168.1.5:80 9 0.000 200 a640bec4a5bfcd847af2c525d95a970e

but nothing about why it chooses default certificate.

So, how can I figure out why ingress chooses wrong tls-secret?

Thanks,
Dmytro

Best Answer

  • chrispokorni
    chrispokorni Posts: 2,419
    Answer ✓

    Hi @dmsheiko,

    The official Kubernetes documentation has a clear note about this requirement:

    Note:
    ... Therefore, hosts in the tls section need to explicitly match the host in the rules section.

    Regards,
    -Chris

Answers

  • Hello @dmsheiko ,

    Double check your "host name" under spec.rules and spec.tls, and also ensure the its matching the CN name and you used the resolve command with same domain name. if there is a mismatch then it will use the default fake certificate.

    If you need further help, please post your ingress file and the command you used to validate so that I can investigate further and help.

    regards,
    Fazlur

  • dmsheiko
    dmsheiko Posts: 28

    Hello Fazlur,

    Overview: pods, services, ingress, secret

    student@master:~$ k get all,ing,secret
    NAME           READY   STATUS    RESTARTS      AGE
    pod/service1   1/1     Running   1 (48m ago)   14h
    pod/service2   1/1     Running   1 (48m ago)   14h
    
    NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
    service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   22d
    service/service1     ClusterIP   10.102.136.184   <none>        80/TCP    14h
    service/service2     ClusterIP   10.96.42.137     <none>        80/TCP    14h
    
    NAME                                       CLASS   HOSTS   ADDRESS    PORTS     AGE
    ingress.networking.k8s.io/secure-ingress   nginx   *       10.3.0.8   80, 443   14h
    
    NAME                    TYPE                DATA   AGE
    secret/secure-ingress   kubernetes.io/tls   2      13h
    

    The pods and services service1 and service2 has been created just with commands:

    $ k run service1 --image=nginx
    $ k run service2 --image=nginx
    $ k expose pod service1 --port=80
    $ k expose pod service2 --port=80
    

    Ingress:

    student@master:~$ k get ing secure-ingress -o yaml
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/"},"creationTimestamp":"2025-01-15T16:53:24Z","generation":2,"name":"secure-ingress","namespace":"default","resourceVersion":"45424","uid":"9c3e0f93-4696-4556-be4b-2aabd09f5b06"},"spec":{"ingressClassName":"nginx","rules":[{"host":"secure-ingress.com","http":{"paths":[{"backend":{"service":{"name":"service1","port":{"number":80}}},"path":"/service1","pathType":"Prefix"},{"backend":{"service":{"name":"service2","port":{"number":80}}},"path":"/service2","pathType":"Prefix"}]}}]},"status":{"loadBalancer":{"ingress":[{"ip":"10.3.0.8"}]}}}
        nginx.ingress.kubernetes.io/rewrite-target: /
      creationTimestamp: "2025-01-15T16:53:24Z"
      generation: 6
      name: secure-ingress
      namespace: default
      resourceVersion: "53142"
      uid: 9c3e0f93-4696-4556-be4b-2aabd09f5b06
    spec:
      ingressClassName: nginx
      rules:
      - http:
          paths:
          - backend:
              service:
                name: service1
                port:
                  number: 80
            path: /service1
            pathType: Prefix
          - backend:
              service:
                name: service2
                port:
                  number: 80
            path: /service2
            pathType: Prefix
      tls:
      - hosts:
        - secure-ingress.com
        secretName: secure-ingress
    status:
      loadBalancer:
        ingress:
        - ip: 10.3.0.8
    

    Secret: certificate

    student@master:~$ k get secrets secure-ingress -o yaml | grep tls.crt | sed 's/  tls.crt: //g' | base64 -d |  openssl x509 -text -noout
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number:
                07:2f:4e:67:a7:01:32:9d:6f:3f:c3:e3:e7:50:a7:c1:1d:15:c1:8f
            Signature Algorithm: sha256WithRSAEncryption
            Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = secure-ingress.com
            Validity
                Not Before: Jan 15 17:32:29 2025 GMT
                Not After : Jan 15 17:32:29 2026 GMT
            Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = secure-ingress.com
            Subject Public Key Info:
                Public Key Algorithm: rsaEncryption
                    RSA Public-Key: (4096 bit)
                    Modulus:
    ... skipped ...
                    Exponent: 65537 (0x10001)
            X509v3 extensions:
                X509v3 Subject Key Identifier:
                    3D:0A:11:E6:BD:16:E1:F9:99:D0:1A:BD:15:CD:4F:F1:48:8F:0F:DA
                X509v3 Authority Key Identifier:
                    keyid:3D:0A:11:E6:BD:16:E1:F9:99:D0:1A:BD:15:CD:4F:F1:48:8F:0F:DA
    
                X509v3 Basic Constraints: critical
                    CA:TRUE
        Signature Algorithm: sha256WithRSAEncryption
    ... skipped ...
    

    Ingress Controller secure port: 31766

    student@master:~$ k get svc -n ingress-nginx ingress-nginx-controller
    NAME                       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
    ingress-nginx-controller   NodePort   10.109.18.140   <none>        80:32633/TCP,443:31766/TCP   14h
    

    For testing I used following curl command from control plane (with output)

    student@master:~$ curl -kv https://secure-ingress.com:31766/service1 --resolve secure-ingress.com:31766:$(curl ifconfig.io)
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100    15  100    15    0     0    161      0 --:--:-- --:--:-- --:--:--   161
    * Added secure-ingress.com:31766:35.208.133.226 to DNS cache
    * Hostname secure-ingress.com was found in DNS cache
    *   Trying 35.208.133.226:31766...
    * TCP_NODELAY set
    * Connected to secure-ingress.com (35.208.133.226) port 31766 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * successfully set certificate verify locations:
    *   CAfile: /etc/ssl/certs/ca-certificates.crt
      CApath: /etc/ssl/certs
    * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    * TLSv1.3 (IN), TLS handshake, Server hello (2):
    * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
    * TLSv1.3 (IN), TLS handshake, Certificate (11):
    * TLSv1.3 (IN), TLS handshake, CERT verify (15):
    * TLSv1.3 (IN), TLS handshake, Finished (20):
    * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
    * TLSv1.3 (OUT), TLS handshake, Finished (20):
    * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
    * ALPN, server accepted to use h2
    * Server certificate:
    *  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
    *  start date: Jan 16 06:28:22 2025 GMT
    *  expire date: Jan 16 06:28:22 2026 GMT
    *  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
    *  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
    * Using HTTP2, server supports multi-use
    * Connection state changed (HTTP/2 confirmed)
    * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
    * Using Stream ID: 1 (easy handle 0x5602c1acb0e0)
      GET /service1 HTTP/2
      Host: secure-ingress.com:31766
      user-agent: curl/7.68.0
      accept: */*
    
    * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
    * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
    * old SSL session ID is stale, removing
    * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
      HTTP/2 200
      date: Thu, 16 Jan 2025 07:20:43 GMT
      content-type: text/html
      content-length: 615
      last-modified: Tue, 26 Nov 2024 15:55:00 GMT
      etag: "6745ef54-267"
      accept-ranges: bytes
      strict-transport-security: max-age=15724800; includeSubDomains
    ... reponse content is skipped ...
     Connection #0 to host secure-ingress.com left intact
    

    Thank you and sorry for lengthy message,
    Dmytro

  • chrispokorni
    chrispokorni Posts: 2,419

    Hi @dmsheiko,

    First I'd check the age of the sources used for this scenario. The vendor seems to have posted these examples between 6 months and 4 years ago. Keeping this in mind, I would take a close look at the official Kubernetes documentation https://kubernetes.io/docs/concepts/services-networking/ingress/#tls, and perhaps the ingress controller official documentation https://kubernetes.github.io/ingress-nginx/user-guide/tls/#tlshttps. They provide samples with up-to-date manifest format and syntax that should be used when defining the required objects.

    Regards,
    -Chris

  • dmsheiko
    dmsheiko Posts: 28

    Well, thank you very much everybody.

    I specified host in a rule like this

    . . .
    spec:
      ingressClassName: nginx
      rules:
      - http:
          paths:
    . . .
        host: secure-ingress.com # ADDED THIS LINE
      tls:
      - hosts:
        - secure-ingress.com
        secretName: secure-ingress
    . . .
    

    and ingress started to use secure-ingress instead of Fake, as I wanted. It may seem that having host in a rule part is essential to make tls work, so I did another experiment and then removed it. I expected that ingress would start to use Fake as before, but it continued to use the right secret (i.e. secure-ingress) despite absent host in a rule.

    If I understand correctly, in order to choose the right rule (and hence service to serve the request) ingress uses HTTP header Host. This works with both plain HTTP and HTTPS protocols. But ingress can't choose the right tls secret with this approach because it has make the right decision on TLS shandshake and there is no HTTP header at this moment. Therefore it uses SNI TLS extention (where client sends desired hostname in TLS negotiation's ClientHello message). So, these mechanisms are completely indepdendent and therefore technically there is no need to have host specified in a single rule. And perhaps that's why tls section is not inside rules in ingress resource. However, it looks like NGINX Ingress Controller has some issues when it tries to convert ingress resource into internal NGINX configuration when it has tls, but doesn't have host in rules.

  • dmsheiko
    dmsheiko Posts: 28

    Hi Chris,

    Ok, what is the point of such requirement? If I have single host why it has to be specified in both tls and rules sections? Or if I expose the same site but with different domain names (e.g. foobar.com, foobar.net, foobar.org, foobar.biz,...) why should I duplicate the same rules for each domain name?

    And also, why did ingress controller use normal (non-Fake) tls secret when I removed host in a rule? Shouldn't it revert to Fake according to the requirement?

    Thanks,
    Dmytro

  • chrispokorni
    chrispokorni Posts: 2,419

    Hi @dmsheiko,

    You may have just uncovered a bug?!?

    Regards,
    -Chris

Categories

Upcoming Training