Lab 16.2.3 - neither proxynode nor k8sServers started

chriswigmore

I've spun up a new node and installed HAProxy on it. I've edited the haproxy.cfg file as shown in the lab instructions, restarted the service and checked the status. I get the following output:
... systemd[1]: Starting HAProxy Load Balancer...
... haproxy[17833]: [WARNING] (17833) : config : 'stats' statement ignored for frontend 'proxynode' as it requires HTTP mode.
... haproxy[17833]: [NOTICE] (17833) : New worker #1 (17835) forked
... systemd[1]: Started HAProxy Load Balancer.
Note: no mention of proxynode nor k8sServers starting, but no error messages to help me work out why not (only the warning about the stats statement - don't know if that's relevant or not). Could someone help me troubleshoot this?

Things I've tried:

• sudo journalctl -xeu haproxy.service (still no error messages)
• Using the haproxy.cfg file from the SOLUTIONS tarball (with appropriate IP tweaks)
• Uninstalling and reinstalling haproxy
• Stopping and starting the service (rather than restarting)
• Running haproxy in the foreground (via sudo haproxy -f /etc/haproxy/haproxy.cfg -db)

chriswigmore

Chris,
Update - I blew away the 22.04 proxy node and tried on a new one with Ubuntu 20.04, and it now works as expected. Since it worked for you on both versions, I suspect there was just something wrong with that particular VM.
Thanks again for your help.

chrispokorni

Hi @chriswigmore,

What type of infrastructure are you using for your cluster nodes (control-plane-1, worker, control-plane-2, control-plane-3, and others if any) and the haproxy load balancer?

With all necessary settings in place, what does your haproxy.cfg look like? Please provide its content in code block to help check for possible typos.

Regards,
-Chris

chriswigmore

Chris, thanks for getting back to me. To answer your questions:

• I'm using AWS EC2s for all 5 nodes (3 cp nodes, 1 worker node, 1 proxy node), with specs as per the guidance (t2.large, 20 gp2 storage, etc)
• My haproxy.cfg is below (apologies - the markdown isn't ideal):

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    tcp
        option  tcplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend proxynode
   bind *:80
   bind *:6443
   stats uri /proxystats
   default_backend k8sServers

backend k8sServers
   balance roundrobin
   server cp1  172.31.36.248:6443 check  #<-- Edit with your IP addresses.
#   server cp2  10.128.0.30:6443 check
#   server cp3  10.128.0.66:6443 check

listen stats
     bind :9999
     mode http
     stats enable
     stats hide-version
     stats uri /stats

chrispokorni

Hi @chriswigmore,

Is your hostname "cp1" or something else? The "cp1", "cp2" and "cp3" strings above should be replaced with the hostnames of the respective control plane nodes, which should be somehow derived from the IP address (based on AWS EC2 naming standards). The IP address edit looks correct though.

Regards,
-Chris

chriswigmore

Thanks for the suggestion, Chris. The hostname is actually 'ip-172-31-36-248'. I tried using that instead of cp1, but still no joy.

chrispokorni

Hi @chriswigmore,

Is the haproxy instance in the same VPC and same SG as your control plane and worker instances?

Regards,
-Chris

chriswigmore

Chris,

Same VPC = yes
Same SG = yes

If it helps, I just tried restarting the haproxy service with the cp node stopped. I saw the following output:

... haproxy[993]: [WARNING]  (993) : Server k8sServers/ip-172-31-36-248 is DOWN, reason: Layer4 connection problem, info: "Connection refused", chec>
... haproxy[993]: [NOTICE]   (993) : haproxy version is 2.4.22-0ubuntu0.22.04.1
... haproxy[993]: [NOTICE]   (993) : path to executable is /usr/sbin/haproxy
... haproxy[993]: [ALERT]    (993) : backend 'k8sServers' has no server available!

So it seems that, if it can't reach the cp node, it lets us know. Which suggests the problem is something else.

chrispokorni

Hi @chriswigmore,

Are your VMs on 22.04 by any chance?

I am trying to spin up a new cluster on AWS, with the recommended OS release though - 20.04, to validate the haproxy config, and to see if I can reproduce your issue.

Regards,
-Chris

chriswigmore

Chris,
The cp node is 20.04 (I first span it up months ago). The proxy node is 22.04.

chrispokorni

Hi @chriswigmore,

I tried to reproduce the issue, but both attempts were in fact successful. I tried haproxy running on Ubuntu 20.04 and 22.04 and nothing unexpected happened.

Did you notice anything strange with the cp node throughout the earlier lab exercises?

Regards,
-Chris

chriswigmore

Chris,
Thanks for investigating. No, I didn't notice anything strange with the cp node - it was well behaved throughout the earlier labs.