Welcome to the Linux Foundation Forum!

Lab 16.2.3 - neither proxynode nor k8sServers started

Options
chriswigmore
chriswigmore Posts: 7
edited May 2023 in LFS258 Class Forum

I've spun up a new node and installed HAProxy on it. I've edited the haproxy.cfg file as shown in the lab instructions, restarted the service and checked the status. I get the following output:
... systemd[1]: Starting HAProxy Load Balancer...
... haproxy[17833]: [WARNING] (17833) : config : 'stats' statement ignored for frontend 'proxynode' as it requires HTTP mode.
... haproxy[17833]: [NOTICE] (17833) : New worker #1 (17835) forked
... systemd[1]: Started HAProxy Load Balancer.
Note: no mention of proxynode nor k8sServers starting, but no error messages to help me work out why not (only the warning about the stats statement - don't know if that's relevant or not). Could someone help me troubleshoot this?

Things I've tried:

  • sudo journalctl -xeu haproxy.service (still no error messages)
  • Using the haproxy.cfg file from the SOLUTIONS tarball (with appropriate IP tweaks)
  • Uninstalling and reinstalling haproxy
  • Stopping and starting the service (rather than restarting)
  • Running haproxy in the foreground (via sudo haproxy -f /etc/haproxy/haproxy.cfg -db)

Best Answer

  • chriswigmore
    chriswigmore Posts: 7
    Answer ✓
    Options

    Chris,
    Update - I blew away the 22.04 proxy node and tried on a new one with Ubuntu 20.04, and it now works as expected. Since it worked for you on both versions, I suspect there was just something wrong with that particular VM.
    Thanks again for your help.

Answers

  • chrispokorni
    chrispokorni Posts: 2,224
    Options

    Hi @chriswigmore,

    What type of infrastructure are you using for your cluster nodes (control-plane-1, worker, control-plane-2, control-plane-3, and others if any) and the haproxy load balancer?

    With all necessary settings in place, what does your haproxy.cfg look like? Please provide its content in code block to help check for possible typos.

    Regards,
    -Chris

  • chriswigmore
    chriswigmore Posts: 7
    edited May 2023
    Options

    Chris, thanks for getting back to me. To answer your questions:

    • I'm using AWS EC2s for all 5 nodes (3 cp nodes, 1 worker node, 1 proxy node), with specs as per the guidance (t2.large, 20 gp2 storage, etc)
    • My haproxy.cfg is below (apologies - the markdown isn't ideal):
    global
            log /dev/log    local0
            log /dev/log    local1 notice
            chroot /var/lib/haproxy
            stats socket /run/haproxy/admin.sock mode 660 level admin
            stats timeout 30s
            user haproxy
            group haproxy
            daemon
    
            # Default SSL material locations
            ca-base /etc/ssl/certs
            crt-base /etc/ssl/private
    
            # Default ciphers to use on SSL-enabled listening sockets.
            # For more information, see ciphers(1SSL). This list is from:
            #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
            ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
            ssl-default-bind-options no-sslv3
    
    defaults
            log     global
            mode    tcp
            option  tcplog
            option  dontlognull
            timeout connect 5000
            timeout client  50000
            timeout server  50000
            errorfile 400 /etc/haproxy/errors/400.http
            errorfile 403 /etc/haproxy/errors/403.http
            errorfile 408 /etc/haproxy/errors/408.http
            errorfile 500 /etc/haproxy/errors/500.http
            errorfile 502 /etc/haproxy/errors/502.http
            errorfile 503 /etc/haproxy/errors/503.http
            errorfile 504 /etc/haproxy/errors/504.http
    
    frontend proxynode
       bind *:80
       bind *:6443
       stats uri /proxystats
       default_backend k8sServers
    
    backend k8sServers
       balance roundrobin
       server cp1  172.31.36.248:6443 check  #<-- Edit with your IP addresses.
    #   server cp2  10.128.0.30:6443 check
    #   server cp3  10.128.0.66:6443 check
    
    listen stats
         bind :9999
         mode http
         stats enable
         stats hide-version
         stats uri /stats
    
  • chrispokorni
    chrispokorni Posts: 2,224
    Options

    Hi @chriswigmore,

    Is your hostname "cp1" or something else? The "cp1", "cp2" and "cp3" strings above should be replaced with the hostnames of the respective control plane nodes, which should be somehow derived from the IP address (based on AWS EC2 naming standards). The IP address edit looks correct though.

    Regards,
    -Chris

  • chriswigmore
    Options

    Thanks for the suggestion, Chris. The hostname is actually 'ip-172-31-36-248'. I tried using that instead of cp1, but still no joy.

  • chrispokorni
    chrispokorni Posts: 2,224
    Options

    Hi @chriswigmore,

    Is the haproxy instance in the same VPC and same SG as your control plane and worker instances?

    Regards,
    -Chris

  • chriswigmore
    Options

    Chris,

    Same VPC = yes
    Same SG = yes

    If it helps, I just tried restarting the haproxy service with the cp node stopped. I saw the following output:

    ... haproxy[993]: [WARNING]  (993) : Server k8sServers/ip-172-31-36-248 is DOWN, reason: Layer4 connection problem, info: "Connection refused", chec>
    ... haproxy[993]: [NOTICE]   (993) : haproxy version is 2.4.22-0ubuntu0.22.04.1
    ... haproxy[993]: [NOTICE]   (993) : path to executable is /usr/sbin/haproxy
    ... haproxy[993]: [ALERT]    (993) : backend 'k8sServers' has no server available!
    

    So it seems that, if it can't reach the cp node, it lets us know. Which suggests the problem is something else.

  • chrispokorni
    chrispokorni Posts: 2,224
    Options

    Hi @chriswigmore,

    Are your VMs on 22.04 by any chance?

    I am trying to spin up a new cluster on AWS, with the recommended OS release though - 20.04, to validate the haproxy config, and to see if I can reproduce your issue.

    Regards,
    -Chris

  • chriswigmore
    Options

    Chris,
    The cp node is 20.04 (I first span it up months ago). The proxy node is 22.04.

  • chrispokorni
    chrispokorni Posts: 2,224
    Options

    Hi @chriswigmore,

    I tried to reproduce the issue, but both attempts were in fact successful. I tried haproxy running on Ubuntu 20.04 and 22.04 and nothing unexpected happened.

    Did you notice anything strange with the cp node throughout the earlier lab exercises?

    Regards,
    -Chris

  • chriswigmore
    Options

    Chris,
    Thanks for investigating. No, I didn't notice anything strange with the cp node - it was well behaved throughout the earlier labs.

Categories

Upcoming Training