Skip to content
    • Welcome to the Linux Foundation Forum!

    Lab 16.2.3 - neither proxynode nor k8sServers started

    Posts: 7
    edited May 2023 in LFS258 Class Forum

    I've spun up a new node and installed HAProxy on it. I've edited the haproxy.cfg file as shown in the lab instructions, restarted the service and checked the status. I get the following output:
    ... systemd[1]: Starting HAProxy Load Balancer...
    ... haproxy[17833]: [WARNING] (17833) : config : 'stats' statement ignored for frontend 'proxynode' as it requires HTTP mode.
    ... haproxy[17833]: [NOTICE] (17833) : New worker #1 (17835) forked
    ... systemd[1]: Started HAProxy Load Balancer.
    Note: no mention of proxynode nor k8sServers starting, but no error messages to help me work out why not (only the warning about the stats statement - don't know if that's relevant or not). Could someone help me troubleshoot this?

    Things I've tried:

    • sudo journalctl -xeu haproxy.service (still no error messages)
    • Using the haproxy.cfg file from the SOLUTIONS tarball (with appropriate IP tweaks)
    • Uninstalling and reinstalling haproxy
    • Stopping and starting the service (rather than restarting)
    • Running haproxy in the foreground (via sudo haproxy -f /etc/haproxy/haproxy.cfg -db)

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Best Answer

    • Posts: 7
      Answer ✓

      Chris,
      Update - I blew away the 22.04 proxy node and tried on a new one with Ubuntu 20.04, and it now works as expected. Since it worked for you on both versions, I suspect there was just something wrong with that particular VM.
      Thanks again for your help.

    Answers

    • Posts: 2,434

      Hi @chriswigmore,

      What type of infrastructure are you using for your cluster nodes (control-plane-1, worker, control-plane-2, control-plane-3, and others if any) and the haproxy load balancer?

      With all necessary settings in place, what does your haproxy.cfg look like? Please provide its content in code block to help check for possible typos.

      Regards,
      -Chris

    • Posts: 7
      edited May 2023

      Chris, thanks for getting back to me. To answer your questions:

      • I'm using AWS EC2s for all 5 nodes (3 cp nodes, 1 worker node, 1 proxy node), with specs as per the guidance (t2.large, 20 gp2 storage, etc)
      • My haproxy.cfg is below (apologies - the markdown isn't ideal):
      1. global
      2.         log /dev/log    local0
      3.         log /dev/log    local1 notice
      4.         chroot /var/lib/haproxy
      5.         stats socket /run/haproxy/admin.sock mode 660 level admin
      6.         stats timeout 30s
      7.         user haproxy
      8.         group haproxy
      9.         daemon
      10.  
      11.         # Default SSL material locations
      12.         ca-base /etc/ssl/certs
      13.         crt-base /etc/ssl/private
      14.  
      15.         # Default ciphers to use on SSL-enabled listening sockets.
      16.         # For more information, see ciphers(1SSL). This list is from:
      17.         #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
      18.         ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
      19.         ssl-default-bind-options no-sslv3
      20.  
      21. defaults
      22.         log     global
      23.         mode    tcp
      24.         option  tcplog
      25.         option  dontlognull
      26.         timeout connect 5000
      27.         timeout client  50000
      28.         timeout server  50000
      29.         errorfile 400 /etc/haproxy/errors/400.http
      30.         errorfile 403 /etc/haproxy/errors/403.http
      31.         errorfile 408 /etc/haproxy/errors/408.http
      32.         errorfile 500 /etc/haproxy/errors/500.http
      33.         errorfile 502 /etc/haproxy/errors/502.http
      34.         errorfile 503 /etc/haproxy/errors/503.http
      35.         errorfile 504 /etc/haproxy/errors/504.http
      36.  
      37. frontend proxynode
      38.    bind *:80
      39.    bind *:6443
      40.    stats uri /proxystats
      41.    default_backend k8sServers
      42.  
      43. backend k8sServers
      44.    balance roundrobin
      45.    server cp1  172.31.36.248:6443 check  #<-- Edit with your IP addresses.
      46. #   server cp2  10.128.0.30:6443 check
      47. #   server cp3  10.128.0.66:6443 check
      48.  
      49. listen stats
      50.      bind :9999
      51.      mode http
      52.      stats enable
      53.      stats hide-version
      54.      stats uri /stats
    • Posts: 2,434

      Hi @chriswigmore,

      Is your hostname "cp1" or something else? The "cp1", "cp2" and "cp3" strings above should be replaced with the hostnames of the respective control plane nodes, which should be somehow derived from the IP address (based on AWS EC2 naming standards). The IP address edit looks correct though.

      Regards,
      -Chris

    • Posts: 7

      Thanks for the suggestion, Chris. The hostname is actually 'ip-172-31-36-248'. I tried using that instead of cp1, but still no joy.

    • Posts: 2,434

      Hi @chriswigmore,

      Is the haproxy instance in the same VPC and same SG as your control plane and worker instances?

      Regards,
      -Chris

    • Posts: 7

      Chris,

      Same VPC = yes
      Same SG = yes

      If it helps, I just tried restarting the haproxy service with the cp node stopped. I saw the following output:

      1. ... haproxy[993]: [WARNING]  (993) : Server k8sServers/ip-172-31-36-248 is DOWN, reason: Layer4 connection problem, info: "Connection refused", chec>
      2. ... haproxy[993]: [NOTICE]   (993) : haproxy version is 2.4.22-0ubuntu0.22.04.1
      3. ... haproxy[993]: [NOTICE]   (993) : path to executable is /usr/sbin/haproxy
      4. ... haproxy[993]: [ALERT]    (993) : backend 'k8sServers' has no server available!

      So it seems that, if it can't reach the cp node, it lets us know. Which suggests the problem is something else.

    • Posts: 2,434

      Hi @chriswigmore,

      Are your VMs on 22.04 by any chance?

      I am trying to spin up a new cluster on AWS, with the recommended OS release though - 20.04, to validate the haproxy config, and to see if I can reproduce your issue.

      Regards,
      -Chris

    • Posts: 7

      Chris,
      The cp node is 20.04 (I first span it up months ago). The proxy node is 22.04.

    • Posts: 2,434

      Hi @chriswigmore,

      I tried to reproduce the issue, but both attempts were in fact successful. I tried haproxy running on Ubuntu 20.04 and 22.04 and nothing unexpected happened.

      Did you notice anything strange with the cp node throughout the earlier lab exercises?

      Regards,
      -Chris

    • Posts: 7

      Chris,
      Thanks for investigating. No, I didn't notice anything strange with the cp node - it was well behaved throughout the earlier labs.

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Quick Links

    Categories

    Upcoming Training