Skip to content
    • Welcome to the Linux Foundation Forum!

    Lab 9.1 (Services and Endpoints) - Master can't curl to endpoints on Worker/Node

    Posts: 5
    in LFS258 Class Forum

    Hi, I was just going through all the labs, got to 9.1 and have hit an interesting problem.

    When I create the NGINX deployment, then create the endpoints using the command:
    kubectl -n accounting expose deployment nginx-one

    It works fine, and I get the endpoints as below:

    1. [username@k8smaster1 ~]$ kubectl -n accounting get endpoints nginx-one 
    2. NAME        ENDPOINTS                                 AGE
    3. nginx-one   192.168.249.49:8080,192.168.249.50:8080   12m

    The problem is that curl will only work from the worker nodes, not the master node

    1. [username@k8smaster1 ~]$ curl -l 192.168.249.49:80
    2. ^C (after several minutes)

    Whereas on the worker:

    1. [username@k8snode1 ~]$ curl -l 192.168.249.49:80
    2. <!DOCTYPE html>
    3. <html>
    4. <head>
    5. <title>Welcome to nginx!</title>
    6. <style>
    7.     body {
    8.         width: 35em;
    9.         margin: 0 auto;
    10.         font-family: Tahoma, Verdana, Arial, sans-serif;
    11.     }
    12. </style>
    13. </head>
    14. <body>
    15. <h1>Welcome to nginx!</h1>
    16. <p>If you see this page, the nginx web server is successfully installed and
    17. working. Further configuration is required.</p>
    18.  
    19. <p>For online documentation and support please refer to
    20. <a href="http://nginx.org/">nginx.org</a>.<br/>
    21. Commercial support is available at
    22. <a href="http://nginx.com/">nginx.com</a>.</p>
    23.  
    24. <p><em>Thank you for using nginx.</em></p>
    25. </body>
    26. </html>

    I assume this has something to do with my networking setup, but since the kubectl commands on the master can return the information, and I can ping back and forth no problem from the two virtual machines, I'm a little stumped as to what would be causing this issue.

    In all honesty, I'm not great on the networking side so perhaps there is something wrong with the calico networking that is setup, but I would really appreciate if anyone could point me in some troubleshooting directions for this.

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Best Answer

    • Posts: 1,000
      Answer ✓

      Hello,

      If you are using Azure there are known problems, and have been for a while.

      I would suggest you use AWS, GCE, Digital Ocean, VirtualBox, VMWare, QEMU/KVM, or bare mental instead, which the labs have been tested upon. The testing is primarily done using GCE vms, but every so often I test on others. Only Azure, so far, has issues.

      Regards,

    Answers

    • Posts: 2,443

      Hi @psyrus,

      Such timeouts are common when the VM to VM (or node to node) networking is not properly configured, outside of the Kubernetes cluster.

      What are you using as infrastructure for your cluster? Are you in the cloud or a local hypervisor?

      Regards,
      -Chris

    • Posts: 5

      Hi @chrispokorni I am using virtual machines hosted in Azure.

      They are on the same VNET, on the same subnet with no networking restrictions in place between the nodes.

      The Azure layer networking has a CIDR range of:
      10.0.22.0/23

      The K8s calico setup is such that:

      1. [username@k8smaster1 ~]$ kubectl get configmaps -n kube-system kubeadm-config -o yaml
      2. apiVersion: v1
      3. data:
      4.   ClusterConfiguration: |
      5.     apiServer:
      6.       extraArgs:
      7.         authorization-mode: Node,RBAC
      8.       timeoutForControlPlane: 4m0s
      9.     apiVersion: kubeadm.k8s.io/v1beta2
      10.     certificatesDir: /etc/kubernetes/pki
      11.     clusterName: kubernetes
      12.     controlPlaneEndpoint: k8smaster:6443
      13.     controllerManager: {}
      14.     dns:
      15.       type: CoreDNS
      16.     etcd:
      17.       local:
      18.         dataDir: /var/lib/etcd
      19.     imageRepository: k8s.gcr.io
      20.     kind: ClusterConfiguration
      21.     kubernetesVersion: v1.20.6
      22.     networking:
      23.       dnsDomain: cluster.local
      24.       podSubnet: 192.168.0.0/16
      25.       serviceSubnet: 10.96.0.0/12
      26.     scheduler: {}
      27.   ClusterStatus: |
      28.     apiEndpoints:
      29.       k8smaster1:
      30.         advertiseAddress: 10.0.22.4
      31.         bindPort: 6443
      32.     apiVersion: kubeadm.k8s.io/v1beta2
      33.     kind: ClusterStatus
      34. kind: ConfigMap
      35. metadata:
      36.   creationTimestamp: "2021-04-08T04:49:09Z"
      37.   managedFields:
      38.   - apiVersion: v1
      39.     fieldsType: FieldsV1
      40.     fieldsV1:
      41.       f:data:
      42.         .: {}
      43.         f:ClusterConfiguration: {}
      44.         f:ClusterStatus: {}
      45.     manager: kubeadm
      46.     operation: Update
      47.     time: "2021-04-08T04:49:09Z"
      48.   name: kubeadm-config
      49.   namespace: kube-system
      50.   resourceVersion: "10235"
      51.   uid: 198fd9ad-8fe7-4b01-93b5-2ecb087dfdb0

      I am curious if the service subnet is the problem. Both the nodes are available from the k8s perspective:

      1. [username@k8smaster1 ~]$ kubectl get nodes
      2. NAME         STATUS   ROLES                  AGE   VERSION
      3. k8smaster1   Ready    control-plane,master   39d   v1.20.6
      4. k8snode1     Ready    <none>                 39d   v1.20.6
    • Posts: 2,443

      Hi @psyrus,

      As long as your networks do not overlap (Pods, Nodes, Services) your cluster should be fine.

      On Azure the network plugin may not behave as expected due to Azure's specific network implementation. For the Calico network plugin you can find Azure specific installation details, or you may search the forum for solutions posted by learners who completed the labs on Azure.

      Regards,
      -Chris

    • Posts: 5

      Thanks @chrispokorni I will check the Azure specific stuff. I hadn't thought of that because I figured it's all virtualized and shouldn't make a difference unless I wanted to take specific advantages of Azure's networking 'special features'. Will post back after some investigation.

    • Posts: 5

      @chrispokorni FYI I am not using AKS-Engine, and instead running directly on virtual machines, and as such there is nothing special that needs to be done from a Calico networking perspective.

      Quoted below for completeness:
      https://docs.projectcalico.org/getting-started/kubernetes/self-managed-public-cloud/azure#other-options-and-tools

      Other options and tools

      Calico networking

      You can also deploy Calico for both networking and policy enforcement. In this mode, Calico uses a VXLAN-based overlay network that masks the IP addresses of the pods from the underlying Azure VNET. This can be useful in large deployments or when running multiple clusters and IP address space is a big concern.

      Unfortunately, aks-engine does not support this mode, so you must use a different tool chain to install and manage the cluster. Some options:

      Use Terraform to provision the Azure networks and VMs, then kubeadm to install the Kubernetes cluster.
      Use Kubespray

      Are there any tips/tricks that I could run (kubectl interrogation commands) or linux level checks that I can do to root out the cause of the "iptables proxy mode" (as far as I can tell) misbehaving?

    • Posts: 5

      Thanks for the information guys, although it doesn't really help with tracking down the problems. There really should be some way to troubleshoot these networking issues from the k8s side, but perhaps that's not what the forum is about. I will just deal with this myself.

    • Posts: 2,443

      Hi @psyrus,

      You may also try one of my prior suggestions:

      search the forum for solutions posted by learners who completed the labs on Azure

      Between the LFS258 and LFD259 forums I am sure that lessons learned and suggestions have been shared for the benefit of future learners.

      Regards,
      -Chris

    • Posts: 2

      Commenting here to say that I am currently facing an identical issue, but on GCE. @psyrus did you ever come up with a solution?

    • Posts: 2,443

      Hi @lhensley,

      When provisioning the GCE instances, did you happen to follow the video guide from the introductory chapter? It also covers the VPC networking requirements and firewalls needed to enable traffic between your instances.

      Regards,
      -Chris

    • Posts: 2
      edited December 2022

      That's probably it. I was initially trying to use local hardware and after pretty quickly coming upon some unexpected behavior decided to use GCE instead - but did not backtrack and follow those steps closely.

    • Posts: 1

      Hi, had the same problem trying to run kubernetes on Azure VMs with Ubuntu Server.
      Got one CP node and two worker ones. No node can reach the pods running on other nodes with curl. I'm using Calico for pod networking. Problem is two fold - in Calico setup and Azure routing (or lack thereof).
      To fix...
      TL;DR: disable bird in calico setup and create Azure routing table to map routes between internal node IPs and VM IPs.

      See full description with picture on stackoverflow: https://stackoverflow.com/questions/60222243/calico-k8s-on-azure-cant-access-pods

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Quick Links

    Categories

    Upcoming Training