Welcome to the Linux Foundation Forum!

Lab 9.1 (Services and Endpoints) - Master can't curl to endpoints on Worker/Node


Hi, I was just going through all the labs, got to 9.1 and have hit an interesting problem.

When I create the NGINX deployment, then create the endpoints using the command:
kubectl -n accounting expose deployment nginx-one

It works fine, and I get the endpoints as below:

[username@k8smaster1 ~]$ kubectl -n accounting get endpoints nginx-one 
NAME        ENDPOINTS                                 AGE
nginx-one,   12m

The problem is that curl will only work from the worker nodes, not the master node

[username@k8smaster1 ~]$ curl -l
^C (after several minutes)

Whereas on the worker:

[username@k8snode1 ~]$ curl -l
<!DOCTYPE html>
<title>Welcome to nginx!</title>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

I assume this has something to do with my networking setup, but since the kubectl commands on the master can return the information, and I can ping back and forth no problem from the two virtual machines, I'm a little stumped as to what would be causing this issue.

In all honesty, I'm not great on the networking side so perhaps there is something wrong with the calico networking that is setup, but I would really appreciate if anyone could point me in some troubleshooting directions for this.

Best Answer

  • serewicz
    serewicz Posts: 1,000
    Answer ✓


    If you are using Azure there are known problems, and have been for a while.

    I would suggest you use AWS, GCE, Digital Ocean, VirtualBox, VMWare, QEMU/KVM, or bare mental instead, which the labs have been tested upon. The testing is primarily done using GCE vms, but every so often I test on others. Only Azure, so far, has issues.



  • chrispokorni
    chrispokorni Posts: 2,165

    Hi @psyrus,

    Such timeouts are common when the VM to VM (or node to node) networking is not properly configured, outside of the Kubernetes cluster.

    What are you using as infrastructure for your cluster? Are you in the cloud or a local hypervisor?


  • psyrus
    psyrus Posts: 5

    Hi @chrispokorni I am using virtual machines hosted in Azure.

    They are on the same VNET, on the same subnet with no networking restrictions in place between the nodes.

    The Azure layer networking has a CIDR range of:

    The K8s calico setup is such that:

    [username@k8smaster1 ~]$ kubectl get configmaps -n kube-system kubeadm-config -o yaml
    apiVersion: v1
      ClusterConfiguration: |
            authorization-mode: Node,RBAC
          timeoutForControlPlane: 4m0s
        apiVersion: kubeadm.k8s.io/v1beta2
        certificatesDir: /etc/kubernetes/pki
        clusterName: kubernetes
        controlPlaneEndpoint: k8smaster:6443
        controllerManager: {}
          type: CoreDNS
            dataDir: /var/lib/etcd
        imageRepository: k8s.gcr.io
        kind: ClusterConfiguration
        kubernetesVersion: v1.20.6
          dnsDomain: cluster.local
        scheduler: {}
      ClusterStatus: |
            bindPort: 6443
        apiVersion: kubeadm.k8s.io/v1beta2
        kind: ClusterStatus
    kind: ConfigMap
      creationTimestamp: "2021-04-08T04:49:09Z"
      - apiVersion: v1
        fieldsType: FieldsV1
            .: {}
            f:ClusterConfiguration: {}
            f:ClusterStatus: {}
        manager: kubeadm
        operation: Update
        time: "2021-04-08T04:49:09Z"
      name: kubeadm-config
      namespace: kube-system
      resourceVersion: "10235"
      uid: 198fd9ad-8fe7-4b01-93b5-2ecb087dfdb0

    I am curious if the service subnet is the problem. Both the nodes are available from the k8s perspective:

    [username@k8smaster1 ~]$ kubectl get nodes
    NAME         STATUS   ROLES                  AGE   VERSION
    k8smaster1   Ready    control-plane,master   39d   v1.20.6
    k8snode1     Ready    <none>                 39d   v1.20.6
  • chrispokorni
    chrispokorni Posts: 2,165

    Hi @psyrus,

    As long as your networks do not overlap (Pods, Nodes, Services) your cluster should be fine.

    On Azure the network plugin may not behave as expected due to Azure's specific network implementation. For the Calico network plugin you can find Azure specific installation details, or you may search the forum for solutions posted by learners who completed the labs on Azure.


  • psyrus
    psyrus Posts: 5

    Thanks @chrispokorni I will check the Azure specific stuff. I hadn't thought of that because I figured it's all virtualized and shouldn't make a difference unless I wanted to take specific advantages of Azure's networking 'special features'. Will post back after some investigation.

  • psyrus
    psyrus Posts: 5

    @chrispokorni FYI I am not using AKS-Engine, and instead running directly on virtual machines, and as such there is nothing special that needs to be done from a Calico networking perspective.

    Quoted below for completeness:

    Other options and tools

    Calico networking

    You can also deploy Calico for both networking and policy enforcement. In this mode, Calico uses a VXLAN-based overlay network that masks the IP addresses of the pods from the underlying Azure VNET. This can be useful in large deployments or when running multiple clusters and IP address space is a big concern.

    Unfortunately, aks-engine does not support this mode, so you must use a different tool chain to install and manage the cluster. Some options:

    Use Terraform to provision the Azure networks and VMs, then kubeadm to install the Kubernetes cluster.
    Use Kubespray

    Are there any tips/tricks that I could run (kubectl interrogation commands) or linux level checks that I can do to root out the cause of the "iptables proxy mode" (as far as I can tell) misbehaving?

  • psyrus
    psyrus Posts: 5

    Thanks for the information guys, although it doesn't really help with tracking down the problems. There really should be some way to troubleshoot these networking issues from the k8s side, but perhaps that's not what the forum is about. I will just deal with this myself.

  • chrispokorni
    chrispokorni Posts: 2,165

    Hi @psyrus,

    You may also try one of my prior suggestions:

    search the forum for solutions posted by learners who completed the labs on Azure

    Between the LFS258 and LFD259 forums I am sure that lessons learned and suggestions have been shared for the benefit of future learners.


  • lhensley

    Commenting here to say that I am currently facing an identical issue, but on GCE. @psyrus did you ever come up with a solution?

  • chrispokorni

    Hi @lhensley,

    When provisioning the GCE instances, did you happen to follow the video guide from the introductory chapter? It also covers the VPC networking requirements and firewalls needed to enable traffic between your instances.


  • lhensley
    lhensley Posts: 2
    edited December 2022

    That's probably it. I was initially trying to use local hardware and after pretty quickly coming upon some unexpected behavior decided to use GCE instead - but did not backtrack and follow those steps closely.

  • pavelhradil

    Hi, had the same problem trying to run kubernetes on Azure VMs with Ubuntu Server.
    Got one CP node and two worker ones. No node can reach the pods running on other nodes with curl. I'm using Calico for pod networking. Problem is two fold - in Calico setup and Azure routing (or lack thereof).
    To fix...
    TL;DR: disable bird in calico setup and create Azure routing table to map routes between internal node IPs and VM IPs.

    See full description with picture on stackoverflow: https://stackoverflow.com/questions/60222243/calico-k8s-on-azure-cant-access-pods


Upcoming Training