Welcome to the Linux Foundation Forum!

Lab 2.3 Unable to Curl

Posts: 9
in LFD259 Class Forum

Hello Experts,

I am trying to do Lab 2.3

ubuntu@k8instance1:~/LFD259/SOLUTIONS/s_02$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
basicpod 1/1 Running 0 5m23s 192.168.1.3 k8instance2

When i try to do curl
curl http://192.168.1.3

I get no out. It connection times out.

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 2,449

    Hi,
    Curl timing out from the master node to the worker node is an indication of a networking issue between your nodes. There may be an infrastructure firewall blocking traffic to some ports, or blocking some protocols. Or you may have firewalls running in the VMs.
    Please make sure all traffic is allowed if you have a custom VPC network and Firewall rule at infra level, and disable all firewalls at the VMs' OS level.
    Regards,
    -Chris

  • Posts: 9

    which particular ports do I have to see? Also when i see the ip address of my minon or worker node, I don't see set as 192.168.1.3, which I get when I do kubectl get pod -wide. I am runnin gthis in a cloud. All outgoing firewall/ports are open.

  • Posts: 2,449

    Have a custom VPC network created for the purpose of this course, attach an all open firewall rule - all ports all protocols all sources/destinations, and spin up your VM instances in this VPC.
    192.168.1.3 is not a node IP, it is the IP of your pod, and it is probably running on the worker node.
    Check any possible firewalls in your VMs as well.
    Regards,
    -Chris

  • Posts: 9

    Hello. I have a all portocols and ports open on my vcn/vpc. The pod is running on my worker node. All fire wall open as well. How to trouble shoot.

  • Posts: 9
    edited March 2019

    .

  • Posts: 2,449

    Hi @stilwalli,
    If the new custom VPC has a new custom firewall rule open to all traffic (all ports, all protocols, all sources/destinations) and your instances have disabled/inactive firewalls, then a good way to troubleshoot the networking between your nodes is to use netcat (nc) and Wireshark on your nodes to determine where is your traffic blocked.
    Providing a snippet of your outputs may also help.
    Regards,
    -Chris

  • Posts: 15

    I have the same issue on Azure VMs.
    Azure Network Security Groups are good - all traffic inbound/outbound is allowed.
    I guess it's smth with Calico setup probably.
    Found here the link to Azure-Vnet CNI plugin, but with it has been installed - networking between nodes becomes completely broken (Pods can't be scheduled on the Workers). So I reverted Azure-Vnet plugin from my Nodes.
    Here is what I have on Master node

    1. sa@ub16:~$ ifconfig
    2. cali02b0884454b Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
    3.           inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
    4.           UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1
    5.           RX packets:1461 errors:0 dropped:0 overruns:0 frame:0
    6.           TX packets:1492 errors:0 dropped:0 overruns:0 carrier:0
    7.           collisions:0 txqueuelen:0
    8.           RX bytes:115291 (115.2 KB)  TX bytes:549569 (549.5 KB)
    9.  
    10. califb8a5d80f8f Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
    11.           inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
    12.           UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1
    13.           RX packets:1450 errors:0 dropped:0 overruns:0 frame:0
    14.           TX packets:1488 errors:0 dropped:0 overruns:0 carrier:0
    15.           collisions:0 txqueuelen:0
    16.           RX bytes:114409 (114.4 KB)  TX bytes:546997 (546.9 KB)
    17.  
    18. docker0   Link encap:Ethernet  HWaddr 02:42:f1:7e:0e:78
    19.           inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
    20.           UP BROADCAST MULTICAST  MTU:1500  Metric:1
    21.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    22.           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    23.           collisions:0 txqueuelen:0
    24.           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
    25.  
    26. eth0      Link encap:Ethernet  HWaddr 00:0d:3a:a2:3b:d7
    27.           inet addr:10.0.0.4  Bcast:10.0.0.255  Mask:255.255.255.0
    28.           inet6 addr: fe80::20d:3aff:fea2:3bd7/64 Scope:Link
    29.           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    30.           RX packets:30029 errors:0 dropped:0 overruns:0 frame:0
    31.           TX packets:39385 errors:0 dropped:0 overruns:0 carrier:0
    32.           collisions:0 txqueuelen:1000
    33.           RX bytes:7448880 (7.4 MB)  TX bytes:24883847 (24.8 MB)
    34.  
    35. lo        Link encap:Local Loopback
    36.           inet addr:127.0.0.1  Mask:255.0.0.0
    37.           inet6 addr: ::1/128 Scope:Host
    38.           UP LOOPBACK RUNNING  MTU:65536  Metric:1
    39.           RX packets:219221 errors:0 dropped:0 overruns:0 frame:0
    40.           TX packets:219221 errors:0 dropped:0 overruns:0 carrier:0
    41.           collisions:0 txqueuelen:1000
    42.           RX bytes:62199452 (62.1 MB)  TX bytes:62199452 (62.1 MB)
    43.  
    44. tunl0     Link encap:IPIP Tunnel  HWaddr
    45.           inet addr:192.168.0.1  Mask:255.255.255.255
    46.           UP RUNNING NOARP  MTU:1440  Metric:1
    47.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    48.           TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
    49.           collisions:0 txqueuelen:1000
    50.           RX bytes:0 (0.0 B)  TX bytes:300 (300.0 B)

    and this is what I have on Worker node

    1. sa@ub16-02:~$ ifconfig
    2. cali3fc1b4ac805 Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
    3.           inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
    4.           UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1
    5.           RX packets:14 errors:0 dropped:0 overruns:0 frame:0
    6.           TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
    7.           collisions:0 txqueuelen:0
    8.           RX bytes:1647 (1.6 KB)  TX bytes:1939 (1.9 KB)
    9.  
    10. docker0   Link encap:Ethernet  HWaddr 02:42:65:65:7e:18
    11.           inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
    12.           UP BROADCAST MULTICAST  MTU:1500  Metric:1
    13.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    14.           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    15.           collisions:0 txqueuelen:0
    16.           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
    17.  
    18. eth0      Link encap:Ethernet  HWaddr 00:0d:3a:a0:a0:f7
    19.           inet addr:10.0.0.5  Bcast:10.0.0.255  Mask:255.255.255.0
    20.           inet6 addr: fe80::20d:3aff:fea0:a0f7/64 Scope:Link
    21.           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    22.           RX packets:38007 errors:0 dropped:0 overruns:0 frame:0
    23.           TX packets:39018 errors:0 dropped:0 overruns:0 carrier:0
    24.           collisions:0 txqueuelen:1000
    25.           RX bytes:19645066 (19.6 MB)  TX bytes:6811361 (6.8 MB)
    26.  
    27. lo        Link encap:Local Loopback
    28.           inet addr:127.0.0.1  Mask:255.0.0.0
    29.           inet6 addr: ::1/128 Scope:Host
    30.           UP LOOPBACK RUNNING  MTU:65536  Metric:1
    31.           RX packets:6294 errors:0 dropped:0 overruns:0 frame:0
    32.           TX packets:6294 errors:0 dropped:0 overruns:0 carrier:0
    33.           collisions:0 txqueuelen:1000
    34.           RX bytes:453032 (453.0 KB)  TX bytes:453032 (453.0 KB)
    35.  
    36. tunl0     Link encap:IPIP Tunnel  HWaddr
    37.           inet addr:192.168.1.1  Mask:255.255.255.255
    38.           UP RUNNING NOARP  MTU:1440  Metric:1
    39.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    40.           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    41.           collisions:0 txqueuelen:1000
    42.           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

    I'm not sure if it's good or not as my networking knowledges are said to say poor.
    Please let me know what I need to check / configure ?
    Any help is appreciated. It actually blocks my Labs (will try to continue work on labs on the only master node for now....)

  • Posts: 2,449

    Hi @vasyhin,
    There is an earlier entry in this forum where @crixo posted a solution to the Calico issue with Kubernetes on Azure.
    Check it out and see if it helps:

    https://forum.linuxfoundation.org/discussion/855882/labs-on-azure#latest

    Regards,
    -Chris

  • Posts: 2

    If you happen to be on GCP, when I got to the curl step in Exercise 2.3 it did not work in my default VPC with default rules.

    Thankfully project calico has documented what solved it for me.

    Just follow the instructions up to the point of 1.2 Setting up GCE networking, then try the curl command again. Works for me.

  • Posts: 2,449

    Because of know issues with the default VPC and default firewall rules, GCE requirements for a custom VPC and firewall rule have been included in Exercise 2.1.

  • Posts: 15

    Eventually I was able to set up Azure VMs to work with Node-To-Pod (on different node) networking.
    The fix was - option#2 from https://docs.projectcalico.org/v3.6/reference/public-cloud/azure#about-calico-on-azure - to use Flannel instead of Calico.
    Simple follow this instructions. Technically you just need to call kubectl apply -f canal.yaml - and that's all - networking is good.
    Master node ifconfig

    1. root@ub16:~# ifconfig
    2. docker0   Link encap:Ethernet  HWaddr 02:42:44:ae:5c:e7
    3.           inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
    4.           UP BROADCAST MULTICAST  MTU:1500  Metric:1
    5.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    6.           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    7.           collisions:0 txqueuelen:0
    8.           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
    9.  
    10. eth0      Link encap:Ethernet  HWaddr 00:0d:3a:a2:3b:d7
    11.           inet addr:10.0.0.4  Bcast:10.0.0.255  Mask:255.255.255.0
    12.           inet6 addr: fe80::20d:3aff:fea2:3bd7/64 Scope:Link
    13.           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    14.           RX packets:217050 errors:0 dropped:0 overruns:0 frame:0
    15.           TX packets:125007 errors:0 dropped:0 overruns:0 carrier:0
    16.           collisions:0 txqueuelen:1000
    17.           RX bytes:79910881 (79.9 MB)  TX bytes:59916636 (59.9 MB)
    18.  
    19. flannel.1 Link encap:Ethernet  HWaddr ee:6a:57:01:1c:81
    20.           inet addr:192.168.0.0  Bcast:0.0.0.0  Mask:255.255.255.255
    21.           inet6 addr: fe80::ec6a:57ff:fe01:1c81/64 Scope:Link
    22.           UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
    23.           RX packets:10 errors:0 dropped:0 overruns:0 frame:0
    24.           TX packets:14 errors:0 dropped:13 overruns:0 carrier:0
    25.           collisions:0 txqueuelen:0
    26.           RX bytes:2238 (2.2 KB)  TX bytes:894 (894.0 B)
    27.  
    28. lo        Link encap:Local Loopback
    29.           inet addr:127.0.0.1  Mask:255.0.0.0
    30.           inet6 addr: ::1/128 Scope:Host
    31.           UP LOOPBACK RUNNING  MTU:65536  Metric:1
    32.           RX packets:538480 errors:0 dropped:0 overruns:0 frame:0
    33.           TX packets:538480 errors:0 dropped:0 overruns:0 carrier:0
    34.           collisions:0 txqueuelen:1000
    35.           RX bytes:149692520 (149.6 MB)  TX bytes:149692520 (149.6 MB)
    36.  
    37. tunl0     Link encap:IPIP Tunnel  HWaddr
    38.           inet addr:192.168.0.1  Mask:255.255.255.255
    39.           UP RUNNING NOARP  MTU:1440  Metric:1
    40.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    41.           TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
    42.           collisions:0 txqueuelen:1000
    43.           RX bytes:0 (0.0 B)  TX bytes:600 (600.0 B)
    44.  

    Master node route -n

    1. root@ub16:~# route -n
    2. Kernel IP routing table
    3. Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    4. 0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0 eth0
    5. 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
    6. 168.63.129.16   10.0.0.1        255.255.255.255 UGH   0      0        0 eth0
    7. 169.254.169.254 10.0.0.1        255.255.255.255 UGH   0      0        0 eth0
    8. 172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
    9. 192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 *
    10. 192.168.1.0     192.168.1.0     255.255.255.0   UG    0      0        0 flannel.1

    Worker node ifconfig

    1. root@ub16-02:~# ifconfig
    2. cali3fc1b4ac805 Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
    3.           inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
    4.           UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1
    5.           RX packets:14 errors:0 dropped:0 overruns:0 frame:0
    6.           TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
    7.           collisions:0 txqueuelen:0
    8.           RX bytes:2546 (2.5 KB)  TX bytes:2264 (2.2 KB)
    9.  
    10. cali840fd25a13b Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
    11.           inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
    12.           UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1
    13.           RX packets:1605 errors:0 dropped:0 overruns:0 frame:0
    14.           TX packets:1662 errors:0 dropped:0 overruns:0 carrier:0
    15.           collisions:0 txqueuelen:0
    16.           RX bytes:126497 (126.4 KB)  TX bytes:605941 (605.9 KB)
    17.  
    18. calif0fcaea2af1 Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
    19.           inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
    20.           UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1
    21.           RX packets:1599 errors:0 dropped:0 overruns:0 frame:0
    22.           TX packets:1678 errors:0 dropped:0 overruns:0 carrier:0
    23.           collisions:0 txqueuelen:0
    24.           RX bytes:126294 (126.2 KB)  TX bytes:606640 (606.6 KB)
    25.  
    26. docker0   Link encap:Ethernet  HWaddr 02:42:f7:28:60:10
    27.           inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
    28.           UP BROADCAST MULTICAST  MTU:1500  Metric:1
    29.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    30.           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    31.           collisions:0 txqueuelen:0
    32.           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
    33.  
    34. eth0      Link encap:Ethernet  HWaddr 00:0d:3a:a0:a0:f7
    35.           inet addr:10.0.0.5  Bcast:10.0.0.255  Mask:255.255.255.0
    36.           inet6 addr: fe80::20d:3aff:fea0:a0f7/64 Scope:Link
    37.           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    38.           RX packets:118396 errors:0 dropped:0 overruns:0 frame:0
    39.           TX packets:87652 errors:0 dropped:0 overruns:0 carrier:0
    40.           collisions:0 txqueuelen:1000
    41.           RX bytes:113745169 (113.7 MB)  TX bytes:11960941 (11.9 MB)
    42.  
    43. flannel.1 Link encap:Ethernet  HWaddr ce:ef:74:d4:d7:06
    44.           inet addr:192.168.1.0  Bcast:0.0.0.0  Mask:255.255.255.255
    45.           inet6 addr: fe80::ccef:74ff:fed4:d706/64 Scope:Link
    46.           UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
    47.           RX packets:14 errors:0 dropped:0 overruns:0 frame:0
    48.           TX packets:10 errors:0 dropped:13 overruns:0 carrier:0
    49.           collisions:0 txqueuelen:0
    50.           RX bytes:894 (894.0 B)  TX bytes:2238 (2.2 KB)
    51.  
    52. lo        Link encap:Local Loopback
    53.           inet addr:127.0.0.1  Mask:255.0.0.0
    54.           inet6 addr: ::1/128 Scope:Host
    55.           UP LOOPBACK RUNNING  MTU:65536  Metric:1
    56.           RX packets:2233 errors:0 dropped:0 overruns:0 frame:0
    57.           TX packets:2233 errors:0 dropped:0 overruns:0 carrier:0
    58.           collisions:0 txqueuelen:1000
    59.           RX bytes:160422 (160.4 KB)  TX bytes:160422 (160.4 KB)
    60.  
    61. tunl0     Link encap:IPIP Tunnel  HWaddr
    62.           inet addr:192.168.1.1  Mask:255.255.255.255
    63.           UP RUNNING NOARP  MTU:1440  Metric:1
    64.           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    65.           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    66.           collisions:0 txqueuelen:1000
    67.           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

    Worker node route -n

    1. root@ub16-02:~# route -n
    2. Kernel IP routing table
    3. Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    4. 0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0 eth0
    5. 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
    6. 168.63.129.16   10.0.0.1        255.255.255.255 UGH   0      0        0 eth0
    7. 169.254.169.254 10.0.0.1        255.255.255.255 UGH   0      0        0 eth0
    8. 172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
    9. 192.168.0.0     192.168.0.0     255.255.255.0   UG    0      0        0 flannel.1
    10. 192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 *
    11. 192.168.1.2     0.0.0.0         255.255.255.255 UH    0      0        0 cali840fd25a13b
    12. 192.168.1.3     0.0.0.0         255.255.255.255 UH    0      0        0 calif0fcaea2af1
    13. 192.168.1.4     0.0.0.0         255.255.255.255 UH    0      0        0 cali3fc1b4ac805
    14.  
  • Posts: 15

    @serewicz I believe it makes sense to mention that for Azure canal.yaml should be used instead of calico.yamlin Lab 2.2. It took me 2 weeks to understand completely what the issue and how it can be fixed

  • Posts: 1,000

    As has been mentioned we do not test or configure for Azure as a lab system.

  • Posts: 2

    Ran thru this issue on 12-14-2022, following https://training.linuxfoundation.org/cm/LFD259/LabSetup-GCE.mp4 for GCE to create firewall rule after the VPN is up. Couldn't get it to work at the first time and scratched out pretty good amount of hair... =/

    Decided to trash the VMs and VPC and all those temp-back-and-forth firewall rules at GCE. Started from scratch and got it to work this time.

    So, the trick is that I have included the firewall rule option during the VPC creation.

    At Firewall rules session, look for a name called "allow-custom" and EDIT it to include both the current subnet and 0.0.0.0/0 and make sure it's "Allow all".

    Nothing needs to be changed inside the VM, I mean I didn't have to mess with the iptables and UFW at all. It just works out of the box.

    I am now a happy camper!

    Have fun and cheers! =)

  • Posts: 1

    I suffered the same (curl hanging) in 2.3 in AWS.
    It turned out my AWS subnet had a security group which allowed TCP traffic, but no UDP traffic. Thus kubernetes operations worked fine (incl. worker node registering itself to the cp, pods getting started, etc.), but "application layer" connectivity didn't work. I can only guess Cilium uses UDP under the hood.

    Remedy: allow all traffic in the subnet - both TCP and UDP.

  • Posts: 2,449

    Hi @grzegon,

    The Overview section of Lab 2.1 does recommend "no firewall" while the AWS demo video from the introductory chapter presents a SG configuration suitable for the lab environment.

    Regards,
    -Chris

  • Posts: 1

    @grzegon said:
    I suffered the same (curl hanging) in 2.3 in AWS.
    It turned out my AWS subnet had a security group which allowed TCP traffic, but no UDP traffic. Thus kubernetes operations worked fine (incl. worker node registering itself to the cp, pods getting started, etc.), but "application layer" connectivity didn't work. I can only guess Cilium uses UDP under the hood.

    Remedy: allow all traffic in the subnet - both TCP and UDP.

    This guy right here is a life saver. I was almost losing my mind because of this

  • Posts: 1

    @grzegon said:
    I suffered the same (curl hanging) in 2.3 in AWS.
    It turned out my AWS subnet had a security group which allowed TCP traffic, but no UDP traffic. Thus kubernetes operations worked fine (incl. worker node registering itself to the cp, pods getting started, etc.), but "application layer" connectivity didn't work. I can only guess Cilium uses UDP under the hood.

    Remedy: allow all traffic in the subnet - both TCP and UDP.

    Great catch! Official Cilium documentation confirm your theory btw; https://docs.cilium.io/en/stable/operations/system_requirements/#firewall-rules

  • Posts: 2,449

    Hi @onur.zengin,

    In the "Course Introduction" chapter of the course, the video titled "IMPORTANT: Using AWS to Set Up the Lab Environment" includes clear instructions for the SG configuration. The SG configuration section starts at timestamp 2:38.

    Regards,
    -Chris

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Quick Links

Categories

Upcoming Training