Welcome to the Linux Foundation Forum!

[Lab 3.4] tcpdump stays empty

Posts: 207
edited April 2022 in LFS258 Class Forum

Hello,
Here is the setting:

  1. $ kubectl get endpoints nginx
  2. NAME ENDPOINTS AGE
  3. nginx 192.168.19.4:80,192.168.86.67:80,192.168.86.69:80 25m
  4. $ kubectl get service nginx
  5. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  6. nginx ClusterIP 10.100.200.161 <none> 80/TCP 27m
  7. $ curl 10.100.200.161:80

Then the curl command sometimes shows the Nginx welcome page, sometimes not. But the tcpdump stays blank. Anything wrong? Both worker and cp are running tcpdump on the tunnel and see nothing:

  1. sudo tcpdump -i tunl0

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 2,451

    Hi @thomas.bucaioni,

    The concerning behavior of the nginx service is the "sometimes not" showing the nginx welcome page. This symptom typically indicates that the nodes are not networked together to Kubernetes' liking. Assuming the nodes are on the same network, this may be a firewall issues, if it blocks required protocols to various ports. Not opening the firewall to all traffic from all sources, all protocols, and to all port destinations as described in the set up videos may cause these types of issues. Are all the control plane pods running? What is the output of

    kubectl get pods -A

    Regards,
    -Chris

  • Hi @chrispokorni
    It could well be my firewall, it's custom... The output of kubectl get pods -A is:

    1. $ kubectl get pods -A
    2. NAMESPACE NAME READY STATUS RESTARTS AGE
    3. default nginx-74d589986c-4ncr8 1/1 Running 1 (6d ago) 6d1h
    4. default nginx-74d589986c-bwfkk 1/1 Running 1 (6d ago) 6d2h
    5. default nginx-74d589986c-l5p5p 1/1 Running 1 (6d ago) 6d2h
    6. kube-system calico-kube-controllers-56fcbf9d6b-bnvxg 1/1 Running 1 (6d ago) 6d4h
    7. kube-system calico-node-gfrl4 0/1 Running 1 (6d ago) 6d4h
    8. kube-system calico-node-rn8pb 0/1 Running 1 6d4h
    9. kube-system coredns-64897985d-9wzkz 1/1 Running 1 (6d ago) 6d6h
    10. kube-system coredns-64897985d-ff8r8 1/1 Running 1 (6d ago) 6d6h
    11. kube-system etcd-dl-dt-03 1/1 Running 6 (6d ago) 6d6h
    12. kube-system kube-apiserver-dl-dt-03 1/1 Running 7 (6d ago) 6d6h
    13. kube-system kube-controller-manager-dl-dt-03 1/1 Running 6 (6d ago) 6d6h
    14. kube-system kube-proxy-dc2dn 1/1 Running 2 (6d ago) 6d6h
    15. kube-system kube-proxy-tkhfr 1/1 Running 2 (6d ago) 6d6h
    16. kube-system kube-scheduler-dl-dt-03 1/1 Running 6 (6d ago) 6d6h

    Otherwise, here is my firewall:

    1. $ cat bin/firewall.sh
    2. #!/bin/sh
    3. #
    4. # firewall.sh
    5.  
    6. # WAN and LAN interfaces
    7. IFACE_LAN=enp2s0
    8. IFACE_WAN=enp0s29f7u7
    9. IFACE_LAN_IP=172.168.1.0/24
    10.  
    11. # Accept all
    12. iptables -t filter -P INPUT ACCEPT
    13. iptables -t filter -P FORWARD ACCEPT
    14. iptables -t filter -P OUTPUT ACCEPT
    15. iptables -t nat -P INPUT ACCEPT
    16. iptables -t nat -P PREROUTING ACCEPT
    17. iptables -t nat -P POSTROUTING ACCEPT
    18. iptables -t nat -P OUTPUT ACCEPT
    19. iptables -t mangle -P INPUT ACCEPT
    20. iptables -t mangle -P PREROUTING ACCEPT
    21. iptables -t mangle -P FORWARD ACCEPT
    22. iptables -t mangle -P POSTROUTING ACCEPT
    23. iptables -t mangle -P OUTPUT ACCEPT
    24.  
    25. # Reset the counters
    26. iptables -t filter -Z
    27. iptables -t nat -Z
    28. iptables -t mangle -Z
    29.  
    30. # Delete all active rules and personalized chains
    31. iptables -t filter -F
    32. iptables -t filter -X
    33. iptables -t nat -F
    34. iptables -t nat -X
    35. iptables -t mangle -F
    36. iptables -t mangle -X
    37.  
    38. # Default policy
    39. iptables -P INPUT DROP
    40. iptables -P FORWARD ACCEPT
    41. iptables -P OUTPUT ACCEPT
    42.  
    43. # Trust ourselves
    44. iptables -A INPUT -i lo -j ACCEPT
    45. #iptables -A INPUT -i lo --dport 6443 -j ACCEPT
    46. #iptables -A INPUT -i lo --sport 6443 -j ACCEPT
    47.  
    48. # Ping
    49. iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
    50. iptables -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
    51. iptables -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
    52.  
    53. # Established connections
    54. iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
    55.  
    56. # SSH
    57. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 22 -j ACCEPT
    58. #iptables -A INPUT -p tcp -i $IFACE_WAN --dport 22 -j ACCEPT
    59.  
    60. #iptables -A INPUT -p tcp -i $IFACE_WAN --sport 3000 -j ACCEPT
    61. #iptables -A INPUT -p tcp -i $IFACE_WAN --dport 3000 -j ACCEPT
    62. #iptables -A INPUT -p tcp -i $IFACE_LAN --dport 3000 -j ACCEPT
    63. #iptables -A INPUT -p tcp -i $IFACE_LAN --sport 3000 -j ACCEPT
    64.  
    65. #iptables -A INPUT -p udp -i $IFACE_WAN --sport 3000 -j ACCEPT
    66. #iptables -A INPUT -p udp -i $IFACE_WAN --dport 3000 -j ACCEPT
    67. #iptables -A INPUT -p udp -i $IFACE_LAN --dport 3000 -j ACCEPT
    68. #iptables -A INPUT -p udp -i $IFACE_LAN --sport 3000 -j ACCEPT
    69.  
    70. # Kubernetes
    71. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 6443 -j ACCEPT
    72. iptables -A INPUT -p udp -i $IFACE_LAN --dport 6443 -j ACCEPT
    73. iptables -A INPUT -p tcp -i $IFACE_LAN --sport 6443 -j ACCEPT
    74. iptables -A INPUT -p udp -i $IFACE_LAN --sport 6443 -j ACCEPT
    75. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 6449 -j ACCEPT
    76. iptables -A INPUT -p udp -i $IFACE_LAN --dport 6449 -j ACCEPT
    77. iptables -A INPUT -p tcp -i $IFACE_LAN --sport 6449 -j ACCEPT
    78. iptables -A INPUT -p udp -i $IFACE_LAN --sport 6449 -j ACCEPT
    79.  
    80. # Dnsmasq
    81. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 53 -j ACCEPT
    82. iptables -A INPUT -p udp -i $IFACE_LAN --dport 53 -j ACCEPT
    83. iptables -A INPUT -p udp -i $IFACE_LAN --dport 67:68 -j ACCEPT
    84.  
    85. # TCP
    86. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 80 -j ACCEPT
    87. iptables -A INPUT -p tcp -i $IFACE_WAN --dport 80 -j ACCEPT
    88. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 443 -j ACCEPT
    89. iptables -A INPUT -p tcp -i $IFACE_WAN --dport 443 -j ACCEPT
    90. iptables -A INPUT -p tcp -i $IFACE_LAN --sport 80 -j ACCEPT
    91. iptables -A INPUT -p tcp -i $IFACE_WAN --sport 80 -j ACCEPT
    92. iptables -A INPUT -p tcp -i $IFACE_LAN --sport 443 -j ACCEPT
    93. iptables -A INPUT -p tcp -i $IFACE_WAN --sport 443 -j ACCEPT
    94.  
    95. # Packet forwarding activation
    96. iptables -t nat -A POSTROUTING -o $IFACE_WAN -s $IFACE_LAN_IP -j MASQUERADE
    97. sysctl -q -w net.ipv4.ip_forward=1
    98.  
    99. # NFS
    100. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 2049 -j ACCEPT
    101. iptables -A INPUT -p tcp -i $IFACE_LAN --sport 2049 -j ACCEPT
    102.  
    103. # Samba
    104. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 445 -j ACCEPT
    105. iptables -A INPUT -p tcp -i $IFACE_LAN --sport 445 -j ACCEPT
    106. iptables -A INPUT -p tcp -i $IFACE_LAN --dport 139 -j ACCEPT
    107. iptables -A INPUT -p tcp -i $IFACE_LAN --sport 139 -j ACCEPT
    108.  
    109. # NTP
    110. iptables -A INPUT -p udp -i $IFACE_LAN --dport 123 -j ACCEPT
    111.  
    112. # Log refused packets
    113. iptables -A INPUT -m limit --limit 2/min -j LOG --log-prefix "IPv4 packet rejected ++ "
    114. iptables -A INPUT -j DROP
    115.  
    116. # Save the configuration
    117. service iptables save
  • Even after flushing the firewall, curl doesn't reach all the nodes:

    1. systemctl stop iptables
    2. systemctl disable iptables
    3. systemctl status iptables
    4. iptables --flush
    5. service iptables save
    6. cat /etc/sysconfig/iptables
  • The CP is on a router connected to a box on one interface, and to the workers on the other interface.
    But the box seems to be in 192.168.x.x, which could interfere with calico?
    If I set up the calico configuration to 182.168.x.x, maybe it goes well

  • Even after changing the calico configuration to:

    1. apiVersion: kubeadm.k8s.io/v1beta2
    2. kind: ClusterConfiguration
    3. kubernetesVersion: 1.22.1
    4. controlPlaneEndpoint: "k8scp:6443"
    5. networking:
    6. podSubnet: 182.168.0.0/16

    the join command proposed is:

    1. kubeadm join 192.168.1.194:6443 --token v9ii23.bz2vgnyxttimr3tu --discovery-token-ca-cert-hash sha256:5c5c0dd3cd3e2a75a27f119cd637ee82fac7b9febb7671cf5272c16d465683ab
  • My router has a name already, dl-dt-03, so I guess during the install I need to replace all the k8scp with the name of the router?

  • So, I've put the worker node on the router and the cp node on the former worker node. Now the service has no endpoint:

    1. $ kubectl get ep nginx
    2. NAME ENDPOINTS AGE
    3. nginx <none> 9m35s

    Apparently, some pods are frozen:

    1. $ kubectl get pods -A
    2. NAMESPACE NAME READY STATUS RESTARTS AGE
    3. default nginx-74d589986c-zpc74 0/1 ContainerCreating 0 13m
    4. default nginx-85b98978db-frgdh 0/1 ContainerCreating 0 20m
    5. kube-system calico-kube-controllers-7c845d499-9l9x9 1/1 Running 0 42m
    6. kube-system calico-node-xgz2k 1/1 Running 0 42m
    7. kube-system calico-node-xwvl6 0/1 Init:0/3 0 37m
    8. kube-system coredns-64897985d-hjn7c 1/1 Running 0 44m
    9. kube-system coredns-64897985d-zgxqp 1/1 Running 0 44m
    10. kube-system etcd-hp-tw-01 1/1 Running 0 45m
    11. kube-system kube-apiserver-hp-tw-01 1/1 Running 0 45m
    12. kube-system kube-controller-manager-hp-tw-01 1/1 Running 0 45m
    13. kube-system kube-proxy-25xmk 0/1 ContainerCreating 0 37m
    14. kube-system kube-proxy-4q728 1/1 Running 0 44m
    15. kube-system kube-scheduler-hp-tw-01 1/1 Running 0 45m
  • Posts: 207
    edited April 2022

    Anyway, for the training, maybe I can run everything from the cp node without worker?

  • Finally, I created two instances at AWS, but the join command gets stuck:

    1. $ kubeadm join k8scp:6443 --token jlb7a6.azs6ad1ocv7nuh75 --discovery-token-ca-cert-hash sha256:0f00ba05e423ad5d51cb18343b9a97c0b0cd73b81ab5a948ee2208d1051085d5 --v=5
    2. I0417 14:30:10.929810 31767 join.go:405] [preflight] found NodeName empty; using OS hostname as NodeName
    3. I0417 14:30:10.930110 31767 initconfiguration.go:116] detected and using CRI socket: /var/run/dockershim.sock
    4. [preflight] Running pre-flight checks
    5. I0417 14:30:10.930376 31767 preflight.go:92] [preflight] Running general checks
    6. I0417 14:30:10.930539 31767 checks.go:245] validating the existence and emptiness of directory /etc/kubernetes/manifests
    7. I0417 14:30:10.930680 31767 checks.go:282] validating the existence of file /etc/kubernetes/kubelet.conf
    8. I0417 14:30:10.930745 31767 checks.go:282] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
    9. I0417 14:30:10.930831 31767 checks.go:106] validating the container runtime
    10. I0417 14:30:10.983222 31767 checks.go:132] validating if the "docker" service is enabled and active
    11. I0417 14:30:10.998636 31767 checks.go:331] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
    12. I0417 14:30:10.998725 31767 checks.go:331] validating the contents of file /proc/sys/net/ipv4/ip_forward
    13. I0417 14:30:10.998790 31767 checks.go:649] validating whether swap is enabled or not
    14. I0417 14:30:10.998856 31767 checks.go:372] validating the presence of executable conntrack
    15. I0417 14:30:10.998909 31767 checks.go:372] validating the presence of executable ip
    16. I0417 14:30:10.998964 31767 checks.go:372] validating the presence of executable iptables
    17. I0417 14:30:10.999005 31767 checks.go:372] validating the presence of executable mount
    18. I0417 14:30:10.999054 31767 checks.go:372] validating the presence of executable nsenter
    19. I0417 14:30:10.999102 31767 checks.go:372] validating the presence of executable ebtables
    20. I0417 14:30:10.999141 31767 checks.go:372] validating the presence of executable ethtool
    21. I0417 14:30:10.999199 31767 checks.go:372] validating the presence of executable socat
    22. I0417 14:30:10.999263 31767 checks.go:372] validating the presence of executable tc
    23. I0417 14:30:10.999311 31767 checks.go:372] validating the presence of executable touch
    24. I0417 14:30:10.999376 31767 checks.go:520] running all checks
    25. I0417 14:30:11.054823 31767 checks.go:403] checking whether the given node name is valid and reachable using net.LookupHost
    26. I0417 14:30:11.057728 31767 checks.go:618] validating kubelet version
    27. I0417 14:30:11.137518 31767 checks.go:132] validating if the "kubelet" service is enabled and active
    28. I0417 14:30:11.148454 31767 checks.go:205] validating availability of port 10250
    29. I0417 14:30:11.148620 31767 checks.go:282] validating the existence of file /etc/kubernetes/pki/ca.crt
    30. I0417 14:30:11.148651 31767 checks.go:432] validating if the connectivity type is via proxy or direct
    31. I0417 14:30:11.148703 31767 join.go:475] [preflight] Discovering cluster-info
    32. I0417 14:30:11.148756 31767 token.go:80] [discovery] Created cluster-info discovery client, requesting info from "k8scp:6443"
    33. I0417 14:30:21.149488 31767 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    34. I0417 14:30:37.057291 31767 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    35.  

    Any idea what goes wrong?

  • So after setting the security group to accept everything, the worker node managed to join. But the cp node says:

    1. # kubectl get nodes
    2. The connection to the server localhost:8080 was refused - did you specify the right host or port?
  • Calico was not running, everything is fine

  • Posts: 2,451

    Hi @thomas.bucaioni,

    In summary, when calico is not running then the cluster is not behaving as expected either. Calico is responsible for managing the pod network, which impacts some of the control plane pods also.

    The IP subnets should be distinct in a cluster, meaning that the pod network (calico's default 192.168.0.0/16), the node network, and eventually the services network (cluster's default 10.96.0.0/12) should not overlap. In local environments it is typical to see the pod and node networks overlap, because many private networks use the 192.168.0.0/x default subnet. This causes issues because all these IP addresses are entered into iptables, where the cluster cannot tell the difference when an IP address represents a pod and when a node.

    The k8scp entry is intended to be an alias only, not a hostname. It will help with chapter 16 on HA. You can build your cluster without the alias, but then ch 16 will require a fresh rebuild, assuming that the instructions from ch 3 are followed as presented.

    For AWS-EC2 and GCP-GCE infrastructures, you can find video guides for each environment's configuration, where VPC's and firewalls/GCs considerations are discussed as well.

    Regards,
    -Chris

  • So, in AWS everything goes well: tcpdump, the load balancer, and even the access from outside the cluster. Fixed

  • Hi @chrispokorni,
    Just saw your answer. Indeed, the reason it didn't work on my baremetal Pcs must be the overlap of Ip ranges... Thanks for confirming
    The cluster is ready for chapter 16 then, no worries.
    Starting back at chapter 1, I saw the videos and I gave it a try at Aws. Everything is clear, it works just perfect now
    Cheers,
    Thomas

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training