Pod network across nodes does not work

I followed the installation procedure of lab 3.1 to 3.3 closely. Everything looks nice, but whenever I try to establish a network connection between a pod on one node and another pod on another node, that does not work. The calico-node pods are up and running. In their logs I don't see any error messages.calicoctl node status
for the cp node results in:
Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.0.0.7 | node-to-node mesh | up | 13:15:03 | Established | +--------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found.
For the worker node, I get
Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.0.0.6 | node-to-node mesh | up | 13:15:03 | Established | +--------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found.
On the cp node ip route
returns:
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.6 metric 100 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.6 168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.6 metric 100 169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.6 metric 100 blackhole 192.168.74.128/26 proto bird 192.168.74.136 dev calie739583d8fa scope link 192.168.74.137 dev cali9270933bb0b scope link 192.168.74.138 dev cali73bd7dd6478 scope link 192.168.74.139 dev cali3344860a0ad scope link 192.168.189.64/26 via 10.0.0.7 dev tunl0 proto bird onlink
On the worker node I see:
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.7 metric 100 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.7 168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.7 metric 100 169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.7 metric 100 192.168.74.128/26 via 10.0.0.6 dev tunl0 proto bird onlink blackhole 192.168.189.64/26 proto bird 192.168.189.76 dev cali3140fc1dafd scope link 192.168.189.77 dev calia35b901ce89 scope link
calicoctl get workloadendpoints -A
returns:
NAMESPACE WORKLOAD NODE NETWORKS INTERFACE accounting nginx-one-575f648647-j2rwh worker2 192.168.189.77/32 calia35b901ce89 accounting nginx-one-575f648647-x5c5c worker2 192.168.189.76/32 cali3140fc1dafd default bb2 k8scp 192.168.74.137/32 cali9270933bb0b kube-system calico-kube-controllers-5f6cfd688c-h29qd k8scp 192.168.74.136/32 calie739583d8fa kube-system coredns-74ff55c5b-69n8g k8scp 192.168.74.139/32 cali3344860a0ad kube-system coredns-74ff55c5b-bngtf k8scp 192.168.74.138/32 cali73bd7dd6478
There is the example from lab 9.1 deployed. In addition I used the pod bb2
containing busybox for debug purposes. The problem became obvious to me, when I tried to curl
the nginx pods. This only works when logged into the worker node.
This is my second cluster. I called the cp node k8scp
and the worker worker2
, as in my first cluster it is still master
and worker
. The issue occurs in both clusters. The first one was set up with docker, the second one with cri-o.
The whole setup runs on VMs on Azure.
Is there anything obvious I missed?
One thing that appears odd to me is that the pods do not get addresses out of the PodCIDR range of the according node. If I do kubectl describe node k8scp |grep PodCIDR
, I get
PodCIDR: 192.168.0.0/24 PodCIDRs: 192.168.0.0/24
The pods on that node are in 192.168.74.128/26
, though, as ip route
shows. Is that normal?
Comments
-
Hi @deissnerk,
Azure is not a recommended or supported environment for labs in this course. However, there are learners who ran lab exercises on Azure and shared their findings in the forum. You may use the search option of the forum to locate them for reference.
Regards,
Chris0 -
Thanks for the quick response @chrispokorni. I suppose I'm running into similar issues as @luis-garza has been describing here.
In the beginning of lab 3.1 it is stated:The labs were written using Ubuntu instances running on GoogleCloudPlatform (GCP). They have been written to be vendor-agnostic so could run on AWS, local hardware, or inside of virtualization to give you the most flexibility and options.
I didn't read this as a clear recommendation. After all it should just be about two Ubuntu VMs in an IP subnet. I was prepared to figure out some Azure specifics on my own, but an incompatibility on this level comes to me as a surprise. A warning in section 3.1 that the components used in the lab might have compatibility issues with other cloud providers would be helpful.
Regards,
Klaus
1 -
Got same problem on AWS:
- all firewalls on cp and worker node disabled
- all input / output traffic enabled
Any help?
0 -
Hi @joov,
On AWS the VPC and Security Group configurations directly impact the cluster networking. If you have not done so already, I would invite you to watch the video "Using AWS to set up labs" found in the introductory chapter of this course. The video outlines important settings needed to enable the networking of your cluster.
Also, when provisioning the second EC2 instance, make sure it is placed in the same VPC subnet, and under the same SG as the first instance.
Regards,
-Chris0 -
I followed the video and got it working already. Thank you.
0
Categories
- All Categories
- 50 LFX Mentorship
- 103 LFX Mentorship: Linux Kernel
- 555 Linux Foundation Boot Camps
- 297 Cloud Engineer Boot Camp
- 119 Advanced Cloud Engineer Boot Camp
- 52 DevOps Engineer Boot Camp
- 54 Cloud Native Developer Boot Camp
- 4 Express Training Courses
- 4 Express Courses - Discussion Forum
- 1.9K Training Courses
- 18 LFC110 Class Forum
- 7 LFC131 Class Forum
- 25 LFD102 Class Forum
- 150 LFD103 Class Forum
- 17 LFD121 Class Forum
- LFD137 Class Forum
- 61 LFD201 Class Forum
- LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 598 LFD259 Class Forum
- 102 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- LFD273 Class Forum
- 2 LFS145 Class Forum
- 24 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 3 LFS203 Class Forum
- 69 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 34 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 140 LFS253 Class Forum
- LFS254 Class Forum
- 1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 92 LFS260 Class Forum
- 130 LFS261 Class Forum
- 32 LFS262 Class Forum
- 79 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 17 LFS267 Class Forum
- 17 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS281 Class Forum
- 221 LFW211 Class Forum
- 168 LFW212 Class Forum
- SKF100 Class Forum
- 902 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 209 Networking
- 101 Printers & Scanners
- 85 Storage
- 761 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 141 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 356 Ubuntu
- 478 Linux System Administration
- 41 Cloud Computing
- 70 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 77 Network Management
- 108 System Management
- 49 Web Management
- 66 Mobile Computing
- 23 Android
- 29 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 536 Off Topic
- 131 Introductions
- 216 Small Talk
- 21 Study Material
- 817 Programming and Development
- 275 Kernel Development
- 508 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 59 All In Program
- 59 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)