Pod network across nodes does not work

I followed the installation procedure of lab 3.1 to 3.3 closely. Everything looks nice, but whenever I try to establish a network connection between a pod on one node and another pod on another node, that does not work. The calico-node pods are up and running. In their logs I don't see any error messages.calicoctl node status
for the cp node results in:
- Calico process is running.
- IPv4 BGP status
- +--------------+-------------------+-------+----------+-------------+
- | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
- +--------------+-------------------+-------+----------+-------------+
- | 10.0.0.7 | node-to-node mesh | up | 13:15:03 | Established |
- +--------------+-------------------+-------+----------+-------------+
- IPv6 BGP status
- No IPv6 peers found.
For the worker node, I get
- Calico process is running.
- IPv4 BGP status
- +--------------+-------------------+-------+----------+-------------+
- | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
- +--------------+-------------------+-------+----------+-------------+
- | 10.0.0.6 | node-to-node mesh | up | 13:15:03 | Established |
- +--------------+-------------------+-------+----------+-------------+
- IPv6 BGP status
- No IPv6 peers found.
On the cp node ip route
returns:
- default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.6 metric 100
- 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.6
- 168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.6 metric 100
- 169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.6 metric 100
- blackhole 192.168.74.128/26 proto bird
- 192.168.74.136 dev calie739583d8fa scope link
- 192.168.74.137 dev cali9270933bb0b scope link
- 192.168.74.138 dev cali73bd7dd6478 scope link
- 192.168.74.139 dev cali3344860a0ad scope link
- 192.168.189.64/26 via 10.0.0.7 dev tunl0 proto bird onlink
On the worker node I see:
- default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.7 metric 100
- 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.7
- 168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.7 metric 100
- 169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.7 metric 100
- 192.168.74.128/26 via 10.0.0.6 dev tunl0 proto bird onlink
- blackhole 192.168.189.64/26 proto bird
- 192.168.189.76 dev cali3140fc1dafd scope link
- 192.168.189.77 dev calia35b901ce89 scope link
calicoctl get workloadendpoints -A
returns:
- NAMESPACE WORKLOAD NODE NETWORKS INTERFACE
- accounting nginx-one-575f648647-j2rwh worker2 192.168.189.77/32 calia35b901ce89
- accounting nginx-one-575f648647-x5c5c worker2 192.168.189.76/32 cali3140fc1dafd
- default bb2 k8scp 192.168.74.137/32 cali9270933bb0b
- kube-system calico-kube-controllers-5f6cfd688c-h29qd k8scp 192.168.74.136/32 calie739583d8fa
- kube-system coredns-74ff55c5b-69n8g k8scp 192.168.74.139/32 cali3344860a0ad
- kube-system coredns-74ff55c5b-bngtf k8scp 192.168.74.138/32 cali73bd7dd6478
There is the example from lab 9.1 deployed. In addition I used the pod bb2
containing busybox for debug purposes. The problem became obvious to me, when I tried to curl
the nginx pods. This only works when logged into the worker node.
This is my second cluster. I called the cp node k8scp
and the worker worker2
, as in my first cluster it is still master
and worker
. The issue occurs in both clusters. The first one was set up with docker, the second one with cri-o.
The whole setup runs on VMs on Azure.
Is there anything obvious I missed?
One thing that appears odd to me is that the pods do not get addresses out of the PodCIDR range of the according node. If I do kubectl describe node k8scp |grep PodCIDR
, I get
- PodCIDR: 192.168.0.0/24
- PodCIDRs: 192.168.0.0/24
The pods on that node are in 192.168.74.128/26
, though, as ip route
shows. Is that normal?
Comments
-
Hi @deissnerk,
Azure is not a recommended or supported environment for labs in this course. However, there are learners who ran lab exercises on Azure and shared their findings in the forum. You may use the search option of the forum to locate them for reference.
Regards,
Chris0 -
Thanks for the quick response @chrispokorni. I suppose I'm running into similar issues as @luis-garza has been describing here.
In the beginning of lab 3.1 it is stated:The labs were written using Ubuntu instances running on GoogleCloudPlatform (GCP). They have been written to be vendor-agnostic so could run on AWS, local hardware, or inside of virtualization to give you the most flexibility and options.
I didn't read this as a clear recommendation. After all it should just be about two Ubuntu VMs in an IP subnet. I was prepared to figure out some Azure specifics on my own, but an incompatibility on this level comes to me as a surprise. A warning in section 3.1 that the components used in the lab might have compatibility issues with other cloud providers would be helpful.
Regards,
Klaus
1 -
Got same problem on AWS:
- all firewalls on cp and worker node disabled
- all input / output traffic enabled
Any help?
0 -
Hi @joov,
On AWS the VPC and Security Group configurations directly impact the cluster networking. If you have not done so already, I would invite you to watch the video "Using AWS to set up labs" found in the introductory chapter of this course. The video outlines important settings needed to enable the networking of your cluster.
Also, when provisioning the second EC2 instance, make sure it is placed in the same VPC subnet, and under the same SG as the first instance.
Regards,
-Chris0 -
I followed the video and got it working already. Thank you.
0
Categories
- All Categories
- 144 LFX Mentorship
- 144 LFX Mentorship: Linux Kernel
- 817 Linux Foundation IT Professional Programs
- 368 Cloud Engineer IT Professional Program
- 184 Advanced Cloud Engineer IT Professional Program
- 83 DevOps Engineer IT Professional Program
- 132 Cloud Native Developer IT Professional Program
- 122 Express Training Courses
- 122 Express Courses - Discussion Forum
- Microlearning - Discussion Forum
- 6.7K Training Courses
- 48 LFC110 Class Forum - Discontinued
- 73 LFC131 Class Forum
- 39 LFD102 Class Forum
- 238 LFD103 Class Forum
- 22 LFD110 Class Forum
- 45 LFD121 Class Forum
- 1 LFD123 Class Forum
- LFD125 Class Forum
- 17 LFD133 Class Forum
- 9 LFD134 Class Forum
- 17 LFD137 Class Forum
- 70 LFD201 Class Forum
- 3 LFD210 Class Forum
- 2 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 1 LFD233 Class Forum
- 3 LFD237 Class Forum
- 23 LFD254 Class Forum
- 721 LFD259 Class Forum
- 110 LFD272 Class Forum
- 3 LFD272-JP クラス フォーラム
- 10 LFD273 Class Forum
- 258 LFS101 Class Forum
- 2 LFS111 Class Forum
- 3 LFS112 Class Forum
- 3 LFS116 Class Forum
- 3 LFS118 Class Forum
- 1 LFS120 Class Forum
- 3 LFS142 Class Forum
- 3 LFS144 Class Forum
- 3 LFS145 Class Forum
- 1 LFS146 Class Forum
- 16 LFS148 Class Forum
- 15 LFS151 Class Forum
- 1 LFS157 Class Forum
- 72 LFS158 Class Forum
- LFS158-JP クラス フォーラム
- 5 LFS162 Class Forum
- 1 LFS166 Class Forum
- 3 LFS167 Class Forum
- 1 LFS170 Class Forum
- 1 LFS171 Class Forum
- 2 LFS178 Class Forum
- 2 LFS180 Class Forum
- 2 LFS182 Class Forum
- 4 LFS183 Class Forum
- 30 LFS200 Class Forum
- 737 LFS201 Class Forum - Discontinued
- 2 LFS201-JP クラス フォーラム
- 21 LFS203 Class Forum
- 118 LFS207 Class Forum
- 2 LFS207-DE-Klassenforum
- LFS207-JP クラス フォーラム
- 302 LFS211 Class Forum
- 55 LFS216 Class Forum
- 54 LFS241 Class Forum
- 43 LFS242 Class Forum
- 37 LFS243 Class Forum
- 13 LFS244 Class Forum
- 6 LFS245 Class Forum
- LFS246 Class Forum
- LFS248 Class Forum
- 113 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 157 LFS253 Class Forum
- LFS254 Class Forum
- 2 LFS255 Class Forum
- 13 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.3K LFS258 Class Forum
- 11 LFS258-JP クラス フォーラム
- 116 LFS260 Class Forum
- 156 LFS261 Class Forum
- 43 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 23 LFS267 Class Forum
- 25 LFS268 Class Forum
- 29 LFS269 Class Forum
- 7 LFS270 Class Forum
- 200 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 2 LFS147 Class Forum
- LFS274 Class Forum
- 3 LFS281 Class Forum
- 18 LFW111 Class Forum
- 262 LFW211 Class Forum
- 179 LFW212 Class Forum
- 15 SKF100 Class Forum
- SKF200 Class Forum
- 2 SKF201 Class Forum
- 791 Hardware
- 199 Drivers
- 68 I/O Devices
- 37 Monitors
- 98 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 85 Storage
- 754 Linux Distributions
- 82 Debian
- 67 Fedora
- 16 Linux Mint
- 13 Mageia
- 23 openSUSE
- 149 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 351 Ubuntu
- 465 Linux System Administration
- 39 Cloud Computing
- 71 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 101 System Management
- 47 Web Management
- 56 Mobile Computing
- 18 Android
- 28 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 366 Off Topic
- 114 Introductions
- 171 Small Talk
- 26 Study Material
- 534 Programming and Development
- 304 Kernel Development
- 223 Software Development
- 1.8K Software
- 212 Applications
- 182 Command Line
- 3 Compiling/Installing
- 405 Games
- 311 Installation
- 79 All In Program
- 79 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)