New training version with calico does not work [LAB 3.3]
In Lab 3.3 at step 20 there is no response from curl method (after some time it returns timeout). It worked with flannel. With calico it seems to block connection to the both Cluster IP and Enpoint IP. In addition ping from master node to worker node works but there is timeout trying to ping Cluster IP or Endpoint IP from master node (with flannel it worked). All steps before were done according to the lab instruction.
Any idea what could be wrong?
Comments
-
Hello,
Thank you for the feedback. Would it be possible to paste the last few commands, and their output? As you mentioned it hangs, but perhaps something in that and the previous output will help debug the problem.
Regards,
0 -
Hello again,
In case you didn't see it, step 20 does have this entry:
...If the curl command times out the pod may be running on the other node. Run the same command on that node and it should work.
Regards,
0 -
Thanks for reply. I have checked curl on the same node as nginx pod is running and then it works. I thought it should work also from another pods (it worked that way whith flannel) especially when trying to use ClusterIP. I suppose this is correct behavior. Is it related with calico network policy? I am asking because I am curious why it is different from flannel.
From node1:
[email protected]:~$ kubectl get svc,ep NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8h service/nginx ClusterIP 10.96.12.64 <none> 80/TCP 8h NAME ENDPOINTS AGE endpoints/kubernetes 10.142.0.2:6443 8h endpoints/nginx 192.168.169.131:80 8h [email protected]:~$ curl 10.96.12.64:80 curl: (7) Failed to connect to 10.96.12.64 port 80: Connection timed out
From node2:
[email protected]:~$ curl 192.168.169.131:80 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
0 -
One way to narrow down the problem is to run wireshark or tcpdump on the interfaces in use. If the curl request leaves the master node, but does not show up on the worker node you would know that something in the middle is blocking the traffic. If the traffic is arriving on the worker node this would indicate an issue with the service or perhaps kube-proxy on the target node.
As Fabio mentioned if you are using GCE you need to ensure that no traffic is being blocked. Other tools like VirtualBox and AWS have a slightly different manner to ensure all traffic is allowed.
0 -
Indeed the problem was with GCE firewall. I saw that the traffic comes out of the node1 but does not arrive to node2. In fact ping and HTTP ports were open so it worked for some scenarios but it does not work for others, eg. complete demo example from lab 4.3 was working partially as there were no images on the page until traffic was open on firewall settings in GCE.
Thanks for helping with this issue.
0 -
Hi,
I had the same problem (yes I think it IS a problem) and I solved it allowing ALL the traffic between the cluster nodes. If you are in GCE like me:
- Tag all the hosts of the cluster with the same network tag, say "k8s".
- Add a firewall rule to allow all the traffic from hosts tagged "k8s" to hosts tagged "k8s".
Let me know if you need more details on it.
Fabio
1 -
Thanks, I solved it allowing All traffic too , I want to know, why this action is necessary ?. before to this action I have allowed All TCP traffic but this didn't work !
Jorge.
0 -
Hi @jitapichab,
Kubernetes is an API driven architecture, and all internal communication takes place via API calls. This implies that each agent will expose a port where it expects requests from other agents.
A firewall rule to allow only TCP traffic will do just that - allow only the TCP traffic. Agents may use other protocols, so you would need to allow all protocols through, not only TCP.
This can be achieved in multiple ways. Tagging instances and setting up firewall rules based on tags is one way. Creating an isolated VPC network, dedicated to you Kubernetes nodes, with a firewall rule to allow all traffic - all protocols, to all ports, from all sources, is another way.
Regards.
-Chris0
Categories
- 10.1K All Categories
- 35 LFX Mentorship
- 88 LFX Mentorship: Linux Kernel
- 504 Linux Foundation Boot Camps
- 279 Cloud Engineer Boot Camp
- 103 Advanced Cloud Engineer Boot Camp
- 48 DevOps Engineer Boot Camp
- 41 Cloud Native Developer Boot Camp
- 2 Express Training Courses
- 2 Express Courses - Discussion Forum
- 1.8K Training Courses
- 17 LFC110 Class Forum
- 5 LFC131 Class Forum
- 19 LFD102 Class Forum
- 148 LFD103 Class Forum
- 13 LFD121 Class Forum
- 61 LFD201 Class Forum
- LFD210 Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- 23 LFD254 Class Forum
- 569 LFD259 Class Forum
- 100 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFS145 Class Forum
- 23 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 1 LFS203 Class Forum
- 45 LFS207 Class Forum
- 298 LFS211 Class Forum
- 53 LFS216 Class Forum
- 46 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 10 LFS244 Class Forum
- 27 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- 131 LFS253 Class Forum
- 997 LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 87 LFS260 Class Forum
- 126 LFS261 Class Forum
- 31 LFS262 Class Forum
- 79 LFS263 Class Forum
- 15 LFS264 Class Forum
- 10 LFS266 Class Forum
- 17 LFS267 Class Forum
- 17 LFS268 Class Forum
- 21 LFS269 Class Forum
- 200 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 212 LFW211 Class Forum
- 153 LFW212 Class Forum
- 899 Hardware
- 217 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 208 Networking
- 101 Printers & Scanners
- 85 Storage
- 749 Linux Distributions
- 88 Debian
- 64 Fedora
- 14 Linux Mint
- 13 Mageia
- 24 openSUSE
- 133 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 355 Ubuntu
- 473 Linux System Administration
- 38 Cloud Computing
- 69 Command Line/Scripting
- Github systems admin projects
- 94 Linux Security
- 77 Network Management
- 108 System Management
- 49 Web Management
- 63 Mobile Computing
- 22 Android
- 27 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 528 Off Topic
- 127 Introductions
- 213 Small Talk
- 20 Study Material
- 794 Programming and Development
- 262 Kernel Development
- 498 Software Development
- 923 Software
- 258 Applications
- 182 Command Line
- 2 Compiling/Installing
- 76 Games
- 316 Installation
- 53 All In Program
- 53 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)