Lab 3.2/3.3 master to node network problem
I am doing the labs in VirtualBox. I had to make a few changes to the instructions to get this far but I wonder if there are other things that I need to do differently for VirtualBox to work.
The VirtualBox NAT network interface lets you access the internet but you cannot access other VMs so each VM has 2 interfaces, a NAT interface to access the internet and a Host Only interface to access it from my host and so they can talk to each other. To make this work, I needed to add the --apiserver-advertise-address=192.168.56.10 option on the kubeadm init command otherwise the second VM could not access the API server. I also needed to disable the firewall using the ufw disable command on both boxes.
I was able to finish Lab 3.2 after making these changes.
On Lab 3.3 step 2, when I run the exec command from my master, I get an error:
jsissom@kmaster:~$ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-768979984b-nfh6r 1/1 Running 0 10m 10.244.1.6 knode1
nginx-768979984b-pqszn 1/1 Running 0 10m 10.244.1.5 knode1
nginx-768979984b-ssdkq 1/1 Running 0 9m 10.244.1.7 knode1
jsissom@kmaster:~$ kubectl exec nginx-768979984b-pqszn -- printenv
error: unable to upgrade connection: pod does not exist
I told it to increase the replicas to 10 to get some running on master and if I try to exec to one running on the master node, I get connected and it works:
jsissom@kmaster:~$ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-768979984b-4b7cc 1/1 Running 0 42s 10.244.0.4 kmaster
nginx-768979984b-7j9k2 1/1 Running 0 42s 10.244.1.8 knode1
nginx-768979984b-9rqfk 1/1 Running 0 42s 10.244.1.11 knode1
nginx-768979984b-cq9r2 1/1 Running 0 42s 10.244.0.6 kmaster
nginx-768979984b-gn9g4 1/1 Running 0 42s 10.244.1.10 knode1
nginx-768979984b-hfgx7 1/1 Running 0 42s 10.244.1.9 knode1
nginx-768979984b-nfh6r 1/1 Running 0 11m 10.244.1.6 knode1
nginx-768979984b-pqszn 1/1 Running 0 11m 10.244.1.5 knode1
nginx-768979984b-ssdkq 1/1 Running 0 10m 10.244.1.7 knode1
nginx-768979984b-vqznn 1/1 Running 0 42s 10.244.0.5 kmaster
jsissom@kmaster:~$ kubectl exec nginx-768979984b-4b7cc -- printenv
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=nginx-768979984b-4b7cc
KUBERNETES_PORT_443_TCP_PORT=443
NGINX_SERVICE_HOST=10.108.68.108
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
NGINX_SERVICE_PORT=80
NGINX_PORT=tcp://10.108.68.108:80
NGINX_PORT_80_TCP=tcp://10.108.68.108:80
NGINX_PORT_80_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
NGINX_PORT_80_TCP_ADDR=10.108.68.108
KUBERNETES_SERVICE_PORT=443
NGINX_PORT_80_TCP_PORT=80
NGINX_VERSION=1.13.12-1~stretch
NJS_VERSION=1.13.12.0.2.0-1~stretch
HOME=/root
jsissom@kmaster:~$ kubectl exec nginx-768979984b-7j9k2 -- printenv
error: unable to upgrade connection: pod does not exist
Is there something I need to change in my environment to allow me to exec into a pod running on my node?
If I continue with the lab, in step 6, if I curl my public IP at the LoadBalancer port over and over, I get connected sometimes, but not others:
jsissom@host ~ $ curl http://kmaster:31166/
curl: (55) getpeername() failed with errno 22: Invalid argument
jsissom@host ~ $ curl http://kmaster:31166/
curl: (55) getpeername() failed with errno 22: Invalid argument
jsissom@host ~ $ curl http://kmaster:31166/
<!DOCTYPE html>
jsissom@host ~ $ curl http://kmaster:31166/
curl: (55) getpeername() failed with errno 22: Invalid argument
jsissom@host ~ $ curl http://kmaster:31166/
<!DOCTYPE html>
Thanks
Jay
Comments
-
Hi Jay,
It is impressive the amount of efort you have invested into this. There are a few earlier posts on VMs running on vbox, but not sure how many labs were completed under this setup.While I cannot relate to what you are experiencing (I ran all the labs on GCP as instructed), reading your post got me to do some more in depth research on vbox networking, and kubernetes networking.
I found a rather complex solution to the VM to VM networking (and I would probably try to simplify it somehow, if possible):
http://www.brianlinkletter.com/how-to-use-virtualbox-to-emulate-a-network/
and some good information on the vbox documentation website:
https://www.virtualbox.org/manual/ch06.htmlAs far as kubernetes, I believe that what you accomplished with kubeadm init ... --apiserver-advertise-address..., may need a similar approach for kube-proxy on each minion. The kubeadm join does not provide any interesting options, but I think that the kube-proxy may need a ...--bind-address... also:
https://v1-9.docs.kubernetes.io/docs/reference/generated/kube-proxy/
I hope some of this helps.
Regards,
-Chris
0 -
It's disappointing that Kubernetes won't work on VirtualBox. I see tutorial after tutorial all over the web showing how to set it up, but I always get the same non-working environment. I had hoped that this course would teach me what I was doing wrong. It's disappointing that it does not. Our production environment has public and private network cards in each server and we would want to specify how the software uses those networks so I'm not confident that I'll be able to make it work there if I can't make it work on VirtualBox.
I would suggest that you put language in your course advertisement that this course requires you to pay for GCE. The advertising info for the course doesn't say that. In fact, in slide 1.7.a it specifically says it will work on VirtualBox which does not seem to be a true statement.
Thanks
0 -
It has been a while since I tried to get Kubernetes to run on virtualbox. Let me attempt the labs and let you know what I have found.
0 -
Hello,
Very strange behavior. I have gone through the labs using VirtualBox on my laptop. I had the same problem at first when exposing the port as a LoadBalancer. It works if you used NodePort instead. Here is where it gets strange, after it works as a NodePort, it will continue to work - on a new, different ports - for LoadBalancer. So there is some strange handover which sticks once NodePort opens or connects it. If you dig into documentation you'll find that a LoadBalancer is supposed to be the same as NodePort, except it send an asyncronus request to the cloud provider to spin up a LoadBalancer. Even if there is not one availalbe it works as a node port in GCE, AWS and OpenStack.
As it works now, even when I switch it back and forth updating the port, I am chalking this up a bug. Please let me know if I can offer more information.
Regards,
0 -
Can you tell me the network configuration you used on VirtualBox?
Did you make sure that nginx was running on both nodes and try exercise 3.3, step 2 against pods running on both nodes? That failed for me even before the load balancer port was attempted.
Were there steps you needed to perform on the base OS that were not included in the labs?
Thanks
0 -
Hello,
The only step I did on the instances, which was not written into the lab, was to disable the firewall with ufw disable. I also ensured virtualbox didn't deny any traffic. Other than virtualbox not handling the initial loadbalancer configureation, but working with later attempts, I followed the steps direct from the lab and saw the expected behavior.
Regards,
0 -
Thanks for trying it. There must be something you are doing that is different. Here's what happens when I try it:
Using VirtualBox 5.2.8
1. Create Ubuntu 16.04-4 Server vm, 1 nat network interface, 2gb RAM - take all defaults
2. ufw disable
3. Start Lab 3.1
4. #11. Fails due to swap being enabled
5. swapoff -a, edit /etc/fstab to comment out swap
6. Continue with Lab 3.1. It completes successfully.7. Create 2nd Ubuntu server, 1 Nat network interface, 2gb RAM, take all defaults
8. ufw disable, disable swap
9. Start Lab 3.2
10. #2. The ethernet name is enp0s3, not ens4
11. Both machines have the same Nat IP address (10.0.2.15) so they cannot communicate with each other.When I tried it originally, I gave the vm's two network cards, one the Nat interface that can connect to the internet and one is a host-only interface so they can connect to each other. This allowed me to install but fails later.
Can you tell me what you did to allow the two VMs to communicate when using the Nat interface?
Thanks
Jay
0 -
I also had to disable swap. I did it without thinking, but remembered once you mentioned it.
Did you make sure the VB interfaces are open and not set to deny all?
Regards,
0 -
When I was trying it originally, I setup 2 network interfaces, 1 was Nat so the VM could access the internet and the 2nd was a host-only network so the 2 vms could communicate.
When I tried it this time, I took the defaults of Virtual Box since you said you didn't do anything special when you tested it. In this case, the two VMs have 1 interface that are Nat and cannot communicate.
Can you please tell me how you configured the network in VB so the VMs can access the internet and communicate to each other? I don't know what you mean by "I also ensured virtualbox didn't deny any traffic.". I don't know if VirtualBox is denying any traffic or not. It assigns the same IP address to both VMs when they are in Nat mode so it is impossible for them to communicate with each other.
0 -
Hi Jay,
I took a swing at this, and this is how far I got in Lab 3:
vBox 5.2.8
1. Created 2 VMs, 2 GB each
2. Adapter 1 ONLY - Bridged interface on each (Allow All)
3. Installed Ubuntu 16 and vbox guest additions
4. ip a (or ifconfig) outputs showed different enp0s3 IPs for each Ubuntu instance: master 10.0.0.50 and second 10.0.0.125
5. ufw status -> inactive (I did not have to do anything here)
6. nc to test inter-node communication, messages were delivered successfully from master to second and second to master
7. swapoff -a
8. commented out swap in /etc/fstabAll steps in labs 3.1, 3.2, 3.3 completed succesfully, except Lab 3.3 Step 6, where I was able to access the nginx page, from a browser on my Host machine, only on the <MASTER-enp0s3-inet-addr>:32056, which in my case was 10.0.0.50:32056.
With NAT only I was not able to get anywhere, because like you said, the VM instances were not able to communicate with each other.
I hope this helps,
-Chris
0 -
Hello,
It sounds like the issue is with networking, and how virtualbox handles connection from one VM to another. I only a casual user of Oracle VirtualBox, you may have some luck asking here: https://forums.virtualbox.org/viewforum.php?f=3
If the Kubernetes (and the labs) works with some virtualbox configurations, but don't in some multi-interface configurrations it would seem to be an issue with virtualbox. As kubernetes is all API driven, it has to be something where the interface expected to transmit or receive is not the one actually being used. I would not be surprised if the "primary" interface was hard-coded and the issue doesn't happen enough for someone to revist the issue in the kubernetes code.
Have you tried using just one interface at first and ensuring the VMs can see each other and work as expected, then adding a second interface for your other needs? This would have the primary interface be all the expected things for kubernetes, then you'd know it works.The later addition of a second interface could be used for your other infrastructure needs. Could be a work-around to the issue.
Regards,
0 -
Using a bridged adapter is problematic because with dhcp the IP address changes from day to day at my workplace.
If I pick a network type where the two VMs can see each other, there is no internet access. If I pick one where they can see the internet, they can't see each other. VirtualBox requires 2 interfaces, unless you use the bridged adapter which means your IP address changes based on your location. Here is the documentation on the networking for Virtual Box: https://www.virtualbox.org/manual/ch06.html
I've come up with a solution to the problem so I'm now able to continue the course.
Thanks
Jay
0 -
Hi jsissom,
Do you mind sharing the solution that you had come up with?Thanks,
Chandra0 -
I ended up having two networks for my VMs. One host-only network for ssh from host machine and a bridge network.
Thanks,
Chandra0
Categories
- All Categories
- 217 LFX Mentorship
- 217 LFX Mentorship: Linux Kernel
- 791 Linux Foundation IT Professional Programs
- 353 Cloud Engineer IT Professional Program
- 178 Advanced Cloud Engineer IT Professional Program
- 82 DevOps Engineer IT Professional Program
- 147 Cloud Native Developer IT Professional Program
- 137 Express Training Courses
- 137 Express Courses - Discussion Forum
- 6.2K Training Courses
- 47 LFC110 Class Forum - Discontinued
- 71 LFC131 Class Forum
- 42 LFD102 Class Forum
- 226 LFD103 Class Forum
- 18 LFD110 Class Forum
- 38 LFD121 Class Forum
- 18 LFD133 Class Forum
- 7 LFD134 Class Forum
- 18 LFD137 Class Forum
- 71 LFD201 Class Forum
- 4 LFD210 Class Forum
- 5 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 2 LFD233 Class Forum
- 4 LFD237 Class Forum
- 24 LFD254 Class Forum
- 697 LFD259 Class Forum
- 111 LFD272 Class Forum
- 4 LFD272-JP クラス フォーラム
- 12 LFD273 Class Forum
- 148 LFS101 Class Forum
- 1 LFS111 Class Forum
- 3 LFS112 Class Forum
- 2 LFS116 Class Forum
- 4 LFS118 Class Forum
- LFS120 Class Forum
- 7 LFS142 Class Forum
- 5 LFS144 Class Forum
- 4 LFS145 Class Forum
- 2 LFS146 Class Forum
- 3 LFS147 Class Forum
- 1 LFS148 Class Forum
- 15 LFS151 Class Forum
- 2 LFS157 Class Forum
- 29 LFS158 Class Forum
- 7 LFS162 Class Forum
- 2 LFS166 Class Forum
- 4 LFS167 Class Forum
- 3 LFS170 Class Forum
- 2 LFS171 Class Forum
- 3 LFS178 Class Forum
- 3 LFS180 Class Forum
- 2 LFS182 Class Forum
- 5 LFS183 Class Forum
- 31 LFS200 Class Forum
- 737 LFS201 Class Forum - Discontinued
- 3 LFS201-JP クラス フォーラム
- 18 LFS203 Class Forum
- 134 LFS207 Class Forum
- 2 LFS207-DE-Klassenforum
- 1 LFS207-JP クラス フォーラム
- 302 LFS211 Class Forum
- 56 LFS216 Class Forum
- 52 LFS241 Class Forum
- 48 LFS242 Class Forum
- 38 LFS243 Class Forum
- 15 LFS244 Class Forum
- 2 LFS245 Class Forum
- LFS246 Class Forum
- 48 LFS250 Class Forum
- 2 LFS250-JP クラス フォーラム
- 1 LFS251 Class Forum
- 152 LFS253 Class Forum
- 1 LFS254 Class Forum
- 1 LFS255 Class Forum
- 7 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.2K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 118 LFS260 Class Forum
- 159 LFS261 Class Forum
- 42 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 24 LFS267 Class Forum
- 22 LFS268 Class Forum
- 30 LFS269 Class Forum
- LFS270 Class Forum
- 202 LFS272 Class Forum
- 2 LFS272-JP クラス フォーラム
- 1 LFS274 Class Forum
- 4 LFS281 Class Forum
- 9 LFW111 Class Forum
- 259 LFW211 Class Forum
- 181 LFW212 Class Forum
- 13 SKF100 Class Forum
- 1 SKF200 Class Forum
- 1 SKF201 Class Forum
- 795 Hardware
- 199 Drivers
- 68 I/O Devices
- 37 Monitors
- 102 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 85 Storage
- 758 Linux Distributions
- 82 Debian
- 67 Fedora
- 17 Linux Mint
- 13 Mageia
- 23 openSUSE
- 148 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 353 Ubuntu
- 468 Linux System Administration
- 39 Cloud Computing
- 71 Command Line/Scripting
- Github systems admin projects
- 93 Linux Security
- 78 Network Management
- 102 System Management
- 47 Web Management
- 63 Mobile Computing
- 18 Android
- 33 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 371 Off Topic
- 114 Introductions
- 174 Small Talk
- 22 Study Material
- 805 Programming and Development
- 303 Kernel Development
- 484 Software Development
- 1.8K Software
- 261 Applications
- 183 Command Line
- 3 Compiling/Installing
- 987 Games
- 317 Installation
- 97 All In Program
- 97 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)