Welcome to the Linux Foundation Forum!

Lab 3.2/3.3 master to node network problem

jsissom
jsissom Posts: 8
edited April 2018 in LFS258 Class Forum

I am doing the labs in VirtualBox.  I had to make a few changes to the instructions to get this far but I wonder if there are other things that I need to do differently for VirtualBox to work.

The VirtualBox NAT network interface lets you access the internet but you cannot access other VMs so each VM has 2 interfaces, a NAT interface to access the internet and a Host Only interface to access it from my host and so they can talk to each other.  To make this work, I needed to add the --apiserver-advertise-address=192.168.56.10 option on the kubeadm init command otherwise the second VM could not access the API server.  I also needed to disable the firewall using the ufw disable command on both boxes.

I was able to finish Lab 3.2 after making these changes.

On Lab 3.3 step 2, when I run the exec command from my master, I get an error:


jsissom@kmaster:~$ kubectl get po -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP           NODE
nginx-768979984b-nfh6r   1/1       Running   0          10m       10.244.1.6   knode1
nginx-768979984b-pqszn   1/1       Running   0          10m       10.244.1.5   knode1
nginx-768979984b-ssdkq   1/1       Running   0          9m        10.244.1.7   knode1

jsissom@kmaster:~$ kubectl exec nginx-768979984b-pqszn -- printenv
error: unable to upgrade connection: pod does not exist

I told it to increase the replicas to 10 to get some running on master and if I try to exec to one running on the master node, I get connected and it works:


jsissom@kmaster:~$ kubectl get po -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-768979984b-4b7cc   1/1       Running   0          42s       10.244.0.4    kmaster
nginx-768979984b-7j9k2   1/1       Running   0          42s       10.244.1.8    knode1
nginx-768979984b-9rqfk   1/1       Running   0          42s       10.244.1.11   knode1
nginx-768979984b-cq9r2   1/1       Running   0          42s       10.244.0.6    kmaster
nginx-768979984b-gn9g4   1/1       Running   0          42s       10.244.1.10   knode1
nginx-768979984b-hfgx7   1/1       Running   0          42s       10.244.1.9    knode1
nginx-768979984b-nfh6r   1/1       Running   0          11m       10.244.1.6    knode1
nginx-768979984b-pqszn   1/1       Running   0          11m       10.244.1.5    knode1
nginx-768979984b-ssdkq   1/1       Running   0          10m       10.244.1.7    knode1
nginx-768979984b-vqznn   1/1       Running   0          42s       10.244.0.5    kmaster

jsissom@kmaster:~$ kubectl exec nginx-768979984b-4b7cc -- printenv
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=nginx-768979984b-4b7cc
KUBERNETES_PORT_443_TCP_PORT=443
NGINX_SERVICE_HOST=10.108.68.108
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
NGINX_SERVICE_PORT=80
NGINX_PORT=tcp://10.108.68.108:80
NGINX_PORT_80_TCP=tcp://10.108.68.108:80
NGINX_PORT_80_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
NGINX_PORT_80_TCP_ADDR=10.108.68.108
KUBERNETES_SERVICE_PORT=443
NGINX_PORT_80_TCP_PORT=80
NGINX_VERSION=1.13.12-1~stretch
NJS_VERSION=1.13.12.0.2.0-1~stretch
HOME=/root

jsissom@kmaster:~$ kubectl exec nginx-768979984b-7j9k2 -- printenv
error: unable to upgrade connection: pod does not exist

Is there something I need to change in my environment to allow me to exec into a pod running on my node?

If I continue with the lab, in step 6, if I curl my public IP at the LoadBalancer port over and over, I get connected sometimes, but not others:


jsissom@host ~ $ curl http://kmaster:31166/
curl: (55) getpeername() failed with errno 22: Invalid argument
jsissom@host ~ $ curl http://kmaster:31166/
curl: (55) getpeername() failed with errno 22: Invalid argument
jsissom@host ~ $ curl http://kmaster:31166/
<!DOCTYPE html>
jsissom@host ~ $ curl http://kmaster:31166/
curl: (55) getpeername() failed with errno 22: Invalid argument
jsissom@host ~ $ curl http://kmaster:31166/
<!DOCTYPE html>

Thanks

Jay

 

Comments

  • chrispokorni
    chrispokorni Posts: 2,372

    Hi Jay, 

    It is impressive the amount of efort you have invested into this. There are a few earlier posts on VMs running on vbox, but not sure how many labs were completed under this setup.

    While I cannot relate to what you are experiencing (I ran all the labs on GCP as instructed), reading your post got me to do some more in depth research on vbox networking, and kubernetes networking. 

    I found a rather complex solution to the VM to VM networking (and I would probably try to simplify it somehow, if possible): 

    http://www.brianlinkletter.com/how-to-use-virtualbox-to-emulate-a-network/

    and some good information on the vbox documentation website:

    https://www.virtualbox.org/manual/ch06.html

    As far as kubernetes, I believe that what you accomplished with kubeadm init ... --apiserver-advertise-address..., may need a similar approach for kube-proxy on each minion. The kubeadm join does not provide any interesting options, but I think that the kube-proxy may need a ...--bind-address... also: 

    https://v1-9.docs.kubernetes.io/docs/reference/generated/kube-proxy/

    I hope some of this helps. 

    Regards, 

    -Chris

  • jsissom
    jsissom Posts: 8

    It's disappointing that Kubernetes won't work on VirtualBox.  I see tutorial after tutorial all over the web showing how to set it up, but I always get the same non-working environment.  I had hoped that this course would teach me what I was doing wrong.  It's disappointing that it does not.  Our production environment has public and private network cards in each server and we would want to specify how the software uses those networks so I'm not confident that I'll be able to make it work there if I can't make it work on VirtualBox.

    I would suggest that you put language in your course advertisement that this course requires you to pay for GCE.  The advertising info for the course doesn't say that.  In fact, in slide 1.7.a it specifically says it will work on VirtualBox which does not seem to be a true statement.

    Thanks

     

  • serewicz
    serewicz Posts: 1,000

    It has been a while since I tried to get Kubernetes to run on virtualbox. Let me attempt the labs and let you know what I have found.

     

     

  • serewicz
    serewicz Posts: 1,000

    Hello,

    Very strange behavior. I have gone through the labs using VirtualBox on my laptop. I had the same problem at first when exposing the port as a LoadBalancer. It works if you used NodePort instead. Here is where it gets strange, after it works as a NodePort, it will continue to work - on a new, different ports - for LoadBalancer. So there is some strange handover which sticks once NodePort opens or connects it.   If you dig into documentation you'll find that a LoadBalancer is supposed to be the same as NodePort, except it send an asyncronus request to the cloud provider to spin up a LoadBalancer. Even if there is not one availalbe it works as a node port in GCE, AWS and OpenStack.  

    As it works now, even when I switch it back and forth updating the port, I am chalking this up a bug. Please let me know if I can offer more information.

    Regards,

     

  • jsissom
    jsissom Posts: 8

    Can you tell me the network configuration you used on VirtualBox?

    Did you make sure that nginx was running on both nodes and try exercise 3.3, step 2 against pods running on both nodes?  That failed for me even before the load balancer port was attempted.

    Were there steps you needed to perform on the base OS that were not included in the labs?

    Thanks

     

  • serewicz
    serewicz Posts: 1,000
    edited April 2018

    Hello,

    The only step I did on the instances, which was not written into the lab, was to disable the firewall with ufw disable.   I also ensured virtualbox didn't deny any traffic.  Other than virtualbox not handling the initial loadbalancer configureation, but working with later attempts, I followed the steps direct from the lab and saw the expected behavior. 

    Regards,

     

  • jsissom
    jsissom Posts: 8

    Thanks for trying it.  There must be something you are doing that is different.  Here's what happens when I try it:

    Using VirtualBox 5.2.8

    1.  Create Ubuntu 16.04-4 Server vm, 1 nat network interface, 2gb RAM - take all defaults

    2.  ufw disable

    3.  Start Lab 3.1

    4.  #11.  Fails due to swap being enabled

    5.  swapoff -a, edit /etc/fstab to comment out swap

    6.  Continue with Lab 3.1.  It completes successfully.

    7.  Create 2nd Ubuntu server, 1 Nat network interface, 2gb RAM, take all defaults

    8.  ufw disable, disable swap

    9.  Start Lab 3.2

    10.  #2.  The ethernet name is enp0s3, not ens4

    11.  Both machines have the same Nat IP address (10.0.2.15) so they cannot communicate with each other.

    When I tried it originally, I gave the vm's two network cards, one the Nat interface that can connect to the internet and one is a host-only interface so they can connect to each other.  This allowed me to install but fails later.

    Can you tell me what you did to allow the two VMs to communicate when using the Nat interface?

    Thanks

    Jay

     

  • serewicz
    serewicz Posts: 1,000
    edited April 2018

    I also had to disable swap. I did it without thinking, but remembered once you mentioned it. 

    Did you make sure the VB interfaces are open and not set to deny all?

    Regards,

  • jsissom
    jsissom Posts: 8
    edited April 2018

    When I was trying it originally, I setup 2 network interfaces, 1 was Nat so the VM could access the internet and the 2nd was a host-only network so the 2 vms could communicate.

    When I tried it this time, I took the defaults of Virtual Box since you said you didn't do anything special when you tested it.  In this case, the two VMs have 1 interface that are Nat and cannot communicate.

    Can you please tell me how you configured the network in VB so the VMs can access the internet and communicate to each other?  I don't know what you mean by "I also ensured virtualbox didn't deny any traffic.".  I don't know if VirtualBox is denying any traffic or not.  It assigns the same IP address to both VMs when they are in Nat mode so it is impossible for them to communicate with each other.

     

     

  • chrispokorni
    chrispokorni Posts: 2,372

    Hi Jay, 

    I took a swing at this, and this is how far I got in Lab 3:

    vBox 5.2.8

    1. Created 2 VMs, 2 GB each

    2. Adapter 1 ONLY - Bridged interface on each (Allow All)

    3. Installed Ubuntu 16 and vbox guest additions

    4. ip a (or ifconfig) outputs showed different enp0s3 IPs for each Ubuntu instance: master 10.0.0.50 and second 10.0.0.125

    5. ufw status -> inactive (I did not have to do anything here)

    6. nc to test inter-node communication, messages were delivered successfully from master to second and second to master

    7. swapoff -a

    8. commented out swap in /etc/fstab

    All steps in labs 3.1, 3.2, 3.3 completed succesfully, except Lab 3.3 Step 6, where I was able to access the nginx page, from a browser on my Host machine, only on the <MASTER-enp0s3-inet-addr>:32056, which in my case was 10.0.0.50:32056.

    With NAT only I was not able to get anywhere, because like you said, the VM instances were not able to communicate with each other.

    I hope this helps, 

    -Chris

  • serewicz
    serewicz Posts: 1,000

    Hello,

    It sounds like the issue is with networking, and how virtualbox handles connection from one VM to another. I only a casual user of Oracle VirtualBox, you may have some luck asking here: https://forums.virtualbox.org/viewforum.php?f=3

    If the Kubernetes (and the labs) works with some virtualbox configurations, but don't in some multi-interface configurrations it would seem to be an issue with virtualbox. As kubernetes is all API driven, it has to be something where the interface expected to transmit or receive is not the one actually being used. I would not be surprised if the "primary" interface was hard-coded and the issue doesn't happen enough for someone to revist the issue in the kubernetes code.   

    Have you tried using just one interface at first and ensuring the VMs can see each other and work as expected, then adding a second interface for your other needs? This would have the primary interface be all the expected things for kubernetes, then you'd know it works.The later addition of a second interface could be used for your other infrastructure needs. Could be a work-around to the issue. 

    Regards,

  • jsissom
    jsissom Posts: 8
    edited April 2018

    Using a bridged adapter is problematic because with dhcp the IP address changes from day to day at my workplace.

    If I pick a network type where the two VMs can see each other, there is no internet access.  If I pick one where they can see the internet, they can't see each other.  VirtualBox requires 2 interfaces, unless you use the bridged adapter which means your IP address changes based on your location.  Here is the documentation on the networking for Virtual Box:  https://www.virtualbox.org/manual/ch06.html

    I've come up with a solution to the problem so I'm now able to continue the course.

    Thanks

    Jay

  • borian
    borian Posts: 2

    Hi jsissom,
    Do you mind sharing the solution that you had come up with?

    Thanks,
    Chandra

  • borian
    borian Posts: 2

    I ended up having two networks for my VMs. One host-only network for ssh from host machine and a bridge network.

    Thanks,
    Chandra

Categories

Upcoming Training