Lab 6.1 External Egress not working (No policies defined yet)

ajain

I am trying follow instruction on Lab 6.1 to test Egress before I setup any policies. I am not able to reach www.linux.com as asked below:

Now, test egress from a container to the outside world. We’ll use the netcat​ command to verify
access to a running webserver on port 80. First, test local access to nginx, then a remote server.
student@ckad-1:~/app2$ kubectl exec -it -c busy secondapp sh
/ $ nc -vz 127.0.0.1 80
127.0.0.1 (127.0.0.1:80) open
/ $ nc -vz www.linux.com 80
www.linux.com (151.101.185.5:80) open

I have re-installed Kubeadm but still cannot reach outside. I have running in virtualbox. One master and 3 nodes. What could I check to enable external access?

Thanks
-Ashish

chrispokorni

Hi Ashish, do you have your vNics open for all traffic in Vbox? I had a similar issue and that solved it for me.
-Chris

ajain

Hi Chris,

Thanks for responding. I am using bridge, so all v instances have ip address from router. I can ping external nodes:

ashish@ashish-ubuntu1:~$ ping www.linux.com
PING n.ssl.fastly.net (151.101.21.5) 56(84) bytes of data.
64 bytes from 151.101.21.5: icmp_seq=1 ttl=57 time=462 ms
64 bytes from 151.101.21.5: icmp_seq=2 ttl=57 time=482 ms
^C

ashish@ashish-ubuntu1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ashish-ubuntu0 Ready 15h v1.11.2
ashish-ubuntu1 Ready master 16h v1.11.1
ashish-ubuntu2 Ready 16h v1.11.1
ashish-ubuntu3 Ready 16h v1.11.1

ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

-Ashish

ajain

I see that pings to ip address of www.linux.com is working. But nc with ip address doesn't work.

ajain

My bad while testing ip address with nc. Egress was block. So now it seems it is DNS issue is reality. Egress/Ingress block are working so I can carry on with my lab. Not sure what to look for resolving DNS issue.

serewicz

Hello,

Could you paste the commands and output which aren't working? Also the output of nslookup to whatever site you are trying to connect to. This will help view the DNS server being used and may help narrow down the issue.

Regards,

ajain

nc -vz www.linux.com 80

/ # nslookup www.linux.com
;; connection timed out; no servers could be reached

Looks like no DNS configuration is getting configured in busybox container.

serewicz

Okay. Yes, it is not able to reach the DNS server. FIrst let's see if all the pods are running. Could you please show the outout of these two commands, kubectl get pods -o wide --all-namespaces and kubectl get svc --all-namespaces. Hopefully your kude-dns service is running. Mine says:

$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 443/TCP 4d
kube-system calico-etcd ClusterIP 10.96.232.136 6666/TCP 4d
kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 4d

`

ajain

Thanks for helping me:

Here is the output:

ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
default secondapp 2/2 Running 3 3h 192.168.173.1 ashish-ubuntu2
kube-system calico-etcd-5hh72 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system calico-node-4dn2q 2/2 Running 1 20h 192.168.0.121 ashish-ubuntu3
kube-system calico-node-8dtts 2/2 Running 1 20h 192.168.0.156 ashish-ubuntu2
kube-system calico-node-j5rc2 2/2 Running 1 19h 192.168.0.183 ashish-ubuntu0
kube-system calico-node-kgxk7 2/2 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 21h 192.168.231.1 ashish-ubuntu1
kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 21h 192.168.231.2 ashish-ubuntu1
kube-system etcd-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system kube-proxy-8xh9m 1/1 Running 0 19h 192.168.0.183 ashish-ubuntu0
kube-system kube-proxy-8xmnd 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system kube-proxy-s6rng 1/1 Running 0 20h 192.168.0.156 ashish-ubuntu2
kube-system kube-proxy-xlzbw 1/1 Running 0 20h 192.168.0.121 ashish-ubuntu3
kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 20h 192.168.231.5 ashish-ubuntu1
metallb-system controller-558b9b86cd-gn7nx 1/1 Running 0 19h 192.168.231.6 ashish-ubuntu1
metallb-system speaker-t48z2 1/1 Running 0 19h 192.168.0.193 ashish-ubuntu1
ashish@ashish-ubuntu0:~/app2$
ashish@ashish-ubuntu0:~/app2$
ashish@ashish-ubuntu0:~/app2$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 443/TCP 21h
default secondapp NodePort 10.106.147.238 80:32000/TCP 6h
kube-system calico-etcd ClusterIP 10.96.232.136 6666/TCP 21h
kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 21h
kube-system kubernetes-dashboard NodePort 10.109.145.247 443:32116/TCP 20h
ashish@ashish-ubuntu0:~/app2$

ajain

Here is busybox resolve file:

ashish@ashish-ubuntu0:~/app2$ kubectl exec -it -c busy secondapp sh
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ #

serewicz

Are you able to use netcat and connect to your DNS service?** nc 10.96.0.10 53** If it hangs it has made a connection, which is good. You'd have to send some hex code with xxd to get a proper response.

If that works, then it could be that your kube-dns is unable to forward the request to an outside source. My first thought is firewalls. If you have opened all the ports inside AWS/GCE - all ports to all instances - then they shouldn't stop anything and the request will use your node or cluster DNS settings. I assume you can ping/nslookup/dig from the host, only the pod is not able to use DNS.

I note you are running metallb-system, a load balancer. This could be intercepting the return DNS forward query and sending it somewhere else or blocking it. Does it work if you don't have that running?

Regards,

ajain

It hangs on nc command. I see coredns have errors in it. I have disabled firewall on all nodes and removed metallb. Just FYI: After installing kubeadm, I couldn't run dashboard till it was deployed on master node. API server was unreachable from other nodes for dashboard pod. Same with metallb. But running busybox on master node doesn't help here.

2018/09/08 12:37:54 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:36874->192.168.0.1:53: i/o timeout
2018/09/08 12:39:27 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:50580->192.168.0.1:53: i/o timeout
2018/09/08 16:18:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:57479->192.168.0.1:53: i/o timeout
2018/09/08 16:18:09 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:58589->192.168.0.1:53: i/o timeout
2018/09/08 16:18:14 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53425->192.168.0.1:53: i/o timeout
2018/09/08 19:11:40 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53883->192.168.0.1:53: i/o timeout
2018/09/08 19:11:40 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:38421->192.168.0.1:53: i/o timeout
2018/09/08 19:11:42 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:51105->192.168.0.1:53: i/o timeout
2018/09/08 19:11:42 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:45913->192.168.0.1:53: i/o timeout
2018/09/08 20:15:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:35092->192.168.0.1:53: i/o timeout
2018/09/08 20:43:19 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:42572->192.168.0.1:53: i/o timeout

ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
default secondapp 2/2 Running 0 48m 192.168.231.7 ashish-ubuntu1
kube-system calico-etcd-5hh72 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system calico-node-4dn2q 2/2 Running 1 21h 192.168.0.121 ashish-ubuntu3
kube-system calico-node-8dtts 2/2 Running 1 21h 192.168.0.156 ashish-ubuntu2
kube-system calico-node-j5rc2 2/2 Running 1 20h 192.168.0.183 ashish-ubuntu0
kube-system calico-node-kgxk7 2/2 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 22h 192.168.231.1 ashish-ubuntu1
kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 22h 192.168.231.2 ashish-ubuntu1
kube-system etcd-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system kube-proxy-8xh9m 1/1 Running 0 20h 192.168.0.183 ashish-ubuntu0
kube-system kube-proxy-8xmnd 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system kube-proxy-s6rng 1/1 Running 0 21h 192.168.0.156 ashish-ubuntu2
kube-system kube-proxy-xlzbw 1/1 Running 0 21h 192.168.0.121 ashish-ubuntu3
kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 21h 192.168.231.5 ashish-ubuntu1
ashish@ashish-ubuntu0:~/app2$

ajain

I have re-installed kubeadm after changing k8sMaster.sh, by adding --feature-gates=CoreDNS=false to kubeadm init. This will use kube-dns instead of coredns.

Still DNS didn't work. Later found one resolution at below link. After this it started to work:

https://github.com/coreos/flannel/issues/983

Copied from above link:
Found the problem which exists in DNS (ip adresses works ok. Helped me config map for dns:

apiVersion: v1
kind: ConfigMap
metadata:
name: kube-dns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
upstreamNameservers: |-
["8.8.8.8", "8.8.4.4"]

serewicz

Thanks for letting us know about the fix. You are/were running flannel instead of Calico then?

ajain

No. I was running Calico. But the DNS was CoreDNS. I just tried Kube-Dns to see if that will make difference. I am not concerned how infrastructure is set but mainly to test/learn functionality. Even Metallb is running, so I am able to test Load Balancer service type. This is all setup in virtualbox.

BTW. Thanks for guiding me towards resolution.

serewicz

Glad its working!

martinaje

Yes its enabled by following the instructions on the documentation, now today a have another problem I turned off my router and started back up but now it seems my router doesn't send any internet connection. Regarding I already talk to the Belkin support Belkin Support expert team. But still, I have been facing the same if you have any assistance about this so please share with me.

serewicz

Not which aspect you were asking about this time. One note, ensure that your VirtualBox network is set to allow all traffic. It defaults to deny. This can be found under the advanced tab for the network.

Regards,