Welcome to the Linux Foundation Forum!

Lab 6.1 External Egress not working (No policies defined yet)

I am trying follow instruction on Lab 6.1 to test Egress before I setup any policies. I am not able to reach www.linux.com as asked below:

Now, test egress from a container to the outside world. We’ll use the netcat​ command to verify
access to a running webserver on port 80. First, test local access to nginx, then a remote server.
student@ckad-1:~/app2$ kubectl exec -it -c busy secondapp sh
/ $ nc -vz 127.0.0.1 80
127.0.0.1 (127.0.0.1:80) open
/ $ nc -vz www.linux.com 80
www.linux.com (151.101.185.5:80) open

I have re-installed Kubeadm but still cannot reach outside. I have running in virtualbox. One master and 3 nodes. What could I check to enable external access?

Thanks
-Ashish

Comments

  • Hi Ashish, do you have your vNics open for all traffic in Vbox? I had a similar issue and that solved it for me.
    -Chris

  • ajain
    ajain Posts: 11

    Hi Chris,

    Thanks for responding. I am using bridge, so all v instances have ip address from router. I can ping external nodes:

    ashish@ashish-ubuntu1:~$ ping www.linux.com
    PING n.ssl.fastly.net (151.101.21.5) 56(84) bytes of data.
    64 bytes from 151.101.21.5: icmp_seq=1 ttl=57 time=462 ms
    64 bytes from 151.101.21.5: icmp_seq=2 ttl=57 time=482 ms
    ^C

    ashish@ashish-ubuntu1:~$ kubectl get nodes
    NAME STATUS ROLES AGE VERSION
    ashish-ubuntu0 Ready 15h v1.11.2
    ashish-ubuntu1 Ready master 16h v1.11.1
    ashish-ubuntu2 Ready 16h v1.11.1
    ashish-ubuntu3 Ready 16h v1.11.1

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide
    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide
    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

    -Ashish

  • ajain
    ajain Posts: 11

    I see that pings to ip address of www.linux.com is working. But nc with ip address doesn't work.

  • ajain
    ajain Posts: 11

    My bad while testing ip address with nc. Egress was block. So now it seems it is DNS issue is reality. Egress/Ingress block are working so I can carry on with my lab. Not sure what to look for resolving DNS issue.

  • serewicz
    serewicz Posts: 1,000

    Hello,

    Could you paste the commands and output which aren't working? Also the output of nslookup to whatever site you are trying to connect to. This will help view the DNS server being used and may help narrow down the issue.

    Regards,

  • ajain
    ajain Posts: 11

    nc -vz www.linux.com 80

    / # nslookup www.linux.com
    ;; connection timed out; no servers could be reached

    Looks like no DNS configuration is getting configured in busybox container.

  • serewicz
    serewicz Posts: 1,000

    Okay. Yes, it is not able to reach the DNS server. FIrst let's see if all the pods are running. Could you please show the outout of these two commands, kubectl get pods -o wide --all-namespaces and kubectl get svc --all-namespaces. Hopefully your kude-dns service is running. Mine says:

    $ kubectl get svc --all-namespaces
    NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    default kubernetes ClusterIP 10.96.0.1 443/TCP 4d
    kube-system calico-etcd ClusterIP 10.96.232.136 6666/TCP 4d
    kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 4d

    `

  • ajain
    ajain Posts: 11

    Thanks for helping me:

    Here is the output:

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    default secondapp 2/2 Running 3 3h 192.168.173.1 ashish-ubuntu2
    kube-system calico-etcd-5hh72 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system calico-node-4dn2q 2/2 Running 1 20h 192.168.0.121 ashish-ubuntu3
    kube-system calico-node-8dtts 2/2 Running 1 20h 192.168.0.156 ashish-ubuntu2
    kube-system calico-node-j5rc2 2/2 Running 1 19h 192.168.0.183 ashish-ubuntu0
    kube-system calico-node-kgxk7 2/2 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 21h 192.168.231.1 ashish-ubuntu1
    kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 21h 192.168.231.2 ashish-ubuntu1
    kube-system etcd-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-8xh9m 1/1 Running 0 19h 192.168.0.183 ashish-ubuntu0
    kube-system kube-proxy-8xmnd 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-s6rng 1/1 Running 0 20h 192.168.0.156 ashish-ubuntu2
    kube-system kube-proxy-xlzbw 1/1 Running 0 20h 192.168.0.121 ashish-ubuntu3
    kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 20h 192.168.231.5 ashish-ubuntu1
    metallb-system controller-558b9b86cd-gn7nx 1/1 Running 0 19h 192.168.231.6 ashish-ubuntu1
    metallb-system speaker-t48z2 1/1 Running 0 19h 192.168.0.193 ashish-ubuntu1
    ashish@ashish-ubuntu0:~/app2$
    ashish@ashish-ubuntu0:~/app2$
    ashish@ashish-ubuntu0:~/app2$ kubectl get svc --all-namespaces
    NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    default kubernetes ClusterIP 10.96.0.1 443/TCP 21h
    default secondapp NodePort 10.106.147.238 80:32000/TCP 6h
    kube-system calico-etcd ClusterIP 10.96.232.136 6666/TCP 21h
    kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 21h
    kube-system kubernetes-dashboard NodePort 10.109.145.247 443:32116/TCP 20h
    ashish@ashish-ubuntu0:~/app2$

  • ajain
    ajain Posts: 11

    Here is busybox resolve file:

    ashish@ashish-ubuntu0:~/app2$ kubectl exec -it -c busy secondapp sh
    / # cat /etc/resolv.conf
    nameserver 10.96.0.10
    search default.svc.cluster.local svc.cluster.local cluster.local
    options ndots:5
    / #

  • serewicz
    serewicz Posts: 1,000

    Are you able to use netcat and connect to your DNS service?** nc 10.96.0.10 53** If it hangs it has made a connection, which is good. You'd have to send some hex code with xxd to get a proper response.

    If that works, then it could be that your kube-dns is unable to forward the request to an outside source. My first thought is firewalls. If you have opened all the ports inside AWS/GCE - all ports to all instances - then they shouldn't stop anything and the request will use your node or cluster DNS settings. I assume you can ping/nslookup/dig from the host, only the pod is not able to use DNS.

    I note you are running metallb-system, a load balancer. This could be intercepting the return DNS forward query and sending it somewhere else or blocking it. Does it work if you don't have that running?

    Regards,

  • ajain
    ajain Posts: 11

    It hangs on nc command. I see coredns have errors in it. I have disabled firewall on all nodes and removed metallb. Just FYI: After installing kubeadm, I couldn't run dashboard till it was deployed on master node. API server was unreachable from other nodes for dashboard pod. Same with metallb. But running busybox on master node doesn't help here.

    2018/09/08 12:37:54 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:36874->192.168.0.1:53: i/o timeout
    2018/09/08 12:39:27 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:50580->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:57479->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:09 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:58589->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:14 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53425->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:40 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53883->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:40 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:38421->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:42 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:51105->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:42 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:45913->192.168.0.1:53: i/o timeout
    2018/09/08 20:15:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:35092->192.168.0.1:53: i/o timeout
    2018/09/08 20:43:19 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:42572->192.168.0.1:53: i/o timeout

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    default secondapp 2/2 Running 0 48m 192.168.231.7 ashish-ubuntu1
    kube-system calico-etcd-5hh72 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system calico-node-4dn2q 2/2 Running 1 21h 192.168.0.121 ashish-ubuntu3
    kube-system calico-node-8dtts 2/2 Running 1 21h 192.168.0.156 ashish-ubuntu2
    kube-system calico-node-j5rc2 2/2 Running 1 20h 192.168.0.183 ashish-ubuntu0
    kube-system calico-node-kgxk7 2/2 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 22h 192.168.231.1 ashish-ubuntu1
    kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 22h 192.168.231.2 ashish-ubuntu1
    kube-system etcd-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-8xh9m 1/1 Running 0 20h 192.168.0.183 ashish-ubuntu0
    kube-system kube-proxy-8xmnd 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-s6rng 1/1 Running 0 21h 192.168.0.156 ashish-ubuntu2
    kube-system kube-proxy-xlzbw 1/1 Running 0 21h 192.168.0.121 ashish-ubuntu3
    kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 21h 192.168.231.5 ashish-ubuntu1
    ashish@ashish-ubuntu0:~/app2$

  • ajain
    ajain Posts: 11

    I have re-installed kubeadm after changing k8sMaster.sh, by adding --feature-gates=CoreDNS=false to kubeadm init. This will use kube-dns instead of coredns.

    Still DNS didn't work. Later found one resolution at below link. After this it started to work:

    https://github.com/coreos/flannel/issues/983

    Copied from above link:
    Found the problem which exists in DNS (ip adresses works ok. Helped me config map for dns:

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: kube-dns
    namespace: kube-system
    labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    data:
    upstreamNameservers: |-
    ["8.8.8.8", "8.8.4.4"]

  • serewicz
    serewicz Posts: 1,000

    Thanks for letting us know about the fix. You are/were running flannel instead of Calico then?

  • ajain
    ajain Posts: 11

    No. I was running Calico. But the DNS was CoreDNS. I just tried Kube-Dns to see if that will make difference. I am not concerned how infrastructure is set but mainly to test/learn functionality. Even Metallb is running, so I am able to test Load Balancer service type. This is all setup in virtualbox.

    BTW. Thanks for guiding me towards resolution.

  • serewicz
    serewicz Posts: 1,000

    Glad its working!

  • martinaje
    martinaje Posts: 1
    edited October 2018

    Yes its enabled by following the instructions on the documentation, now today a have another problem I turned off my router and started back up but now it seems my router doesn't send any internet connection. Regarding I already talk to the Belkin support Belkin Support expert team. But still, I have been facing the same if you have any assistance about this so please share with me.

  • serewicz
    serewicz Posts: 1,000

    Not which aspect you were asking about this time. One note, ensure that your VirtualBox network is set to allow all traffic. It defaults to deny. This can be found under the advanced tab for the network.

    Regards,

Categories

Upcoming Training