Welcome to the new Linux Foundation Forum!

Lab 6.1 External Egress not working (No policies defined yet)

I am trying follow instruction on Lab 6.1 to test Egress before I setup any policies. I am not able to reach www.linux.com as asked below:

Now, test egress from a container to the outside world. We’ll use the netcat​ command to verify
access to a running webserver on port 80. First, test local access to nginx, then a remote server.
[email protected]:~/app2$ kubectl exec -it -c busy secondapp sh
/ $ nc -vz 127.0.0.1 80
127.0.0.1 (127.0.0.1:80) open
/ $ nc -vz www.linux.com 80
www.linux.com (151.101.185.5:80) open

I have re-installed Kubeadm but still cannot reach outside. I have running in virtualbox. One master and 3 nodes. What could I check to enable external access?

Thanks
-Ashish

Comments

  • Hi Ashish, do you have your vNics open for all traffic in Vbox? I had a similar issue and that solved it for me.
    -Chris

  • ajainajain Posts: 11

    Hi Chris,

    Thanks for responding. I am using bridge, so all v instances have ip address from router. I can ping external nodes:

    [email protected]:~$ ping www.linux.com
    PING n.ssl.fastly.net (151.101.21.5) 56(84) bytes of data.
    64 bytes from 151.101.21.5: icmp_seq=1 ttl=57 time=462 ms
    64 bytes from 151.101.21.5: icmp_seq=2 ttl=57 time=482 ms
    ^C

    [email protected]:~$ kubectl get nodes
    NAME STATUS ROLES AGE VERSION
    ashish-ubuntu0 Ready 15h v1.11.2
    ashish-ubuntu1 Ready master 16h v1.11.1
    ashish-ubuntu2 Ready 16h v1.11.1
    ashish-ubuntu3 Ready 16h v1.11.1

    [email protected]:~/app2$ kubectl get pods -o wide
    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

    [email protected]:~/app2$ kubectl get pods -o wide
    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

    -Ashish

  • ajainajain Posts: 11

    I see that pings to ip address of www.linux.com is working. But nc with ip address doesn't work.

  • ajainajain Posts: 11

    My bad while testing ip address with nc. Egress was block. So now it seems it is DNS issue is reality. Egress/Ingress block are working so I can carry on with my lab. Not sure what to look for resolving DNS issue.

  • Hello,

    Could you paste the commands and output which aren't working? Also the output of nslookup to whatever site you are trying to connect to. This will help view the DNS server being used and may help narrow down the issue.

    Regards,

  • ajainajain Posts: 11

    nc -vz www.linux.com 80

    / # nslookup www.linux.com
    ;; connection timed out; no servers could be reached

    Looks like no DNS configuration is getting configured in busybox container.

  • Okay. Yes, it is not able to reach the DNS server. FIrst let's see if all the pods are running. Could you please show the outout of these two commands, kubectl get pods -o wide --all-namespaces and kubectl get svc --all-namespaces. Hopefully your kude-dns service is running. Mine says:

    $ kubectl get svc --all-namespaces
    NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    default kubernetes ClusterIP 10.96.0.1 443/TCP 4d
    kube-system calico-etcd ClusterIP 10.96.232.136 6666/TCP 4d
    kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 4d

    `

  • ajainajain Posts: 11

    Thanks for helping me:

    Here is the output:

    [email protected]:~/app2$ kubectl get pods -o wide --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    default secondapp 2/2 Running 3 3h 192.168.173.1 ashish-ubuntu2
    kube-system calico-etcd-5hh72 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system calico-node-4dn2q 2/2 Running 1 20h 192.168.0.121 ashish-ubuntu3
    kube-system calico-node-8dtts 2/2 Running 1 20h 192.168.0.156 ashish-ubuntu2
    kube-system calico-node-j5rc2 2/2 Running 1 19h 192.168.0.183 ashish-ubuntu0
    kube-system calico-node-kgxk7 2/2 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 21h 192.168.231.1 ashish-ubuntu1
    kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 21h 192.168.231.2 ashish-ubuntu1
    kube-system etcd-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-8xh9m 1/1 Running 0 19h 192.168.0.183 ashish-ubuntu0
    kube-system kube-proxy-8xmnd 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-s6rng 1/1 Running 0 20h 192.168.0.156 ashish-ubuntu2
    kube-system kube-proxy-xlzbw 1/1 Running 0 20h 192.168.0.121 ashish-ubuntu3
    kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 20h 192.168.231.5 ashish-ubuntu1
    metallb-system controller-558b9b86cd-gn7nx 1/1 Running 0 19h 192.168.231.6 ashish-ubuntu1
    metallb-system speaker-t48z2 1/1 Running 0 19h 192.168.0.193 ashish-ubuntu1
    [email protected]:~/app2$
    [email protected]:~/app2$
    [email protected]:~/app2$ kubectl get svc --all-namespaces
    NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    default kubernetes ClusterIP 10.96.0.1 443/TCP 21h
    default secondapp NodePort 10.106.147.238 80:32000/TCP 6h
    kube-system calico-etcd ClusterIP 10.96.232.136 6666/TCP 21h
    kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 21h
    kube-system kubernetes-dashboard NodePort 10.109.145.247 443:32116/TCP 20h
    [email protected]:~/app2$

  • ajainajain Posts: 11

    Here is busybox resolve file:

    [email protected]:~/app2$ kubectl exec -it -c busy secondapp sh
    / # cat /etc/resolv.conf
    nameserver 10.96.0.10
    search default.svc.cluster.local svc.cluster.local cluster.local
    options ndots:5
    / #

  • Are you able to use netcat and connect to your DNS service?** nc 10.96.0.10 53** If it hangs it has made a connection, which is good. You'd have to send some hex code with xxd to get a proper response.

    If that works, then it could be that your kube-dns is unable to forward the request to an outside source. My first thought is firewalls. If you have opened all the ports inside AWS/GCE - all ports to all instances - then they shouldn't stop anything and the request will use your node or cluster DNS settings. I assume you can ping/nslookup/dig from the host, only the pod is not able to use DNS.

    I note you are running metallb-system, a load balancer. This could be intercepting the return DNS forward query and sending it somewhere else or blocking it. Does it work if you don't have that running?

    Regards,

  • ajainajain Posts: 11

    It hangs on nc command. I see coredns have errors in it. I have disabled firewall on all nodes and removed metallb. Just FYI: After installing kubeadm, I couldn't run dashboard till it was deployed on master node. API server was unreachable from other nodes for dashboard pod. Same with metallb. But running busybox on master node doesn't help here.

    2018/09/08 12:37:54 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:36874->192.168.0.1:53: i/o timeout
    2018/09/08 12:39:27 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:50580->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:57479->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:09 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:58589->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:14 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53425->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:40 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53883->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:40 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:38421->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:42 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:51105->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:42 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:45913->192.168.0.1:53: i/o timeout
    2018/09/08 20:15:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:35092->192.168.0.1:53: i/o timeout
    2018/09/08 20:43:19 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:42572->192.168.0.1:53: i/o timeout

    [email protected]:~/app2$ kubectl get pods -o wide --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    default secondapp 2/2 Running 0 48m 192.168.231.7 ashish-ubuntu1
    kube-system calico-etcd-5hh72 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system calico-node-4dn2q 2/2 Running 1 21h 192.168.0.121 ashish-ubuntu3
    kube-system calico-node-8dtts 2/2 Running 1 21h 192.168.0.156 ashish-ubuntu2
    kube-system calico-node-j5rc2 2/2 Running 1 20h 192.168.0.183 ashish-ubuntu0
    kube-system calico-node-kgxk7 2/2 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 22h 192.168.231.1 ashish-ubuntu1
    kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 22h 192.168.231.2 ashish-ubuntu1
    kube-system etcd-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-8xh9m 1/1 Running 0 20h 192.168.0.183 ashish-ubuntu0
    kube-system kube-proxy-8xmnd 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-s6rng 1/1 Running 0 21h 192.168.0.156 ashish-ubuntu2
    kube-system kube-proxy-xlzbw 1/1 Running 0 21h 192.168.0.121 ashish-ubuntu3
    kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 21h 192.168.231.5 ashish-ubuntu1
    [email protected]:~/app2$

  • ajainajain Posts: 11

    I have re-installed kubeadm after changing k8sMaster.sh, by adding --feature-gates=CoreDNS=false to kubeadm init. This will use kube-dns instead of coredns.

    Still DNS didn't work. Later found one resolution at below link. After this it started to work:

    https://github.com/coreos/flannel/issues/983

    Copied from above link:
    Found the problem which exists in DNS (ip adresses works ok. Helped me config map for dns:

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: kube-dns
    namespace: kube-system
    labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    data:
    upstreamNameservers: |-
    ["8.8.8.8", "8.8.4.4"]

  • Thanks for letting us know about the fix. You are/were running flannel instead of Calico then?

  • ajainajain Posts: 11

    No. I was running Calico. But the DNS was CoreDNS. I just tried Kube-Dns to see if that will make difference. I am not concerned how infrastructure is set but mainly to test/learn functionality. Even Metallb is running, so I am able to test Load Balancer service type. This is all setup in virtualbox.

    BTW. Thanks for guiding me towards resolution.

  • Glad its working!

  • martinajemartinaje Posts: 1
    edited October 2018

    Yes its enabled by following the instructions on the documentation, now today a have another problem I turned off my router and started back up but now it seems my router doesn't send any internet connection. Regarding I already talk to the Belkin support Belkin Support expert team. But still, I have been facing the same if you have any assistance about this so please share with me.

  • serewiczserewicz Posts: 506

    Not which aspect you were asking about this time. One note, ensure that your VirtualBox network is set to allow all traffic. It defaults to deny. This can be found under the advanced tab for the network.

    Regards,

Sign In or Register to comment.