Welcome to the Linux Foundation Forum!

Lab 6.1 External Egress not working (No policies defined yet)

ajain
ajain Posts: 11
in LFD259 Class Forum

I am trying follow instruction on Lab 6.1 to test Egress before I setup any policies. I am not able to reach www.linux.com as asked below:

Now, test egress from a container to the outside world. We’ll use the netcat​ command to verify
access to a running webserver on port 80. First, test local access to nginx, then a remote server.
student@ckad-1:~/app2$ kubectl exec -it -c busy secondapp sh
/ $ nc -vz 127.0.0.1 80
127.0.0.1 (127.0.0.1:80) open
/ $ nc -vz www.linux.com 80
www.linux.com (151.101.185.5:80) open

I have re-installed Kubeadm but still cannot reach outside. I have running in virtualbox. One master and 3 nodes. What could I check to enable external access?

Thanks
-Ashish

Comments

  • chrispokorni
    chrispokorni Posts: 2,605

    Hi Ashish, do you have your vNics open for all traffic in Vbox? I had a similar issue and that solved it for me.
    -Chris

  • ajain
    ajain Posts: 11

    Hi Chris,

    Thanks for responding. I am using bridge, so all v instances have ip address from router. I can ping external nodes:

    ashish@ashish-ubuntu1:~$ ping www.linux.com
    PING n.ssl.fastly.net (151.101.21.5) 56(84) bytes of data.
    64 bytes from 151.101.21.5: icmp_seq=1 ttl=57 time=462 ms
    64 bytes from 151.101.21.5: icmp_seq=2 ttl=57 time=482 ms
    ^C

    ashish@ashish-ubuntu1:~$ kubectl get nodes
    NAME STATUS ROLES AGE VERSION
    ashish-ubuntu0 Ready 15h v1.11.2
    ashish-ubuntu1 Ready master 16h v1.11.1
    ashish-ubuntu2 Ready 16h v1.11.1
    ashish-ubuntu3 Ready 16h v1.11.1

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide
    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide
    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    secondapp 2/2 Running 2 2h 192.168.239.133 ashish-ubuntu0

    -Ashish

  • ajain
    ajain Posts: 11

    I see that pings to ip address of www.linux.com is working. But nc with ip address doesn't work.

  • ajain
    ajain Posts: 11

    My bad while testing ip address with nc. Egress was block. So now it seems it is DNS issue is reality. Egress/Ingress block are working so I can carry on with my lab. Not sure what to look for resolving DNS issue.

  • ajain
    ajain Posts: 11

    nc -vz www.linux.com 80

    / # nslookup www.linux.com
    ;; connection timed out; no servers could be reached

    Looks like no DNS configuration is getting configured in busybox container.

  • ajain
    ajain Posts: 11

    Thanks for helping me:

    Here is the output:

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    default secondapp 2/2 Running 3 3h 192.168.173.1 ashish-ubuntu2
    kube-system calico-etcd-5hh72 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system calico-node-4dn2q 2/2 Running 1 20h 192.168.0.121 ashish-ubuntu3
    kube-system calico-node-8dtts 2/2 Running 1 20h 192.168.0.156 ashish-ubuntu2
    kube-system calico-node-j5rc2 2/2 Running 1 19h 192.168.0.183 ashish-ubuntu0
    kube-system calico-node-kgxk7 2/2 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 21h 192.168.231.1 ashish-ubuntu1
    kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 21h 192.168.231.2 ashish-ubuntu1
    kube-system etcd-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-8xh9m 1/1 Running 0 19h 192.168.0.183 ashish-ubuntu0
    kube-system kube-proxy-8xmnd 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-s6rng 1/1 Running 0 20h 192.168.0.156 ashish-ubuntu2
    kube-system kube-proxy-xlzbw 1/1 Running 0 20h 192.168.0.121 ashish-ubuntu3
    kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 21h 192.168.0.193 ashish-ubuntu1
    kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 20h 192.168.231.5 ashish-ubuntu1
    metallb-system controller-558b9b86cd-gn7nx 1/1 Running 0 19h 192.168.231.6 ashish-ubuntu1
    metallb-system speaker-t48z2 1/1 Running 0 19h 192.168.0.193 ashish-ubuntu1
    ashish@ashish-ubuntu0:~/app2$
    ashish@ashish-ubuntu0:~/app2$
    ashish@ashish-ubuntu0:~/app2$ kubectl get svc --all-namespaces
    NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    default kubernetes ClusterIP 10.96.0.1 443/TCP 21h
    default secondapp NodePort 10.106.147.238 80:32000/TCP 6h
    kube-system calico-etcd ClusterIP 10.96.232.136 6666/TCP 21h
    kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 21h
    kube-system kubernetes-dashboard NodePort 10.109.145.247 443:32116/TCP 20h
    ashish@ashish-ubuntu0:~/app2$

  • ajain
    ajain Posts: 11

    Here is busybox resolve file:

    ashish@ashish-ubuntu0:~/app2$ kubectl exec -it -c busy secondapp sh
    / # cat /etc/resolv.conf
    nameserver 10.96.0.10
    search default.svc.cluster.local svc.cluster.local cluster.local
    options ndots:5
    / #

  • ajain
    ajain Posts: 11

    It hangs on nc command. I see coredns have errors in it. I have disabled firewall on all nodes and removed metallb. Just FYI: After installing kubeadm, I couldn't run dashboard till it was deployed on master node. API server was unreachable from other nodes for dashboard pod. Same with metallb. But running busybox on master node doesn't help here.

    2018/09/08 12:37:54 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:36874->192.168.0.1:53: i/o timeout
    2018/09/08 12:39:27 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:50580->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:57479->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:09 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:58589->192.168.0.1:53: i/o timeout
    2018/09/08 16:18:14 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53425->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:40 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:53883->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:40 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:38421->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:42 [ERROR] 2 www.linux.com. A: unreachable backend: read udp 192.168.231.1:51105->192.168.0.1:53: i/o timeout
    2018/09/08 19:11:42 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:45913->192.168.0.1:53: i/o timeout
    2018/09/08 20:15:04 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:35092->192.168.0.1:53: i/o timeout
    2018/09/08 20:43:19 [ERROR] 2 www.linux.com. AAAA: unreachable backend: read udp 192.168.231.1:42572->192.168.0.1:53: i/o timeout

    ashish@ashish-ubuntu0:~/app2$ kubectl get pods -o wide --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
    default secondapp 2/2 Running 0 48m 192.168.231.7 ashish-ubuntu1
    kube-system calico-etcd-5hh72 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system calico-kube-controllers-74b888b647-9vzcc 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system calico-node-4dn2q 2/2 Running 1 21h 192.168.0.121 ashish-ubuntu3
    kube-system calico-node-8dtts 2/2 Running 1 21h 192.168.0.156 ashish-ubuntu2
    kube-system calico-node-j5rc2 2/2 Running 1 20h 192.168.0.183 ashish-ubuntu0
    kube-system calico-node-kgxk7 2/2 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system coredns-78fcdf6894-59lv8 1/1 Running 0 22h 192.168.231.1 ashish-ubuntu1
    kube-system coredns-78fcdf6894-rphc2 1/1 Running 0 22h 192.168.231.2 ashish-ubuntu1
    kube-system etcd-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-apiserver-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-controller-manager-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-8xh9m 1/1 Running 0 20h 192.168.0.183 ashish-ubuntu0
    kube-system kube-proxy-8xmnd 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kube-proxy-s6rng 1/1 Running 0 21h 192.168.0.156 ashish-ubuntu2
    kube-system kube-proxy-xlzbw 1/1 Running 0 21h 192.168.0.121 ashish-ubuntu3
    kube-system kube-scheduler-ashish-ubuntu1 1/1 Running 0 22h 192.168.0.193 ashish-ubuntu1
    kube-system kubernetes-dashboard-9b67c5f9f-lwhpw 1/1 Running 0 21h 192.168.231.5 ashish-ubuntu1
    ashish@ashish-ubuntu0:~/app2$

  • ajain
    ajain Posts: 11

    I have re-installed kubeadm after changing k8sMaster.sh, by adding --feature-gates=CoreDNS=false to kubeadm init. This will use kube-dns instead of coredns.

    Still DNS didn't work. Later found one resolution at below link. After this it started to work:

    https://github.com/coreos/flannel/issues/983

    Copied from above link:
    Found the problem which exists in DNS (ip adresses works ok. Helped me config map for dns:

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: kube-dns
    namespace: kube-system
    labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    data:
    upstreamNameservers: |-
    ["8.8.8.8", "8.8.4.4"]

  • ajain
    ajain Posts: 11

    No. I was running Calico. But the DNS was CoreDNS. I just tried Kube-Dns to see if that will make difference. I am not concerned how infrastructure is set but mainly to test/learn functionality. Even Metallb is running, so I am able to test Load Balancer service type. This is all setup in virtualbox.

    BTW. Thanks for guiding me towards resolution.

  • martinaje
    martinaje Posts: 1
    edited October 2018

    Yes its enabled by following the instructions on the documentation, now today a have another problem I turned off my router and started back up but now it seems my router doesn't send any internet connection. Regarding I already talk to the Belkin support Belkin Support expert team. But still, I have been facing the same if you have any assistance about this so please share with me.

Categories

Upcoming Training