Welcome to the Linux Foundation Forum!

Hostname resolution on nginx/ubuntu container (excerise 7.2, step#11)

Posts: 11
edited May 2019 in LFD259 Class Forum

Hi Guys,

While doing exercise 7.2, step#11, i am getting following error while updating apt repositories:

  1. root@thirdpage:/# apt-get update
  2. Err:1 http://deb.debian.org/debian stretch InRelease Temporary failure resolving 'deb.debian.org'
  3. Err:2 http://security.debian.org/debian-security stretch/updates InRelease Temporary failure resolving 'security.debian.org'

My container's resolv.conf have following content:

  1. root@thirdpage:/# cat /etc/resolv.conf
  2. nameserver 10.96.0.10
  3. search default.svc.cluster.local svc.cluster.local cluster.local localdomain
  4. options ndots:5

NOTE My Host OS /etc/network/interfaces file is configured as:

  1. baqai@lfdtru1804n01:/etc/init.d$ cat /etc/network/interfaces
  2. # This file describes the network interfaces available on your system
  3. # and how to activate them. For more information, see interfaces(5).
  4.  
  5. source /etc/network/interfaces.d/*
  6.  
  7. # The loopback network interface
  8. auto lo
  9. iface lo inet loopback
  10.  
  11. # The primary network interface
  12. auto ens32
  13. iface ens32 inet dhcp
  14.  

NOTE: I am doing all these exercises on local VMs created on VMWare Workstation.

Appreciate your help..

Thanks.
Furqan Baqai

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Answers

  • Posts: 1,000

    Hello Furqan,

    Definitely a DNS issue. We do not test using VMWare though.

    Having said that, you did test and found that 10.96.0.10 was the listed DNS server inside the container as I see in the /etc/resolv.conf file. This is most likely the IP address of the kubernetes service, found with kubectl get svc -o wide.

    Do apt-get commands work inside the master VM? Same for the worker node? What IP address do they use for DNS?

    You may want to use kubectl get pods -o wide and kubectl get svc to find that IP address. Also ensure that the coredns and calico-node pods are properly running.

    Regards,

  • Posts: 11

    Hi Serewicz,

    Thank you for your response.

    Responding to your questions inline:
    1. Does apt-get commands on the host work ?
    Yes apt-get is working and ping to a DNS is working as well
    2. Output of kubectl get pods -o wide:

    1. baqai@lfdtru1804n01:~$ kubectl get svc -o wide
    2. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
    3. kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21d <none>
    4. nginx ClusterIP 10.100.147.5 <none> 443/TCP 16d io.kompose.service=nginx
    5. registry ClusterIP 10.102.89.186 <none> 5000/TCP 16d io.kompose.service=registry
    6. secondapp LoadBalancer 10.102.242.149 <pending> 80:32000/TCP 34h example=second
    7. thirdpage NodePort 10.107.220.47 <none> 80:31325/TCP 24h example=third
    1. Output of kubectl get svc:
    1. baqai@lfdtru1804n01:~$ kubectl get svc
    2. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    3. kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21d
    4. nginx ClusterIP 10.100.147.5 <none> 443/TCP 16d
    5. registry ClusterIP 10.102.89.186 <none> 5000/TCP 16d
    6. secondapp LoadBalancer 10.102.242.149 <pending> 80:32000/TCP 34h
    7. thirdpage NodePort 10.107.220.47 <none> 80:31325/TCP 24h
    8.  
    1. Also ensure that the coredns and calico-node pods are properly running.
      I checked if the pods are running by issuing command kubectl get -n kube-system podsand found out that the pods are seeemd to be working. Not sure if it is something to do with their configuration. Please find the relevant output
    1. baqai@lfdtru1804n01:~$ kubectl get -n kube-system pods
    2. NAME READY STATUS RESTARTS AGE
    3. calico-node-k5tv2 2/2 Running 48 21d
    4. calico-node-zwm9z 2/2 Running 56 21d
    5. coredns-86c58d9df4-62h6j 1/1 Running 28 21d
    6. coredns-86c58d9df4-fzhk6 1/1 Running 28 21d
    7. etcd-lfdtru1804n01 1/1 Running 29 21d
    8. kube-apiserver-lfdtru1804n01 1/1 Running 31 21d
    9. kube-controller-manager-lfdtru1804n01 1/1 Running 29 21d
    10. kube-proxy-8cs5r 1/1 Running 25 21d
    11. kube-proxy-qfkcc 1/1 Running 28 21d
    12. kube-scheduler-lfdtru1804n01 1/1 Running 29 21d
    13. traefik-ingress-controller-qhv4s 1/1 Running 1 24h
    14. traefik-ingress-controller-vv8nh 1/1 Running 1 24h
    15.  

    Thanks in advance for your help...

    Regards,
    Furqan Baqai

  • Posts: 11

    Hi Guys,

    I think i found out the issue. As i am running all labs on VMWare workstation, kubernetes core DNS services are pointing to my vmware gateway for DNS resolution which is evidently failing. Please check below logs:

    1. baqai@lfdtru1804n01:~$ kubectl logs --namespace=kube-system coredns-86c58d9df4-62h6j
    2. .:53
    3. 2019-05-20T17:54:20.286Z [INFO] CoreDNS-1.2.6
    4. 2019-05-20T17:54:20.286Z [INFO] linux/amd64, go1.11.2, 756749c
    5. CoreDNS-1.2.6
    6. linux/amd64, go1.11.2, 756749c
    7. [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
    8. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:47806->192.168.159.2:53: i/o timeout
    9. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:49965->192.168.159.2:53: i/o timeout
    10. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:58770->192.168.159.2:53: i/o timeout
    11. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:37151->192.168.159.2:53: i/o timeout
    12. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:42153->192.168.159.2:53: i/o timeout
    13. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:46402->192.168.159.2:53: i/o timeout
    14. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:35809->192.168.159.2:53: i/o timeout
    15. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:40866->192.168.159.2:53: i/o timeout
    16. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:33442->192.168.159.2:53: i/o timeout
    17. [ERROR] plugin/errors: 2 325794039783830212.4647384507963668824. HINFO: unreachable backend: read udp 192.168.0.163:54784->192.168.159.2:53: i/o timeout
    18. [ERROR] plugin/errors: 2 www.linuxfoundation.org. AAAA: unreachable backend: read udp 192.168.0.163:47129->192.168.159.2:53: i/o timeout
    19. [ERROR] plugin/errors: 2 www.linuxfoundation.org. A: unreachable backend: read udp 192.168.0.163:47656->192.168.159.2:53: i/o timeout
    20. [ERROR] plugin/errors: 2 www.linuxfoundation.org. A: unreachable backend: read udp 192.168.0.163:56256->192.168.159.2:53: i/o timeout
    21. [ERROR] plugin/errors: 2 www.linuxfoundation.org. AAAA: unreachable backend: read udp 192.168.0.163:42880->192.168.159.2:53: i/o timeout
    22. [ERROR] plugin/errors: 2 www.linuxfoundation.org. A: unreachable backend: read udp 192.168.0.163:60223->192.168.159.2:53: i/o timeout
    23. [ERROR] plugin/errors: 2 www.linuxfoundation.org. AAAA: unreachable backend: read udp 192.168.0.163:57832->192.168.159.2:53: i/o timeout
    24. [ERROR] plugin/errors: 2 www.linuxfoundation.org. A: unreachable backend: read udp 192.168.0.163:56050->192.168.159.2:53: i/o timeout
    25. [ERROR] plugin/errors: 2 www.linuxfoundation.org. AAAA: unreachable backend: read udp 192.168.0.163:33869->192.168.159.2:53: i/o timeout
    26.  

    Appreciate if somebody help us out to resolve this.

  • Posts: 1,000

    Hello,

    Debugging DNS can definitely be a task. There are several configurations which can effect resolution, some like routing, may not actually be a DNS issue as much as an overall network issue.

    I'm glad you looked at the logs, I tried to post here suggesting that but the message was blocked for some reason. From the output it looks like the UDP requests are going from 192.168.0.163:47806 to 192.168.159.2:53 with a timeout. Either the forwarding in VMWare is not properly configured, it is a routing issue, or perhaps a firewall issue.

    I don't know much of anything about how VMWare sets up the network between VMs by default. In the systems we support and use like GCE and AWS this problem is typically a firewall issue. That either we have to open all ports, or as in the case with VirtualBox we need to activate every interface to use promiscuous mode so as to not drop some packets.

    Regards,

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training