Issue on lab 9.3: Working with CoreDNS

andrea.calvario

Here again for another issue on the 9.3 lab: my pod cannot resolve addresses.
After a lot of problems, as described in my past discussion, I recreate the entire cluster, so I'm now able to go ahead, but, reached this lab (9.3) I found another problem. The feeling is that the coredns doesn't work correctly, I restart them, but the problem persists.
I provide here some information and configuration, hoping that someone can help me to identify the problem.

Status of the coredns pods

**kubectl get pods --all-namespaces | grep coredns**
kube-system   coredns-6b4b86d8bf-2r222                   1/1     Running   0          12m
kube-system   coredns-6b4b86d8bf-nqj2m                   1/1     Running   0          12m

my /etc/resolve.conf file on the host

nameserver 127.0.0.53
options edns0 trust-ad
search homenet.telecomitalia.it

/etc/resolve.conf file on nettool pod

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local homenet.telecomitalia.it
options ndots:5

Output of the apt update command executed in the pod

[email protected]:/# apt update
Err:1 http://security.ubuntu.com/ubuntu focal-security InRelease         
  Temporary failure resolving 'security.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease                   
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done        
Building dependency tree       
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

Log from the coredns pods

**kubectl logs coredns-6b4b86d8bf-2r222 --namespace kube-system **
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:50073->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:53444->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:33483->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:60055->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:45734->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:58277->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:48417->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:45403->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:43683->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 2411310349111695803.1156763954485922933. HINFO: read udp 192.168.140.134:52890->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:40467->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:55114->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:54780->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:32826->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:34987->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:57937->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:54716->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:37758->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:48427->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:39343->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:55263->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:51377->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:35208->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:45064->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com. A: read udp 192.168.140.134:36542->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com. A: read udp 192.168.140.134:59127->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:52336->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:55511->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:58522->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:48809->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:59664->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:44874->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com. A: read udp 192.168.140.134:50931->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:34514->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:54530->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:45994->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:60092->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:39119->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com.homenet.telecomitalia.it. A: read udp 192.168.140.134:59702->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.140.134:45259->192.168.1.1:53: i/o timeout

Can someone help me to understand where I mistaking?

Thanks in advance!

Andy74

chrispokorni

Hi @andrea.calvario,

The issue you are experiencing at this point seems to be related to how your Internet Service Provider operates. It is trying to locate *.ubuntu.com on homenet.telecomitalia.it domain.

With a simple google search I was able to find a forum post that should provide you with a solution around the DNS issues with telecomitalia. I hope it helps to fix your host's resolv.conf file:

https://forum.ubuntu-it.org/viewtopic.php?t=428583

Regards,
-Chris

paristiz

Hi,

I am experiencing the same issue but the logs don't show the same error related with ISP (until I can understand). Is there something I can do?

These are the logs from both CoreDNS pods:

[email protected]:~$ kubectl -n kube-system logs coredns-f9fd979d6-2nd2n
.:53
[INFO] plugin/reload: Running configuration MD5 = 823e90b886243896bc5fc311d1fe380f
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:36158->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:37468->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:33847->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:58102->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:56101->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:35935->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:49304->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:37233->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:58338->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 5854889205053853889.4992786380578661315. HINFO: read udp 192.168.54.36:56906->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com. A: read udp 192.168.54.36:44621->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:39647->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:43970->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:49128->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:43476->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:54778->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:35449->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:32906->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.36:39382->192.168.18.1:53: i/o timeout
[email protected]:~$ kubectl -n kube-system logs coredns-f9fd979d6-sbmlw
.:53
[INFO] plugin/reload: Running configuration MD5 = 823e90b886243896bc5fc311d1fe380f
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[ERROR] plugin/errors: 2 7925864582003874829.5346898514191063070. HINFO: read udp 192.168.54.39:49955->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 7925864582003874829.5346898514191063070. HINFO: read udp 192.168.54.39:47710->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 7925864582003874829.5346898514191063070. HINFO: read udp 192.168.54.39:36716->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 7925864582003874829.5346898514191063070. HINFO: read udp 192.168.54.39:60608->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 7925864582003874829.5346898514191063070. HINFO: read udp 192.168.54.39:39340->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 7925864582003874829.5346898514191063070. HINFO: read udp 192.168.54.39:57446->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 7925864582003874829.5346898514191063070. HINFO: read udp 192.168.54.39:42763->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com. A: read udp 192.168.54.39:53249->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.39:42660->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.39:32993->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com. A: read udp 192.168.54.39:54913->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 security.ubuntu.com. A: read udp 192.168.54.39:35481->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.39:59322->192.168.18.1:53: i/o timeout
[ERROR] plugin/errors: 2 archive.ubuntu.com. A: read udp 192.168.54.39:48382->192.168.18.1:53: i/o timeout

As you can see I am working with virtual machines, so far, I haven´t had DNS issues.

Any information will be appreciated

Pablo A.

serewicz

Hello,

Did you read through the link provided, about checking your VM DNS settings? Are your VMs able to resolve DNS names? Are all of the pods running when you run kubectl get pods --all-namespaces ?

Regards,

RonaldHeirbaut

I had the same issue running on debian 10. I installed the resolvconf package and changed the DNS on the masternode from my local DNS (192.168.1.1) to external DNS (1.1.1.1). And of course a static IP configured on the node. That solved it for me.

paristiz

Hi, thank you for your answers. I had already checked the pods out and they were always running. Finally, I followed the RonaldHeirbaut recommendation and downloaded the resolvconf package on the masternode and after that, my pod was able to download the dig and other tools needed for the practice.