nameserver 10.96.0.10 not responding

teggy

Regarding exercice 9.3, I was getting issues to run the testing pod created by nettool.yaml. If I do as instructed, I can't get 'apt update' to work.

After some googling, I have changed the nameserver and added the following to nettool.yaml:
dnsPolicy: "None"
dnsConfig:
nameservers:

8.8.8.8
This tweak has helped me move forward to get curl and dnsutils installed and hence get ping working. But when I try to ping 10.96.0.10 (which was the default nameserver had I not changed the nameserver) from the pod, the ping fails. ping 10.96.0.10 also fails when I run it directly from the master node host.

I understand for, at least for the rest of the exercice, I need to use 10.96.0.10 as dns. So how can I fix this issue ? Thanks for any light you can provide

Kind regards,
Teggy

teggy

----8<----------------
Couple of investigations done so far
----8<----------------

coredns pods seems to be ok:

14:46 teggy_info@lab-3-1-1:~ $ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
accounting nginx-one-5b5d85886b-2dpw2 1/1 Running 0 3h55m
accounting nginx-one-5b5d85886b-nrngj 1/1 Running 0 3h55m
default nett 1/1 Running 0 7m35s
default ubuntu 1/1 Running 0 36m
kube-system calico-kube-controllers-77c4b7448-pgz7p 1/1 Running 10 41d
kube-system calico-node-lhd49 1/1 Running 17 241d
kube-system calico-node-zrtqf 1/1 Running 15 239d
kube-system coredns-f9fd979d6-xxfcg 1/1 Running 10 41d
kube-system coredns-f9fd979d6-zsxgp 1/1 Running 10 41d

14:49 teggy_info@lab-3-1-1:~ $ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 241d

14:49 teggy_info@lab-3-1-1:~ $ kubectl describe service kubernetes
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations:
Selector:
Type: ClusterIP
IP: 10.96.0.1
Port: https 443/TCP
TargetPort: 6443/TCP
Endpoints: 10.132.0.5:6443
Session Affinity: None
Events:

14:50 teggy_info@lab-3-1-1:~ $ sudo netstat -nlp |grep 6443
tcp6 0 0 :::6443 :::* LISTEN 3421/kube-apiserver

14:50 teggy_info@lab-3-1-1:~ $ ping 10.132.0.5
PING 10.132.0.5 (10.132.0.5) 56(84) bytes of data.
64 bytes from 10.132.0.5: icmp_seq=1 ttl=64 time=0.066 ms
^C
— 10.132.0.5 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.064/0.065/0.066/0.001 ms

14:51 teggy_info@lab-3-1-1:~ $ ping lab-3-1-1
PING lab-3-1-1.europe-west1-b.c.lfs258-lfs258.internal (10.132.0.5) 56(84) bytes of data.
64 bytes from k8smaster (10.132.0.5): icmp_seq=1 ttl=64 time=0.072 ms
^C
— lab-3-1-1.europe-west1-b.c.lfs258-lfs258.internal ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2044ms
rtt min/avg/max/mdev = 0.059/0.063/0.072/0.010 ms

chrispokorni

Hi @teggy,

Not all services respond to ping. The fact that ping returns no response from 10.96.0.10 does not mean that your container cannot talk to it when needed. In the lab exercise there are no references to ping. A different command is used instead dig - are you able to run dig and see expected the outputs?

Regards,
-Chris

timothyaw

I can verify that teggy is correct, I had to add Google DNS (8.8.8.8) to the yaml file. Once installed I can ping and I can telnet (telnet 10.96.0.10 53). I get responses from both. If I do a dig @10.96.0.10 redhat.com, in the answer section I get "WARNING: recursion requested but not available" Any ideas? FYI - I'm running the cluster on personal vm's. Cloud isn't involved.

chrispokorni

Hi @timothyaw,

A hypervisor will play some role in the networking configuration of your hosts, which will impact the overall cluster networking behavior. On GCE, where the labs have been tested, no additional config options were needed for the DNS exercise.

What local hypervisor are you using and how is the host networking configured by the hypervisor?

Regards,
-Chris

timothyaw

I'm using KVM. I'm using the default network that is setup when you install KVM. I have ip forwarding turned on in the kernel.

teggy

Hi @chrispokorni ,

Thanks for seeing into that issue. You're right about ping; I should have thought about that....

I've tested once again but this time with dig and it's not working any better

From the container which I've forced to use 8.8.8.8, I'm running the following:

root@ubuntu:/etc# cat /etc/resolv.conf
nameserver 8.8.8.8
root@ubuntu:/etc# dig @10.96.0.10

; <<>> DiG 9.16.1-Ubuntu <<>> @10.96.0.10
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Another test which I've run is that I've updated resolv.conf and set the same info as other pods:

root@ubuntu:/etc# cat /etc/resolv.conf
#nameserver 8.8.8.8
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local europe-west1-b.c.lfs258-lfs258.internal c.lfs258-lfs258.internal google.internal
options ndots:5
root@ubuntu:/etc# dig

; <<>> DiG 9.16.1-Ubuntu <<>>
;; global options: +cmd
;; connection timed out; no servers could be reached

root@ubuntu:/etc# dig @10.96.0.10

; <<>> DiG 9.16.1-Ubuntu <<>> @10.96.0.10
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Any idea what could be wrong ? (in the meantime, I'll go once again through the pre-req and see if I've left something out. I don't think so since I have able to run the exercices so far...)

For info, I'm running these nodes on GCE...

chrispokorni

Hi @teggy,

This is a strange behavior. I tried replicating it on GCE, but everything worked as described in the lab exercise. In the past I did experience different, yet strange and unexpected behaviors from GCE networking, which I resolved with a custom VPC and firewall rule specifically set for the VMs of this class.

How is your VPC setup, and what firewall rule(s) do you have associated with it?

Regards,
-Chris

chrispokorni

Hi @timothyaw,

Each KVM networking type has its possible limitations, so it helps knowing how yours is setup.
Is there any benefit from the enabled IP forwarding? I can't remember ever needing to enable it and all worked as expected.

Regards,
-Chris

teggy

Hello @chrispokorni

Thanks for your time looking into that issue. I have been able to move forward, the issue was about the pre-requisites and specially the VPC configuration. I've got these set up correctly and things are working fine now.

Kind regards,
Teggy

rzadzins

Hi @teggy
Would you mind sharing the details of what you had to change in the VPC please?