nameserver 10.96.0.10 not responding

Regarding exercice 9.3, I was getting issues to run the testing pod created by nettool.yaml. If I do as instructed, I can't get 'apt update' to work.
After some googling, I have changed the nameserver and added the following to nettool.yaml:
dnsPolicy: "None"
dnsConfig:
nameservers:
8.8.8.8
This tweak has helped me move forward to get curl and dnsutils installed and hence get ping working. But when I try to ping 10.96.0.10 (which was the default nameserver had I not changed the nameserver) from the pod, the ping fails. ping 10.96.0.10 also fails when I run it directly from the master node host.
I understand for, at least for the rest of the exercice, I need to use 10.96.0.10 as dns. So how can I fix this issue ? Thanks for any light you can provide
Kind regards,
Teggy
Comments
----8<----------------
Couple of investigations done so far
----8<----------------
coredns pods seems to be ok:
14:46 [email protected]:~ $ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
accounting nginx-one-5b5d85886b-2dpw2 1/1 Running 0 3h55m
accounting nginx-one-5b5d85886b-nrngj 1/1 Running 0 3h55m
default nett 1/1 Running 0 7m35s
default ubuntu 1/1 Running 0 36m
kube-system calico-kube-controllers-77c4b7448-pgz7p 1/1 Running 10 41d
kube-system calico-node-lhd49 1/1 Running 17 241d
kube-system calico-node-zrtqf 1/1 Running 15 239d
kube-system coredns-f9fd979d6-xxfcg 1/1 Running 10 41d
kube-system coredns-f9fd979d6-zsxgp 1/1 Running 10 41d
14:49 [email protected]:~ $ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 241d
14:49 [email protected]:~ $ kubectl describe service kubernetes
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations:
Selector:
Type: ClusterIP
IP: 10.96.0.1
Port: https 443/TCP
TargetPort: 6443/TCP
Endpoints: 10.132.0.5:6443
Session Affinity: None
Events:
14:50 [email protected]:~ $ sudo netstat -nlp |grep 6443
tcp6 0 0 :::6443 :::* LISTEN 3421/kube-apiserver
14:50 [email protected]:~ $ ping 10.132.0.5
PING 10.132.0.5 (10.132.0.5) 56(84) bytes of data.
64 bytes from 10.132.0.5: icmp_seq=1 ttl=64 time=0.066 ms
^C
— 10.132.0.5 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.064/0.065/0.066/0.001 ms
14:51 [email protected]:~ $ ping lab-3-1-1
PING lab-3-1-1.europe-west1-b.c.lfs258-lfs258.internal (10.132.0.5) 56(84) bytes of data.
64 bytes from k8smaster (10.132.0.5): icmp_seq=1 ttl=64 time=0.072 ms
^C
— lab-3-1-1.europe-west1-b.c.lfs258-lfs258.internal ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2044ms
rtt min/avg/max/mdev = 0.059/0.063/0.072/0.010 ms
Hi @teggy,
Not all services respond to
ping
. The fact thatping
returns no response from10.96.0.10
does not mean that your container cannot talk to it when needed. In the lab exercise there are no references toping
. A different command is used insteaddig
- are you able to rundig
and see expected the outputs?Regards,
-Chris
I can verify that teggy is correct, I had to add Google DNS (8.8.8.8) to the yaml file. Once installed I can ping and I can telnet (telnet 10.96.0.10 53). I get responses from both. If I do a dig @10.96.0.10 redhat.com, in the answer section I get "WARNING: recursion requested but not available" Any ideas? FYI - I'm running the cluster on personal vm's. Cloud isn't involved.
Hi @timothyaw,
A hypervisor will play some role in the networking configuration of your hosts, which will impact the overall cluster networking behavior. On GCE, where the labs have been tested, no additional config options were needed for the DNS exercise.
What local hypervisor are you using and how is the host networking configured by the hypervisor?
Regards,
-Chris
I'm using KVM. I'm using the default network that is setup when you install KVM. I have ip forwarding turned on in the kernel.
Hi @chrispokorni ,
Thanks for seeing into that issue. You're right about ping; I should have thought about that....
I've tested once again but this time with dig and it's not working any better
From the container which I've forced to use 8.8.8.8, I'm running the following:
Another test which I've run is that I've updated resolv.conf and set the same info as other pods:
Any idea what could be wrong ? (in the meantime, I'll go once again through the pre-req and see if I've left something out. I don't think so since I have able to run the exercices so far...)
For info, I'm running these nodes on GCE...
Hi @teggy,
This is a strange behavior. I tried replicating it on GCE, but everything worked as described in the lab exercise. In the past I did experience different, yet strange and unexpected behaviors from GCE networking, which I resolved with a custom VPC and firewall rule specifically set for the VMs of this class.
How is your VPC setup, and what firewall rule(s) do you have associated with it?
Regards,
-Chris
Hi @timothyaw,
Each KVM networking type has its possible limitations, so it helps knowing how yours is setup.
Is there any benefit from the enabled IP forwarding? I can't remember ever needing to enable it and all worked as expected.
Regards,
-Chris
Hello @chrispokorni
Thanks for your time looking into that issue. I have been able to move forward, the issue was about the pre-requisites and specially the VPC configuration. I've got these set up correctly and things are working fine now.
Kind regards,
Teggy
Hi @teggy
Would you mind sharing the details of what you had to change in the VPC please?