Welcome to the Linux Foundation Forum!

nameserver 10.96.0.10 not responding

teggy
teggy Posts: 6
in LFS258 Class Forum

Regarding exercice 9.3, I was getting issues to run the testing pod created by nettool.yaml. If I do as instructed, I can't get 'apt update' to work.

After some googling, I have changed the nameserver and added the following to nettool.yaml:
dnsPolicy: "None"
dnsConfig:
nameservers:

8.8.8.8
This tweak has helped me move forward to get curl and dnsutils installed and hence get ping working. But when I try to ping 10.96.0.10 (which was the default nameserver had I not changed the nameserver) from the pod, the ping fails. ping 10.96.0.10 also fails when I run it directly from the master node host.

I understand for, at least for the rest of the exercice, I need to use 10.96.0.10 as dns. So how can I fix this issue ? Thanks for any light you can provide

Kind regards,
Teggy

Comments

  • teggy
    teggy Posts: 6

    ----8<----------------
    Couple of investigations done so far
    ----8<----------------

    coredns pods seems to be ok:

    14:46 teggy_info@lab-3-1-1:~ $ kubectl get pods --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE
    accounting nginx-one-5b5d85886b-2dpw2 1/1 Running 0 3h55m
    accounting nginx-one-5b5d85886b-nrngj 1/1 Running 0 3h55m
    default nett 1/1 Running 0 7m35s
    default ubuntu 1/1 Running 0 36m
    kube-system calico-kube-controllers-77c4b7448-pgz7p 1/1 Running 10 41d
    kube-system calico-node-lhd49 1/1 Running 17 241d
    kube-system calico-node-zrtqf 1/1 Running 15 239d
    kube-system coredns-f9fd979d6-xxfcg 1/1 Running 10 41d
    kube-system coredns-f9fd979d6-zsxgp 1/1 Running 10 41d

    14:49 teggy_info@lab-3-1-1:~ $ kubectl get svc
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    kubernetes ClusterIP 10.96.0.1 443/TCP 241d

    14:49 teggy_info@lab-3-1-1:~ $ kubectl describe service kubernetes
    Name: kubernetes
    Namespace: default
    Labels: component=apiserver
    provider=kubernetes
    Annotations:
    Selector:
    Type: ClusterIP
    IP: 10.96.0.1
    Port: https 443/TCP
    TargetPort: 6443/TCP
    Endpoints: 10.132.0.5:6443
    Session Affinity: None
    Events:

    14:50 teggy_info@lab-3-1-1:~ $ sudo netstat -nlp |grep 6443
    tcp6 0 0 :::6443 :::* LISTEN 3421/kube-apiserver

    14:50 teggy_info@lab-3-1-1:~ $ ping 10.132.0.5
    PING 10.132.0.5 (10.132.0.5) 56(84) bytes of data.
    64 bytes from 10.132.0.5: icmp_seq=1 ttl=64 time=0.066 ms
    ^C
    — 10.132.0.5 ping statistics —
    2 packets transmitted, 2 received, 0% packet loss, time 1007ms
    rtt min/avg/max/mdev = 0.064/0.065/0.066/0.001 ms

    14:51 teggy_info@lab-3-1-1:~ $ ping lab-3-1-1
    PING lab-3-1-1.europe-west1-b.c.lfs258-lfs258.internal (10.132.0.5) 56(84) bytes of data.
    64 bytes from k8smaster (10.132.0.5): icmp_seq=1 ttl=64 time=0.072 ms
    ^C
    — lab-3-1-1.europe-west1-b.c.lfs258-lfs258.internal ping statistics —
    3 packets transmitted, 3 received, 0% packet loss, time 2044ms
    rtt min/avg/max/mdev = 0.059/0.063/0.072/0.010 ms

  • chrispokorni
    chrispokorni Posts: 2,605

    Hi @teggy,

    Not all services respond to ping. The fact that ping returns no response from 10.96.0.10 does not mean that your container cannot talk to it when needed. In the lab exercise there are no references to ping. A different command is used instead dig - are you able to run dig and see expected the outputs?

    Regards,
    -Chris

  • timothyaw
    timothyaw Posts: 15
    edited November 2020

    I can verify that teggy is correct, I had to add Google DNS (8.8.8.8) to the yaml file. Once installed I can ping and I can telnet (telnet 10.96.0.10 53). I get responses from both. If I do a dig @10.96.0.10 redhat.com, in the answer section I get "WARNING: recursion requested but not available" Any ideas? FYI - I'm running the cluster on personal vm's. Cloud isn't involved.

  • chrispokorni
    chrispokorni Posts: 2,605

    Hi @timothyaw,

    A hypervisor will play some role in the networking configuration of your hosts, which will impact the overall cluster networking behavior. On GCE, where the labs have been tested, no additional config options were needed for the DNS exercise.

    What local hypervisor are you using and how is the host networking configured by the hypervisor?

    Regards,
    -Chris

  • timothyaw
    timothyaw Posts: 15
    edited November 2020

    I'm using KVM. I'm using the default network that is setup when you install KVM. I have ip forwarding turned on in the kernel.

  • teggy
    teggy Posts: 6
    edited November 2020

    Hi @chrispokorni ,

    Thanks for seeing into that issue. You're right about ping; I should have thought about that....

    I've tested once again but this time with dig and it's not working any better :(

    From the container which I've forced to use 8.8.8.8, I'm running the following:

    root@ubuntu:/etc# cat /etc/resolv.conf
nameserver 8.8.8.8
root@ubuntu:/etc# dig @10.96.0.10

; <<>> DiG 9.16.1-Ubuntu <<>> @10.96.0.10
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

    Another test which I've run is that I've updated resolv.conf and set the same info as other pods:

    root@ubuntu:/etc# cat /etc/resolv.conf
#nameserver 8.8.8.8
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local europe-west1-b.c.lfs258-lfs258.internal c.lfs258-lfs258.internal google.internal
options ndots:5
root@ubuntu:/etc# dig

; <<>> DiG 9.16.1-Ubuntu <<>>
;; global options: +cmd
;; connection timed out; no servers could be reached

root@ubuntu:/etc# dig @10.96.0.10

; <<>> DiG 9.16.1-Ubuntu <<>> @10.96.0.10
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

    Any idea what could be wrong ? (in the meantime, I'll go once again through the pre-req and see if I've left something out. I don't think so since I have able to run the exercices so far...)

    For info, I'm running these nodes on GCE...

  • chrispokorni
    chrispokorni Posts: 2,605

    Hi @teggy,

    This is a strange behavior. I tried replicating it on GCE, but everything worked as described in the lab exercise. In the past I did experience different, yet strange and unexpected behaviors from GCE networking, which I resolved with a custom VPC and firewall rule specifically set for the VMs of this class.

    How is your VPC setup, and what firewall rule(s) do you have associated with it?

    Regards,
    -Chris

  • chrispokorni
    chrispokorni Posts: 2,605

    Hi @timothyaw,

    Each KVM networking type has its possible limitations, so it helps knowing how yours is setup.
    Is there any benefit from the enabled IP forwarding? I can't remember ever needing to enable it and all worked as expected.

    Regards,
    -Chris

  • teggy
    teggy Posts: 6

    Hello @chrispokorni

    Thanks for your time looking into that issue. I have been able to move forward, the issue was about the pre-requisites and specially the VPC configuration. I've got these set up correctly and things are working fine now.

    Kind regards,
    Teggy

  • rzadzins
    rzadzins Posts: 13

    Hi @teggy
    Would you mind sharing the details of what you had to change in the VPC please?

Categories

Upcoming Training