Welcome to the Linux Foundation Forum!

Lab 3.3 Cluster IP access

kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx ClusterIP 10.103.189.38 80/TCP 8m
curl 10.103.189.38:80

Using the cluster ip as above, unable to access the curl from master node, but I can access from worker node.
Is there anything I am missing here ? Thanks.

Comments

  • serewicz
    serewicz Posts: 1,000

    Hello,

    My first thought is there may be some firewall blocking access. Did you have a rule to allow all traffic?

    What are you using to run the labs?

    Does the command time out or other error?

    Please make sure all the pods are running with kubectl get pod --all-namespaces Are there any pods or containers not running?

    Regards,

  • Hi,
    Are you still working on GCP? Do you have a firewall rule to allow all traffic (all ports, all sources, all protocols)?
    If on AWS, you would need a similar firewal rule.
    This may be required to be able to complete this lab and subsequent labs as well.
    Regards,
    -Chris

  • Thank you.
    Added below and it starts working. Thanks.

    Name Type Targets Filters Protocols / ports Action Priority Network
    allowall Ingress Apply to all IP ranges: 0.0.0.0/0 all Allow 1000 default

  • Yes I have to say I spent ages trying to troubleshoot on lab 8 until I read this post :rage:
    After adding a firewall rule to allow all ingress traffic, and adding to the master and worker nodes, I'm back on track.

  • Hi @EdwardQuick ,
    I am glad you found the solution helpful.
    Did you encounter any similar issues in labs prior to Lab 8?
    Thanks,
    -Chris

  • I can't recall to be honest Chris. This is on GCP by the way. Perhaps I've got a poor memory but was there a LAB on how to set up the hosts on GCP and did it mention adding firewall rules?

  • Hi @EdwardQuick ,
    For some reason, I remember some instructions to open all ports for traffic, but I may be thinking of another course. On the K8s.io documentation site there are some notes about port ranges used by K8s, but last time I checked that list was incomplete (creating fw rules for those specific port ranges still produced errors). So for the purpose of these labs, I just opened all ports to all protocols from all sources.
    -Chris

  • serewicz
    serewicz Posts: 1,000

    I will update the introduction chapter to make it clear that ports should be opened to avoid firewall issues.

    Regards,

  • is it updated in the introduction chapter? I didn't find it and spent hours with the local IP tables rule until I found this post.. :s

  • fcioanca
    fcioanca Posts: 1,880

    It will be included in the next course version, but it has not yet been done.

  • I also faced this problem. I really didn't make it work, even I tried the firewall rule mentioned in this post. At last, I followed the kuernetes-the-hard-way to set up a working cluster. Could you provide more specific commands?

    The following is the glcoud commands I used to create network and instances on GCP. What did I miss?

    $ gcloud compute networks create cka --subnet-mode custom
    $ gcloud compute networks subnets create kubernetes --network cka --range 10.240.0.0/24
    $ gcloud compute firewall-rules create cka-external --allow tcp:22,tcp:6443,icmp --network cka --source-ranges 0.0.0.0/0
    $ gcloud compute firewall-rules create cka-internal --network cka --allow tcp,udp,icmp --source-ranges 192.168.0.0/16,10.240.0.0/16
    $ gcloud compute instances create controller-1 --async --boot-disk-size 200GB --can-ip-forward --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud --machine-type n1-standard-1 --private-network-ip 10.240.0.11 --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring --subnet kubernetes --tags cka,controller
    $ gcloud compute instances create worker-1 --async --boot-disk-size 200GB --can-ip-forward --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud --machine-type n1-standard-1 --private-network-ip 10.240.0.21 --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring --subnet kubernetes --tags cka,worker
    
  • Hi,
    Kubernetes uses many port ranges for agent-to-agent communication and for external communication. As I mentioned above, create a FW rule to allow all traffic: all protocols, all ports, all sources. The GCP documentation on firewall rules shows a sample that you can use to create this. If you are using the console, just navigate to VPC network -> Firewall rules and from there you can create the same rule.
    Regards,
    -Chris

  • Thanks, Chris. It worked.
    Kubernetes-the-hard-way works fine with such firewall rule setting, but our lesson doesn't.
    The only difference here I can inspect is the network plugin. Kubernetes-the-hard-way uses GCP route and our lesson uses Calico.
    I doubt we can make it work with much less permissive firewall rule. Maybe, just add BGP protocol permission is enough.

  • Hello!
    I Have the same issue. I am using 2 virtual machines in Azure, and curl is not responding:
    Some useful info:

    $ kubectl get svc nginx-7db75b8b78-8gtzs
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    nginx-7db75b8b78-8gtzs ClusterIP 10.111.205.50 80/TCP 5h45m
    $ curl 10.111.205.50:80

    At this point nothing happens until timeout. I Already opened all ports in Azure portal (inbound / outbound rules).
    Master and node hosts have Internet access, but seems that pods doesn't. Or at least, can't resolve DNS, because I can pin google's IP, but can't to google.com:

    **/ # ping 172.217.168.206
    PING 172.217.168.206 (172.217.168.206): 56 data bytes
    64 bytes from 172.217.168.206: seq=0 ttl=48 time=3.478 ms
    64 bytes from 172.217.168.206: seq=1 ttl=48 time=3.292 ms
    ^C
    --- 172.217.168.206 ping statistics ---
    2 packets transmitted, 2 packets received, 0% packet loss
    round-trip min/avg/max = 3.292/3.385/3.478 ms
    / #
    / # ping google.com
    ^C
    **

    Also, all pods are running:

    $ kubectl get pod --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE
    default bosybox-56bbfc6d5c-627wp 1/1 Running 1 6h5m
    default nginx-7db75b8b78-8gtzs 1/1 Running 0 24h
    kube-system calico-node-2k4qt 2/2 Running 2 3d23h
    kube-system calico-node-bkbc4 2/2 Running 2 4d
    kube-system coredns-869f847d58-ldtm9 1/1 Running 1 4d2h
    kube-system coredns-869f847d58-nfdjq 1/1 Running 1 4d2h
    kube-system etcd-demian-test 1/1 Running 1 3d23h
    kube-system kube-apiserver-demian-test 1/1 Running 1 3d23h
    kube-system kube-controller-manager-demian-test 1/1 Running 1 3d23h
    kube-system kube-proxy-6ddgr 1/1 Running 1 4d2h
    kube-system kube-proxy-gp2t4 1/1 Running 1 3d23h
    kube-system kube-scheduler-demian-test 1/1 Running 1 3d23h

    any ideas? Thanks

  • Hi @demian711 ,
    You can try to restart your DNS pods, and see if that resolves your DNS issue.
    If not, then here is a tutorial on how to setup DNS for Kubernetes on Azure:
    https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/azure.md
    Regards,
    -Chris

Categories

Upcoming Training