Lab 3.3 Cluster IP access

RaveendraGuntupalli

kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx ClusterIP 10.103.189.38 80/TCP 8m
curl 10.103.189.38:80

Using the cluster ip as above, unable to access the curl from master node, but I can access from worker node.
Is there anything I am missing here ? Thanks.

serewicz

Hello,

My first thought is there may be some firewall blocking access. Did you have a rule to allow all traffic?

What are you using to run the labs?

Does the command time out or other error?

Please make sure all the pods are running with kubectl get pod --all-namespaces Are there any pods or containers not running?

Regards,

chrispokorni

Hi,
Are you still working on GCP? Do you have a firewall rule to allow all traffic (all ports, all sources, all protocols)?
If on AWS, you would need a similar firewal rule.
This may be required to be able to complete this lab and subsequent labs as well.
Regards,
-Chris

RaveendraGuntupalli

Thank you.
Added below and it starts working. Thanks.

Name Type Targets Filters Protocols / ports Action Priority Network
allowall Ingress Apply to all IP ranges: 0.0.0.0/0 all Allow 1000 default

EdwardQuick

Yes I have to say I spent ages trying to troubleshoot on lab 8 until I read this post
After adding a firewall rule to allow all ingress traffic, and adding to the master and worker nodes, I'm back on track.

chrispokorni

Hi @EdwardQuick ,
I am glad you found the solution helpful.
Did you encounter any similar issues in labs prior to Lab 8?
Thanks,
-Chris

EdwardQuick

I can't recall to be honest Chris. This is on GCP by the way. Perhaps I've got a poor memory but was there a LAB on how to set up the hosts on GCP and did it mention adding firewall rules?

chrispokorni

Hi @EdwardQuick ,
For some reason, I remember some instructions to open all ports for traffic, but I may be thinking of another course. On the K8s.io documentation site there are some notes about port ranges used by K8s, but last time I checked that list was incomplete (creating fw rules for those specific port ranges still produced errors). So for the purpose of these labs, I just opened all ports to all protocols from all sources.
-Chris

serewicz

I will update the introduction chapter to make it clear that ports should be opened to avoid firewall issues.

Regards,

gotojeff

is it updated in the introduction chapter? I didn't find it and spent hours with the local IP tables rule until I found this post..

fcioanca

It will be included in the next course version, but it has not yet been done.

blackpiglet

I also faced this problem. I really didn't make it work, even I tried the firewall rule mentioned in this post. At last, I followed the kuernetes-the-hard-way to set up a working cluster. Could you provide more specific commands?

The following is the glcoud commands I used to create network and instances on GCP. What did I miss?

$ gcloud compute networks create cka --subnet-mode custom
$ gcloud compute networks subnets create kubernetes --network cka --range 10.240.0.0/24
$ gcloud compute firewall-rules create cka-external --allow tcp:22,tcp:6443,icmp --network cka --source-ranges 0.0.0.0/0
$ gcloud compute firewall-rules create cka-internal --network cka --allow tcp,udp,icmp --source-ranges 192.168.0.0/16,10.240.0.0/16
$ gcloud compute instances create controller-1 --async --boot-disk-size 200GB --can-ip-forward --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud --machine-type n1-standard-1 --private-network-ip 10.240.0.11 --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring --subnet kubernetes --tags cka,controller
$ gcloud compute instances create worker-1 --async --boot-disk-size 200GB --can-ip-forward --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud --machine-type n1-standard-1 --private-network-ip 10.240.0.21 --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring --subnet kubernetes --tags cka,worker

chrispokorni

Hi,
Kubernetes uses many port ranges for agent-to-agent communication and for external communication. As I mentioned above, create a FW rule to allow all traffic: all protocols, all ports, all sources. The GCP documentation on firewall rules shows a sample that you can use to create this. If you are using the console, just navigate to VPC network -> Firewall rules and from there you can create the same rule.
Regards,
-Chris

blackpiglet

Thanks, Chris. It worked.
Kubernetes-the-hard-way works fine with such firewall rule setting, but our lesson doesn't.
The only difference here I can inspect is the network plugin. Kubernetes-the-hard-way uses GCP route and our lesson uses Calico.
I doubt we can make it work with much less permissive firewall rule. Maybe, just add BGP protocol permission is enough.

demian711

Hello!
I Have the same issue. I am using 2 virtual machines in Azure, and curl is not responding:
Some useful info:

$ kubectl get svc nginx-7db75b8b78-8gtzs
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-7db75b8b78-8gtzs ClusterIP 10.111.205.50 80/TCP 5h45m
$ curl 10.111.205.50:80

At this point nothing happens until timeout. I Already opened all ports in Azure portal (inbound / outbound rules).
Master and node hosts have Internet access, but seems that pods doesn't. Or at least, can't resolve DNS, because I can pin google's IP, but can't to google.com:

**/ # ping 172.217.168.206
PING 172.217.168.206 (172.217.168.206): 56 data bytes
64 bytes from 172.217.168.206: seq=0 ttl=48 time=3.478 ms
64 bytes from 172.217.168.206: seq=1 ttl=48 time=3.292 ms
^C
--- 172.217.168.206 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 3.292/3.385/3.478 ms
/ #
/ # ping google.com
^C
**

Also, all pods are running:

$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default bosybox-56bbfc6d5c-627wp 1/1 Running 1 6h5m
default nginx-7db75b8b78-8gtzs 1/1 Running 0 24h
kube-system calico-node-2k4qt 2/2 Running 2 3d23h
kube-system calico-node-bkbc4 2/2 Running 2 4d
kube-system coredns-869f847d58-ldtm9 1/1 Running 1 4d2h
kube-system coredns-869f847d58-nfdjq 1/1 Running 1 4d2h
kube-system etcd-demian-test 1/1 Running 1 3d23h
kube-system kube-apiserver-demian-test 1/1 Running 1 3d23h
kube-system kube-controller-manager-demian-test 1/1 Running 1 3d23h
kube-system kube-proxy-6ddgr 1/1 Running 1 4d2h
kube-system kube-proxy-gp2t4 1/1 Running 1 3d23h
kube-system kube-scheduler-demian-test 1/1 Running 1 3d23h

any ideas? Thanks

chrispokorni

Hi @demian711 ,
You can try to restart your DNS pods, and see if that resolves your DNS issue.
If not, then here is a tutorial on how to setup DNS for Kubernetes on Azure:
https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/azure.md
Regards,
-Chris