Welcome to the Linux Foundation Forum!

Problem with Exercise 3.3: Access from Outside the Cluster

sudeepbatra1975
sudeepbatra1975 Posts: 15
edited March 2018 in LFS258 Class Forum

Please check this status :

esudbat@kube-master:~$ kubectl get svc

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE

kubernetes   ClusterIP      10.96.0.1        <none>        443/TCP        40d

nginx        LoadBalancer   10.106.224.107   <pending>     80:30619/TCP   26m



Step 6 : External access via the public IP and port doesnt work.

Any suggestion ?

Comments

  • serewicz
    serewicz Posts: 1,000
    edited March 2018

    Hello,

    First check the access to the high port on the node itself. First use loopback, then the 10.106.224.107 IP. This may narrow down if the issue is a firewall problem.  Remember that only the high-port is accessable with services.  The nc command can be helpful to troubleshoot, such as: nc 10.106.224.107 30619    

    If the nc command works using 127.0.0.1 but not the 10. IP it probably is a firewall issue.   If you load a webserver on the host node, can you access that server via the public IP?

    Regards,

     

  • devagari
    devagari Posts: 1

    Hello serewicz,

    Can you please list down the steps, to fix the firewall issue. All my services are woking inside cluster but when I try to test ouside , get no response. Even my kubernetes dashboard is not running becuse of this, It will be great help. 

  • serewicz
    serewicz Posts: 1,000
    edited March 2018

    Hello devagari,

    The iptables command would be where I started, something like sudo iptables -vL  and see if there are any rules which would drop or reject the expected traffic. The use of a LOG target early in the rules can be helpful so you can see the packet enter, then you can find which rule is dropping, rejecting or perhaps even sending the packet to an unexpected place. 

    Chances are the packet is dopped on INPUT instead of OUTPUT, so one easy way to check is to add as the first rule to ACCEPT all traffic from the sending host, something like sudo iptables -I INPUT -s <ip of sending node here> -j ACCEPT  Then try the curl, wget or HTTP commands again and see if it works.  You can learn more about iptables here: https://www.howtogeek.com/177621/the-beginners-guide-to-iptables-the-linux-firewall/ 

    Did it work using loopback/127.0.0.1 but not the exterior IP address?  Are you using virtualbox or a cloud provider for your instances, like AWS or GCE?

    Regards,

  • sudeepbatra1975
    sudeepbatra1975 Posts: 15
    edited March 2018

    Using GCE :

    Definitely this interface is too slow and almost wastage of time- I wish LF moves to Slack- otherwise this is too slow and not so useful :

    Here is my output :

    esudbat@kube-master:~$ kubectl get svc nginx

    NAME      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE

    nginx     LoadBalancer   10.106.224.107   <pending>     80:30619/TCP   4d

    esudbat@kube-master:~$ curl 104.196.99.153:30619

    ^C

    esudbat@kube-master:~$ kubectl get svc nginx

    NAME      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE

    nginx     LoadBalancer   10.106.224.107   <pending>     80:30619/TCP   4d

    esudbat@kube-master:~$ curl 10.106.224.107:8080

    ^C

    esudbat@kube-master:~$ curl 10.106.224.107:80

    <!DOCTYPE html>

    <html>

    <head>

    <title>Welcome to nginx!</title>

    <h1>Welcome to nginx!</h1>

    <p>If you see this page, the nginx web server is successfully installed and

    working. Further configuration is required.</p>

    <truncated>

    esudbat@node1:~$ sudo iptables -A INPUT -p tcp --dport 30619 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

    esudbat@node1:~$ sudo iptables -A OUTPUT -p tcp --sport 30619 -m conntrack --ctstate ESTABLISHED -j ACCEPT

     

    esudbat@kube-master:~$ curl 104.196.99.153:30619

    Keeps waiting - no response here....

    LinuxFoundation Guide is not user-friendly.

     

  • chrispokorni
    chrispokorni Posts: 2,376
    edited March 2018

    Hi, I also use GCE for the labs in this course. I remember having a similar issue with the access from outside the cluster, and after a little bit of google-shooting :) I realized it was a GCE firewall issue. I added another firewall rule and after that I was able to complete the exercise. Hope this helps.

    Good luck!

    -Chris

  • sudeepbatra1975
    sudeepbatra1975 Posts: 15
    edited March 2018

    Yes I did that- I suppose you are referring this link.

    https://cloud.google.com/compute/docs/tutorials/basic-webserver-apache

    And also tested using a different VM with webserver- that all works. But still couldnt figure out the issue..

    I tried flushing all the iptables , still doesnt work, strange. I think there is some firewall issue locally on the node vm- which I am unable to decipher.

  • serewicz
    serewicz Posts: 1,000
    edited March 2018

    Hello Sudeep,

    Did you attempt to access the port using the loopback IP address?  As  Chris pointed out GCE, which runs SDN on our behalf, has a firewall by default. One thing you may try is to log into the console and add a firewall rule to GCE that opens up all ports, from all source IP addresses. If the curl traffic works once you add the rule, you know it was the cause.  

    From your example, the 104.196.99.153 is your public IP address. Which would indicate traffic would navigate throught the Google SDN to get to your node. The firewall would not be inside the node, but inside google's network.

    Do you have a rule inside GCE which allows all traffic to your nodes?

    Regards,

     

  • sudeepbatra1975
    sudeepbatra1975 Posts: 15
    edited March 2018

    Ok - I added an allow-all firewall rule to allow all trafffic from 0.0.0.0/0, and now it did work. Thanks.

  • serewicz
    serewicz Posts: 1,000
    edited March 2018

    Glad to hear it Sudeep. Thanks for letting us know.

  • SudeepBatra
    SudeepBatra Posts: 1

    I am able to curl from within all the GCE network but the external access doesnt seem to be working.

    esudbat@node1:~$ curl 35.231.45.60:30438

    <!DOCTYPE html>

    <html>

    <head>

    <title>Welcome to nginx!</title>

    I tried the same rules on the GCE, allow-all but strangely cant access from outside.

    chrispokorni , Can you confirm the rules you created in GCE ? To allow all or a particular port.

  • chrispokorni
    chrispokorni Posts: 2,376
    edited March 2018

    For simplicity, I created a rule to allow all tcp traffic (rather than allowing a specific port), and only then I was able to access from the outside on the [nodeIP]:[nodePort], and I verified access on all running nodes.

    -Chris

    PS: The new rule I created was in the same project where my nodes were (in my case a custom project created only for the purpose of lfs258 where I run all the master/worker nodes)

     

     

  • serewicz
    serewicz Posts: 1,000
    edited March 2018

    Hello Sudeep,

    If you have added an allow-all rule to GCE and still cannot gain access to your Pod, perhaps the block is not in GCE.

    Like Chris I had opened all ports from all source IPs. To test which rules are actually necessary I returned to the GCE firewall page and added only the port exposed by the service, or 30494 in my case. When I removed the all-traffic firewall rule traffic my request for the page timed out. When I added only tcp:30494 I was able see the Welcome to nginx! page again.  From this testing the only necessary rule is the particular port being exposed, which will change with each time you run the kubectl expose command.  Please note that it took about minute for the rule to actually take affect after selecting the save button in the GCE console. 

    Could there be a corportate firewall or proxy which is blocking the high ports?

    Regards,

  • jyothsna271
    jyothsna271 Posts: 1

    The issue is definitely due to the corporate firewall. I tested it outside the corporate network and it worked.

Categories

Upcoming Training