Welcome to the new Linux Foundation Forum!

Lab 3 now on GCP. Can't get nodes to curl nginx.

btanouebtanoue Posts: 53
edited April 11 in LFS258 Class Forum

Master Node Output:
[email protected]:~$ kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx ClusterIP 10.101.100.68 80/TCP 9m32s
[email protected]:~$ kubectl get ep nginx
NAME ENDPOINTS AGE
nginx 192.168.2.2:80 9m37s
[email protected]:~$ kubectl describe pod nginx-7db75b8b78-j5dq7 |grep Node:
Node: kubeworker/10.142.0.9

Worker Node:
[email protected]:~$ sudo tcpdump -i tunl0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tunl0, link-type RAW (Raw IP), capture size 262144 bytes

Master Node
[email protected]:~$ curl 10.101.100.86:80

This just hangs.

[email protected]:~$ curl 192.168.2.2:80

Hangs as well.

Any suggestion for debug?

Comments

  • btanouebtanoue Posts: 53

    Nodes seem OK:

    NAME STATUS ROLES AGE VERSION
    kubemaster Ready master 4h26m v1.13.1
    kubeworker Ready 55m v1.13.1

  • btanouebtanoue Posts: 53

    Yaml file: first.yaml.
    I think I edited it OK.

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    generation: 1
    labels:
    app: nginx
    name: nginx
    namespace: default
    spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
    matchLabels:
    app: nginx
    strategy:
    rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
    type: RollingUpdate
    template:
    metadata:
    creationTimestamp: null
    labels:
    app: nginx
    spec:
    containers:
    - image: nginx
    imagePullPolicy: Always
    name: nginx
    _ ports:
    - containerPort: 80
    protocol: TCP _
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    dnsPolicy: ClusterFirst
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    terminationGracePeriodSeconds: 30

  • chrispokornichrispokorni Posts: 243

    Hi,

    Check the IP in your curl command, and compare with the ClusterIP, there is a typo in your curl to 10.101...

    However, that is not why your curls are hanging now. Similar curl issues have been reported in earlier discussions and solutions posted as well. Feel free to check earlier discussions before reporting an issue because chances are it may have already been reported and solutioned.

    By not being able to curl to your endpoint (from the master node to a pod running on the worker node) indicates that there is a networking issue between your nodes. It could be a firewall in your Ubuntu OS (check ufw, apparmor, ...) or a networking firewall issue at GCE level. Do you have a custom VPC network? Do you have an allow-all (all-open) firewall rule? Your VMs inside the new VPC network?

    The nodes listing looks ok. I hope your pods are running as expected as well.

    I also hope your YAML file is properly formatted/indented, because the way I see it above it would clearly fail. There are also some underscores ("_") which should not be there. Check your file for accuracy.

    Regards,
    -Chris

  • btanouebtanoue Posts: 53

    For the Yaml, the copy and paste didn't work well....

  • btanouebtanoue Posts: 53

    I didn't do anything special but create a VM on GCE. I didn't touch firewalls or anything like that. Just following the directions....again.

  • chrispokornichrispokorni Posts: 243

    In case you missed it, in section 3.1 Overview, an info box labeled !Very Important addresses the firewall for GCP. Please review it.

    Also, a day or two ago, I provided detailed instructions for you to follow when transitioned from vbox to GCP. In case you missed those too, here they are, again:

    When setting up your VMs in the cloud keep in mind the initial networking requirements - nodes need to talk to each other and talk to the internet. For this purpose, create a new custom VPC network (do not go with the predefined VPCs), assign to it a new custom firewall rule which allows all traffic (all protocols, all ports, from all sources, to all destinations) and provision your VMs inside this custom network.

    All these instructions are provided to guide you towards successfull completion of the lab exercises. Please read them carefully before applying them. They are equally important for the initial environemt setup and for the overall Kubernetes cluster behavior. A misconfigured environment leads to issues in Kubernetes - as you already experienced, and in some cases they may be quite difficult to troubleshoot.

    Good luck!
    -Chris

  • btanouebtanoue Posts: 53

    The Typeo was there because I tried to run the command again and re-generate output. I ran it with the 10.10 ip and it was the same.

  • btanouebtanoue Posts: 53

    Oh, Let me check the yaml.

  • btanouebtanoue Posts: 53

    Ignore the formatting, I can't get it to paste right, but there weren't any underscores in the yaml. I double checked.

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    generation: 1
    labels:
    app: nginx
    name: nginx
    namespace: default
    spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
    matchLabels:
    app: nginx
    strategy:
    rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
    type: RollingUpdate
    template:
    metadata:
    creationTimestamp: null
    labels:
    app: nginx
    spec:
    containers:
    - image: nginx
    imagePullPolicy: Always
    name: nginx
    ports:
    - containerPort: 80
    protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    dnsPolicy: ClusterFirst
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    terminationGracePeriodSeconds: 30

  • fcioancafcioanca Posts: 287

    You can use the Code block option in the ribbon to paste formatted code. Besides the yaml file, you need to ensure your environment and networking is set up properly from the start (follow the instructions in Lab 2.1, also repeated by Chris in a previous post in this thread).

  • btanouebtanoue Posts: 53

    OK, since I switched from VB to GCP, I did miss the networking change.

    I don't know how to create the rule in GCP. I'm really new to all of this so the directions don't even have a link on how to do that.

    "If using GCP you can add a rule to the project which allows all traffic to all ports."

  • btanouebtanoue Posts: 53

    OK. I tried to add a VPC and clicked all the kubernetes rules that were there and called the network kubernetes-network.
    I restarted the VMs just in case.

    Now I can't connect to the cluster:

    kubectl get svc nginx
    The connection to the server localhost:8080 was refused - did you specify the right host or port?

  • btanouebtanoue Posts: 53

    Disregard the previous mail. Wrong node...grrr.

  • btanouebtanoue Posts: 53

    OK, I still can't curl.
    I'd like to request an admin to contact me and webex so I can show you my GCP setup and Kubernetes setup. I don't really know what else I can do now. I've been stuck in Chapter 3 for a week.

  • coopcoop Posts: 284

    I'm not a moderator on this forum, but I am afraid one-on-one live support and tutoring is not available at this kind of price point.

    I think the moderators have been helping as much as they can as fast as they can. This is not real time support and when posting 4 or 5 messages in less than an hour , you can't expect an immediate response. Hopefully, with moderator support your problems will be solved.

    In my experience, most of the problems in the course forums I do moderate come from not careful enough reading of instructions, or cutting and pasting from pdf's which can butcher the characters such as underscores and other special characters. So please take a fresh look :wink:

  • serewiczserewicz Posts: 500

    We do have a video available with details of how to set up the GCE lab environment. Perhaps it would show where the setup is incorrect.

  • fcioancafcioanca Posts: 287

    Check the online resources as the video is available there - page 1.8. Course Resources has the instructions on how to access them (same location as the files you are using for labs).

  • btanouebtanoue Posts: 53

    @serewicz said:
    We do have a video available with details of how to set up the GCE lab environment. Perhaps it would show where the setup is incorrect.

    That would be awesome! Where is it?

  • btanouebtanoue Posts: 53

    @fcioanca said:
    Check the online resources as the video is available there - page 1.8. Course Resources has the instructions on how to access them (same location as the files you are using for labs).

    Thanks.

  • btanouebtanoue Posts: 53

    So, a lot of pain was self inflicted. I learned a lot in how GCE works and how the VPC works.
    Since my original plan was to use VB for all the labs, I shifted midway to GCE.

    When I did that, I forgot that I needed the VPC until @serewicz made the comment above and then it all started to make sense. I forgot about the video since I was hyper focused on VB at that time and when I moved, I just forgot. I'm trying to take this between meetings and daily work etc. Still my bad.

    I also learned that you can't move a VM in GCE between networks even if you shut it down. The edit didn't seem to let me do it so I had to recreate a VM to make sure it was on the right Kubernetes network since I couldn't figure out how to move it.

    After that, everything started to work. I got the LoadBalancer up and traffic was flowing. It was pretty magnificent.

    I'm still a little confused right now on how the Endpoints work. I understand the service is what creates the IP and it connects to an EP I think. I assume as we go through the rest of the material and lab I'll learn more about how it all connects.

    I'm so happy it is working. Thank you all for the help and push in the right direction.

Sign In or Register to comment.