Welcome to the Linux Foundation Forum!

500 Internal Error message when connecting to LINKERD.

Todd Gill
Todd Gill Posts: 22
in Cloud & Containers Training

500 Internal Server Error
rpc error: code = Unavailable des = connection error: des= "error reading server preface: raw-read top 192.168.0.232:44340->10.103.80.224:8085: use of closed network connection"

Comments

  • chrispokorni
    chrispokorni Posts: 2,627

    Hi @Todd Gill,

    What course/step produces this behavior? Are all installation, configuration, and injection steps followed as presented in the lab guide prior to this error?

    Regards,
    -Chris

  • Todd Gill
    Todd Gill Posts: 22

    Yes, all the steps were followed and I had everything working.

    I am on GCP, and after each lesson, I need to suspend the VMs to keep costs down.
    I think something happened after I restarted the VMs.

    1) I checked the linkerd process running , killed it and tried restarting it .. but didn't work.

    student@cp:~$ ps aux | grep linkerd
    2103 946647 0.0 0.6 1320680 50524 ? Ssl Aug24 0:30 ./web -linkerd-metrics-api-addr=metrics-api.linkerd-viz.svc.cluster.local:8085 -cluster-domain=cluster.local -controller-namespace=linkerd -log-level=info -log-format=plain -enable-pprof=false

    I am connecting at : http://34.94.26.233:31500/namespaces

    I can view the webpage, but it still flashes the 500 Internal Server Error at the bottom of the page.

  • Todd Gill
    Todd Gill Posts: 22

    A 500 Internal Server Error
    rpc error: code = Unavailable des= connection error: des= "error reading server preface: raw-read top 192.168.0.232:44322->10.103.80.224:8085: use of closed network connection"
    http://34.94.26.233:31500/api/tps-reports?resource_type=namespace&namespace=all&tcp_stats=true&window=1m

  • chrispokorni
    chrispokorni Posts: 2,627

    Hi @Todd Gill,

    Perhaps answering these questions would help:

    1. What course is this?
    2. Which Lab exercise and which Step number is attempted when this behavior is observed?

    Regards,
    -Chris

  • Todd Gill
    Todd Gill Posts: 22

    Kubernetes Fundamentals (LFS258)

    Lab 11.2. Ingress Controller

    Step one wants you to create two deployments in the default namespaces, which I did using
    lesson : Lab 10.1. Deploy a New Service. , two deployments of nginx.

  • Todd Gill
    Todd Gill Posts: 22

    I am currently in this step and the VM is hanging .....

  • Todd Gill
    Todd Gill Posts: 22
    edited August 25

    After it times out .... it returns a 504 error.....

  • chrispokorni
    chrispokorni Posts: 2,627

    Hi @Todd Gill,

    With the introduction of the hosted lab environment for LFS258 we highly recommend using it to practice the hands-on lab exercises.

    Since the service mesh seems to be the one impacted, I'd recommend to revisit the web deployment and the linkerd-viz service configurations and validate the initial configuration edits were preserved between GCE instances restarts. Check for linkerd viz dashboard instances that may be running redundantly in the background - ensure a single dashboard background process exists.

    If the service mesh stops you from making progress, you can completely uninstall linkerd and re-install and reconfigure it.

    Also, check the state of your cluster, to ensure your nodes are ready and all application pods are running. Check for a network policy resource - remove it if it exists.

    Regards,
    -Chris

  • Todd Gill
    Todd Gill Posts: 22
    edited August 26

    Hi Chris,

    Thank you for the checklist. My cluster control plane pods are all "Running" and in a good state.
    The nodes are also "Ready".

    I took your advice to uninstall Linkerd, but even that was a web of "Uninjections" and finally deleting the namespace with the pods that needed to be "Uninjected". After that, I was able to run :

    linkerd uninstall | kubectl delete -f - ..... and delete the control plane and then commands to delete ingress controller using :

    helm list -A
    helm uninstall -n

    Now, I am back to ground 0 and ready to rebuild the ingress controller with linkerd service mesh.

    I totally get what you mean by using the hosted lab environment, it would save me a lot of time and headaches. However, I have learned so much more breaking and fixing things. :) This time I won't suspend my cluster in GCP until I can better understand linkerd's GUI structure.

    I'll be starting the lab again, soon. Thank you for the help!

    Regards,

    Todd

  • chrispokorni
    chrispokorni Posts: 2,627

    Hi @Todd Gill,

    I do see the learning advantage when you control the environment.

    Keep in mind that while the service mesh is used to monitor the traffic of the "injected" applications, it is not a requirement for the ingress exercise. The ingress exercise can be completed independently of the service mesh exercise, but without the web UI to validate traffic sources and destinations (and by simply skipping the few "inject" commands throughout the ingress exercise).

    Regards,
    -Chris

Categories

Upcoming Training