Welcome to the Linux Foundation Forum!

Chapter 8: can't deploy echo-server due to "input/output error"

I'm following the latest version of the labs, with a local K8S cluster of Ubuntu 24.04.1, as recommended. In chapter 8, step 6, when I run

  1. helm upgrade -i tester ealenn/echo-server --debug

Helm says that everything is fine:

  1. Release "tester" has been upgraded. Happy Helming!

but the new pod goes into CrashLoopBackOff, and when I open logs, the only thing I see is:

  1. exec /usr/local/bin/node: input/output error

I've tried searching for this error everywhere to no avail. How can I troubleshoot this?
I've attached the full helm output.
Thanks in advance!

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Answers

  • I've checked that the nodes have enough RAM and disk space, and that I have no leftovers from previous exercises

  • Posts: 2,434

    Hi @PetroKazmirchuk,

    After the" tester" release is newly deployed, what are the events displayed by describing the tester-echo-server Pod?

    Regards,
    -Chris

  • $ k describe pod tester-echo-server-79f4957797-pvjrr

    1. Events:
    2. Type Reason Age From Message
    3. ---- ------ ---- ---- -------
    4. Normal Scheduled 41s default-scheduler Successfully assigned default/tester-echo-server-79f4957797-pvjrr to ubuntu-w
    5. Normal Pulled 17s (x3 over 40s) kubelet Container image "ealen/echo-server:0.6.0" already present on machine
    6. Normal Created 17s (x3 over 40s) kubelet Created container echo-server
    7. Normal Started 17s (x3 over 39s) kubelet Started container echo-server
    8. Warning BackOff 11s (x4 over 37s) kubelet Back-off restarting failed container echo-server in pod tester-echo-server-79f4957797-pvjrr_default(599d903c-e9e3-4ead-b29a-ef74f444c4da)
    9.  

    btw further in the chapter, Apache deployment works fine. So, in principle it is not a blocker, but I want to know how to troubleshoot such problems in real life. E.g. do the same deployment but instead of starting Node, just open a shell and try starting Node manually? but also disable readiness&liveness probes, so that K8S doesn't restart the container while I'm inside?

  • Posts: 2,434

    Hi @PetroKazmirchuk,

    As you suggested, you could remove the probes to possibly prevent the container from being restarted. But that may not be an indication as to why it fails. Rebooting the Node may or may not necessarily help, considering that other applications are scheduled to the Node and deploy successfully; this could also produce disruptions for other applications running on that particular Node. Exec-ing into the Pod's container (with kubectl exec), or simply extracting the Pod's container logs (with kubectl logs) may also help to further troubleshoot the issue. Deleting the Pod and allowing its controller to spawn a replacement could also help, considering it were not a naked Pod.

    Regards,
    -Chris

  • I can't exec into a dead pod, that's the problem

    1. $ k exec -it tester-echo-server-79f4957797-scmdv -- /bin/sh
    2. error: Internal error occurred: unable to upgrade connection: container not found ("echo-server")

    ChatGPT advises to export the the pod definition to YAML, replace the entrypoint with smth approx.command: [/bin/sh -c sleep] and run a new pod with the patched definition, but:

    • this is way too cumbersome, I can't imagine DevOps people doing this on production systems on a regular basis
    • it doesn't work anyway: exec /bin/sh: exec format error

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training