Chapter 8: can't deploy echo-server due to "input/output error"

PetroKazmirchuk

I'm following the latest version of the labs, with a local K8S cluster of Ubuntu 24.04.1, as recommended. In chapter 8, step 6, when I run

helm upgrade -i tester ealenn/echo-server --debug

Helm says that everything is fine:

Release "tester" has been upgraded. Happy Helming!

but the new pod goes into CrashLoopBackOff, and when I open logs, the only thing I see is:

exec /usr/local/bin/node: input/output error

I've tried searching for this error everywhere to no avail. How can I troubleshoot this?
I've attached the full helm output.
Thanks in advance!

PetroKazmirchuk

I've checked that the nodes have enough RAM and disk space, and that I have no leftovers from previous exercises

chrispokorni

Hi @PetroKazmirchuk,

After the" tester" release is newly deployed, what are the events displayed by describing the tester-echo-server Pod?

Regards,
-Chris

PetroKazmirchuk

$ k describe pod tester-echo-server-79f4957797-pvjrr

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  41s                default-scheduler  Successfully assigned default/tester-echo-server-79f4957797-pvjrr to ubuntu-w
  Normal   Pulled     17s (x3 over 40s)  kubelet            Container image "ealen/echo-server:0.6.0" already present on machine
  Normal   Created    17s (x3 over 40s)  kubelet            Created container echo-server
  Normal   Started    17s (x3 over 39s)  kubelet            Started container echo-server
  Warning  BackOff    11s (x4 over 37s)  kubelet            Back-off restarting failed container echo-server in pod tester-echo-server-79f4957797-pvjrr_default(599d903c-e9e3-4ead-b29a-ef74f444c4da)

btw further in the chapter, Apache deployment works fine. So, in principle it is not a blocker, but I want to know how to troubleshoot such problems in real life. E.g. do the same deployment but instead of starting Node, just open a shell and try starting Node manually? but also disable readiness&liveness probes, so that K8S doesn't restart the container while I'm inside?

chrispokorni

Hi @PetroKazmirchuk,

As you suggested, you could remove the probes to possibly prevent the container from being restarted. But that may not be an indication as to why it fails. Rebooting the Node may or may not necessarily help, considering that other applications are scheduled to the Node and deploy successfully; this could also produce disruptions for other applications running on that particular Node. Exec-ing into the Pod's container (with kubectl exec), or simply extracting the Pod's container logs (with kubectl logs) may also help to further troubleshoot the issue. Deleting the Pod and allowing its controller to spawn a replacement could also help, considering it were not a naked Pod.

Regards,
-Chris

PetroKazmirchuk

I can't exec into a dead pod, that's the problem

$ k exec -it tester-echo-server-79f4957797-scmdv -- /bin/sh
error: Internal error occurred: unable to upgrade connection: container not found ("echo-server")

ChatGPT advises to export the the pod definition to YAML, replace the entrypoint with smth approx.command: [/bin/sh -c sleep] and run a new pod with the patched definition, but:

• this is way too cumbersome, I can't imagine DevOps people doing this on production systems on a regular basis
• it doesn't work anyway: exec /bin/sh: exec format error