Welcome to the Linux Foundation Forum!

Chapter 4 Quiz 4.3 answer appears to be indeterminate

TBBle
TBBle Posts: 13
in LFD259 Class Forum

Given the question:

If a Pod tries to use more memory than the node has, it will be evicted. True or False?

the claimed correct answer is

True

but I could not reproduce this effect.

Taking the stress.yaml I ended up with after the exercise, and changing the container resources and args to achieve the described behaviour on my 2-core 8Gi AWS instance, I get

      containers:
      - image: vish/stress
        imagePullPolicy: Always
        name: stressmeout
        resources:
          limits:
            cpu: "3"
            memory: "16Gi"
          requests:
            cpu: "0.5"
            memory: "4Gi"
        args:
        - -cpus
        - "2"
        - -mem-total
        - "16Gi"
        - -mem-alloc-size
        - "500Mi"
        - -mem-alloc-sleep
        - "1s"

Deploying this, the container is continuously dying due to system-local OOM, but the pod is never evicted, i.e. it does not go into the Failed state. It does eventually go into CrashLoopBackOff.

Reading the docs, the documented answer here appears to be "Maybe", because it seems to depend if kubelet gets there before the OOM killer. In my case it never happened due to the size of my mem-alloc-size, there was never a point where the kubelet had memory pressure, as the OOMKiller killed it.

Updating that Deployment with -mem-alloc-size of 50Mi still didn't trigger it. I suspect a longer sleep between allocations would, i.e. hanging around inside the eviction threshold for more than 10 seconds, but that'll take a while to run.

Given the content of the chapter, I would have expected the question to be about the requests or limits being greater than the node has, rather than the runtime usage, since that wasn't really touched upon at all in the exercises.

Answers

  • mkevinmchugh
    mkevinmchugh Posts: 17

    @TBBle the value I derived from the exercise was fiddling around with the sizing and repeating that over and over gaining a certain confidence in debugging

    i see your point…. I just got something else out of the exercise … since the course seems to have secondary teaching goals that are not stated… I assume I had achieved the learning goal

    :-)

  • TBBle
    TBBle Posts: 13

    To be clear, the exercise itself was fine, I was talking about the end-of-chapter review quiz question.

    I used the final state of the exercise to try to get the result claimed in the quiz answer key, and failing that, looked in the k8s, where I found an answer that did not match the review quiz answer.

  • yang.wang11
    yang.wang11 Posts: 5

    update with below and have a try

              requests:
            cpu: "3"
            memory: "16Gi"
  • TBBle
    TBBle Posts: 13
    edited February 2022

    @yang.wang11 said:
    update with below and have a try

              requests:
            cpu: "3"
            memory: "16Gi"

    With only 8Gibytes of RAM on the nodes, that configuration means it won't ever be scheduled, so will never have the chance to use too much memory and be evicted/OOM killed.

  • ricardofrsimoes
    ricardofrsimoes Posts: 2

    It's a bad question in my opinion. Too loosely phrased.

    From k8s docs: "If a container exceeds its memory request and the node that it runs on becomes short of memory overall, it is likely that the Pod the container belongs to will be evicted."

    Also: "If the node where a Pod is running has enough of a resource available, it's possible (and allowed) for a container to use more resource than its request for that resource specifies."

    So, i can deduct that, tldr: Eviction happens due to node pressure, not just “memory usage”: "However, terminations only happen when the kernel detects memory pressure."

    https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Categories

Upcoming Training