Welcome to the Linux Foundation Forum!

diskPressure on LAB 7 5 - lfs258

luigicucciolillo
luigicucciolillo Posts: 17
in LFS258 Class Forum

hi @chrispokorni ,
hope all good there, here still losing my mind with the k8s dream...
Today i have a new problem to push here:
DiskPressure on the worker node and, consequently, pods are getting evicted once created.
here the output of few cmds.

Starting the lab

lc@control-plane-1-lfs258-n2:~$ kubectl get pod
NAME           READY   STATUS    RESTARTS   AGE
ds-one-7nzr7   0/1     Evicted   0          0s
ds-one-mcq26   0/1     Evicted   0          0s
lc@control-plane-1-lfs258-n2:~$ kubectl get pod
NAME           READY   STATUS    RESTARTS   AGE
ds-one-jjj4w   0/1     Evicted   0          14s
ds-one-s4srn   0/1     Evicted   0          14s
lc@control-plane-1-lfs258-n2:~$ kubectl get pod
NAME           READY   STATUS    RESTARTS   AGE
ds-one-rqznk   0/1     Evicted   0          0s
ds-one-vshnk   0/1     Evicted   0          0s

Checking out

disk on the control plane

lc@control-plane-1-lfs258-n2:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       8.7G  8.0G  656M  93% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           1.6G  3.6M  1.6G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs         56K   24K   27K  48% /sys/firmware/efi/efivars
/dev/sda16      881M  114M  705M  14% /boot
/dev/sda15      105M  6.2M   99M   6% /boot/efi
shm              64M  4.0K   64M   1% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a5ea1d57a1963a265c31afc7a15b8c5104a80bb9f83c37957efe17eb99c291cf/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/0c07e3d3d9db1a6ba519439e395e69b99f2cf679bc0dadbac46e8a9cfb025ba2/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/61d9a1076b784422f788dbecfacd76b798bf916747f5c9fdbb09aa176a23f86d/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a8d83a775f2fc159f9b077ed701853fbfaf83507c9121ecd6eaf4afc655bf78b/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/b290e2eeec9f5722423b272acae89ce7e01e09d92dc799e71eec3489a456612b/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c6b55453ce04a21a56666b16c47775dd794ec7e27a4e36570830361ba72f823f/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/6db43ea13ee85bf8456e693456cba89cb7ac5bcadd187a3ae8dfe8bfa589f667/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/ad88b8ceae5b1307b9f153f8908745287d83747175028eb0977b4b4b72c80ac2/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/aca659e3fa5c82141726b3f4c06b561db682c6089792c280f256b6de56b4257f/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/694d7d434a6e81fcf1d4cd2a1b54d199648fcf9dedf384ca3356d6081a72fd6f/shm
tmpfs           794M   12K  794M   1% /run/user/1002

Last events showed up

22s         Warning   FreeDiskSpaceFailed       node/cp            Failed to garbage collect required amount of images. Attempted to free 1169802035 bytes, but only found 0 bytes eligible to free.
5m22s       Warning   FreeDiskSpaceFailed       node/cp            Failed to garbage collect required amount of images. Attempted to free 1169638195 bytes, but only found 0 bytes eligible to free.
6m52s       Normal    Scheduled                 pod/ds-one-dsnj8   Successfully assigned default/ds-one-dsnj8 to cp
6m53s       Warning   Evicted                   pod/ds-one-dsnj8   The node had condition: [DiskPressure].
6m53s       Warning   Evicted                   pod/ds-one-6cd8h   The node had condition: [DiskPressure].
6m52s       Normal    Scheduled                 pod/ds-one-6cd8h   Successfully assigned default/ds-one-6cd8h to wn1
10m         Warning   FreeDiskSpaceFailed       node/cp            Failed to garbage collect required amount of images. Attempted to free 1169462067 bytes, but only found 0 bytes eligible to free.

Disk on the worker node

lc@wn1-lfs258:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       8.7G  7.5G  1.3G  86% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           783M  2.7M  780M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs         56K   24K   27K  48% /sys/firmware/efi/efivars
/dev/sda16      881M  114M  705M  14% /boot
/dev/sda15      105M  6.2M   99M   6% /boot/efi
shm              64M  4.0K   64M   1% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/4419450b610a516daceea9e899e1aaebdf4f566d7f261eaa9cf05740297b4f49/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/6e93cbd50d80d4c6019e607495a5f830ee98c6b5beab4560cb03e0948707ba36/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/897379d7b78c76844c4ac9a5640233b257017b8fbe2ff246780839834425cc08/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/ba648dd983220648839e6f263fbeaef9687140d52792427094f2bc67cc44b57a/shm
tmpfs           392M   12K  392M   1% /run/user/1001

showing disk pressure from kubectl describe

  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Tue, 29 Jul 2025 08:53:31 +0000   Tue, 29 Jul 2025 08:53:31 +0000   CiliumIsUp                   Cilium is running on this node
  MemoryPressure       False   Wed, 20 Aug 2025 15:14:04 +0000   Tue, 29 Jul 2025 08:52:46 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         True    Wed, 20 Aug 2025 15:14:04 +0000   Wed, 20 Aug 2025 05:52:24 +0000   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure          False   Wed, 20 Aug 2025 15:14:04 +0000   Tue, 29 Jul 2025 08:52:46 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Wed, 20 Aug 2025 15:14:04 +0000   Tue, 29 Jul 2025 08:53:28 +0000   KubeletReady                 kubelet is posting ready status

I already tried to Clean images & sandboxes, Clear orphaned pod data and emptyDirs and restart kubelet.
But no changes, still DiskPressure happening.

any suggestions?

Comments

  • chrispokorni
    chrispokorni Posts: 2,605

    Hi @luigicucciolillo,

    What are the sizes of the v-disks attached to each VM? If they are 10 GB or less, you should expect storage related issues.

    Regards,
    -Chris

  • luigicucciolillo
    luigicucciolillo Posts: 17

    hi @chrispokorni ,
    all good.
    I just completed the LAB 7.5,
    solved the DiskPressure by adding more space on the vdisk and enlarged the fs on the two nodes.

    just one think that i would like to notify, maybe a typo:

    • step 15, the output of:
      student@cp:˜$ kubectl describe po ds-two-10khc |grep Image:
      is supposed to be:
      Image: nginx:1.15.1
      i got:
      Image: nginx:1.22.1

    • step 19, the output of:
      student@cp:˜$ kubectl describe po ds-two-8lx7k |grep Image:
      is supposed to be:
      Image: nginx:1.16.1-alpine
      i got:
      Image: nginx:1.26-alpine

    Apart that, amazing!
    hopefully gonna complete soon...

Categories

Upcoming Training