Welcome to the Linux Foundation Forum!

Taints (Lab 2.2)

Hello,
I have some little doubts.

When I create the cluster (lab 2.2), my nodes are ready.
When I try to remove the taints, the procedure is unsuccessful for node.kubernetes.io/disk-pressure- taint.

Why this thing? What am I doing wrong?

Thanks so much

Comments

  • serewicz
    serewicz Posts: 1,000

    Hello,

    If you look at the output you'll notice it says "disk pressure". Meaning the nodes do not meet the requirements for Kubernetes to run. What are you using to run the labs, CPU/Memory/Disk type and size?

    Regards,

  • Hi @MariangelaPetraglia,

    I recently noticed similar node behaviors with GCE VM instances configured with 10 GB disks, when I installed additional cluster management tools that were otherwise not part of the course lab material.

    You could run sudo du / -h -d 1 on each VM to see the sizes of all high level directories, and drill down into the ones that seem to take up too much disk space.

    Regards,
    -Chris

  • Hi @serewicz
    The following output is master node :

    mary@master:~$ sudo lshw -short
    [sudo] password for mary: 
    H/W path      Device      Class       Description
    =================================================
                              system      VirtualBox
    /0                        bus         VirtualBox
    /0/0                      memory      128KiB BIOS
    /0/1                      memory      7898MiB System memory
    /0/2                      processor   Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
    /0/100                    bridge      440FX - 82441FX PMC [Natoma]
    /0/100/1                  bridge      82371SB PIIX3 ISA [Natoma/Triton II]
    /0/100/1.1                storage     82371AB/EB/MB PIIX4 IDE
    /0/100/2                  display     SVGA II Adapter
    /0/100/3      enp0s3      network     82540EM Gigabit Ethernet Controller
    /0/100/4                  generic     VirtualBox Guest Service
    /0/100/5                  multimedia  82801AA AC'97 Audio Controller
    /0/100/6                  bus         KeyLargo/Intrepid USB
    /0/100/6/1    usb1        bus         OHCI PCI host controller
    /0/100/6/1/1              input       USB Tablet
    /0/100/7                  bridge      82371AB/EB/MB PIIX4 ACPI
    /0/100/d                  storage     82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode]
    /0/3          scsi0       storage     
    /0/3/0.0.0    /dev/cdrom  disk        CD-ROM
    /0/4          scsi2       storage     
    /0/4/0.0.0    /dev/sda    disk        31GB VBOX HARDDISK
    /0/4/0.0.0/1  /dev/sda1   volume      10238MiB EXT4 volume
    /1            docker0     network     Ethernet interface
    mary@master:~$ uname -a
    Linux master 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
    mary@master:~$ free
                  total        used        free      shared  buff/cache   available
    Mem:        8088556     1154048     5746168       31680     1188340     6647276
    Swap:        483800           0      483800
    
    

    The following output is worker node (is the same of master):

    mary@worker1:~$ sudo lshw -short
    [sudo] password for mary: 
    H/W path      Device      Class       Description
    =================================================
                              system      VirtualBox
    /0                        bus         VirtualBox
    /0/0                      memory      128KiB BIOS
    /0/1                      memory      7898MiB System memory
    /0/2                      processor   Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
    /0/100                    bridge      440FX - 82441FX PMC [Natoma]
    /0/100/1                  bridge      82371SB PIIX3 ISA [Natoma/Triton II]
    /0/100/1.1                storage     82371AB/EB/MB PIIX4 IDE
    /0/100/2                  display     SVGA II Adapter
    /0/100/3      enp0s3      network     82540EM Gigabit Ethernet Controller
    /0/100/4                  generic     VirtualBox Guest Service
    /0/100/5                  multimedia  82801AA AC'97 Audio Controller
    /0/100/6                  bus         KeyLargo/Intrepid USB
    /0/100/6/1    usb1        bus         OHCI PCI host controller
    /0/100/6/1/1              input       USB Tablet
    /0/100/7                  bridge      82371AB/EB/MB PIIX4 ACPI
    /0/100/d                  storage     82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode]
    /0/3          scsi0       storage     
    /0/3/0.0.0    /dev/cdrom  disk        CD-ROM
    /0/4          scsi2       storage     
    /0/4/0.0.0    /dev/sda    disk        31GB VBOX HARDDISK
    /0/4/0.0.0/1  /dev/sda1   volume      10238MiB EXT4 volume
    /1            docker0     network     Ethernet interface
    mary@worker1:~$ uname -a
    Linux worker1 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
    mary@worker1:~$ free
                  total        used        free      shared  buff/cache   available
    Mem:        8088556     1124936     5865968       38576     1097652     6671776
    Swap:        483800           0      483800
    
  • Hi @chrispokorni

    On master node that command produces following output:

    mary@master:~$ sudo du / -h -d 1
    du: cannot access '/run/user/1000/gvfs': Permission denied
    1,6M    /run
    3,8G    /snap
    3,5G    /usr
    12M /sbin
    40K /root
    385M    /home
    127M    /boot
    13M /bin
    1,2G    /lib
    226M    /opt
    80K /tmp
    4,0K    /mnt
    4,0K    /srv
    16K /lost+found
    4,0K    /cdrom
    4,0K    /lib64
    du: cannot access '/proc/12488/task/12488/fd/4': No such file or directory
    du: cannot access '/proc/12488/task/12488/fdinfo/4': No such file or directory
    du: cannot access '/proc/12488/fd/3': No such file or directory
    du: cannot access '/proc/12488/fdinfo/3': No such file or directory
    0   /proc
    3,3G    /var
    8,0K    /media
    13M /etc
    0   /sys
    0   /dev
    13G /
    

    On worker node that command produces following output:

    mary@worker1:~$ sudo du / -h -d 1
    du: cannot access '/run/user/1000/gvfs': Permission denied
    1,6M    /run
    3,8G    /snap
    3,3G    /usr
    12M /sbin
    40K /root
    381M    /home
    114M    /boot
    13M /bin
    878M    /lib
    226M    /opt
    80K /tmp
    4,0K    /mnt
    4,0K    /srv
    16K /lost+found
    4,0K    /cdrom
    4,0K    /lib64
    du: cannot access '/proc/8722/task/8722/fd/4': No such file or directory
    du: cannot access '/proc/8722/task/8722/fdinfo/4': No such file or directory
    du: cannot access '/proc/8722/fd/3': No such file or directory
    du: cannot access '/proc/8722/fdinfo/3': No such file or directory
    0   /proc
    3,1G    /var
    8,0K    /media
    13M /etc
    0   /sys
    0   /dev
    13G /
    
  • Thanks for the detailed outputs @MariangelaPetraglia.

    It seems that the 10 GB volumes assigned to each VBox VM may not be sufficient. Maybe going up to 15 GB volumes per VM would help.

    Regards,
    -Chris

  • serewicz
    serewicz Posts: 1,000

    Hello,

    While du shows disk usage this information is useless when determining if the disk is full. Please run df -h

    From the output 3.8G  /snap, 3,3G  /usr, 3.1G /var you are out of space if you chose a 10G disk.

    Why you have this much in those directories I'm unsure. What did you do that is not in the lab guide? What did you install using snap which is notorious for wasting space?

    A new cluster shows this on my 2cpu/7.5G/10Gdisk node:

    Filesystem   Size Used Avail Use% Mounted on
    udev      3.7G   0 3.7G  0% /dev
    tmpfs      746M 1.9M 744M  1% /run
    /dev/sda1    9.6G 6.1G 3.5G 64% /
    tmpfs      3.7G   0 3.7G  0% /dev/shm
    tmpfs      5.0M   0 5.0M  0% /run/lock
    tmpfs      3.7G   0 3.7G  0% /sys/fs/cgroup
    /dev/sda15   105M 3.6M 101M  4% /boot/efi
    /dev/loop0    30M  30M   0 100% /snap/snapd/8790
    /dev/loop1    56M  56M   0 100% /snap/core18/1885
    /dev/loop3   126M 126M   0 100% /snap/google-cloud-sdk/147
    /dev/loop4   126M 126M   0 100% /snap/google-cloud-sdk/148
    tmpfs      746M   0 746M  0% /run/user/1001
    


    What does yours show?

    Tim

  • MariangelaPetraglia
    edited September 2020

    Hi @chrispokorni
    Actually, node disks are dynamically allocated and their capacity is 30 GB. Could dynamic allocation be the problem?
    Below there is the screenshot (first and third).

    Hi @serewicz
    this is the execution of df -h command on node:

    mary@master:~$ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            3,9G     0  3,9G   0% /dev
    tmpfs           790M  1,6M  789M   1% /run
    /dev/sda1       9,8G  9,1G  195M  98% /
    tmpfs           3,9G     0  3,9G   0% /dev/shm
    tmpfs           5,0M  4,0K  5,0M   1% /run/lock
    tmpfs           3,9G     0  3,9G   0% /sys/fs/cgroup
    /dev/loop0       55M   55M     0 100% /snap/core18/1754
    /dev/loop2      3,8M  3,8M     0 100% /snap/gnome-system-monitor/127
    /dev/loop3      256M  256M     0 100% /snap/gnome-3-34-1804/36
    /dev/loop1      162M  162M     0 100% /snap/gnome-3-28-1804/128
    /dev/loop4      1,0M  1,0M     0 100% /snap/gnome-logs/100
    /dev/loop5       15M   15M     0 100% /snap/gnome-characters/399
    /dev/loop6      2,5M  2,5M     0 100% /snap/gnome-calculator/748
    /dev/loop7       63M   63M     0 100% /snap/gtk-common-themes/1506
    /dev/loop8      161M  161M     0 100% /snap/gnome-3-28-1804/116
    /dev/loop9       97M   97M     0 100% /snap/core/9804
    /dev/loop10      56M   56M     0 100% /snap/core18/1885
    /dev/loop12     2,3M  2,3M     0 100% /snap/gnome-system-monitor/148
    /dev/loop11      97M   97M     0 100% /snap/core/9436
    /dev/loop13      45M   45M     0 100% /snap/gtk-common-themes/1440
    /dev/loop14     4,3M  4,3M     0 100% /snap/gnome-calculator/544
    /dev/loop15     1,0M  1,0M     0 100% /snap/gnome-logs/81
    /dev/loop16     384K  384K     0 100% /snap/gnome-characters/550
    tmpfs           790M   36K  790M   1% /run/user/121
    tmpfs           790M   32K  790M   1% /run/user/1000
    
  • serewicz
    serewicz Posts: 1,000

    Hello,

    If you look at your root file system you will notice it shows as 98% full. While the disk may be dynamically allocated, the OS using the disk is not. The file system thinks /dev/sda1 is 9.8G, not 30G

    I would either make a larger disk without dynamic allocation or not add all the stuff not mentioned in the course. Snap installed software tends to take up a lot of space.

    Regards,

  • Ok,
    I will.

    Thank you
    Regards

Categories

Upcoming Training