Taints (Lab 2.2)

MariangelaPetraglia

Hello,
I have some little doubts.

When I create the cluster (lab 2.2), my nodes are ready.
When I try to remove the taints, the procedure is unsuccessful for node.kubernetes.io/disk-pressure- taint.

Why this thing? What am I doing wrong?

Thanks so much

chrispokorni

Hi @MariangelaPetraglia,

I recently noticed similar node behaviors with GCE VM instances configured with 10 GB disks, when I installed additional cluster management tools that were otherwise not part of the course lab material.

You could run sudo du / -h -d 1 on each VM to see the sizes of all high level directories, and drill down into the ones that seem to take up too much disk space.

Regards,
-Chris

MariangelaPetraglia

Hi @serewicz
The following output is master node :

mary@master:~$ sudo lshw -short
[sudo] password for mary: 
H/W path      Device      Class       Description
=================================================
                          system      VirtualBox
/0                        bus         VirtualBox
/0/0                      memory      128KiB BIOS
/0/1                      memory      7898MiB System memory
/0/2                      processor   Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
/0/100                    bridge      440FX - 82441FX PMC [Natoma]
/0/100/1                  bridge      82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1                storage     82371AB/EB/MB PIIX4 IDE
/0/100/2                  display     SVGA II Adapter
/0/100/3      enp0s3      network     82540EM Gigabit Ethernet Controller
/0/100/4                  generic     VirtualBox Guest Service
/0/100/5                  multimedia  82801AA AC'97 Audio Controller
/0/100/6                  bus         KeyLargo/Intrepid USB
/0/100/6/1    usb1        bus         OHCI PCI host controller
/0/100/6/1/1              input       USB Tablet
/0/100/7                  bridge      82371AB/EB/MB PIIX4 ACPI
/0/100/d                  storage     82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode]
/0/3          scsi0       storage     
/0/3/0.0.0    /dev/cdrom  disk        CD-ROM
/0/4          scsi2       storage     
/0/4/0.0.0    /dev/sda    disk        31GB VBOX HARDDISK
/0/4/0.0.0/1  /dev/sda1   volume      10238MiB EXT4 volume
/1            docker0     network     Ethernet interface
mary@master:~$ uname -a
Linux master 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
mary@master:~$ free
              total        used        free      shared  buff/cache   available
Mem:        8088556     1154048     5746168       31680     1188340     6647276
Swap:        483800           0      483800

The following output is worker node (is the same of master):

mary@worker1:~$ sudo lshw -short
[sudo] password for mary: 
H/W path      Device      Class       Description
=================================================
                          system      VirtualBox
/0                        bus         VirtualBox
/0/0                      memory      128KiB BIOS
/0/1                      memory      7898MiB System memory
/0/2                      processor   Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
/0/100                    bridge      440FX - 82441FX PMC [Natoma]
/0/100/1                  bridge      82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1                storage     82371AB/EB/MB PIIX4 IDE
/0/100/2                  display     SVGA II Adapter
/0/100/3      enp0s3      network     82540EM Gigabit Ethernet Controller
/0/100/4                  generic     VirtualBox Guest Service
/0/100/5                  multimedia  82801AA AC'97 Audio Controller
/0/100/6                  bus         KeyLargo/Intrepid USB
/0/100/6/1    usb1        bus         OHCI PCI host controller
/0/100/6/1/1              input       USB Tablet
/0/100/7                  bridge      82371AB/EB/MB PIIX4 ACPI
/0/100/d                  storage     82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode]
/0/3          scsi0       storage     
/0/3/0.0.0    /dev/cdrom  disk        CD-ROM
/0/4          scsi2       storage     
/0/4/0.0.0    /dev/sda    disk        31GB VBOX HARDDISK
/0/4/0.0.0/1  /dev/sda1   volume      10238MiB EXT4 volume
/1            docker0     network     Ethernet interface
mary@worker1:~$ uname -a
Linux worker1 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
mary@worker1:~$ free
              total        used        free      shared  buff/cache   available
Mem:        8088556     1124936     5865968       38576     1097652     6671776
Swap:        483800           0      483800

MariangelaPetraglia

Hi @chrispokorni

On master node that command produces following output:

mary@master:~$ sudo du / -h -d 1
du: cannot access '/run/user/1000/gvfs': Permission denied
1,6M    /run
3,8G    /snap
3,5G    /usr
12M /sbin
40K /root
385M    /home
127M    /boot
13M /bin
1,2G    /lib
226M    /opt
80K /tmp
4,0K    /mnt
4,0K    /srv
16K /lost+found
4,0K    /cdrom
4,0K    /lib64
du: cannot access '/proc/12488/task/12488/fd/4': No such file or directory
du: cannot access '/proc/12488/task/12488/fdinfo/4': No such file or directory
du: cannot access '/proc/12488/fd/3': No such file or directory
du: cannot access '/proc/12488/fdinfo/3': No such file or directory
0   /proc
3,3G    /var
8,0K    /media
13M /etc
0   /sys
0   /dev
13G /

On worker node that command produces following output:

mary@worker1:~$ sudo du / -h -d 1
du: cannot access '/run/user/1000/gvfs': Permission denied
1,6M    /run
3,8G    /snap
3,3G    /usr
12M /sbin
40K /root
381M    /home
114M    /boot
13M /bin
878M    /lib
226M    /opt
80K /tmp
4,0K    /mnt
4,0K    /srv
16K /lost+found
4,0K    /cdrom
4,0K    /lib64
du: cannot access '/proc/8722/task/8722/fd/4': No such file or directory
du: cannot access '/proc/8722/task/8722/fdinfo/4': No such file or directory
du: cannot access '/proc/8722/fd/3': No such file or directory
du: cannot access '/proc/8722/fdinfo/3': No such file or directory
0   /proc
3,1G    /var
8,0K    /media
13M /etc
0   /sys
0   /dev
13G /

chrispokorni

Thanks for the detailed outputs @MariangelaPetraglia.

It seems that the 10 GB volumes assigned to each VBox VM may not be sufficient. Maybe going up to 15 GB volumes per VM would help.

Regards,
-Chris

MariangelaPetraglia

Hi @chrispokorni
Actually, node disks are dynamically allocated and their capacity is 30 GB. Could dynamic allocation be the problem?
Below there is the screenshot (first and third).

Hi @serewicz
this is the execution of df -h command on node:

mary@master:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3,9G     0  3,9G   0% /dev
tmpfs           790M  1,6M  789M   1% /run
/dev/sda1       9,8G  9,1G  195M  98% /
tmpfs           3,9G     0  3,9G   0% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs           3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/loop0       55M   55M     0 100% /snap/core18/1754
/dev/loop2      3,8M  3,8M     0 100% /snap/gnome-system-monitor/127
/dev/loop3      256M  256M     0 100% /snap/gnome-3-34-1804/36
/dev/loop1      162M  162M     0 100% /snap/gnome-3-28-1804/128
/dev/loop4      1,0M  1,0M     0 100% /snap/gnome-logs/100
/dev/loop5       15M   15M     0 100% /snap/gnome-characters/399
/dev/loop6      2,5M  2,5M     0 100% /snap/gnome-calculator/748
/dev/loop7       63M   63M     0 100% /snap/gtk-common-themes/1506
/dev/loop8      161M  161M     0 100% /snap/gnome-3-28-1804/116
/dev/loop9       97M   97M     0 100% /snap/core/9804
/dev/loop10      56M   56M     0 100% /snap/core18/1885
/dev/loop12     2,3M  2,3M     0 100% /snap/gnome-system-monitor/148
/dev/loop11      97M   97M     0 100% /snap/core/9436
/dev/loop13      45M   45M     0 100% /snap/gtk-common-themes/1440
/dev/loop14     4,3M  4,3M     0 100% /snap/gnome-calculator/544
/dev/loop15     1,0M  1,0M     0 100% /snap/gnome-logs/81
/dev/loop16     384K  384K     0 100% /snap/gnome-characters/550
tmpfs           790M   36K  790M   1% /run/user/121
tmpfs           790M   32K  790M   1% /run/user/1000

MariangelaPetraglia

Ok,
I will.

Thank you
Regards