Taints (Lab 2.2)

MariangelaPetraglia

Hello,
I have some little doubts.

When I create the cluster (lab 2.2), my nodes are ready.
When I try to remove the taints, the procedure is unsuccessful for node.kubernetes.io/disk-pressure- taint.

Why this thing? What am I doing wrong?

Thanks so much

serewicz

Hello,

If you look at the output you'll notice it says "disk pressure". Meaning the nodes do not meet the requirements for Kubernetes to run. What are you using to run the labs, CPU/Memory/Disk type and size?

Regards,

chrispokorni

Hi @MariangelaPetraglia,

I recently noticed similar node behaviors with GCE VM instances configured with 10 GB disks, when I installed additional cluster management tools that were otherwise not part of the course lab material.

You could run sudo du / -h -d 1 on each VM to see the sizes of all high level directories, and drill down into the ones that seem to take up too much disk space.

Regards,
-Chris

MariangelaPetraglia

Hi @serewicz
The following output is master node :

mary@master:~$ sudo lshw -short
[sudo] password for mary: 
H/W path      Device      Class       Description
=================================================
                          system      VirtualBox
/0                        bus         VirtualBox
/0/0                      memory      128KiB BIOS
/0/1                      memory      7898MiB System memory
/0/2                      processor   Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
/0/100                    bridge      440FX - 82441FX PMC [Natoma]
/0/100/1                  bridge      82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1                storage     82371AB/EB/MB PIIX4 IDE
/0/100/2                  display     SVGA II Adapter
/0/100/3      enp0s3      network     82540EM Gigabit Ethernet Controller
/0/100/4                  generic     VirtualBox Guest Service
/0/100/5                  multimedia  82801AA AC'97 Audio Controller
/0/100/6                  bus         KeyLargo/Intrepid USB
/0/100/6/1    usb1        bus         OHCI PCI host controller
/0/100/6/1/1              input       USB Tablet
/0/100/7                  bridge      82371AB/EB/MB PIIX4 ACPI
/0/100/d                  storage     82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode]
/0/3          scsi0       storage     
/0/3/0.0.0    /dev/cdrom  disk        CD-ROM
/0/4          scsi2       storage     
/0/4/0.0.0    /dev/sda    disk        31GB VBOX HARDDISK
/0/4/0.0.0/1  /dev/sda1   volume      10238MiB EXT4 volume
/1            docker0     network     Ethernet interface
mary@master:~$ uname -a
Linux master 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
mary@master:~$ free
              total        used        free      shared  buff/cache   available
Mem:        8088556     1154048     5746168       31680     1188340     6647276
Swap:        483800           0      483800

The following output is worker node (is the same of master):

mary@worker1:~$ sudo lshw -short
[sudo] password for mary: 
H/W path      Device      Class       Description
=================================================
                          system      VirtualBox
/0                        bus         VirtualBox
/0/0                      memory      128KiB BIOS
/0/1                      memory      7898MiB System memory
/0/2                      processor   Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
/0/100                    bridge      440FX - 82441FX PMC [Natoma]
/0/100/1                  bridge      82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1                storage     82371AB/EB/MB PIIX4 IDE
/0/100/2                  display     SVGA II Adapter
/0/100/3      enp0s3      network     82540EM Gigabit Ethernet Controller
/0/100/4                  generic     VirtualBox Guest Service
/0/100/5                  multimedia  82801AA AC'97 Audio Controller
/0/100/6                  bus         KeyLargo/Intrepid USB
/0/100/6/1    usb1        bus         OHCI PCI host controller
/0/100/6/1/1              input       USB Tablet
/0/100/7                  bridge      82371AB/EB/MB PIIX4 ACPI
/0/100/d                  storage     82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode]
/0/3          scsi0       storage     
/0/3/0.0.0    /dev/cdrom  disk        CD-ROM
/0/4          scsi2       storage     
/0/4/0.0.0    /dev/sda    disk        31GB VBOX HARDDISK
/0/4/0.0.0/1  /dev/sda1   volume      10238MiB EXT4 volume
/1            docker0     network     Ethernet interface
mary@worker1:~$ uname -a
Linux worker1 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
mary@worker1:~$ free
              total        used        free      shared  buff/cache   available
Mem:        8088556     1124936     5865968       38576     1097652     6671776
Swap:        483800           0      483800

MariangelaPetraglia

Hi @chrispokorni

On master node that command produces following output:

mary@master:~$ sudo du / -h -d 1
du: cannot access '/run/user/1000/gvfs': Permission denied
1,6M    /run
3,8G    /snap
3,5G    /usr
12M /sbin
40K /root
385M    /home
127M    /boot
13M /bin
1,2G    /lib
226M    /opt
80K /tmp
4,0K    /mnt
4,0K    /srv
16K /lost+found
4,0K    /cdrom
4,0K    /lib64
du: cannot access '/proc/12488/task/12488/fd/4': No such file or directory
du: cannot access '/proc/12488/task/12488/fdinfo/4': No such file or directory
du: cannot access '/proc/12488/fd/3': No such file or directory
du: cannot access '/proc/12488/fdinfo/3': No such file or directory
0   /proc
3,3G    /var
8,0K    /media
13M /etc
0   /sys
0   /dev
13G /

On worker node that command produces following output:

mary@worker1:~$ sudo du / -h -d 1
du: cannot access '/run/user/1000/gvfs': Permission denied
1,6M    /run
3,8G    /snap
3,3G    /usr
12M /sbin
40K /root
381M    /home
114M    /boot
13M /bin
878M    /lib
226M    /opt
80K /tmp
4,0K    /mnt
4,0K    /srv
16K /lost+found
4,0K    /cdrom
4,0K    /lib64
du: cannot access '/proc/8722/task/8722/fd/4': No such file or directory
du: cannot access '/proc/8722/task/8722/fdinfo/4': No such file or directory
du: cannot access '/proc/8722/fd/3': No such file or directory
du: cannot access '/proc/8722/fdinfo/3': No such file or directory
0   /proc
3,1G    /var
8,0K    /media
13M /etc
0   /sys
0   /dev
13G /

chrispokorni

Thanks for the detailed outputs @MariangelaPetraglia.

It seems that the 10 GB volumes assigned to each VBox VM may not be sufficient. Maybe going up to 15 GB volumes per VM would help.

Regards,
-Chris

serewicz

Hello,

While du shows disk usage this information is useless when determining if the disk is full. Please run df -h

From the output 3.8G  /snap, 3,3G  /usr, 3.1G /var you are out of space if you chose a 10G disk.

Why you have this much in those directories I'm unsure. What did you do that is not in the lab guide? What did you install using snap which is notorious for wasting space?

A new cluster shows this on my 2cpu/7.5G/10Gdisk node:

Filesystem   Size Used Avail Use% Mounted on
udev      3.7G   0 3.7G  0% /dev
tmpfs      746M 1.9M 744M  1% /run
/dev/sda1    9.6G 6.1G 3.5G 64% /
tmpfs      3.7G   0 3.7G  0% /dev/shm
tmpfs      5.0M   0 5.0M  0% /run/lock
tmpfs      3.7G   0 3.7G  0% /sys/fs/cgroup
/dev/sda15   105M 3.6M 101M  4% /boot/efi
/dev/loop0    30M  30M   0 100% /snap/snapd/8790
/dev/loop1    56M  56M   0 100% /snap/core18/1885
/dev/loop3   126M 126M   0 100% /snap/google-cloud-sdk/147
/dev/loop4   126M 126M   0 100% /snap/google-cloud-sdk/148
tmpfs      746M   0 746M  0% /run/user/1001

What does yours show?

Tim

MariangelaPetraglia

Hi @chrispokorni
Actually, node disks are dynamically allocated and their capacity is 30 GB. Could dynamic allocation be the problem?
Below there is the screenshot (first and third).

Hi @serewicz
this is the execution of df -h command on node:

mary@master:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3,9G     0  3,9G   0% /dev
tmpfs           790M  1,6M  789M   1% /run
/dev/sda1       9,8G  9,1G  195M  98% /
tmpfs           3,9G     0  3,9G   0% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs           3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/loop0       55M   55M     0 100% /snap/core18/1754
/dev/loop2      3,8M  3,8M     0 100% /snap/gnome-system-monitor/127
/dev/loop3      256M  256M     0 100% /snap/gnome-3-34-1804/36
/dev/loop1      162M  162M     0 100% /snap/gnome-3-28-1804/128
/dev/loop4      1,0M  1,0M     0 100% /snap/gnome-logs/100
/dev/loop5       15M   15M     0 100% /snap/gnome-characters/399
/dev/loop6      2,5M  2,5M     0 100% /snap/gnome-calculator/748
/dev/loop7       63M   63M     0 100% /snap/gtk-common-themes/1506
/dev/loop8      161M  161M     0 100% /snap/gnome-3-28-1804/116
/dev/loop9       97M   97M     0 100% /snap/core/9804
/dev/loop10      56M   56M     0 100% /snap/core18/1885
/dev/loop12     2,3M  2,3M     0 100% /snap/gnome-system-monitor/148
/dev/loop11      97M   97M     0 100% /snap/core/9436
/dev/loop13      45M   45M     0 100% /snap/gtk-common-themes/1440
/dev/loop14     4,3M  4,3M     0 100% /snap/gnome-calculator/544
/dev/loop15     1,0M  1,0M     0 100% /snap/gnome-logs/81
/dev/loop16     384K  384K     0 100% /snap/gnome-characters/550
tmpfs           790M   36K  790M   1% /run/user/121
tmpfs           790M   32K  790M   1% /run/user/1000

serewicz

Hello,

If you look at your root file system you will notice it shows as 98% full. While the disk may be dynamically allocated, the OS using the disk is not. The file system thinks /dev/sda1 is 9.8G, not 30G

I would either make a larger disk without dynamic allocation or not add all the stuff not mentioned in the course. Snap installed software tends to take up a lot of space.

Regards,

MariangelaPetraglia

Ok,
I will.

Thank you
Regards