LFS201 - Chapter 13 Discussion // Questions // Banter
Regarding the OOM killer, we tried to use stress-ng to fill the memory and get the oom killer to work.
I had suggested to use dd as follows:
heiko@LM20-heiko:~$ dd if=/dev/urandom of=/tmp/bigfile bs=4096
dd: error writing '/tmp/bigfile': No space left on device
6346336+0 records in
6346335+0 records out
25994588160 bytes (26 GB, 24 GiB) copied, 451.941 s, 57.5 MB/s
heiko@LM20-heiko:~$ free
total used free shared buff/cache available
Mem: 49351760 17871876 3011112 117328 28468772 30762084
Swap: 999420 0 999420
I guess in my case the root file system filled before I hit the memory limit. My VM has 48Gig and the entire file system is only ~50Gig. Should have run the VM with less memory.
But in general this method also seems to work, but it takes more time.
Another observation, during the Thursday session, when using stress-ng we saw that the oom killer actually terminated the stress-ng program.
If you look at:
heiko@LM20-heiko:~$ cat /proc/sys/vm/oom_kill_allocating_task
0
"If set, the oom-killer kills the task that triggered the out of memory situation, rather than trying to select the best one." That might explain why the stress-ng program was killed.
Comments
-
The lab example 13.1 using stress-ng -m 8 t 10 doesn't work for me at all, even if I increase the numbers.
The following command option, however, works fine:
stress-ng --vm 10 --vm-bytes 100% -t 60s
Jun 27 23:04:35 LM20-heiko kernel: [ 1594.357508] oom-kill:constraint=CONSTRAINT
_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/
user-1000.slice/session-c1.scope,task=stress-ng-vm,pid=4153,uid=1000
Jun 27 23:04:35 LM20-heiko kernel: [ 1594.357525] Out of memory: Killed process
4153 (stress-ng-vm) total-vm:126712kB, anon-rss:72736kB, file-rss:0kB, shmem-rss
:60kB, UID:1000 pgtables:216kB oom_score_adj:1000
Jun 27 23:04:35 LM20-heiko kernel: [ 1594.360546] oom_reaper: reaped process 415
3 (stress-ng-vm), now anon-rss:0kB, file-rss:0kB, shmem-rss:60kB
Jun 27 23:04:35 LM20-heiko stress-ng: memory (MB): total 1984.47, free 97.17, sh
ared 94.01, buffer 0.18, swap 0.00, free swap 0.001 -
Here's a little C program that we use in some other courses to devour and then release the memory. Just run it with one argument (the memory to eat in MB) or with no argument figures out how much memory you have and uses that.
c8:/tmp>cat lab_wastemem.c /* simple program to defragment memory, J. Cooperstein 2/04-1/2020 */ #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <string.h> #include <sys/sysinfo.h> #include <signal.h> #define MB (1024*1024) #define BS 16 /* will allocate BS*MB at each step */ #define CHUNK (MB*BS) #define QUIT_TIME 20 void quit_on_timeout(int sig) { printf("\n\nTime expired, quitting\n"); exit(EXIT_SUCCESS); } int main(int argc, char **argv) { struct sysinfo si; int j, m; char *c; /* get total memory on the system */ sysinfo(&si); m = si.totalram / MB; printf("Total System Memory in MB = %d MB\n", m); m = (9 * m) / 10; /* drop 10 percent */ printf("Using somewhat less: %d MB\n", m); if (argc == 2) { m = atoi(argv[1]); printf("Choosing instead mem = %d MB\n", m); } signal(SIGALRM, quit_on_timeout); printf("Will quite in QUIT_TIME seconds if no normal termination\n"); alarm(QUIT_TIME); for (j = 0; j <= m; j += BS) { /* yes we know this is a memory leak, no free, * that's the idea! */ c = malloc(CHUNK); /* just fill the block with j over and over */ memset(c, j, CHUNK); printf("%8d", j); fflush(stdout); } printf("\n\n Quitting and releasing memory\n"); exit(EXIT_SUCCESS); }You can compile with "gcc -o lab_wastemem.c lab_wastemem.c" and run with "./lab_wastememc." (or any other name you want.
This started out as a one line program in C (really) but evolved to this.
1 -
Thanks coop, will try that too.
0 -
Hi @heiko_s , that's good! Did you try that on CentOS or Ubuntu LST?
Regards,
Luis.0 -
@luisviveropena said:
Hi @heiko_s , that's good! Did you try that on CentOS or Ubuntu LST?Regards,
Luis.As to stress-ng, I tried it on Ubuntu 20.04 (actually Linux Mint 20, but it should be the same for all purposes). As I've mentioned, it doesn't allow me to fill the memory. Tried it also after reducing the VM memory. I will give it a go on Centos.
I haven't tried the C program yet.
1 -
I was also not able to fill it up on CentOS. I will have to revisit this one..
0 -
Hi @moulinath ,
I have a vm guest with 2GB ram, and the closer I was to exhaust memory was using these following commands:
stress-ng --vm 1 --vm-bytes 100% -t 60s
stress-ng --vm 1 --vm-bytes 2g -t 60s
Regards,
Luis.0 -
@heiko_s said:
Regarding the OOM killer, we tried to use stress-ng to fill the memory and get the oom killer to work.I had suggested to use dd as follows:
heiko@LM20-heiko:~$ dd if=/dev/urandom of=/tmp/bigfile bs=4096
dd: error writing '/tmp/bigfile': No space left on device
6346336+0 records in
6346335+0 records out
25994588160 bytes (26 GB, 24 GiB) copied, 451.941 s, 57.5 MB/sI must correct myself:
dd if=... of=/tmp/...will only work if /tmp is mounted as a tmpfs. Else dd just writes to disk, not memory.CentOS 8 does not mount /tmp as tmpfs, nor does Ubuntu 20.04. Need to install 7 and 18.04 respectively to check.
However, you can use the dd method with** /dev/shm**:
dd if=/dev/zero of=/dev/shm/manyzeros bs=1G count=10You don't need to be superuser or King Kong to write to /dev/shm. The above command should do it (unless you got more than 10 Gig memory - so adjust to taste).
You can check if /dev/shm will work:
df -Th | grep /dev/shmtmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm
Bingo.
0 -
As an aside, you can turn off mounting tmp as a RAM disk with:
sudo systemctl mask tmp.mount
and then reboot. I always do this. I've only seen this being the default on Fedora (using tmp in RAM) and I personally find it a bad idea. If the drive /tmp is mounted on is an SSD, it is already rather fast and the gains by using RAM do not outweigh the problems if you use /tmp in such a way that might fill memory. (I do, because I download ISO files there sometimes etc.)
1 -
Hi @heiko_s ,
I found a C code that works faster than stress-ng. Perhaps I forgot to disable swap, because I did it on Ubuntu 18 and stress-ng took some time before killing the process, triggering the OOM Killer, and restarting the process (but it worked anyway).
So, I tried the following C code and it worked very fast (it's necessary to disable swap as well):
#include <stdio.h> #include <stdlib.h> #include <string.h> #define PAGE_SZ (1<<12) int main() { int i; int gb = 2; // memory to consume in GB for (i = 0; i < ((unsigned long)gb<<30)/PAGE_SZ ; ++i) { void *m = malloc(PAGE_SZ); if (!m) break; memset(m, 0, 1); } printf("allocated %lu MB\n", ((unsigned long)i*PAGE_SZ)>>20); getchar(); return 0; }Then compile it as "gcc memory_exhaust.c -o memory_exhaust", and then run it

Many regards,
Luis.0 -
hi, I have a question about this lab... how do you know which process get clobbered first? The /var/log/syslog only displays 2 or 3 lines with less info and dmesg is spewing messages with some processes "DENIED".
0 -
Hi @TemiAworanti ,
The offending process -the one that's consuming more memory, or causing memory exhaustion- will be terminated first.
Many regards,
Luis.0
Categories
- All Categories
- 175 LFX Mentorship
- 175 LFX Mentorship: Linux Kernel
- 745 Linux Foundation IT Professional Programs
- 372 Cloud Engineer IT Professional Program
- 168 Advanced Cloud Engineer IT Professional Program
- 73 DevOps IT Professional Program - Discontinued
- 3 DevOps & GitOps IT Professional Program
- 98 Cloud Native Developer IT Professional Program
- 7.6K Training Courses & Learning Paths
- AI & ML Training
- Blockchain & Decentralized Identity Training
- Cloud & Containers Training
- Cybersecurity Training
- DevOps & Site-Reliability Training
- Linux Kernel Development Training
- Networking Training
- Open Source Best Practice Training
- System Administration Training
- System Engineering Training
- Web & Application Development Training
- 2 LFD103-JP クラス フォーラム
- 4 LFD210-CN Class Forum
- 764 LFD259 Class Forum
- 681 LFS101 Class Forum
- 2 LFS158-JP クラス フォーラム
- 162 LFS207 Class Forum
- 3 LFS207-DE-Klassenforum
- 4 LFS207-JP クラス フォーラム
- 61 LFS241 Class Forum
- 52 LFS242 Class Forum
- 42 LFS243 Class Forum
- 19 LFS244 Class Forum
- 4 LFS250-JP クラス フォーラム
- 166 LFS253 Class Forum
- 19 LFS256 Class Forum
- 1.4K LFS258 Class Forum
- 165 LFS261 Class Forum
- 26 LFS267 Class Forum
- 792 Hardware
- 202 Drivers
- 68 I/O Devices
- 37 Monitors
- 95 Multimedia
- 173 Networking
- 91 Printers & Scanners
- 87 Storage
- 768 Linux Distributions
- 81 Debian
- 67 Fedora
- 22 Linux Mint
- 13 Mageia
- 24 openSUSE
- 150 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 356 Ubuntu
- 465 Linux System Administration
- 31 Cloud Computing
- 73 Command Line/Scripting
- Github systems admin projects
- 98 Linux Security
- 78 Network Management
- 101 System Management
- 46 Web Management
- 106 Mobile Computing
- 18 Android
- 73 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 392 Off Topic
- 121 Introductions
- 181 Small Talk
- 29 Study Material
- 945 Programming and Development
- 310 Kernel Development
- 617 Software Development
- 977 Software
- 369 Applications
- 182 Command Line
- 5 Compiling/Installing
- 68 Games
- 317 Installation
- Archived
- 2 LFD140 Class Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)
