LFS201 - Chapter 13 Discussion // Questions // Banter

heiko_s

Regarding the OOM killer, we tried to use stress-ng to fill the memory and get the oom killer to work.

I had suggested to use dd as follows:

heiko@LM20-heiko:~$ dd if=/dev/urandom of=/tmp/bigfile bs=4096
dd: error writing '/tmp/bigfile': No space left on device
6346336+0 records in
6346335+0 records out
25994588160 bytes (26 GB, 24 GiB) copied, 451.941 s, 57.5 MB/s

heiko@LM20-heiko:~$ free
total used free shared buff/cache available
Mem: 49351760 17871876 3011112 117328 28468772 30762084
Swap: 999420 0 999420

I guess in my case the root file system filled before I hit the memory limit. My VM has 48Gig and the entire file system is only ~50Gig. Should have run the VM with less memory.

But in general this method also seems to work, but it takes more time.

Another observation, during the Thursday session, when using stress-ng we saw that the oom killer actually terminated the stress-ng program.

If you look at:
heiko@LM20-heiko:~$ cat /proc/sys/vm/oom_kill_allocating_task
0

"If set, the oom-killer kills the task that triggered the out of memory situation, rather than trying to select the best one." That might explain why the stress-ng program was killed.

heiko_s

The lab example 13.1 using stress-ng -m 8 t 10 doesn't work for me at all, even if I increase the numbers.

The following command option, however, works fine:

stress-ng --vm 10 --vm-bytes 100% -t 60s

Jun 27 23:04:35 LM20-heiko kernel: [ 1594.357508] oom-kill:constraint=CONSTRAINT
_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/
user-1000.slice/session-c1.scope,task=stress-ng-vm,pid=4153,uid=1000
Jun 27 23:04:35 LM20-heiko kernel: [ 1594.357525] Out of memory: Killed process
4153 (stress-ng-vm) total-vm:126712kB, anon-rss:72736kB, file-rss:0kB, shmem-rss
:60kB, UID:1000 pgtables:216kB oom_score_adj:1000
Jun 27 23:04:35 LM20-heiko kernel: [ 1594.360546] oom_reaper: reaped process 415
3 (stress-ng-vm), now anon-rss:0kB, file-rss:0kB, shmem-rss:60kB
Jun 27 23:04:35 LM20-heiko stress-ng: memory (MB): total 1984.47, free 97.17, sh
ared 94.01, buffer 0.18, swap 0.00, free swap 0.00

coop

Here's a little C program that we use in some other courses to devour and then release the memory. Just run it with one argument (the memory to eat in MB) or with no argument figures out how much memory you have and uses that.

c8:/tmp>cat lab_wastemem.c

/* simple program to defragment memory, J. Cooperstein 2/04-1/2020
 */

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/sysinfo.h>
#include <signal.h>

#define MB (1024*1024)
#define BS 16           /* will allocate BS*MB at each step */
#define CHUNK (MB*BS)
#define QUIT_TIME 20
void quit_on_timeout(int sig)
{
    printf("\n\nTime expired, quitting\n");
    exit(EXIT_SUCCESS);
}   

int main(int argc, char **argv)
{
    struct sysinfo si;
    int j, m;
    char *c;

    /* get total memory on the system */
    sysinfo(&si);
    m = si.totalram / MB;
    printf("Total System Memory in MB = %d MB\n", m);
    m = (9 * m) / 10;   /* drop 10 percent */
    printf("Using somewhat less: %d MB\n", m);

    if (argc == 2) {
        m = atoi(argv[1]);
        printf("Choosing instead mem = %d MB\n", m);
    }

    signal(SIGALRM, quit_on_timeout);
    printf("Will quite in QUIT_TIME seconds if no normal termination\n");
    alarm(QUIT_TIME);

    for (j = 0; j <= m; j += BS) {
        /* yes we know this is a memory leak, no free,
         * that's the idea!
         */
        c = malloc(CHUNK);
        /* just fill the block with j over and over */
        memset(c, j, CHUNK);
        printf("%8d", j);
        fflush(stdout);
    }
    printf("\n\n    Quitting and releasing memory\n");
    exit(EXIT_SUCCESS);
}

You can compile with "gcc -o lab_wastemem.c lab_wastemem.c" and run with "./lab_wastememc." (or any other name you want.

This started out as a one line program in C (really) but evolved to this.

heiko_s

Thanks coop, will try that too.

luisviveropena

Hi @heiko_s , that's good! Did you try that on CentOS or Ubuntu LST?

Regards,
Luis.

heiko_s

@luisviveropena said:
Hi @heiko_s , that's good! Did you try that on CentOS or Ubuntu LST?

Regards,
Luis.

As to stress-ng, I tried it on Ubuntu 20.04 (actually Linux Mint 20, but it should be the same for all purposes). As I've mentioned, it doesn't allow me to fill the memory. Tried it also after reducing the VM memory. I will give it a go on Centos.

I haven't tried the C program yet.

moulinath

I was also not able to fill it up on CentOS. I will have to revisit this one..

luisviveropena

Hi @moulinath ,

I have a vm guest with 2GB ram, and the closer I was to exhaust memory was using these following commands:

stress-ng --vm 1 --vm-bytes 100% -t 60s

stress-ng --vm 1 --vm-bytes 2g -t 60s

Regards,
Luis.

heiko_s

@heiko_s said:
Regarding the OOM killer, we tried to use stress-ng to fill the memory and get the oom killer to work.

I had suggested to use dd as follows:

heiko@LM20-heiko:~$ dd if=/dev/urandom of=/tmp/bigfile bs=4096
dd: error writing '/tmp/bigfile': No space left on device
6346336+0 records in
6346335+0 records out
25994588160 bytes (26 GB, 24 GiB) copied, 451.941 s, 57.5 MB/s

I must correct myself: dd if=... of=/tmp/... will only work if /tmp is mounted as a tmpfs. Else dd just writes to disk, not memory.

CentOS 8 does not mount /tmp as tmpfs, nor does Ubuntu 20.04. Need to install 7 and 18.04 respectively to check.

However, you can use the dd method with** /dev/shm**:

dd if=/dev/zero of=/dev/shm/manyzeros bs=1G count=10

You don't need to be superuser or King Kong to write to /dev/shm. The above command should do it (unless you got more than 10 Gig memory - so adjust to taste).

You can check if /dev/shm will work:

df -Th | grep /dev/shm

tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm

Bingo.

coop

As an aside, you can turn off mounting tmp as a RAM disk with:

sudo systemctl mask tmp.mount

and then reboot. I always do this. I've only seen this being the default on Fedora (using tmp in RAM) and I personally find it a bad idea. If the drive /tmp is mounted on is an SSD, it is already rather fast and the gains by using RAM do not outweigh the problems if you use /tmp in such a way that might fill memory. (I do, because I download ISO files there sometimes etc.)

luisviveropena

Hi @heiko_s ,

I found a C code that works faster than stress-ng. Perhaps I forgot to disable swap, because I did it on Ubuntu 18 and stress-ng took some time before killing the process, triggering the OOM Killer, and restarting the process (but it worked anyway).

So, I tried the following C code and it worked very fast (it's necessary to disable swap as well):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define PAGE_SZ (1<<12)

int main() {
    int i;
    int gb = 2; // memory to consume in GB

    for (i = 0; i < ((unsigned long)gb<<30)/PAGE_SZ ; ++i) {
        void *m = malloc(PAGE_SZ);
        if (!m)
            break;
        memset(m, 0, 1);
    }
    printf("allocated %lu MB\n", ((unsigned long)i*PAGE_SZ)>>20);
    getchar();
    return 0;
}

---

Then compile it as "gcc memory_exhaust.c -o memory_exhaust", and then run it

Many regards,
Luis.

TemiAworanti

hi, I have a question about this lab... how do you know which process get clobbered first? The /var/log/syslog only displays 2 or 3 lines with less info and dmesg is spewing messages with some processes "DENIED".

luisviveropena

Hi @TemiAworanti ,

The offending process -the one that's consuming more memory, or causing memory exhaustion- will be terminated first.

Many regards,
Luis.