how to automatically kill processes that exhaust memory

I do scientific and numerical programming as a hobby, and it is common for me to write programs that use a huge amount of memory. I also don't use swap because the drop in performance usually just isn't worth the few extra GB to me.

The current behavior on Manjaro is to lock up and just generally behave badly when running out of memory. The servers I use at my job have something that very cleanly terminates processes that cause the server to run out of memory, but I don't know what that is.

I saw the comments here but this didn't offer a clear solution.

Is there some sort of daemon or kernel module that I can use to automatically kill programs that use too much memory? This behavior would be vastly preferable to the system locking up and having to reboot anyway.

There are a couple of daemons (aur):

  • nohang-git
  • oomd & -git

Install, enable reboot and see if they help.

1 Like

I thought that was the default behavior. It seems some factor may prevent the kernel's OOM-killer from triggering, for instance high swap consumption -- which is weird in your case since you don't seem to use swap.

The kernel's OOM handling is pretty terrible. It's there as a "last resort" but can do things like also try to clear GPU RAM... :man_facepalming:

EarlyOOM, oomd, etc. are better intermediate options and less likely to break the system in trying to free some memory.

2 Likes

Guess how we prevent our servers from running out of memory?

We enable SQRT(RAM) of swap space on a Tier-0 SSD to allow the Linux KOOMK¹ to do its work!

Without any kind of swap, it's the developer's responsibility to ensure the machine does not run out of RAM, so guess what happens if you don't? The system will just lock up!

So you have 2 possibilities here:

  1. Don't allocate more RAM than you have available:

    • Don't forget that the total amount of free memory is not equal to the largest chunk of contiguous memory when you do a malloc()
      I.E. Only allocate half of free memory in one go and when half of available is smaller than what you need, your application (but not your system) has run out of memory.
  2. Enable SQRT(RAM) of swap so the KOOMK¹ can do its work...

    • No, it doesn't matter any more if you use a swap file or a swap partition nowadays

Sincerely,

Your SysAdmin²

Note 1: The Kernel Out Of Memory Killer :stuck_out_tongue_winking_eye: :crazy_face:
Note 2: No, I'm not yours specifically (that I know of), but you get the point...

:innocent:

1 Like

It works quite well and it's now in the Arch community repo. I made the mistake of using cmake --build . --parallel $(nproc) and compiling REGoth ate up all 16GB RAM + 4GB swap (rarely ever used). Early OOM saved the day and killed a few applications so I could cancel building. I didn't exactly do the math when I read "expect up to 2GB of RAM usage per core", heh.

3 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Forum kindly sponsored by