[Solved] Display warning message or kill program before system runs out of memory

I have a problem that my system freezes completely when it runs out of memory. When this happens there is nothing to do than holding down the power button. The magic SysRq doesn't work either. This is as described here at the forum (see links at the bottom).

My problem is that the program Shotcut consumes loads of memory, which in return crashes everything, despite having 32 GB of memory on the system. I don't think changing swappiness can help me.

It would be nice to have a little program that:

  • When having X % memory left display a warning message on the screen.
  • When having Y % memory left kill the program consuming most memory.

Does anyone know of such a tool that can be used? Is it an OK idea? I would rather having a program auto-killed than having to do a hard reset of my system.

System spec

[klevstul@silentGamerMjr ~]$ inxi -Fxzc0
System:    Host: silentGamerMjr Kernel: 5.6.15-1-MANJARO x86_64 bits: 64 compiler: gcc v: 10.1.0 Desktop: i3 4.18.1 
           Distro: Manjaro Linux 
Machine:   Type: Desktop Mobo: MSI model: Z87-G45 GAMING (MS-7821) v: 1.0 serial: <filter> UEFI: American Megatrends v: 1.9 
           date: 07/21/2014 
CPU:       Topology: Quad Core model: Intel Core i7-4770S bits: 64 type: MT MCP arch: Haswell rev: 3 L2 cache: 8192 KiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 49631 
           Speed: 1772 MHz min/max: 800/3900 MHz Core speeds (MHz): 1: 1300 2: 1300 3: 1300 4: 1300 5: 1300 6: 1300 7: 1301 
           8: 1302 
Graphics:  Device-1: NVIDIA GP106 [GeForce GTX 1060 6GB] driver: nvidia v: 440.82 bus ID: 01:00.0 
           Display: x11 server: X.Org 1.20.8 driver: nvidia 
           resolution: 1920x1080~60Hz, 1920x1080~60Hz, 1920x1200~60Hz, 2560x1600~60Hz 
           OpenGL: renderer: GeForce GTX 1060 6GB/PCIe/SSE2 v: 4.6.0 NVIDIA 440.82 direct render: Yes 
Audio:     Device-1: Intel 8 Series/C220 Series High Definition Audio vendor: Micro-Star MSI driver: snd_hda_intel v: kernel 
           bus ID: 00:1b.0 
           Device-2: NVIDIA GP106 High Definition Audio driver: snd_hda_intel v: kernel bus ID: 01:00.1 
           Device-3: Logitech G933 Wireless Headset Dongle type: USB driver: hid-generic,snd-usb-audio,usbhid bus ID: 2-1:2 
           Device-4: Audio-Technica Logitech G933 Gaming Wireless Headset type: USB driver: snd-usb-audio bus ID: 2-3.1:5 
           Device-5: NAE R-14PM type: USB driver: hid-generic,snd-usb-audio,usbhid bus ID: 2-9:7 
           Sound Server: ALSA v: k5.6.15-1-MANJARO 
Network:   Device-1: Qualcomm Atheros Killer E220x Gigabit Ethernet vendor: Micro-Star MSI driver: alx v: kernel port: d000 
           bus ID: 03:00.0 
           IF: enp3s0 state: up speed: 100 Mbps duplex: full mac: <filter> 
           IF-ID-1: tun0 state: unknown speed: 10 Mbps duplex: full mac: N/A 
Drives:    Local Storage: total: 5.68 TiB used: -9205357636301939160 
           ID-1: /dev/sda vendor: Intel model: SSDSC2BW240A4 size: 223.57 GiB 
           ID-2: /dev/sdb vendor: Seagate model: ST2000DM001-1ER164 size: 1.82 TiB 
           ID-3: /dev/sdc vendor: Seagate model: ST2000DM001-1ER164 size: 1.82 TiB 
           ID-4: /dev/sdd vendor: Western Digital model: WD2003FZEX-00SRLA0 size: 1.82 TiB 
Partition: ID-1: / size: 184.87 GiB used: 30.94 GiB (16.7%) fs: ext4 dev: /dev/dm-0 
           ID-2: swap-1 size: 34.44 GiB used: 13.2 MiB (0.0%) fs: swap dev: /dev/dm-1 
Sensors:   System Temperatures: cpu: 37.0 C mobo: N/A gpu: nvidia temp: 46 C 
           Fan Speeds (RPM): N/A gpu: nvidia fan: 26% 
Info:      Processes: 260 Uptime: 19h 30m Memory: 31.31 GiB used: 20.44 GiB (65.3%) Init: systemd Compilers: gcc: 10.1.0 
           Shell: bash v: 5.0.17 inxi: 3.0.37 
[klevstul@silentGamerMjr ~]$ cat /sys/fs/cgroup/memory/memory.swappiness
60

Related posts

There are a few AUR packages, I use nohang-git.

There is also oomd, but I have never used it. As to all the fancy bells and whistles, I don't know any.

3 Likes

It happens on my laptop too. I just thought it was getting too old. I'll have to check this out.

Thanks a lot for the tip. I've no installed nohang (nohang-git) and it seems to do the trick! Just what I was looking for.

Side notes

I had 32 GB of swap on my system. I needed to remove/turn off swap ($ sudo swapoff /dev/dm-1) for nohang to work. As long as I used swap files nohang --memtest crashed my system. With swapping disabled it worked just fine:

[klevstul@silentGamerMjr ~]$ nohang --memload
Enter the numbers 899896 to confirm that you are not a robot: 899896
--------------------------------------------------------------------
Warning! The process will consume memory until 40 MiB of memory
(MemAvailable + SwapFree) remain free, and it will be terminated via SIGUSR1
at the end. This may cause the system to freeze and processes to terminate.
Do you want to continue? [No/Yes] Yes
Memory consumption has started!
MemAvailable: 1491 MiB, SwapFree: 0 MiB           
Terminated

On Manjaro the config files are located at:
/usr/share/nohang/ /etc/nohang/ [updated]

I enabled nohang with GUI notifications:

$ sudo systemctl enable nohang-desktop
$ sudo systemctl start nohang-desktop

[klevstul@silentGamerMjr ~]$ cat /sys/fs/cgroup/memory/memory.swappiness
60

I recommend you to low the memory swappiness to a value of 5 or less,to only use swap when you are nearly full in ram usage,also change the vfs_cache_pressure to 50 (100 is the default).

1 Like

Thanks. I tried using those settings. However, as long as I use swapping my system crashes. I need to disable swapping completely for nohang to kick in before a total system freeze. Not sure why but swapping just doesn't seem to work.

Maybe try reducing the swap,I don't know how much do you use but 32 GB is too much IO on the disk,nohang prevent when you are nearly full of ram and swap,but it doesn't prevent freeze by too much IO on the disk by swapping.Maybe try 8 GB of swap and test with nohang.

2 Likes

@Wollie

Hiya. I see that you removed the prepending "[SOLVED]" in front of the title. I've been marking all my solved issues with this tag for the purpose of increasing readability. I searched the forum now and found the following issue:

It seems like some sort of conclusion that "[SOLVED]" is not to be prepended. Am I right? If that is the conclusion I will stop doing this from now on.

OK, I've now used GParted and reduced my swap size to 5 GB. I will retry crashing my system by running out of memory and report back.

I don't see a problem, as long as the solved button is ticked.

1 Like

Brilliant tip. Reducing the swap file (to 5 GB) worked very well. Thank you!

[klevstul@silentGamerMjr ~]$ nohang --memload
Enter the numbers 079661 to confirm that you are not a robot: 079661
--------------------------------------------------------------------
Warning! The process will consume memory until 40 MiB of memory
(MemAvailable + SwapFree) remain free, and it will be terminated via SIGUSR1
at the end. This may cause the system to freeze and processes to terminate.
Do you want to continue? [No/Yes] Yes
Memory consumption has started!
MemAvailable: 95 MiB, SwapFree: 430 MiB              
Terminated
1 Like

If it's not a problem I'll keep using this tag, as I personally like it due to readability.

Does not setting swappiness to such low values (under 10) create the danger of avoiding swap too much and thus creating issues similar to what OP was already experiencing?

1 Like

I would recommend Olive video editor over Shotcut. Olive is light on the resources and very robust for an open source offering.

1 Like

If you don't set swapiness low values (like the 60 by default),you don't avoid it,actually you start swapping early,with a value of 60 you start swapping when 40% of the RAM is full,for me that doesn't make much sense and create innecesary IO on the disk because you have plenty room on the ram before getting out of ram.

I understand how swappiness works.
My point is that values under 10 are considered dangerous to the extent that they can create out of memory situations by avoiding swap too aggressively and thus resulting in 'freezes' etc.

EDIT - but ah, oh, they have 32GB of RAM ... so I guess in that case a rather low percentage can be reasonable. Maybe something as low as ~3% . But I again reiterate that just 'set this to a low number' is not a good default approach. As a rule 10 is as low as you should go unless you have a large amount of memory ... 16GB+

Ohh yes,in that perspective yeah low values can be bad,but maybe for system with 4GB+ vm.swappiness of 5 can be good,otherwise then a value of 10 I suppose.

Thanks for the tip. Never heard about Olive video editor before. I see it's in alpha. I'll try it out but is it stable enough for using it professionally?

1 Like

@RaTiX @cscs

Thanks for this additional information. I have another Manjaro system with 4 GB RAM and will redo this on that system. However, then I will use 10 for swappiness.

Hi! I am nohang author.

I had 32 GB of swap on my system.

Of course, the larger the size of the swap space, the more time is needed to fill it.

As long as I used swap files nohang --memtest crashed my system.

Perhaps the default settings are not suitable for you. nohang can respond to high memory pressure during heavy swapping.
Nohang does not immediately respond to intense swapping, the reaction begins in about a minute.

If the swap space is disabled, nohang responds quickly, as it responds to available memory and does not wait for the swap to fill.

On Manjaro the config files are located at:

/etc/nohang/nohang.conf
/etc/nohang/nohang-desktop.conf

In /usr/share/nohang/ located only default versions of these files.

Here are the defaults that respond to intense swap reactions:

psi_excess_duration = 30
soft_threshold_max_psi  = 40

You can decrease the value of these config keys to make the reaction to intense swapping more aggressive. For example,

psi_excess_duration = 0
soft_threshold_max_psi  = 20
3 Likes

Forum kindly sponsored by