Run out of memory when download big files using Firefox

Dear All,

I am now using Firefox to download a big .ISO file. It seems that Firefox store the the partially download file in memory rather than disk. This file is over 4GB, but sadly my computer memory is only 4GB. I tried several times and the task finished every time when I downloaded 3.2GB of this file. I noticed that when the download task is almost finish, my computer memory is almost used up, and when the task finished, the memory usage suddenly drop down to normal, i.e. around 800MB. So I think this might be the lack of memory.

I tried to google this problem using keywords "Linux, Firefox, download, big file, out of memory", but I got little useful answer. I tried to download in windows, and there was no such problem.

So here is my question:
Is this some kind of Linux feature, or it is just Firefox feature. In other words, will I get the same problem when I use other download method like curl or wget?
How can I avoid this problem, do I need to change system settings or just Firefox settings?

It didn't prompt any error message when the task abort, so I don't know if there's any log I should provide. Please let me know if you need any information.

Thanks a lot!

Could be

  • lack of swap space
  • tmp is in ram rather than disk
2 Likes

The best way is to right click on the link, then select "Save link as" or "Save as" (I don't use Firefox in english). I don't know how you do it, but it looks like raw data of the file fill your memory, it's safer not using left click.

Please post the output of:

mount | grep tmp

so that we can verify that temporary storage is on disk and not memory...

and also post the output of:

free --human.

Please use the blockquotes to post text as per screenshot below:
Screenshot_20200217_192813

Thanks for your remind.@linux-aarhus I think it's because the /tmp dir is in ram. Here are the output of mount | grep tmp and free -h

dev on /dev type devtmpfs (rw,nosuid,relatime,size=1950812k,nr_inodes=487703,mode=755)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=392100k,mode=700,uid=1000,gid=1001)
              total        used        free      shared  buff/cache   available
Mem:          3.7Gi       2.1Gi       155Mi       258Mi       1.5Gi       1.2Gi
Swap:         8.2Gi        48Mi       8.2Gi

@Fabby

And here is the output of df -h

Filesystem      Size  Used Avail Use% Mounted on
dev             1.9G     0  1.9G   0% /dev
run             1.9G  1.2M  1.9G   1% /run
/dev/sda2       450G  182G  245G  43% /
tmpfs           1.9G   95M  1.8G   5% /dev/shm
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           1.9G   62M  1.9G   4% /tmp
/dev/sda1       300M  280K  300M   1% /boot/efi
tmpfs           383M   32K  383M   1% /run/user/1000

It seems like that the tmp dir only has 1.9G space, but I remember that I can download more than 1.9G. :dizzy_face:

I know that mount /tmp on ram can improve performance, but is there a way to mount /tmp on disk temporarily and safely? Or can I manually change Firefox temp file location? (Because in this case right click chose "save as" and left click are the same. @Pouet)

1 Like

tmp is often mounted to tmpfs and it defaults to half system ram.

This matches your statement on system ram.

Maybe - You can comment the line in fstab and thus disable tmp on ram - reboot for the change to take effect.

From the output of mount - it appears you are using systemd-swap and free says you are merely not touching your swap space.

You could try disabling systemd-swap - but - I really have no idea what is going on.


That is kind of weird.

While my system is not comparable - as to memory (no system below 8G) - I have just tested Firefox and transferred an 8GB ISO from my web server to my workstation

2 Likes

Hmm, I doubt it. :thinking:
Could we see :

df -Th
cat /etc/default/grub
cat /etc/fstab
cat /etc/systemd/swap.conf
swapon -s
sudo blkid | grep swap

Problem solved! I checked the /etc/fstab file, but it only contains the information about mounting /,/boot/efi and swap. Then I double checked the wiki, here's some information in tmpfs entry:

Under systemd, /tmp is automatically mounted as a tmpfs even though no entry is specified in /etc/fstab . To disable the automatic mount, mask the tmp.mount systemd unit.

After masking this unit, file will be stored on block device. But one needs to do some further steps to clean /tmpfolder automatically:

Files will no longer be stored in a tmpfs, but on the block device instead. The /tmp contents will now be preserved between reboots, which might not be the desired behavior. To regain the previous behavior and clean the /tmp folder automatically when restarting, consider using tmpfiles.d(5)

As for the swap, I think it might because I changed default swappiness from 60 to 10, so the system rarely use swap. But the system should use swap when it has to, right? I think maybe if I change the size of tmpfs larger than RAM, then it will use swap, since tmpfs doesn't know whether it's using ram or swap. Please let me know if I'm wrong.

That's quite long haha~

$ df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
dev            devtmpfs  1.9G     0  1.9G   0% /dev
run            tmpfs     1.9G  1.3M  1.9G   1% /run
/dev/sda2      ext4      450G  185G  242G  44% /
tmpfs          tmpfs     1.9G   80M  1.8G   5% /dev/shm
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs          tmpfs     1.9G   63M  1.9G   4% /tmp
/dev/sda1      vfat      300M  280K  300M   1% /boot/efi
tmpfs          tmpfs     383M   24K  383M   1% /run/user/1000


$ cat /etc/default/grub  (I leave out the comment lines)
GRUB_DEFAULT=saved
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="quiet apparmor=1 security=apparmor resume=UUID=8d454fe1-5a49-4484-8815-ee90f4e35bab nomce udev.log_priority=3"
GRUB_CMDLINE_LINUX=""
GRUB_SAVEDEFAULT=true
GRUB_PRELOAD_MODULES="part_gpt part_msdos"
GRUB_TERMINAL_INPUT=console
GRUB_GFXMODE=auto
GRUB_GFXPAYLOAD_LINUX=keep
GRUB_DISABLE_RECOVERY=true
GRUB_COLOR_NORMAL="light-gray/black"
GRUB_COLOR_HIGHLIGHT="green/black"
GRUB_THEME="/usr/share/grub/themes/manjaro/theme.txt"

$ cat /etc/fstab
UUID=D219-DFCC                            /boot/efi      vfat    umask=0077 0 2
UUID=3aa0a10c-d428-4c68-9584-174389300213 /              ext4    defaults,noatime 0 1
UUID=8d454fe1-5a49-4484-8815-ee90f4e35bab swap           swap    defaults,noatime 0 2

$ swapon -s
Filename				Type		Size	Used	Priority
/dev/sda3                              	partition	8626272	54608	-2

$ sudo blkid | grep swap
/dev/sda3: UUID="8d454fe1-5a49-4484-8815-ee90f4e35bab" TYPE="swap" PARTUUID="edd6c5c7-da40-4c03-aef3-b07ac0f2d500"

And there's no /etc/systemd/swap.conf file.

So my doubts were right.
You don't use systemd-swap but have a partition.
strange thing : df doesn't show sda3

lsblk -f should tell it all.

Do you hibernate ?

1 Like

It's not because of the swappiness? Sure, I use hibernate. But when it comes to hibernate, it seems that using hibernate and hibrid-sleep, the recover speed is the same, both are slow, even if I didn't cut out power supply.

Here's the output of lsblk -f

NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda                                                                           
├─sda1 vfat   FAT32       D219-DFCC                             299.1M     0% /boot/efi
├─sda2 ext4   1.0         3aa0a10c-d428-4c68-9584-174389300213  241.8G    41% /
└─sda3 swap   1           8d454fe1-5a49-4484-8815-ee90f4e35bab                [SWAP]
sr0         

Normally, does df show swap?

If you hibernate, you cannot use systemd-swap.
I never grasped the utility of hibernation ^^

Low swappiness maximizes the use of RAM and that is good, no reason it creates a problem like yours.

What are your Firefox extensions ?

grep -oP '},"name":"\K[^"]+' ~/.mozilla/firefox/*.*default*/addons.json

Sorry I don't understand, what do you mean if I hibernate, I cannot use systemd-swap? Can I use systemd-swap without changing any setting just don't hibernate? Or what should I do to forbid hibernate and use systemd-swap? Because now I think the resume speed is too slow so I would rather shut it down.

Here's my addons

To Google Translate
OneTab
User-Agent Switcher and Manager
English (US) Language Pack
English United States Dictionary
OneNote Web Clipper
Vim Vixen

Once again, I never grasped which interest hibernation had.
Yes, systemd-swap doesn't cope with it.

Here is what I do (click on the english flag to have it) : https://dolys.fr/forums/topic/optimiser-gestion-ram-swap-linux/

The advantages of hibernate are:

  • I'm sitting on the train and hear my station being called: Push Fn+F7, slam the lid shut, take the metro, push the power button and I'm right back where I was without using any power.
  • I've left my laptop unplugged from power and someone asks me to "just have a 5 minute chat" which ends up being 1 hour. I come back to my laptop, plug in the power and I'm right back where I was.

Especially in the last scenario, much better than without hibernation::

  • hard shutdown
  • EXT4 and NTFS journals being rolled back
  • I've just lost even my last autosaves...

:smiling_imp: :innocent: :grin:

P.S. Vrai en 2016 mais aujourd'hui avec les paramètres TRIM qui existent... Plus vraiment. :wink:

1 Like

Bref, aucun intérêt (surtout comparé aux avantages de systemd-swap :stuck_out_tongue: )

---> for those who didn't graps this post : https://www.deepl.com/translator (also available as browser add-on)

1 Like

I guess it depends if "Always ask..." is checked in the download setting.
I don't understand how /tmp is used, every time i download a file, there's a temporary file with .part extension next to the file to download, in the same directory.

You might try to disable memory cache in about:config by setting browser.cache.memory.enable to false (not sure if it's used for downloads).

By the way, you can get rid of vim vixen, why do you need a M$ add-on ? (OneNote)
You can add ublock origin !!!

Because I'm using OneNote among different machines. :joy:

Thanks guys! Based on your descriptions and disscussions, I think I know what to do. :laughing:

Cheers!

Forum kindly sponsored by