Decrease dirty bytes for more reliable USB transfer

I had an issue where transferring files to (not sure if same happens to from) USB drives would hang, and often not complete. The transfer/completion rates were also falsely reported.

A common solution to this is adjusting the value of dirty_bytes (whatever that is) by running this command on startup:

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

Which I am currently doing and can confirm has solved the issue for me. The default settings for these variables(?) is generally considered inadequate for modern 64-bit operating systems.

Related articles:


This is a quite serious issue and as a by-product introduces many other isseus with USB drives on linux (like incomplete file transfers, and usb devices 'hanging' so you cannot eject them, and an inability to cancel running file transfers, or rather, it takes a very long time after you hit cancel for it to close), there are even reported cases of this issue slowing/freezing the entire system I'm surprised it hasn't been fixed by now. Especially considering how easy it is to fix, and how old it is.

Update: Further clarification after further research.

The reason the default settings are considered inadequate for modern 64-bit Operating Systems has to do with the amounts of RAM they typically have, when these settings were created for the kernel, they were only created with 8-64MB of RAM in mind, and seems to operate half-decently up to 1GB of RAM.

These settings, when misconfigured on 64-bit operating systems with large amounts of memory (Confirmed for 12GB+, presumed to affect 2GB+, further testing needed) can cause more serious issues than the aesthetic ones that I mentioned earlier, there have been reports of system-hangs related to this issue. This issue does not seem to affect Manjaro, so it has presumably be fixed in the scheduler we use (BFQ) but it still mis-reports the file transfer completion rate, as well as the transfer speeds of the transfer in question, which, while it seemingly doesn't impede function, is still worth looking more deeply into. Such issues are generally unacceptable for a modern consumer that happens to use USB Drives for any reason.

It is likely that adjustments to the scheduler configuration could be an alternative solution to fix the issue.

There is also a possibility that setting vm_highmem_is_dirtyable to 1 would solve the issue. It was a suggested solution to make dirty memory behave the same way on 64-bit systems as on 32-bit ones, which seemingly has not been enabled by default.

In 2013, Linus Torvalds said this about the issue:

Right. The percentage notion really goes back to the days when we
typically had 8-64 megabytes of memory So if you had a 8MB machine
you wouldn't want to have more than one megabyte of dirty data, but if
you were "Mr Moneybags" and could afford 64MB, you might want to have
up to 8MB dirty!!

Things have changed.

So I would suggest we change the defaults. Or pwehaps make the rule be
that "the ratio numbers are 'ratio of memory up to 1GB'", to make the
semantics similar across 32-bit HIGHMEM machines and 64-bit machines.

The modern way of expressing the dirty limits are to give the actual
absolute byte amounts, but we default to the legacy ratio mode..

Linus

Update 2: Out of concerns for possible performance impact of changing these settings, I did some simple benchmarking in a below post, there are more benchmarks if you scroll further down.

There seems to be a real possibility that contrary to my fears, adjusting this setting can improve drive (write) performance noticably, even on fast drives.

8 Likes

Thanks for this. I wasn't getting any freeze or incomplete transfers, but I did notice USB transfers would seem to hang for a while at 100% before completing. That doesn't happen with the settings you suggested.

What is the default? Why is it inadequate? Who is generally considering this? [citation needed]

Kernels change over time, so reading an article from 2013 and an SO question from 2015 won't necessarily be representative of the current state of the kernel. For example, is this behaviour limited to certain kernels?


Reading the LWN article and linked mailing list posts I suspect setting a byte-based limit is not the correct way (which value do you pick?), and instead should look at a kernel config option:

If this is already active then there should already be a reasonable default value in place.

3 Likes

I don't know if vm_highmem_is_dirtyable is set (I am pretty sure the default for that one is 0 from a google search...) and I don't know how to check.

But it is very clear from my post, and the next post after, that the default values are not reasonable as is considering the issues with large file transfers to slow media (usbs) with manjaro's current defaults.

The default settings are:

vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 1500
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 1500
vm.dirtytime_expire_seconds = 43200

And here's a guide for how to optimize it for specific scenarios. Judging by that, a good configuration would be setting dirty_ratio to 80 and dirty_background_ratio to 5, to optimize for both. (the ratios deal in percentages whereas bytes deal in hard values. I believe the values I set in the OP were 16MB and 48MB respectively, which is better I'm not sure, but I'm leaning towards the hard values rather than percentages.)

I believe that the problem is that because I have a large amount of memory, 10% and 20% are relatively large values. On a system with 4GB of RAM, 10% would be 400MB, so the values would be much more sane, 400MB & 800MB respectively, it's still too high, but might work. On my 16GB computer though, it fluctuates between 1.6GB and 3.2GB, which are multiple times larger values. Now maybe setting vm_highmem_is_dirtyable to 1 could fix this whole thing but I have no idea how to do that so I can't test it.

On the other hand, setting a hard value of say 100MB and 200MB (close to the values we'd get for 1GB memory) might be a good middle ground. Which is exactly what Linus Torvalds suggested where you quoted him.

The default settings are clearly optimized for 32-bit systems with 1GB RAM, the system requirements for Manjaro are higher than that, meaning everyone using it is likely to have this problem.

vm_highmem_is_dirtyable seems like a workaround that makes it read everything above 1GB ram as if it were 1GB ram (in other words, to give us 10% and 20% of 1GB of ram, e.g. 100MB and 200MB as a hard limit) and since we can assume that every user will have above 1GB of RAM on manjaro, it seems like a safe bet to set the default settings like so:

echo 107374182 > /proc/sys/vm/dirty_background_bytes
echo 214748365 > /proc/sys/vm/dirty_bytes

Seems like a suitable solution. This is still a sizable cache, without being too big. So I decided to test it, and what I found, disappoitningly, is that this does not solve the issue. It's better, it is MUCH better than the default, for sure, but this is still large enough so that the slow file transfer to USB is falsely reported as fast for a few seconds before the speed starts correcting itself.

I have found the most acceptable compromise between the extreme low suggested in the op, and the extreme limit of 100/200mb is to set them like this:

echo 50331648 > /proc/sys/vm/dirty_background_bytes
echo 107374182 > /proc/sys/vm/dirty_bytes

Setting the values to 48 and 100mb.

Essentially, what htis means is that the file transfer function in the file manager will falsely report ~80MB of the transfer as finished before it actually is, however such a small amount is not quite an unbearable wait and if I recall correctly, windows has similar issues where you need to wait a few seconds after a large file is transferred to usb (so it's not just linux).

For a USB drive with ~5MB/s transfer rate, this means a 10 second wait after a large transfer is finished. Personally, I think the values suggested in the OP are completely reasonable and I will be sticking to them. The cache has no need to be bigger than this as far as I am aware, and the settings in the OP give me mostly accurate reports for file transfers to slow mediums.

I imagine some testing might be appropriate to see what kind of effects setting these values to what's in the op vs what's default might be smart to see what kind of performance effects it has on SSD and HDD/internal volume performance. I have noticed no difference personally, but maybe it is just because I haven't been looking.

3 Likes

I also have another update, according to this github post

This issue is caused by bugs in the I/O scheduler. Not sure if tweaking the I/O scheduler as suggested is worth it for this though.

Perhaps something worth looking into.

Instead of setting the dirty bytes, it's IMHO enough to specify the ratio alone.

Personally I use vm.dirty_ratio=10 and vm.dirty_background_ratio=5, i.e. half the default.
Keep in mind that size of available RAM plays an important role, so you can't just use a specific one-size-fits-em-all default.

Shouldn't be made default though (yet).

As I explained in an above post, these ratios are nowhere near low enough. Even if it's just 1% and 2% it is still too high for people with 16+GB of RAM. It would probably be adequate for 8GB though (it would be tolerable at least)

I have 32 GB of RAM, and the ratio settings above are just fine here.
I do have fast NVMe storage though.

I see, so did you properly plug in one of those slow USB2 Drives, and transfer a 200+MB file to it?

When you do, does your file manager correctly report the transfer rates? And when the file manager shows the file fully transferred, does it not hang for a few seconds?

What happens if you do this with a 1GB file?

Can you clarify 'what' this important role of available RAM for writeback cache is?

Most of my USB sticks are USB3 (though still terribly slow).
I'll have to check it again, but I don't remember any hangs.

The dirty ratios are depending on available free RAM. If you have 6 GB of available free RAM and a dirty ratio of 10, a maximum of 600 MB (10% of 6 GB) can be dirty before being written out to disk.

I admit though that I'm not sure anymore, it's been some time I investigated VM settings. It's slightly more complicated than I described above.

The process is best checked when copying a file and checking the output of watch -n1 cat /proc/meminfo

You explained what the cache is, I was asking you (to fully clarify) what 'important role' this cache serves where it needs to have several gigabytes of my RAM available to it, and why it can't use the small amounts suggested in the OP? What would be the possible negative side-effects of it?

I'm not at all criticising your OP, in fact I think it's very interesting and reasonable.
I was just thinking that there isn't a single value that works on all configurations, as some of these settings depend on the amount of RAM.

No, I didn't explain cache, but dirty ratio, albeit my explanation is not 100% correct.
Again, I have to look it up again in my notes.
When I find something, I'll post it here.

EDIT: Here's some very basic info from Red Hat:

@cestarian, I've now read everything again, and your proposals are good IMHO.
I think we're trying to achieve the same thing in a different way.

We can either use dirty_bytes, like you did, and specify a concrete value for that, or alternatively, we use dirty_ratio, which does the same thing but uses a ratio in regard to available RAM to get the number of bytes.
Both are mutually exclusive, if ratio is set, bytes will be ignored and vice-versa.

In any way, thank you for reporting this, it's an interesting topic that needs more discussion.
Your solution is IMHO appropriate for the majority of users.

The problem with those VM settings is that they concern every kind of storage, not only slow USB sticks, and perhaps they are not really appropriate for fast storage.

Would you be interested in running some benchmarks?
I can help you with that.
Cheers.

Is this the same across all kernels?

Changing default values has to be done very carefully as what works for you at the moment with your hardware can introduce issues for others.

I'm not saying we shouldn't make a change here - I want to make certain that it's the right change.

2 Likes

I would assume that it is the same across all kernels, at least the ratios. But since that's just an assumption, to check you just run this command: sysctl -a | grep dirty, this will show you all the related settings for the system in question.

I agree that testing is important, which is why I did a lot of it earlier to see how different values affected the issue at hand, and at least on my hardware in kernel 4.18 (stable branch), any values higher than what was suggested in the OP leads to issues, the higher the values, the greater the issues, I observed a similar behavior on my laptop which has 12GB of RAM as well, but I have not tested it on 8, 4 or 2GB RAM setups where it might behave differently, I dont think there's a need to test below that, because while the officially recommended specs for manjaro only include 1GB of RAM, that seems to be outdated information, as even right out the gate I am using just under 1GB of RAM (>970MB) with no applications besides the default (+ keyboard drivers/ckb-next) running. Therefore it seems like manjaro should update that data to 2GB RAM, of course, you can still run the system on swap, but these are 'recommended' specs. Personally I go over 2GB quite fast just by using a browser, but it might be sufficient with another (browser (not firefox or chromium)) among with some other tweaks... Nobody on modern hardware has less than 4GB anyways, I wouldn't recommend manjaro to anyone with less than 4GB personally.

Back to the point however, I intend to do some benchmarking to see the differences, I only have a so fast SSD however, it's just a regular SATA one with ~500MBs Read/Write limits. So someone else will be needed to test the even faster types of storage. It should be worth mentioning however that my SSD had no problems with the default settings, only my slow USB2 Drive, it's safe to assume that USB3 Drives which have several times faster rates than USB2 ones, can handle higher dirty cache settings better. Since a lot of USB2 drives are still in circulation and they are commonly used, I feel that at least for now, they need to behave reasonably. It's OK, if large transfers need 1-5 extra seconds to complete, but anything beyond that is a bit much.

I should also note that functionally speaking, higher values do not ACTUALLY affect the transfer rates to the SSD, the transfer speeds are always roughly the same for this USB2 device no matter the settings, it's mostly a matter of how the I/O Scheduler reports it (at least that's what I think is happening). But the previous reports of system hangs, and freezeups, when transferring files to slow media (which I have not personally accounted, but not tested for either) when these settings are misconfigured (by misconfigured I'm pretty sure it means set too high) should give us plenty of cause for investigating this carefully.

The only benchmarking I know about and have access to on linux for drive read/writes however is using dd, @anon23612428 if you know any more advanced testing suites I could use it would be appreciated.

Edit: Further update, there's a third option called ratelimit_pages that is related to the issue and will most likely need to be adjusted accordingly based on what settings we use for dirty bytes and dirty background bytes. I cannot find this setting however.

I also made some updates to the OP.

I think you mean decrease dirty bytes, not increase. The ratio method sets a very high dirty bytes number on 1+ GB systems.

I noticed this huge dirty bytes issues in other ways when I started using linux. And I agree, the kernel defaults are not a good setting for modern systems.

"Ratio of memory up to 1GB" is a really good idea. Why was that patch never put into the kernel? Linus is a smart guy! :smile_cat:

1 Like

Good catch :smiley: I changed it.

And yes, why indeed wasn't it... Did someone forget? or was there an actual reason...

1 Like

This is the comment to the patch by Jan Kara. I think it ended there.

There will always be people who have loads which benefit from setting dirty
limits high but I agree they are minority. The reason why we left the
limits at what they are now despite them having less and less sence is that
we didn't want to break user expectations. If we cap the dirty limits as
you suggest, I bet we'll get some user complaints and "don't break users"
policy thus tells me we shouldn't do such changes :wink:

So the patch wasn't applied because it would change the expected behavior of systems. With the acknowledgement that it's not a good setting for most people today.

Might be worth spinning up a custom kernel with this patch applied for some testing.....

1 Like

Really it's up to you do do the legwork seeing as you've made the proposal.

Forum kindly sponsored by