[Solved] ZFS makes my system hang randomly after some hours

Have you run a scrub recently? What's the pool health like?

zpool status -v
  pool: tank
 state: ONLINE
  scan: resilvered 11,8T in 13h51m with 0 errors on Wed Jan  2 02:00:28 2019
config:

	NAME                                              STATE     READ WRITE CKSUM
	tank                                              ONLINE       0     0     0
	  raidz2-0                                        ONLINE       0     0     0
	    sdx_crypt10                                   ONLINE       0     0     0
	    sdx_crypt11                                   ONLINE       0     0     0
	    sdx_crypt7                                    ONLINE       0     0     0
	    sdx_crypt6                                    ONLINE       0     0     0
	cache
	  nvme-Samsung_SSD_960_EVO_500GB_S3EUNX0J215828E  ONLINE       0     0     0

errors: No known data errors

I will run another scrub now and see if it does anything. I fear though this will take for ages with the current system lockup behavior.

1 Like

Probably also worth checking drive health with smartctl.

Recently one disk of one of my three-disk RAID0 ZFS arrays showed platter seek errors which triggered a system hang during access but not during a scrub.

They are all healty, no end-to-end errors, no reallocated sectors.

1 Like

The last thing to check before diving into ZFS itself will be to see if the recent 0.7.13 release fixes the issue.

I'll have a look at updating the Manjaro packages, unless Phil gets to them first (though they will be in unstable first anyway).

1 Like

Good to hear a new version is around the corner. Always a good shot.
I wonder though. I am on 4.19 kernel as this is the "recommended" kernel. Would you suggest me switching to 4.20? Could this help?

Not likely to make any difference with ZFS as it's a "third-party" kernel module.

But - it may still be worth trying.

Also, check whether you're still using noop/none as the disk I/O scheduler; ZFS should set it automatically for pools but it's worth making sure.

1 Like

How do I do that?

Is this the right way?

[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/sda/queue/scheduler 
[mq-deadline] kyber bfq bfq-mq none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/sdb/queue/scheduler 
[mq-deadline] kyber bfq bfq-mq none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/sdc/queue/scheduler 
[mq-deadline] kyber bfq bfq-mq none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/sdd/queue/scheduler 
[mq-deadline] kyber bfq bfq-mq none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/sde/queue/scheduler 
[mq-deadline] kyber bfq bfq-mq none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/dm-0/queue/scheduler 
none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/dm-1/queue/scheduler 
none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/dm-2/queue/scheduler 
none
[I-KNOW-YOU wwksxbpo.default]# cat /sys/block/dm-3/queue/scheduler 
none

I use luks encryption so I think dm-0/1/2/3 should be the disks used for zfs. But the underlying physical disks have [mq-deadline]

1 Like

I think you could be onto something:

https://blog.mthode.org/posts/2014/Feb/zfs-performance-on-luks/

Also talks about the scheduler not to be correctly set on luks devices.
So I will try to figure out how to set it to "none" for the physical devices and hope the best.

1 Like

https://wiki.archlinux.org/index.php/improving_performance#Changing_I.2FO_scheduler

1 Like

Thanks for the link!

Ok, I created /etc/udev/rules.d/70-ioschedulers.rules

And in there I added my 4 drives like this:

ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_SERIAL_SHORT}=="ZA21RL4P", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_SERIAL_SHORT}=="ZA20SV9S", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_SERIAL_SHORT}=="ZA2242TX", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_SERIAL_SHORT}=="ZA2246CJ", ATTR{queue/scheduler}="none"

I got the ID_SERIAL_SHORT by running:

udevadm info -n /dev/sde | grep ID_SERIAL_SHORT

I will report back, if this has helped or not.
Thanks for the great support until now Jonathon!
Much appreciated.

UPDATE: Previously I forgot the last comma in each line and I also had to rename the file to 70-ioschedulers.rules instead of 60-ioschedulers.rulse. I assume that the ENV parameter is not available at 60. I am not sure if 70 is the most appropriate value, but it works for me.

2 Likes

Looking good so far. Fingers crossed.

Scrub finished uninterrupted. Returned no errors again.

I am using it since I changed the scheduler the same way that caused the system to crush after a few hours quite reliably. And nothing so far.

Surely too early to call. And I bet it crashes right in the moment I press "Reply". But my hope is up :slight_smile:

1 Like

Did it crash yet? :smile:

1 Like

Haha. I had to wait two minutes to be sure. But not so far \o/

1 Like

It crashed again, but most likely due to my stupidity. I did not really test my udev rules and they did not work properly after reboot. I updated my post about it accordingly. So now I have to wait again and see, if it is really stable.

1 Like

No crash anymore so far. I consider this as solved. Thanks!!

1 Like

The issue is back, crushing my PC once per day :frowning_face:

I had a few random crashes when I tried to use Bluetooth in all this time since I fixed it, but now it seems to crash while I play "Darkest Dungeon". I only do this and listen to youtube. Beside that there were some Manjaro updates but not directly related to ZFS.

Darkest Dungeon saves quite aggressively to my home which is on the ZFS. I will now try moving this data to by root filesystem to see if this goes away. But this really sucks...

And yes, I checked the disk schedulers are still at none

Crashes involving Bluetooth and a game don't sound related to ZFS...

Unless you've enabled dedup or have a very large ARC with little RAM, disk I/O won't be an issue - unless your disks are generating hardware errors. I've seen systems taken down (or partially depending on which process segfaults) by a page fault when trying to write to swap to a bad block/failing disk.

It could even be bad cabling or EM interference within the case, e.g. a poorly-shielded USB3 port causing issues with SATA.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Forum kindly sponsored by