NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
RN3312 BTRFS operations are completely hung
We have owned the RN3312 for a bit over 6 months, and all was seemingly fine. However, things went downhill recently and now pretty much the entire BTRFS partition is completely unusable at this point. Even leaving the NAS offline and just trying to do whatever internal metadata cleanup by itself in a reasonable time is not enough to recover.
What has happened is a combination of the Bit Rot Protection / COW + Compression + Snapshots being turned on, on a partition used for file backups, and image backups (Veeam) for a single, large, fileserver. BTRFS is NOT production ready for such a setup, I firmly believe this option should be removed from the UI, or a huge warning displayed.
Everything was going great until the first snapshots needed to be deleted, where I ran into the problem of btrfs-cleaner taking up 100% CPU. Symptoms: the admin UI would lock up on any file operation in certain directories. Directory accesses would hang forever, even over SMB. Of course all the backups to the NAS were timing out. I eventually was able to delete the snapshots by hard rebooting the system and removing them before btrfs-cleaner got too bad.
But now, I have the problem where btrfs-transacti is taking up 100% CPU. I have left the system sitting offline for a week just spinning at 100% CPU (!), and there is no visible improvement - EVERY BTRFS operation still hangs, no matter what I try. There is little disk activity, it is not thrashing - makes me think there is something wrong in the internals of BTRFS, or that the CPU is too underpowered to handle the amount of storage metadata operations.
top - 12:30:29 up 1:39, 2 users, load average: 115.21, 112.48, 99.52
Tasks: 334 total, 2 running, 332 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.6 us, 23.0 sy, 0.0 ni, 72.1 id, 1.6 wa, 0.0 hi, 2.7 si, 0.0 st
KiB Mem: 8113792 total, 2673896 used, 5439896 free, 4404 buffers
KiB Swap: 2093052 total, 0 used, 2093052 free. 1980036 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3740 root 20 0 0 0 0 R 100.0 0.0 93:52.62 btrfs-tran+
1 root 20 0 136632 6868 5144 S 0.0 0.1 0:02.45 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreaddadmin@archive:/data$ iostat
Linux 4.4.68.x86_64.1 (archive) 07/11/2017 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.55 0.00 25.73 1.56 0.00 72.15
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 3.26 55.82 57.19 337578 345828
sdb 3.25 55.07 57.08 333044 345196
sdc 3.45 74.32 57.02 449448 344808
sdd 3.28 53.95 57.21 326224 345936
sde 3.26 57.73 57.09 349104 345252
sdf 3.28 54.89 56.87 331951 343908
md0 1.68 27.57 39.28 166740 237520
md1 0.02 0.19 0.00 1172 0
md127 9.38 243.50 69.42 1472516 419788 < not a lot of activity...I have tried starting a balance to fix fragmentation, I believe there are operations blocking it inside the kernel, but even at -dusage=0 I gave up after giving it the weekend to do its thing. Trying to look for evidence is fragmented files is horrendously slow. But it is very bad now:
admin@archive:/data$ ls *** hangs forever ***
My hope at this point is to try and mount the system read-only and recover data onto a USB drive, the share with data is around 8 TB which might just fit after a couple of days/weeks? of copying... Then figuring out some way to drop the share? and rebuild it without selecting the 'Bit Rot Protection' or 'Compression' options. Hopefully I don't have to resort to copying the NAS to something else and wiping it - there is about 14 TB of data on it currently, and I don't have that much capacity available anywhere else...
After going through this and after lots of research, I see lots of horror stories showing that BTRFS is extremely fragile and not ready for prime time. I believe it is reckless for Netgear to base a NAS on such an unproven FS. The features are not worth it if they explode in spectacular fashion after a couple of months.
- Symptoms include btrfs-transacti and btrfs-endio-wri taking up a lot of CPU time (in spikes, possibly triggered by syncs). You can use filefrag to locate heavily fragmented files (may not work correctly with compression).
...
- "a balance on 2TB of data that was heavily snapshotted - it took 3 months"
- "when I have to do balances ... I delete all the snapshots and allow a few months for the balance to finish"
https://btrfs.wiki.kernel.org/index.php/Gotchas
We are running version 6.7.4. We currently have 6 x 8 TB in X-RAID (certified drives.) I struggle to think what would happen if we filled up all 12 slots...
Are there any other operations anyone from support wants to try before I start wiping it? Unfortunately our 90-day free support has expired before any of this happened, so I am left venting in public...
4 Replies
Replies have been turned off for this discussion
kevinb_vr wrote:What has happened is a combination of the Bit Rot Protection / COW + Compression + Snapshots being turned on, on a partition used for file backups, and image backups (Veeam) for a single, large, fileserver. BTRFS is NOT production ready for such a setup, I firmly believe this option should be removed from the UI, or a huge warning displayed.
Veeam already provides block level versioning. It's counter productive to take snapshots on Veeam backups. Veeam incremental images are compressed, so BTRFS Compression is most likely also unnecessary. Bit Rot Protection implies fragmentation, so running defrag + balance on schedule is advisable.
If btrfs-cleaner is currently taking 100% (a load average of 115.21 is huge), I don't think it's a good idea to try start BTRFS balance or defrag on top of that. Power-cycling (in normal mode or read-only mode via the Boot Menu) risks BTRFS corruption, but you may have no other choice at some point. What do you see in dmesg?
If the following command gives an output (logged in as root), please paste it here (replace "data" by the name of your volume if different):
btrfs fi us /data
BTRFS has been hugely improved during the last few years. And there were many "bug" fixes and implementation improvements with new firmware version on the ReadyNAS. I don't think half of the old threads you've read through are still applicable. I remember about the introduction of BTRFS quotas on 6.4.0 impacting heavily fragmented volumes, with some improvements on 6.4.1. There was also some other firmwares where the BTRFS balance was longer than expected.
In general, I tend to recommend disabling the BTRFS quotas at the volume level (in the settings of the volume) if you notice slowness in this type of tasks.
After doing some assessment on the metadata allocation, etc., you have a few options, which are mostly available after a power cycle (at the risk of BTRFS corruption). Booting the NAS in Volume Read-Only mode via the Boot Menu could allow you to perform a full backup of the data and update the F/W to 6.7.5. After that, you can try to reboot in normal mode, immediately disable the quotas in the settings of the volume, etc. But let's check dmesg and btrfs fi us /data first.
You can also do paid Support with NETGEAR, with a Per Incident contract or a yearly contract.
BTRFS is mostly misunderstood and misused.
I think that Bit Rot Protection, Compression and Snapshots are disabled by default when creating a new share (at least on recent firmwares). Before "checking all the boxes and starting everything", one would be expected to do some research to understand what is the feature and its implications/limitations. The biggest issue with ReadyNAS and BTRFS is that this type of information is not very easy to access. I do think that the documentation could be much better and the implications of each feature explained. It shouldn't be expected from users to be experts in every software components used on the NAS.
https://kb.netgear.com/26091/Bit-rot-Protection-and-Copy-on-Write-COW-in-depth
I couldn't find a single NETGEAR KB article explaining about huge metadata allocation, and you can't see it from the GUI either.
It's also not possible to control the level the balance runs at from the GUI (for example to run a balance on empty metadata chunks).
kevinb_vr wrote:
We have owned the RN3312 for a bit over 6 months
I struggle to think what would happen if we filled up all 12 slots...The correct product name of your ReadyNAS is RR3312.
Hi, thanks for your reply!
The biggest problem I guess I see is there is no way to go 'back' after creating the volume, and then it getting into a state where it is pretty messed up. In the ReadyNAS admin screen, the options to remove compression or bit rot protection don't do anything, as the admin web UI just hangs spinning forever as it is waiting for something in the background.
I consider myself somewhat experienced with Linux administration (though maybe not so much with btrfs...) this was one of the reasons I liked the ReadyNAS line so much, a nice UI frontend plus solid Linux internals! But I wouldn't expect an average NAS user to be able to do this level of troubleshooting to try and get things going smoothly again. I also think it is not fair to the support team or to the users to require a support contract to fix these sorts of issues... that's why I lean on some sort of a UI or documentation fix to prevent them in the first place...
This share is used for both File Level backups (using BackupAssist, which makes rsnapshot-looking backup directories, and uses NTFS hard links?) and also for block/incremental backups using Veeam, both backing up a single fileserver. So this is why the Compression was turned on originally, I admit it would probably have been better to setup a separate share instead of just a separate directory once Veeam needed a backup target to use...
After checking dmesg, looks pretty clean, except I guess it might have been attempting to drop an old snapshot at boot still? There are only some hung-task notifications which seem to have resolved themselves... See full dmesg log here: https://pastebin.com/9mbkZe67
btrfs-transacti did keep going and going, however around 8 hours later it looks like it finally finished whatever it was doing after over 500 load avg. But then readynasd got unblocked and is now at 100%, locking up the UI and not allowing web access (sigh...) so I am going to try rebooting and see if I can get a clean startup.
root@archive:/data# btrfs fi us /data Overall: Device size: 36.36TiB Device allocated: 15.24TiB Device unallocated: 21.13TiB Device missing: 0.00B Used: 14.52TiB Free (estimated): 21.71TiB (min: 11.15TiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 232.53MiB) Data,single: Size:14.95TiB, Used:14.37TiB /dev/md127 14.95TiB Metadata,DUP: Size:145.50GiB, Used:80.66GiB /dev/md127 291.00GiB System,DUP: Size:8.00MiB, Used:1.97MiB /dev/md127 16.00MiB Unallocated: /dev/md127 21.13TiBroot@archive:/data# btrfs fi df /data Data, single: total=14.95TiB, used=14.37TiB System, DUP: total=8.00MiB, used=1.97MiB Metadata, DUP: total=145.50GiB, used=80.66GiB ** maybe 80 GB is OK for 14 TB of space used? ** GlobalReserve, single: total=512.00MiB, used=232.53MiB
root@archive:/data# btrfs fi usage /data Overall: Device size: 36.36TiB Device allocated: 15.24TiB Device unallocated: 21.13TiB Device missing: 0.00B Used: 14.52TiB Free (estimated): 21.71TiB (min: 11.15TiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 3.00MiB) Data,single: Size:14.95TiB, Used:14.37TiB /dev/md127 14.95TiB Metadata,DUP: Size:145.50GiB, Used:80.66GiB /dev/md127 291.00GiB System,DUP: Size:8.00MiB, Used:1.97MiB /dev/md127 16.00MiB Unallocated: /dev/md127 21.13TiBThe other thing is, I don't exactly know what readynasd is looking for itself - is it OK to disable quotas by just going ahead and running 'btrfs quota disable /data'? and not mess up the ReadyNAS UI? or should I try harder to go through the UI?
Is a reboot performed with the command 'reboot' from SSH or over serial? Would you happen to know if there is any output after [ OK ] Reached target Shutdown. ? Has it unmounted all the filesystems yet or should I just let it keep trying? This is via the serial console btw. Not sure if it is a real hang, systemd quirk, or is just busy trying to perform that last 'sync' before unmounting.
Thanks for your help, I really appreciate it!
- (Sorry for the quick reply.)
(used) 80GB of Metadata is huge! And it's heavily unbalanced, 145GB allocated. In DUP mode, that gives 291GB of Metadata on the volume.
In comparison, on my 1.81TiB backup volume containing 1.20TiB of data, the metadata used is 368.75MiB (but it is low).
Huge Metadata, heavy fragmentation, many snapshots, BTRFS quotas enabled on the volume, that's why the NAS can't keep up.
You can disable the quotas with the command for now and do it from the GUI later (to update readynasd database).
'reboot' is not clean. Use 'rn_shutdown' for graceful shutdown or 'rn_shutdown -r' for graceful reboot. I'm not sure that you will be able to shutdown/reboot with a system command if the system is in that state though.All right, seems to be becoming more manageable. I was able to disable quotas through the GUI on the volume, and then also compression, and CoW on the share after a reboot last night and letting it sit for a while, and today I have been balancing the metadata and data to try and clean up some of the usage.
The 'reboot' command does work after a long while, it just must take a bit to unmount the partition. At that point rn_shutdown didn't do anything, as readynasd was not accepting commands (or web ui interaction...)
Balancing the metadata brought total metadata usage down almost 100 GB... maybe still high at 52 GB but it is much better. Running:
root@archive:~# for u in `seq 30 10 80`; do echo $u; btrfs balance start -musage=$u /data; done
root@archive:/data# btrfs fi df /data Data, single: total=14.60TiB, used=12.38TiB System, DUP: total=32.00MiB, used=1.97MiB Metadata, DUP: total=58.00GiB, used=52.17GiB GlobalReserve, single: total=512.00MiB, used=0.00B
Then the data balancing is still running... might just let that run in the background and start re-enabling backups.
Thanks again...
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
