NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade
Since I upgraded my RN102 to 6.4.0 last night I frequently have trouble reaching the admin interface, and my laptop has reported not being able to reach Time Machine.
When I login by ssh and use "top", I can see that btrfs-cleaner is stuck at 100% CPU. This can last up to 10 minutes at a time. I only installed about 12h ago and the btrfs-cleaner process already has over 85 minutes of CPU time, that's more than 20 times more than the second most long-running process, readynasd which is at 4 minutes.
After a while, btrfs-cleaner drops back to 0% CPU and I can access the NAS normally.
My question: is it normal to have such high load of btrfs-cleaner so frequently? Is there any configuration setting that influences this behaviour. I never noticed this problem under the previous version, 6.2.5.
Cheers,
Joachim
In case it matters, my NAS is equipped with 2x 4TB drives in RAID0 with about 40% in use.
It seems like I found a way to get rid of the hanging btrfs-cleaner: After switching off quota
btrfs quota disable /data
btrfs-cleaner did never show up again (in top).
(After booting I usualy had about a minute to issue the command before btrfs-cleaner did spawn and prevented the command to respond.)
I did not intentionally enable quota - i guess it was enabled by default.
Good luck,
HansW
PS: Maybe the problem is known by Netgear - in their "FAQs on recently released firmware 6.4.0" (see link on top of page) they talk about "... this is likely due to the ReadyNAS performing a quota check".
Some websites also describe problems with quota and btrfs (Rockstor forum), other sites cite the quota support in btrfs as experimental (but these might be out of date).
I did not extensive testing after diabling quota that way - I'm happy that I can acces my files again ...
31 Replies
I'm seeing the same exact problem. The admin interface is worthless and times out when that service is running. I was copying files via the admin interface and they froze for 30 minutes before I cancelled the copy. This all started with the 6.4.0 update.
Hi Joachim,
Did you recently delete a very large file or a lot of files? Has there been any delete request like this or similar sent to the volume?
btrfs-cleaner is used when there is a large file(s) deletion.
Alex
Did it remove a bunch of your old snapshots? If you have snapshots and Smart Snapshot Management cleans up these snapshots, btrfs-cleaner will be busy wiping away those snapshots. In 6.4.0, we upgrade to the latest version of snapshots (which is why you can't downgrade from 6.4.0), so this could also be related.
Is there any way to turn on logging for btrfs-cleaner to find out what it's up to? As it is, I don't know why it seems to be taking longer than before since my usage profile hasn't changed that much since upgrading...
Hi Alex,
most of the data traffic on my RN102 is from TimeMachine on a couple of macs, but the load hasn't changed since before the update.
I'm still getting very high load from btrfs-cleaner several times a day, so much so that TimeMachine gives an error message saying the backup disk isn't available. The CPU time after 6 days is now up to a record 1270 minutes for btrfs-cleaner with readynasd at a mere 36 minutes, according to "top".
Joachim
This has started happening with my RN316, also after the 6.4.0 upgrade. In my case, the activity seems to be triggered by any attempt to use the Plex server app, although that could also be coincidental. Logging in with ssh, top shows btrfs-cleaner maxed out at 99/100%, starving any other processes of CPU time. Admin page becomes unresponsive, and attempts to contact the plex app stall. A few times it has died down by itself after 5/10 minutes or so, twice it's gone on for a lot longer and I've given up and rebooted. It has happened at various times of the day, late evening, now happening before lunch.
Even if the process is going through and cleaning up after snapshots, should it really be able to max out the CPU, and prevent other processes from running?
Cheers,
Andrew
Hi,
i disabled dropbox backup and now btrfs-cleaner use 10/15% of cpu. I do not know if there is a relationship between the two but now the NAS working
I'm in the same boat. Just want to chime-in, lest anybody think this is a rare problem with 6.4.0.
Just now, I've had a browser window open, trying to connect to the admin pages (all I've seen for the past 90 minutes is the splash screen, after authenticating). Simultaneously, I've had a ssh terminal window open, running top on the NAS. At the exact moment that the btrfs-cleaner process went to 0% CPU, the admin page finally popped-up.
- Same here with me. System: ReadyNAS 102 It might be the first reboot after the update. Uptime 2:08 hrs, load average: 8.45, 9.00, 9.97 Top shows "120:18.59 btrfs-cleaner" - quite a long time for a single process. Hoping it's not "cleaning" all my 2x3TB HDDs ;-) Admin page keeps reloading (constantly posting to "ddbroker") Copying to the NAS ("cp" over NFS4) is not returning (canelled). Directory listing in several directories either (not over NFS, not locally). The syslogs showing quite often "readynasd[3359]: thread create fail, errno=12" I will leave it running, checking tomorrow whether "btrfs-cleaner" calmed down.
Short after the las reboot (the NAS was not even responding to ssh) the btrfs-cleaner restarted - taking up to 97% of CPU.
Acces to the files (via NFS) is not possible any more, even a ls never returns.
It might be related to the deletion of a snapshot (the ReadyNas-Logs were showing an according message after the reboot), but it should not be that it leaves the NAS unusable.
It never happened befor firmware 6.4.0, Usage of the NAS did not change in any way.
Unfortunately there seems to be no documentation for btrfs-cleaner and no way to find out what it's really doing ...
Hello, Netgear *knock* *knock* *knock* - any comments on this?
Regards, HansW
Well - I'm not too sure but it seems that any file-access keeps hanging and "deadlocks" with btrfs-cleaner.
The NAS did not even shutdown completely: After shutdown -r it stopped responding to ping very quickly but never switched off - and did not respond to the power-button any more.
The only way to reboot was to disconnect it from power ... :smileysad:
It seems like I found a way to get rid of the hanging btrfs-cleaner: After switching off quota
btrfs quota disable /data
btrfs-cleaner did never show up again (in top).
(After booting I usualy had about a minute to issue the command before btrfs-cleaner did spawn and prevented the command to respond.)
I did not intentionally enable quota - i guess it was enabled by default.
Good luck,
HansW
PS: Maybe the problem is known by Netgear - in their "FAQs on recently released firmware 6.4.0" (see link on top of page) they talk about "... this is likely due to the ReadyNAS performing a quota check".
Some websites also describe problems with quota and btrfs (Rockstor forum), other sites cite the quota support in btrfs as experimental (but these might be out of date).
I did not extensive testing after diabling quota that way - I'm happy that I can acces my files again ...
HansWeihnacht wrote:It seems like I found a way to get rid of the hanging btrfs-cleaner: After switching off quota
btrfs quota disable /data
Thanks a lot Hans, this sounds like the first sensible, workable solution in this thread. I don't have access to my NAS until November, but this is the first thing I'll try when I do.
Joachim
I have no idea what side-effects it may bring - I'm not too deep into the details of BTRFS.
What I noticed so far:
* Read- and write-acces as before
* Snapshot creation and deletion works as before (according to ReadyNAS-Logs)
* btrfs-cleaner never used noticeable CPU since (logging in intervals of 15min)
* ReadyNAS Admin-Page shows data-usage only "Snapshots" and "Free" (no "Data" any more, but I'm not sure wheter that's related to switching off quota - might be, because tha data on disk tends to be "old")
HansWeihnacht wrote:It seems like I found a way to get rid of the hanging btrfs-cleaner: After switching off quota
btrfs quota disable /data
btrfs-cleaner did never show up again (in top).
Hi Hans,
like you, I haven't enabled quota myself and they're not used in any of my shares.
I would've tried the "quota disable" command if it had been obvious whether it destroys all of the qgroups on the volume (visible using "qgroup show /data"). Is this documented anywhere for btrfs?
It's also not clear how to re-enable quota if it requires re-creating all the qgroups as described on the wiki. Is it documented anywhere how ReadyNAS OS requires qgroups to be set up?
In the end, by the time I got access to my NAS 6.4.1-T21 beta was out, so I installed it and haven't had problems since (after 10h of uptime, btrfs-cleaner is still below 7s of CPU time). So I'll consider this fixed if the beta ever turns into a release.
Cheers,
joki
I see a similar issue, but not matching the descriptions exactly. In my case, top doesn't show btrfs-cleaner using 100% CPU, but if I install iotop, I see the process eating all of the available disk IO, meaning that most of the CPU is spent in iowait. This causes any other disk-bound process to run extremely slowly; I see similar usability symptoms as others (slow FrontView response, failed NAS mounting, file reads, etc).
This has been persistent under 6.4.0 and the latest 6.4.1 RC3 (running RC3 and observing this as I type):
Total DISK READ: 759.25 K/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1938 be/4 root 381.48 K/s 503.70 K/s 0.00 % 97.83 % [btrfs-cleaner] 1939 be/4 root 374.07 K/s 500.00 K/s 0.00 % 96.80 % [btrfs-transacti]
Apparently 6.4.1 is now out and has fixed this issue...Upgraded to 6.4.1
Btrfscleaner stopped using 100% cpu but now I have kworker/u4:4 using 98% or more
And one of the volumes shows Calculating instead of free space
Hi Airat,
Sorry to hear that...
6.4.1 also fixed my btrfs-cleaner issue, but I have not encountered the kworker process issue that you are having...
So far, 6.4.1 seems stable on my side, but I am keeping an eye on it and will watch the kworker process as well.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
