btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

[Joufflu]

Hello,

After 6.4.0 upgrade btrfs-cleaner process always uses 100% cpu even after many uptime hours.

SMB shares are ok (but slow)

Frontview times out after authentication.

Top command (ssh is ok) :

top - 16:33:54 up  2:51,  1 user,  load average: 14.25, 14.00, 11.14
PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
2524 root      20   0     0    0    0 R  96.4  0.0 135:53.96 btrfs-cleaner

Any ideas ?

Thank you

[TonyKL]

I wish I could help but I am in the same situation, my ReadyNAS is now useless due to this upgrade.

[spotcatbug]

I'm also unable to use the NAS, although it appears to be fully booted. I just ssh'd in to check it out. Indeed, btrfs-cleaner is (apparently) hard at work.

In my case, if I wait for about 90 minutes after boot, the NAS becomes usable again. I suspect the btrfs-cleaner process will not be active at that point. I will confirm in about an hour (hopefully).

UPDATE: As suspected, when btrfs-cleaner stopped, the NAS became fully usable. Also, I did an immediate reboot to see if btrfs-cleaner would go wild again - it didn't. The NAS seems to be working normally for now.

[TonyKL]

Thanks for the info, I have had the box with the cleaner stopped but as soon as I touch the filesystem it's off again.  It must think I'm really dirty to need to spend all day cleaning.  I have about 6TB of data.

I'm trying to clean up and delete old snapshots but ever one I delete freezes my NAS for a good 10 minutes +

[spotcatbug]

I'm not seeing that same symptom, where touching the filesystem sets off btrfs-cleaner again. I've been using the NAS all morning (since btrfs-cleaner stopped) without incident. I'm keeping an eye on a running top command, just to see if it pops in there again, but so far it's OK.

This wouldn't be the first time I had the NAS go from unusable to usable since 6.4.0. I've had it do this a few times now. This time is different, however; it's remained usable through reboot. I did try the USB recovery and that kept it from happening through one reboot. I want to reboot again, to see if it continues to be usable, but I have work to do, so that will have to wait until later today.

[JasperA]

This upgrade is giving me a terrible headache. Is there a way to disable btrfs-cleaner temporarily until there is a fix?

The NAS is completely useless now, especially for NFS conntections to my VMware cluster. NFS is unavailable most of the time during the cleaner causing my VM's to remount filesystems read-only.

Please fix this as soon as possible Netgear!

[Joufflu]

btrfs-cleaner finally calm down on my NAS after hours of power on

I now have access to Frontend : 1TB data, 165GB snapshots

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      2524 88.6  0.0      0     0 ?        S    Oct11 1618:06 [btrfs-cleaner]

[spotcatbug]

@I wrote:

I want to reboot again, to see if it continues to be usable, but I have work to do, so that will have to wait until later today.

---

So I booted it up this morning, as per usual. btrfs-cleaner is at it again and the NAS is unusable... again.

[TonyKL]

Finally I think my btrfs-cleaner has stopped.  

It could just be because I had 5TB of data that it took ages to munge through, or it could be the fact that I deleted all my snapshots that has stopped it.  Maybe we just need to be a little more patient?

[JasperA]

Mine has stopped yesterday, so I fired up my vSphere environment again, using NFS shares from my NAS.

Unfortunately the NAS became useless again over night and this morning btrfs-cleaner was running again. Probably because of the Smart Snapshot Manager removing a bunch of snapshots.

I'm now facing a few options: Disable quota (how?), Disable snapshotting, Disable or kill btrfs-cleaner (impossible?), ...

Okay, I must admit; throwing the NAS out of the window really came to mind as one of the options. This is driving me insane.

[ChristineT]

Good afternoon ReadNAS Community,

Thank you all for contributing to this thread. We are currently investigating this issue and have found in some cases this is likely due to the ReadyNAS performing a quota check. The ReadyNAS may take some time doing a quota check. This quota check should only run once after upgrading as long as it is able to complete successfully. Rebooting your chassis might cause the quota check to restart from the beginning. Please allow it some time to finish.

Also, btrfs-cleaner is commonly invoked after Smart Snapshot Management prunes older snapshots. ReadyNAS commonly prunes older snapshots based on its snapshot schedule. If your ReadyNAS is restarting btrfs-cleaner every time you restart and consuming 100% CPU endlessly, or if the device is not behaving as notated here, please contact NETGEAR Support.

Thank you,

Christine

NETGEAR Community

[TonyKL]

For the record, I don't have any quota enabled and still btrfs-cleaner got busy.  For me I think it was snapshot related and cleanup.

After 3 days all seems well inside my ReadyNAS.

Thanks for the continued support of this great NAS.

[JasperA]

Seriously TonyKL, your NAS was unusable for 3 days and you call this 'a great NAS'?

I disabled (or so I think) the quota support, but now it's 'kworker' taking up lots and lots of CPU, rendering my NAS useless again after over 4 days of trouble.

Good to see Netgear is looking in to this, but this version should have never been released like this.

Even the manpage of btrfs states the quota system is not stable (see https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota)

[TonyKL]

Up until this release yes, it's be a great NAS.  I am running a ReadyNAS Ultra 4 (legacy hardware) and have been for well over 5 years and it's been peachy.

I agree, with this issue the update should not have been released (without any prior warning) but these things happen sometimes.

Although the forced switch to btrfs might not have been the best decision.

[Joufflu]

---

ChristineT a écrit :
The ReadyNAS may take some time doing a quota check. This quota check should only run once after upgrading as long as it is able to complete successfully.

---

Hello,

My RN102 is up for about 3 days.

btrfs-cleaner is still frequently using 100% cpu which leads to an unavailable web interface.

top - 10:17:27 up 2 days, 20:35,  1 user,  load average: 2.58, 2.38, 1.86
Tasks: 137 total,   2 running, 135 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 90.5 sy,  0.0 ni,  0.0 id,  9.5 wa,  0.0 hi,  0.0 si, 
0.0 st
KiB Mem:    508804 total,   485396 used,    23408 free,     4040 buffers
KiB Swap:   523964 total,      748 used,   523216 free,   229488 cached

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+ COMMAND
2524 root      20   0     0    0    0 R  90.2  0.0   2065:41 btrfs-cleaner

34h hours of btrfs-cleaner activity within 68h of uptime.

I think that it's more than "a quota scan takes place during boot" as described in the release notes.

[JasperA]

Just to give you an update... btrfs-cleaner is still (after 6 days) destroying my NAS every night.

In the afternoon everything seems fine. So I start up all VM's that depend on NFS from the NAS. But each and every morning everything is crashed due to btrfs-cleaner.

Perhaps I should just buy a different NAS. I'll put 'NO BTRFS' on my checklist.

[StephenB]

---

@JasperA wrote:

Just to give you an update... btrfs-cleaner is still (after 6 days) destroying my NAS every night.

---

Have you contacted support?

Are snapshots disabled on the shares used by the VMs?

[JasperA]

Only support I seem to be able to get is from the Community.

Snapshots are all disabled.

btrfs-cleaner is STILL running and STILL making my NAS completely useless... after a very frustrating week with no updates from Netgear.

The update to 6.4.0 completely bricked my ReadyNAS 516. :-(((

[JasperA]

Update: during the small amount of time the web interface was responsive I was able to take another look at the Snapshot settings of the VMware volume. It was set to 'Daily' so I set it to 'Never'. I checked some other volumes and did the same there.

Now, btrfs-cleaner is quiet again and the NAS is sort of responsive.

I'm not cheering too soon; probably the btrfs-cleaner process will go wild again tonight.

I'll update you tomorrow.

But if this helps; the only thing 6.4.0 got me was the removal of the snapshot feature...

[JasperA]

...

..

.

btrfs-cleaner killed everything... again...

RIP ReadyNAS. I'll put you on ebay and go for a Synology or ANYTHING without btrfs.

[Joufflu]

I think that automatic snapshot prune does not work on my RN102.

According to this KB :

http://kb.netgear.com/app/answers/detail/a_id/23353/~/what-are-basic-snapshot-concepts-i-need-to-und...

"Daily snapshots are kept for four weeks."

But I see many snapshots older than one month on shares configured with daily snapshots.

It seem to keep a monthly backup and never delete it.

I never made manual snapshots.

I'm cleaning old snapshots but it's really long on large shares.

[StephenB]

---

@Joufflu wrote:
 

According to this KB :

http://kb.netgear.com/app/answers/detail/a_id/23353/~/what-are-basic-snapshot-concepts-i-need-to-und...

"Daily snapshots are kept for four weeks."

But I see many snapshots older than one month on shares configured with daily snapshots.

It seem to keep a monthly backup and never delete it.

---

That is the way it is designed to work.  Snapshots are "thinned" down to 1/month, but that monthly one is not deleted.

[AlexPe]

Hi Jasper,

Would you like to participate in the solution for this issue? I would like to gain the log data from your system here: 

Send me logs put "ATTN: ALEX btrfs-cleaner issue"

Then, once I have the log data, I would like to work with one our developers in determining what is happening on your system that sparks the btrfs-cleaner process that is consuming so much of the 516's resources. In order to accomplish this, I will need to set up remote access to your system for one of our developers. Please send me a PM if you are interested in this.

If not I understand, but seeing the system in this state is paramount to its resolution. 

Alex

[JasperA]

Thank you Alex, I will do that.

[AlexPe]

Dear Joufflu,

• Hourly snapshots are kept for 48 hours.
• Daily snapshots are kept for four weeks.
• Weekly snapshots are kept for eight weeks.

Do you have any that are older than eight weeks? So the way snapshots work is it still takes weekly snapshots if you have daily snapshots selected and if you have hourly, it will take all three at selected intervals.

So you might have weekly snapshots that are still present. The key to checking is to see when the oldest snapshots were taken and see if the dates are synonymous with an 8 week interval.

Thanks

Alex