Reply

btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Joufflu
Aspirant

btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Hello,

 

After 6.4.0 upgrade btrfs-cleaner process always uses 100% cpu even after many uptime hours.

SMB shares are ok (but slow)

Frontview times out after authentication.

 

Top command (ssh is ok) :

 

top - 16:33:54 up  2:51,  1 user,  load average: 14.25, 14.00, 11.14
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2524 root 20 0 0 0 0 R 96.4 0.0 135:53.96 btrfs-cleaner

Any ideas ?

Thank you

Message 1 of 40
TonyKL
Guide

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

I wish I could help but I am in the same situation, my ReadyNAS is now useless due to this upgrade.

Message 2 of 40
spotcatbug
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

 

I'm also unable to use the NAS, although it appears to be fully booted. I just ssh'd in to check it out. Indeed, btrfs-cleaner is (apparently) hard at work.

 

In my case, if I wait for about 90 minutes after boot, the NAS becomes usable again. I suspect the btrfs-cleaner process will not be active at that point. I will confirm in about an hour (hopefully).

 

UPDATE: As suspected, when btrfs-cleaner stopped, the NAS became fully usable. Also, I did an immediate reboot to see if btrfs-cleaner would go wild again - it didn't. The NAS seems to be working normally for now.

 

Message 3 of 40
TonyKL
Guide

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Thanks for the info, I have had the box with the cleaner stopped but as soon as I touch the filesystem it's off again.  It must think I'm really dirty to need to spend all day cleaning.  I have about 6TB of data.

 

I'm trying to clean up and delete old snapshots but ever one I delete freezes my NAS for a good 10 minutes +

Message 4 of 40
spotcatbug
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

 

I'm not seeing that same symptom, where touching the filesystem sets off btrfs-cleaner again. I've been using the NAS all morning (since btrfs-cleaner stopped) without incident. I'm keeping an eye on a running top command, just to see if it pops in there again, but so far it's OK.

 

This wouldn't be the first time I had the NAS go from unusable to usable since 6.4.0. I've had it do this a few times now. This time is different, however; it's remained usable through reboot. I did try the USB recovery and that kept it from happening through one reboot. I want to reboot again, to see if it continues to be usable, but I have work to do, so that will have to wait until later today.

 

Message 5 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

This upgrade is giving me a terrible headache. Is there a way to disable btrfs-cleaner temporarily until there is a fix?

The NAS is completely useless now, especially for NFS conntections to my VMware cluster. NFS is unavailable most of the time during the cleaner causing my VM's to remount filesystems read-only.

 

Please fix this as soon as possible Netgear!

Message 6 of 40
Joufflu
Aspirant

Re : btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

btrfs-cleaner finally calm down on my NAS after hours of power on

I now have access to Frontend : 1TB data, 165GB snapshots

 

 

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      2524 88.6  0.0      0     0 ?        S    Oct11 1618:06 [btrfs-cleaner]

 

Message 7 of 40
spotcatbug
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

 

@I wrote:

 

I want to reboot again, to see if it continues to be usable, but I have work to do, so that will have to wait until later today.

 


 

So I booted it up this morning, as per usual. btrfs-cleaner is at it again and the NAS is unusable... again.

 

Message 8 of 40
TonyKL
Guide

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Finally I think my btrfs-cleaner has stopped.  

 

It could just be because I had 5TB of data that it took ages to munge through, or it could be the fact that I deleted all my snapshots that has stopped it.  Maybe we just need to be a little more patient?

Message 9 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Mine has stopped yesterday, so I fired up my vSphere environment again, using NFS shares from my NAS.

Unfortunately the NAS became useless again over night and this morning btrfs-cleaner was running again. Probably because of the Smart Snapshot Manager removing a bunch of snapshots.

 

I'm now facing a few options: Disable quota (how?), Disable snapshotting, Disable or kill btrfs-cleaner (impossible?), ...

 

Okay, I must admit; throwing the NAS out of the window really came to mind as one of the options. This is driving me insane.

Message 10 of 40
ChristineT
Admin

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Good afternoon ReadNAS Community,

 

Thank you all for contributing to this thread. We are currently investigating this issue and have found in some cases this is likely due to the ReadyNAS performing a quota check. The ReadyNAS may take some time doing a quota check. This quota check should only run once after upgrading as long as it is able to complete successfully. Rebooting your chassis might cause the quota check to restart from the beginning. Please allow it some time to finish.

 

Also, btrfs-cleaner is commonly invoked after Smart Snapshot Management prunes older snapshots. ReadyNAS commonly prunes older snapshots based on its snapshot schedule. If your ReadyNAS is restarting btrfs-cleaner every time you restart and consuming 100% CPU endlessly, or if the device is not behaving as notated here, please contact NETGEAR Support.

 

Thank you,

 

Christine

NETGEAR Community

 

 

Message 11 of 40
TonyKL
Guide

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

For the record, I don't have any quota enabled and still btrfs-cleaner got busy.  For me I think it was snapshot related and cleanup.

After 3 days all seems well inside my ReadyNAS.

 

Thanks for the continued support of this great NAS.

Message 12 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Seriously TonyKL, your NAS was unusable for 3 days and you call this 'a great NAS'?

I disabled (or so I think) the quota support, but now it's 'kworker' taking up lots and lots of CPU, rendering my NAS useless again after over 4 days of trouble.

Good to see Netgear is looking in to this, but this version should have never been released like this.

Even the manpage of btrfs states the quota system is not stable (see https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota)

Message 13 of 40
TonyKL
Guide

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Up until this release yes, it's be a great NAS.  I am running a ReadyNAS Ultra 4 (legacy hardware) and have been for well over 5 years and it's been peachy.

 

I agree, with this issue the update should not have been released (without any prior warning) but these things happen sometimes.

 

Although the forced switch to btrfs might not have been the best decision.

Message 14 of 40
Joufflu
Aspirant

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade


ChristineT a écrit :
The ReadyNAS may take some time doing a quota check. This quota check should only run once after upgrading as long as it is able to complete successfully.

 


Hello,

My RN102 is up for about 3 days.

btrfs-cleaner is still frequently using 100% cpu which leads to an unavailable web interface.

 

 

 

top - 10:17:27 up 2 days, 20:35,  1 user,  load average: 2.58, 2.38, 1.86
Tasks: 137 total,   2 running, 135 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 90.5 sy,  0.0 ni,  0.0 id,  9.5 wa,  0.0 hi,  0.0 si, 
0.0 st
KiB Mem:    508804 total,   485396 used,    23408 free,     4040 buffers
KiB Swap:   523964 total,      748 used,   523216 free,   229488 cached

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+ COMMAND
2524 root      20   0     0    0    0 R  90.2  0.0   2065:41 btrfs-cleaner

 

34h hours of btrfs-cleaner activity within 68h of uptime.

I think that it's more than "a quota scan takes place during boot" as described in the release notes.

 

 

Message 15 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Just to give you an update... btrfs-cleaner is still (after 6 days) destroying my NAS every night.

In the afternoon everything seems fine. So I start up all VM's that depend on NFS from the NAS. But each and every morning everything is crashed due to btrfs-cleaner.

 

Perhaps I should just buy a different NAS. I'll put 'NO BTRFS' on my checklist.

Message 16 of 40
StephenB
Guru

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade


@JasperA wrote:

Just to give you an update... btrfs-cleaner is still (after 6 days) destroying my NAS every night.

 


Have you contacted support?

 

Are snapshots disabled on the shares used by the VMs?

Message 17 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Only support I seem to be able to get is from the Community.

Snapshots are all disabled.

btrfs-cleaner is STILL running and STILL making my NAS completely useless... after a very frustrating week with no updates from Netgear.

The update to 6.4.0 completely bricked my ReadyNAS 516. :-(((

Message 18 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Update: during the small amount of time the web interface was responsive I was able to take another look at the Snapshot settings of the VMware volume. It was set to 'Daily' so I set it to 'Never'. I checked some other volumes and did the same there.

 

Now, btrfs-cleaner is quiet again and the NAS is sort of responsive.

 

I'm not cheering too soon; probably the btrfs-cleaner process will go wild again tonight.

I'll update you tomorrow.

 

But if this helps; the only thing 6.4.0 got me was the removal of the snapshot feature...

Message 19 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

...

..

.

 

btrfs-cleaner killed everything... again...

 

RIP ReadyNAS. I'll put you on ebay and go for a Synology or ANYTHING without btrfs.

Message 20 of 40
Joufflu
Aspirant

Re : btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

I think that automatic snapshot prune does not work on my RN102.

 

According to this KB :

http://kb.netgear.com/app/answers/detail/a_id/23353/~/what-are-basic-snapshot-concepts-i-need-to-und...

"Daily snapshots are kept for four weeks."

 

But I see many snapshots older than one month on shares configured with daily snapshots.

It seem to keep a monthly backup and never delete it.

I never made manual snapshots.

 

I'm cleaning old snapshots but it's really long on large shares.

Message 21 of 40
StephenB
Guru

Re : btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade


@Joufflu wrote:
 

According to this KB :

http://kb.netgear.com/app/answers/detail/a_id/23353/~/what-are-basic-snapshot-concepts-i-need-to-und...

"Daily snapshots are kept for four weeks."

 

But I see many snapshots older than one month on shares configured with daily snapshots.

It seem to keep a monthly backup and never delete it.

 


That is the way it is designed to work.  Snapshots are "thinned" down to 1/month, but that monthly one is not deleted.

Message 22 of 40
AlexPe
NETGEAR Moderator

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Hi Jasper,

 

Would you like to participate in the solution for this issue? I would like to gain the log data from your system here: 

Send me logs put "ATTN: ALEX btrfs-cleaner issue"

 

Then, once I have the log data, I would like to work with one our developers in determining what is happening on your system that sparks the btrfs-cleaner process that is consuming so much of the 516's resources. In order to accomplish this, I will need to set up remote access to your system for one of our developers. Please send me a PM if you are interested in this.

 

If not I understand, but seeing the system in this state is paramount to its resolution. 

 

Alex

Message 23 of 40
JasperA
Apprentice

Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Thank you Alex, I will do that.

Message 24 of 40
AlexPe
NETGEAR Moderator

Re : btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade

Dear Joufflu,

 

  • Hourly snapshots are kept for 48 hours.
  • Daily snapshots are kept for four weeks.
  • Weekly snapshots are kept for eight weeks.

Do you have any that are older than eight weeks? So the way snapshots work is it still takes weekly snapshots if you have daily snapshots selected and if you have hourly, it will take all three at selected intervals.

 

So you might have weekly snapshots that are still present. The key to checking is to see when the oldest snapshots were taken and see if the dates are synonymous with an 8 week interval.

 

Thanks

 

Alex

Message 25 of 40
Discussion stats
  • 39 replies
  • 5023 views
  • 5 kudos
  • 9 in conversation
Announcements