× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade

mdgm-ntgr
NETGEAR Employee Retired

Re: btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade

What should have been done was to let the system do what it needs to do to clean things up.

 

It can take quite some time, especially if volume maintenance hasn't been done.

Message 26 of 32

Re: btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade

After rebooting (pulling the cable, power-down-button had no effect) the system welcomed me with "Resyncing started for Volume data".

After that it was quite busy (15-min-load went up to 8.1, kworker using 90% CPU), later it was mainly the processes systemd-journal, md127_resync and md127_raid1. About 6 hours later load was back at 0.3, no process taking more CPU than usual.

 

The admin page no shows all used space as "data" (no space used by "snapshots", but as there is not much change in the data, the - existing - snapshots may take not much space).

 

So far: Looks good, thanks Netgear.

 

What I'd like to know: What was the problem? And which "rare" circumstances led to the complete inaccessibility of the device?

 

Regards, HansW

Message 27 of 32
joki
Tutor

Re: btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade


@HansWeihnacht wrote:

It seems like I found a way to get rid of the hanging btrfs-cleaner: After switching off quota

btrfs quota disable /data

btrfs-cleaner did never show up again (in top).

 

 

Hi Hans,

 

like you, I haven't enabled quota myself and they're not used in any of my shares.

 

I would've tried the "quota disable" command if it had been obvious whether it destroys all of the qgroups on the volume (visible using "qgroup show /data"). Is this documented anywhere for btrfs?

 

It's also not clear how to re-enable quota if it requires re-creating all the qgroups as described on the wiki. Is it documented anywhere how ReadyNAS OS requires qgroups to be set up?

 

In the end, by the time I got access to my NAS 6.4.1-T21 beta was out, so I installed it and haven't had problems since (after 10h of uptime, btrfs-cleaner is still below 7s of CPU time). So I'll consider this fixed if the beta ever turns into a release.

 

Cheers,

joki

Message 28 of 32
cwoodfield
Aspirant

Re: btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade

I see a similar issue, but not matching the descriptions exactly. In my case, top doesn't show btrfs-cleaner using 100% CPU, but if I install iotop, I see the process eating all of the available disk IO, meaning that most of the CPU is spent in iowait. This causes any other disk-bound process to run extremely slowly; I see similar usability symptoms as others (slow FrontView response, failed NAS mounting, file reads, etc).

 

This has been persistent under 6.4.0 and the latest 6.4.1 RC3 (running RC3 and observing this as I type):

Total DISK READ:     759.25 K/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                        
 1938 be/4 root      381.48 K/s  503.70 K/s  0.00 % 97.83 % [btrfs-cleaner]
 1939 be/4 root      374.07 K/s  500.00 K/s  0.00 % 96.80 % [btrfs-transacti]
Message 29 of 32

Re: btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade


Apparently 6.4.1 is now out and has fixed this issue...
Message 30 of 32
Airat
Aspirant

Re: btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade

Upgraded to 6.4.1

Btrfscleaner stopped using 100% cpu but now I have kworker/u4:4 using 98% or more

And one of the volumes shows Calculating instead of free space

Message 31 of 32

Re: btrfs-cleaner frequently stuck at 100% CPU after 6.4.0 upgrade

Hi Airat,

 

Sorry to hear that...

6.4.1 also fixed my btrfs-cleaner issue, but I have not encountered the kworker process issue that you are having...

So far, 6.4.1 seems stable on my side, but I am keeping an eye on it and will watch the kworker process as well.

Message 32 of 32
Top Contributors
Discussion stats
Announcements