Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade
In my own case, my 516 was running 6.4 for 5 days before I ran into problems. By then the initial quota calculation was long done and I could already see the size of snapshots displayed in the admin UI. What I observed triggering that caused my first symptom was scheduled balance of /data and a simultaneous Time Machine backup. I eventually got past this by stopping apps and canceling the balance (via ssh).
Subsequently, I noticed cleaner periodically chewing up lots of CPU and I/O. I expected to see cleaner because I have snapshots of various freuqencies enabled on my shares (incl /home). But it's just gotten to be too much of an impact when it runs. Basically, none of my shares or apps responds.
Now I have taken the step of disabling all snapshot schedules and removing all snapshots from the system. We're talking hundreds of snapshots removed within 20 minutes. So of course cleaner is quite busy now. But once it's done, I hope to have a happy NAS again.
I will be happy once these bugs are resolved and we can have our functionality back. On the off chance that logs will be helpful to Netgear, I'll provide mine as well. If there is additional diagnosis we can do to assist, please let us know. It sounds like we have several people in the forums affected, and might be willing to render aid in order to revive their NAS'es. 🙂
Re : btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade
AlexPe a écrit :
Weekly snapshots are kept for eight weeks.
Do you have any that are older than eight weeks? So the way snapshots work is it still takes weekly snapshots if you have daily snapshots selected and if you have hourly, it will take all three at selected intervals.
Hello,
On shares with auto daily snapshots there were (before I deleted them manually) :
- a snapshot per day on the last 4 weeks
- a snapshot per week between the last 4 to 8 weeks
- a snapshot per month (before this 8 weeks period)
as described by StephenB
I deleted all snapshots (165GB).
On a large share (480GB now), btrfs-cleaner worked 20-25 minutes to clean each snapshot.
Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade
My NAS is inaccessible for over 7 days in a row now, constantly 100% busy doing whatever btrfs-cleaner is doing. The /data volume does not appear on my admin page anymore.
I am running out of options and out of hope for this NAS.
This will leave your data intact. Check if problems are solved.
In my case they were not.
When I was finally able to access the admin page the first thing I did was disable snapshots.
This stopped the creation of new shapshots but dit not stop the removal of snapshots once an hour.
This kept the NAS 100% occupied for aprox 10 minutes every hour, but it gave me time to create a backup of all the data.
"Backup done (and checked!) I did a Factory reset, this WILL destroy all data on the disk.
After this I recreated the shares making sure to disable the snapshot-function.
Then I had to copy back all the data.
Overall this process took me 3 days!
But now I have a NAS that is once again up and running.
Ariane"
While I would not wish this to be the solution, it would work to resolve. The cleaner service is likely stuck on a rather large file or large quantity of files. We've been really trying to connect to a system that is having this issue to diagnose the issue directly. The log files we have obtained, point to systems that have been unmaintained. Since you have started the system from factory defaults, I'd suggest running scheduled maintenance now so that the file system stays maintained for the future. Volume Maintenance Guidance
Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade
I am deleting all snapshots by hand, one by one, from the web interface. That is... in the rare time I get it to respond at all.
When that's done, I think I have a system that should work again, more or less. I expect btrfs-cleaner to be busy as hell for a day or 10 after I'm finished. I won't mention that it's stupid I'm not able to select multiple snapshots and delete them all.
All this misery caused by beta software, adding features I don't need and enabling it by default. I'm not a guinea pig for testing this software, I had a business depending on a rock solid NAS. Which my ReadyNAS 516 obviously is not (anymore).
I'll keep you posted. The solution Ariane is giving us won't work for me, because I can't backup all my data to another device. It's a terrible shame that the only option to get a somehow working NAS again is to disable the snapshot feature. Too bad Netgear is giving up on this problem so easily. My problem exists for almost a month now with no real solution to look forward to. I owned three ReadNAS'ses, but the 516 is definately the last one I've bought.
Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade
You can select multiple snapshots and delete them. This certainly works with Chrome (I just did it). Use ctrl-click, or shift-click on the first and shift-click again on the last.
Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade
Thank you! Will try that as soon as I get some time at the web interface. Which I tried to restart, but is not running at all since I tried. The restart spawns '/bin/systemd-tty-ask-password-agent --watch' which times out.
Re: btrfs-cleaner always stuck at 100% CPU after 6.4.0 upgrade
Well, all my snapshots are gone now. Hurray!
The NAS seemed to perform a little bit this morning, giving me some hope.
But as I expected, btrfs-cleaner is sky high again in the evening. Not sure what it's doing, but obviously I deleted quite a number of snapshots. I'll let it run for the next days or weeks. (...)
With an uptime of 10 days and 11 hours, btrfs-cleaner took 7 days and 10 hours of CPU time.
