This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
In earlier releases (< 6.7.0), the target extent size was 256KB. That meant that the defrag operation would only attempt to defrag chunks smaller than 256KB. That was very fast, of course, but it didn't really do much defragging. The default changed from 256KB to 32MB in 6.7.0, which made defrag operations take much longer because were were now attempting to defrag any chunks smaller than 32MB instead of 256KB. 6.7.5 brings that down somewhere in the middle, and sets the target extent size to 4MB.
In earlier releases (< 6.7.0), the target extent size was 256KB. That meant that the defrag operation would only attempt to defrag chunks smaller than 256KB. That was very fast, of course, but it didn't really do much defragging. The default changed from 256KB to 32MB in 6.7.0, which made defrag operations take much longer because were were now attempting to defrag any chunks smaller than 32MB instead of 256KB. 6.7.5 brings that down somewhere in the middle, and sets the target extent size to 4MB.
That's good info! I should try it and see how it behaves on mine now. When I last tried (6.7.1 or 6.7.2, I can't recall) it just dragged the whole box down. Even if I only run it once ever 3 or 4 months, it just tanks it... and I got stuff running! 😉
I'll give it a try on 6.7.5 now and see. By chance, have any changes been made that would improve the scrub performance? Or at least having scrub run in such a way that it too doesn't hog every last bit of performance for itself?
Thing is... I don't particularly mind if these operations take a while... volume is over 30TB now. But what I mind is when running them makes doing anything else so slow it's not worthwhile.
btaroli wrote: By chance, have any changes been made that would improve the scrub performance? Or at least having scrub run in such a way that it too doesn't hog every last bit of performance for itself?
Thing is... I don't particularly mind if these operations take a while... volume is over 30TB now. But what I mind is when running them makes doing anything else so slow it's not worthwhile.
I just logon and use htop & change the nice down a few levels.
@btaroli, what have you found is affecting performance? The scrub process already sets its ionice level to the lowest (idle) priority, although it's not obvious from normal top output since the I/O priority only gets changed on the child thread that actually does the I/O. From our testing, the CPU isn't taxed much at all by scrub, so the I/O priority is the only thing that should really matter.
It's been a bit since I tried it, so I'll try again and observe.
The symptoms from the past included other processes on the host (like PLEX, DVBLink, which can themselves be pretty CPU intensive) failing to get enough cycle time to function normally. Indeed, it would get to the point that trying for normal playback just wasn't worth it. I'd either wait to watch or kill the scrub.
The primary symptom I observed in the past is that other services running on the NAS would be starved for CPU once the scrub began. DVBLink, PLEX, ZNC are examples. In the case of PLEX, this would result in frequent complaints that the server wasn't powerful enough for transcoding, but only while the scrub was running. ZNC, a very lightweight proces, would repeatedly stall and get into reconnection loops. Invariably, to keep other things running, I had to kill the scrub. Once killed, everything began responding normally again.
So to test this i had a program already running via PLEX AND kicked off the scrub. About two hours in I've only had one complaint about bandwidth. som screen captures of top running.
it looks like the kernel workers are using higher CPU now (up to 69% apiece instead of 40-50 as earlier), and its starving PLEX. DVBLink also can't begin a playback of a video I had watched earlier in the evening without any issue. it does seem to have managed to do a recording during this time though. ZNC also seems to be staying connected, but it's fairly lightweight too.
Whatever process priority the scrub is running with, it is not properly yielding to heavier tasks. I dare not thin kwhat will happen when a Time Machine backup starts. Indeed I'd better kill the scrub. In the past, the impact to AFP (which is extremly CPU heavy during Time Machine backups) winds up invalidating my backup archive. Very annoying.
But Skywalker said 'The scrub process already sets its ionice level to the lowest (idle) priority, although it's not obvious from normal top output since the I/O priority only gets changed on the child thread that actually does the I/O.'
You may want to install the App htop (much nicer), then instead of top run htop, F2-setup/columns - add IO_PRIORITY {need to scroll the list down} (& maybe the IO R/W rates too), make sure K H commands are showing the kernel & user threads, then click on the IO column twice to sort reverse (B0 - best-effort 0 is mega priority, B7 low, id idle).
[I'm not on 6.7.5 yet so can't check, still waiting for defrag & full backup to finish...]
I'd also keep an eye on memeroy, if you are running all that exrta stuff it could be a constraint, swapping?
I'm certainly happy to try it again later tonight. But you're kind of missing something obvious. Who cares what ionice is when there's no appreciable iowait time? This is CPU bottleneck not IO. But sure I'll try it again later.
Something else has to be going on. You can definitely see a fair amount of iowait during other activities, and when there is beavy IO for balance, defrag, scrub I expect to see at least some. But when it's zero iowait and si/hi, then I have to wonder where the cycles are going. Mind you, I have a fairly large RAM. But I expect that filesystem ops at this later going to be generating a fair amount of I/O. So perhaps the activity isn't beating exposed through the usual counters too can see? Not sure. The symptoms and effects on other apps are fairly predictable however.
OK... finally getting around to trying this. htop installed fine, but then I stumbled across a different issue. Can't get the F9 and F8 keys working in Terminal on macOS. Tried a few options and finally gave this up.
But I notice in watching the process list that there appear to be THREE "btrfs scrub" processes in the list, with sequential process ID's. When I ionice'd the first, it came back as class 0 (none). I ionice'd it to class 3 (idle). Then when I checked the other two, they both came back as 3/idle.
I've now got transcodes running in PLEX and DVBLink running simultaneously and they seem to be working. DVBLink is using ffmpeg (much more heavy on CPU than PLEX) and it occasionally pauses playback but it doesn't die as it was before.
Since I also run btrfs on Linux, I looked at the btrfs-scrub manpage and found that it HAS a -c option to specify an ionice class when invoking it. But I don't see that in the CMD of these processes. Perhaps it expects to ionice them separately. I do confirm that the btrfs version included in 6.7.5 has this option available.
# btrfs scrub
usage: btrfs scrub <command> [options] <path>|<device>
btrfs scrub start [-BdqrRf] [-c ioprio_class -n ioprio_classdata] <path>|<device>
Start a new scrub. If a scrub is already running, the new one fails.
btrfs scrub cancel <path>|<device>
Cancel a running scrub
btrfs scrub resume [-BdqrR] [-c ioprio_class -n ioprio_classdata] <path>|<device>
Resume previously canceled or interrupted scrub
btrfs scrub status [-dR] <path>|<device>
Show status of running or finished scrub
verify checksums of data and metadata
So how is it that this process is supposed to be classified for 3/idle? Whatever it is, it doesn't seem to actually happen.
Model: RN528X|ReadyNAS 528X - Premium Performance Business Data Storage 8-Bay
what needs root? huh? To make function keys work? Well, I ssh as root to the NAS if that's what you mean. 🙂
As for btrfs scrub and prio, I can tell you that it surely doesn't run idle on Linux by default. So to be sure I set the class when I invoke it there.
I notice that htop was showing three entries for btrfs scrub but top shows just one. I suspect threads or lwp. I didn't have to ionice all three. Just doing the first (which was the one top also showed) was effective.
> what needs root? huh? To make function keys work? Well, I ssh as root to the NAS if that's what you mean.
Yes that's what I mean, AFAIK to nice other processes it needs root permissions.
Did you insotall the Netgeap htop app? or via Linux?
I did the former, maybe it sets things up better for the NAS??
> As for btrfs scrub and prio
Well, hopefully Netgear are listening.
> I notice that htop was showing three entries for btrfs scrub but top shows just one
As I said above
run htop, F2-setup/columns - add IO_PRIORITY {need to scroll the list down} (& maybe the IO R/W rates too), make sure K H commands are showing the kernel & user threads, then click on the IO column twice to sort reverse (B0 - best-effort 0 is mega priority, B7 low, id idle).
If you had K & H it shows all threads. There is prob. a similar command for top.
Anyway, looks like you have a workaround.
BTW - I'm just a user here with a little knowledge.
I'll be upgrading some time soon, then I can confirm your findings.
Yeah, that's not especially helpful for Terminal on macOS, which I pointed out -- even if I run a Windows VM it's still got to filter through the macOS, etc. It's also moot since I was able to get the information I needed and make the ionice change.
I should also add now that I see Time Machine backups are completing successfully as well, as the scrub continues -- 8.4% now (of 32.7TB). So this ioprio class change IS what is needed. What isn't clear is why 6.7.5 doesn't set the ioclass for scrub, and I'm hoping that 6.8 might help address this.
My scrub is now at 28.82% and I continue to see decent behavior of applications/services running on the NAS, even for transcoding. The key is definitely to ensure that scrub is runing in "idle" (class 3) IO priority.
The curious thing is that the kworker/u8:* processes (showing iopro B4) are consuming 40-70% CPU but the btrfs scrub process/threads seem to be reporting all the actual I/O, and are running in "idle" (class 3) ioprio after I ran "ionice -p #### -c 3" on the process. I'm curious why it is the kworkers are so busy in the CPU. I expect it's something being dispatched from the scrub, but it seems awfully cpu heavy compared to what I've seen on Fedora running a btrfs scrub.
As for htop, I found that I *could* use the Fn keys correctly running in a Linux VM on my Mac (via VMware Fusion), and the GNOME Terminal app did also support mouse clicking on the function items in the bottom row of the display. I was also able to do this via the Remoter Pro app on my iPhone. Once done, I also noticed that htop stored the alterations in ~/.config/htop/htoprc. I noticed this and saw that the changes were visible in the macOS Terminal app then (as long as I don't have to change them further).
Was reminded of an older thread on this subject (from 6.6.1) that really is probably a better place for further updates on the scrub issue, since this thread was really for something else. will post further updates over there.
