× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: 6.6.1 Scrub Hammers CPU

btaroli
Prodigy

Re: 6.6.1 Scrub Hammers CPU

Did you verify the scrub is still running? How much data is in your volume? If memory serves, it only scrubs data and not free space. I'll have to check action of -A as well, because it seems you picked up kernel threads and I wonder if it includes use space stuff too? Not familiar with that option and can't check until my toddler is down for his nap.
Message 26 of 37
Laserbait
Luminary

Re: 6.6.1 Scrub Hammers CPU

Yes, scrub is running, currently 4.85% completed.

Message 27 of 37
btaroli
Prodigy

Re: 6.6.1 Scrub Hammers CPU

https://manpages.debian.org/stretch/procps/ps.1.en.html

 

-A is a synonym for -a, selecting all processes

 

What you seem to be missing is an option for output format. I generally use -f, but there are others.

 

If you only see the name of the program being run, then ALL Btrfs commands show up as "btrfs". You need to see the command arguments in order to know which function is being called, such as scrub.

 

So try "ps -Af"

Message 28 of 37
btaroli
Prodigy

Re: 6.7.5 Scrub Hammers CPU

Got onto this subject in a thread on 6.7.5 but was recently reminded this one was still out there and "unanswered". Wanted to refresh this thread and get back on a single topic. Other thread was https://forum1.netgear.com/t5/ReadyNAS-Beta-Release/ReadyNAS-OS-6-Software-Version-6-7-5-Question

 

I was encouraged by some changes @Skywalker mentioned had happened in 6.7.5, wondering if the scrub behavior may have changed. Initial attempts suggested similar impact to several services on the NAS, including DVBLink, PLEX, AV, and Time Machine.

 


@Skywalker wrote:
@btaroli ... The scrub process already sets its ionice level to the lowest (idle) priority, although it's not obvious from normal top output since the I/O priority only gets changed on the child thread that actually does the I/O.  From our testing, the CPU isn't taxed much at all by scrub, so the I/O priority is the only thing that should really matter.

 

 In testing, I found still that these processes were starved (esp for video transcode by DVBLink and PLEX), presumably for CPU. It was suggested by @Michael_Oz that I install htop to get a better look, and this did help. As @Skywalker indicated, btrfs scrub has a couple of threads. But htop wasn't showing these consistenly coming up as in idle ioprio class. Indeed, btrfs scrub has the -c and -n options specifically to set ioprio on the scrub. It's not clear how it's being done now, but it doesn't seem to be having the intended effect.

 

# btrfs scrub
usage: btrfs scrub <command> [options] <path>|<device>

    btrfs scrub start [-BdqrRf] [-c ioprio_class -n ioprio_classdata] <path>|<device>
        Start a new scrub. If a scrub is already running, the new one fails.
    btrfs scrub cancel <path>|<device>
        Cancel a running scrub
    btrfs scrub resume [-BdqrR] [-c ioprio_class -n ioprio_classdata] <path>|<device>
        Resume previously canceled or interrupted scrub
    btrfs scrub status [-dR] <path>|<device>
        Show status of running or finished scrub

verify checksums of data and metadata

What I found was that once I had set "idle" (ionice -c 3 -p ####) on all three btrfs scrub processes/threads THEN I began to see that PLEX, DVBLink, Time Machine were all able to complete their work normally without impact.

 

I was playing a bit more with htop and sorting on various columns. When I sorted reverse on IO_PRIORITY, though, I noticed something really curious and may actually expose why it's important that btrfs scrub must run fully in idle ioprio in order to ensure no impact to services. In particular, I noticed the following processes running in low priority classes:

 

idle

   btrfs scrub

B7

   clamd
   PLEX (background process)
   watchdog

B6

   DVBLink (ffmpeg)

B4

   kworker/u8:*
   DVBLink (server)
PLEX (server)
PLEX (transcoder) Dropbox Transmission md*_raid6 dbus_daemon ksoftirqd/*

It looks very much like services (including clamd) are using ioprio in B6 and B7, and might get starved if something like btrfs scrub (or any other background maintenance process) wasn't running in idle class. Once btrfs scrub was confirmed (and reported by htop) has being in idle class it stopped interfering with other processes.

 

Indeed when I look at the values for nice and ioprio that processes are using, I wonder if some of them don't need to be ratcheted down in priority quite as far as they are. Too many things set down in B6 and B7 might created problems, no? Have to leave room for things that really do belong down there. 😄 Think I'll whisper over in PLEX and DVBLink forums on that subject.

Model: RN528X|ReadyNAS 528X - Premium Performance Business Data Storage 8-Bay
Message 29 of 37
Skywalker
NETGEAR Expert

Re: 6.7.5 Scrub Hammers CPU

btrfs scrub has 3 threads: (1) the master thread, that just waits for completion, (2) the thread that does the actual scrub I/O work, and (3) the status update thread that wakes up every 5 seconds to grab the latest status info from the kernel.  The brfs utility only sets the I/O priority (ionice) level on the I/O thread.  You can verify that by looking at iotop output.  Setting the other two threads to idle priority has essentially no effect.  It's likely that the variations in the quality of the other services you saw after changing the I/O priority on threads 1 and 3 was coincidental.  If the services getting affected are starved for CPU, then changing I/O priority isn't going to matter much.

 

The scrub operation of course has to do a lot of I/O.  It also involves calculating checksums for every data and metadata block and compares that to the checksum stored on disk.  So that can certainly chew up some CPU.  That work would be done by a kworker thread, so setting nice levels or I/O priority on the btrfs scrub process isn't going to affect those processes.

 

It's still unclear to me what functions of DVBLink, PLEX, AV, and Time Machine are getting starved in your case.  Is it transcoding?  That's a CPU-intensive operation that doesn't do too much I/O, which would suggest there's contention for CPU resources.  Or is it writing large files?  Writing small files?  Reading large files?  Small files?

 

Message 30 of 37
btaroli
Prodigy

Re: 6.6.1 Scrub Hammers CPU

In the case of PLEX and DVBLink it's definitely the transcoding that was most affected. Recordings and other operations seemed ok.

Time Machine (really the AFP file service) would get so starved for cycles that my backup would corrupt and get flagged to be replaced (as TM is want to do) whenever it attempted to run during a scrub.

I fully appreciate the de ription of the scrub processes you present here, but I can tell you that until I forced them into idle with ionice I was having exactly the problems described. And once having done it the behaviors stopped.

I was rather hope to let the scrub finish, since this value has never been scrubbed fully, and it's 5 days in now at about 55%. But I'm happy to restart the process to compare the ioprio on the breads as reported and detail/verify the outcome of the tweaks. Been through a few iterations during this run. But I am in a state now where things are humming along nicely, even if the scrub is taking a while -- which is fine by me.

The affect on the other processes (whether it's CPU, IO, or a combination of both) is real and extremely annoying, however. So something does need adjustment here.

I do observed on Fedora that btrfs scrub behaves as described on the btrfs wiki and your description. The base process is in ioprio B4, as well as one of the threads, the thread doing the effective IO is in idle. Course on this system effective CPU utilization is under 40% (four threads) and iowait is almost nil. Debian is of course quite different in how kernel threads operate, but this is what I see on fedora.

I'll restart the scrub on the NAS later tonight and see what I see there to compare. Not totally apples and apples, but I think it's odd how btrfs stopped getting in the way of other services until it was ionice'd directly.

I'll report back once I've had a chance to observe it again without manual adjustments.
Message 31 of 37
Skywalker
NETGEAR Expert

Re: 6.6.1 Scrub Hammers CPU

It's worth noting that the resources consumed by btrfs scrub can vary widely depending on the files it's scrubbing at the time.  Very fragmented files will result in a lot more I/O wait time, and more CPU consumption.  Also, a ReadyNAS volume scrub also includes a MD RAID scrub, which adds to the I/O load.  MD is generally pretty good at yielding to other I/O consumers, but it's still a minimum 30MB/sec used for that plus the CPU usage for parity calculation.

Message 32 of 37
Michael_Oz
Luminary

Re: 6.6.1 Scrub Hammers CPU

Still on 6.7.4.

Started scrub.

Three threads as described above, main process & two threads (shown with H command to htop)

 

 PPID   PID USER         IO  IORR  IOWR IO PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
    1 29003 root      41462 41462     0 B3  19  -1 32180   200    16 S  5.6  0.0  0:26.28 `- btrfs scrub start /N316AR6
    1 29005 root          0     0     0 B3  19  -1 32180   200    16 S  0.0  0.0  0:00.03 |  `- btrfs
    1 29004 root      41462 41462     0 id  19  -1 32180   200    16 D  5.6  0.0  0:26.26 |  `- btrfs

I'm reading that as the IO/IORR/IOWR & CPU usage is accounted for in the main process too, rather than both doing I/O & CPU.

Note active thread is idle iopriority, nice -1 (as most things spawned from readynas seem to inherit -1).

 

However, sort by CPU, (K command to show kernal threads), everything with CPU > 0.

 

 PPID   PID USER         IO  IORR  IOWR IO PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
    2 29066 root          0     0     0 B4  20   0     0     0     0 R 39.9  0.0  3:52.37 kworker/u8:10
    2 29043 root          0     0     0 B4  20   0     0     0     0 R 38.2  0.0  4:47.15 kworker/u8:9
    2 28656 root          0     0     0 B4  20   0     0     0     0 R 37.0  0.0  4:37.22 kworker/u8:2
    2 28793 root          0     0     0 B4  20   0     0     0     0 R 37.0  0.0  4:42.18 kworker/u8:3
    2 30086 root          0     0     0 B4  20   0     0     0     0 R 35.8  0.0  1:05.82 kworker/u8:14
    2 28903 root       57.2  57.2     0 B4  20   0     0     0     0 S 33.4  0.0  4:37.18 kworker/u8:8
    2 28071 root          0     0     0 B4  20   0     0     0     0 R 33.4  0.0  4:27.62 kworker/u8:0
    2 29095 root       19.2  19.2     0 B4  20   0     0     0     0 R 31.6  0.0  1:44.06 kworker/u8:13
    2 28902 root          0     0     0 B4  20   0     0     0     0 R 31.0  0.0  4:56.74 kworker/u8:7
    2  1515 root          0     0     0 B4  17  -1     0     0     0 R 25.6  0.0 35h04:21 md127_raid6
    2 27507 root          0     0     0 B4  20   0     0     0     0 R 17.9  0.0  4:31.39 kworker/u8:1
    2 29067 root          0     0     0 B4  20   0     0     0     0 R 17.9  0.0  4:30.13 kworker/u8:11
    1 29003 root      45490 45490     0 B3  19  -1 32180   200    16 S  6.0  0.0  1:12.96 btrfs scrub start /N316AR6
    1 29004 root      45517 45517     0 id  19  -1 32180   200    16 D  6.0  0.0  1:12.87 btrfs
    2 29001 root          0     0     0 ??  39  19     0     0     0 R  3.0  0.0  2:22.40 md127_resync
28517 28522 root          0     0     0 B4  20   0 29460  3688  3004 R  3.0  0.2  0:56.68 htop
    2  1402 root          0     0     0 B0   0 -20     0     0     0 S  1.8  0.0  0:54.53 kworker/1:1H
    2  1401 root          0     0     0 B0   0 -20     0     0     0 S  1.2  0.0  0:50.00 kworker/0:1H
    2  1359 root          0     0     0 B0   0 -20     0     0     0 S  0.6  0.0  3:29.12 kworker/2:1H
    2  1389 root          0     0     0 B0   0 -20     0     0     0 S  0.6  0.0  3:34.20 kworker/3:1H
    2     7 root          0     0     0 B4  20   0     0     0     0 R  0.6  0.0  4:10.65 rcu_sched
    1  3703 nut           0     0     0 B4  20   0 17240  1296   932 S  0.6  0.1  1:31.42 /lib/nut/usbhid-ups -a UPS

Note all the CPU chewed up by kworker threads, Nice 0, doing little IO. (kworker IO increased later, similar numbers as above, but more threads doing IO)

 

@btaroli's media processes (infering from the above iopriority B7/B6 - I don't know what they are doing) will have lesser nice values, and be CPU constrained. Those at B4 will have Nice 0, round-robin with all those kworker's, so ~1/12th CPU (?5/17th?).

 

How changing the iopriority of the other two threads can change this I can't fathom ATM.

 

Perhaps media threads with a 'interactive' workload should run nice -1 for now.

 

Longer term there should be a gap between current -1 processes (readynas etc) and default worker thread priority (currently 0 ), so intermidiate priority things can fit in the middle??

 

I'll repeat this on 6.7.5 when I get around to it, I'm currently juggling 10TB disks upgrading...nothing happens fast...

 

 

 

 

 

 

Message 33 of 37
Michael_Oz
Luminary

Re: 6.6.1 Scrub Hammers CPU

p.s. I run encryption, so CPU may be higher in my case. (?)

Message 34 of 37
Skywalker
NETGEAR Expert

Re: 6.6.1 Scrub Hammers CPU

Yes, the crypto functions are executed by kworkers.

Message 35 of 37
btaroli
Prodigy

Re: 6.6.1 Scrub Hammers CPU

Well, this really repeats other information already shared. Most importantly that the kworker threads are monopolizing ALL the CPU's with "kernel" usage during this process. It was explained more than once -- not our first time at this rodeo -- that this wasn't supposed to be an issue because the btrfs process was nice'd. If, however, the kernel overhead pushes the CPUs so hard nothing else of lower priority can get time that's an issue.

 

As for the behavior of the processes I've been watching, CPU load for PLEX tends to scale with the resolution and bitrate of the media; so television is generally more efficient than movies. Indeed, I noticed more (but brief) pauses with movies. DVBLink is much heavier on the CPU because it relies upon ffmpeg for transcoding. Consequently, it demonstrates impact from the CPU bottleneck much more quickly. Time Machine (netatalk) seems to have shorter spurts of CPU, across three or four processes.

 

From an IO perspective, PLEX and DVBLink are very light. Time Machine (netatalk) has longer periods where I observed it getting to 20-30 MB/sec during a backup I triggered manually during the scrub. Interestingly, the scrub process seemed also to range between 20-35 MB/sec peak... even when the netatalk was pushing the same. Of course, the scrub is doing nearly 100% read activity whereas the netatalk was doing mostly writes (for the backup).

 

Honestly, though, I think the issue is CPU. My DVBLink has had a background transcode running (to "save" a recording off for keeping) over 24 hours, at nice 5 and B5 ioprio. It's IO rate is nominal, <5000, but it's CPU is high... 350-370% This process didn't die during the scrub, but it was clearly starved.

 

So I'll poke some more, but I honestly believe that it's CPU at issue here. Perhaps shifting the scrub to lower ioprio constrains it from getting data more rapidly, but I find it hard to imagine given that there weren't generally other heavy IO sources.

 

So, ultimately, I have to wonder... is there a way to protect other services on the NAS from the kworkers doing their level best to hammer the CPU? From @Skywalker's notes, the kernel processes are likely busy computing checksums to verify and perhaps there is also mdadm overhead. But is there a way to either (a) pin them to certain CPU threads or (b) constrain their level of activity to ensure other processes can get time? Clearly, you don't want to nice kernel threads... but I'm not sure what other levers are available there.

Message 36 of 37
Laserbait
Luminary

Re: 6.6.1 Scrub Hammers CPU

In my case, it's using so much CPU when scrub is running, my Veeam Endpoint backups fail.

 

VeeamFail.png

Message 37 of 37
Top Contributors
Discussion stats
  • 36 replies
  • 6134 views
  • 1 kudo
  • 9 in conversation
Announcements