6.6.1 Scrub Hammers CPU

Got onto this subject in a thread on 6.7.5 but was recently reminded this one was still out there and "unanswered". Wanted to refresh this thread and get back on a single topic. Other thread was https://forum1.netgear.com/t5/ReadyNAS-Beta-Release/ReadyNAS-OS-6-Software-Version-6-7-5-Question

I was encouraged by some changes Skywalker mentioned had happened in 6.7.5, wondering if the scrub behavior may have changed. Initial attempts suggested similar impact to several services on the NAS, including DVBLink, PLEX, AV, and Time Machine.

---

Skywalker wrote:
@btaroli ... The scrub process already sets its ionice level to the lowest (idle) priority, although it's not obvious from normal top output since the I/O priority only gets changed on the child thread that actually does the I/O.  From our testing, the CPU isn't taxed much at all by scrub, so the I/O priority is the only thing that should really matter.

---

 In testing, I found still that these processes were starved (esp for video transcode by DVBLink and PLEX), presumably for CPU. It was suggested by Michael_Oz that I install htop to get a better look, and this did help. As Skywalker indicated, btrfs scrub has a couple of threads. But htop wasn't showing these consistenly coming up as in idle ioprio class. Indeed, btrfs scrub has the -c and -n options specifically to set ioprio on the scrub. It's not clear how it's being done now, but it doesn't seem to be having the intended effect.

# btrfs scrub
usage: btrfs scrub <command> [options] <path>|<device>

    btrfs scrub start [-BdqrRf] [-c ioprio_class -n ioprio_classdata] <path>|<device>
        Start a new scrub. If a scrub is already running, the new one fails.
    btrfs scrub cancel <path>|<device>
        Cancel a running scrub
    btrfs scrub resume [-BdqrR] [-c ioprio_class -n ioprio_classdata] <path>|<device>
        Resume previously canceled or interrupted scrub
    btrfs scrub status [-dR] <path>|<device>
        Show status of running or finished scrub

verify checksums of data and metadata

What I found was that once I had set "idle" (ionice -c 3 -p ####) on all three btrfs scrub processes/threads THEN I began to see that PLEX, DVBLink, Time Machine were all able to complete their work normally without impact.

I was playing a bit more with htop and sorting on various columns. When I sorted reverse on IO_PRIORITY, though, I noticed something really curious and may actually expose why it's important that btrfs scrub must run fully in idle ioprio in order to ensure no impact to services. In particular, I noticed the following processes running in low priority classes:

idle

   btrfs scrub

B7

   clamd
   PLEX (background process)
   watchdog

B6

   DVBLink (ffmpeg)

B4

   kworker/u8:*
   DVBLink (server)
   PLEX (server)
   PLEX (transcoder)
   Dropbox
   Transmission
   md*_raid6
   dbus_daemon
   ksoftirqd/*

It looks very much like services (including clamd) are using ioprio in B6 and B7, and might get starved if something like btrfs scrub (or any other background maintenance process) wasn't running in idle class. Once btrfs scrub was confirmed (and reported by htop) has being in idle class it stopped interfering with other processes.

Indeed when I look at the values for nice and ioprio that processes are using, I wonder if some of them don't need to be ratcheted down in priority quite as far as they are. Too many things set down in B6 and B7 might created problems, no? Have to leave room for things that really do belong down there. :D Think I'll whisper over in PLEX and DVBLink forums on that subject.