NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.

Forum Discussion

fastfwd's avatar
fastfwd
Virtuoso
Jan 24, 2025
Solved

Astonishingly slow scrub

ReadyNAS Pro running OS 6.10.10

6 x 4TB Seagate IronWolf drives, RAID5

ClamAV disabled

 

So this is something I've never seen before...

 

I started a BTRFS scrub on December 26. It ran fine at first, but then slowed dramatically. So I cancelled it and rebooted, and then started another scrub on December 28.

 

After ten days, I checked and it had scrubbed less than 500GB.  It's now been running for almost 4 weeks and has scrubbed a total of 750GB.

 

At this rate, it'll take almost 3 years to complete the scrub.

 

Drives are healthy. There's no other user process using the CPU or the drives (and CPU and drive temperatures are normal). The scrub hasn't found any errors:

 

 

 

root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 251:54:16
        total bytes scrubbed: 485.68GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 285:45:37
        total bytes scrubbed: 506.37GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 296:23:26
        total bytes scrubbed: 515.62GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 322:45:40
        total bytes scrubbed: 531.30GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 357:28:19
        total bytes scrubbed: 548.72GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 381:34:39
        total bytes scrubbed: 563.77GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 428:35:48
        total bytes scrubbed: 590.56GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 444:57:14
        total bytes scrubbed: 603.67GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 463:21:50
        total bytes scrubbed: 617.57GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 476:12:11
        total bytes scrubbed: 628.88GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 518:16:21
        total bytes scrubbed: 663.11GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 622:38:23
        total bytes scrubbed: 741.74GiB with 0 errors
root@NAS3:~# btrfs scrub status /data
scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
        scrub started at Sat Dec 28 23:37:38 2024, running for 635:34:18
        total bytes scrubbed: 752.52GiB with 0 errors

 

 

 

 

Htop shows that the CPU is spending essentially 100% of its time in kernel space, with 4 kworker threads each taking about 50% of a CPU core's time (dual-core CPU).

 

Has anyone else ever seen this?  Any ideas for fixing it?

 

EDITED to add:

 

Quota is disabled.

I defragged immediately before starting this scrub.  Defrag took a couple days.

I've scrubbed this NAS before (although maybe with an earlier OS than 6.10.10), and it was slow -- maybe a week or so -- but never this slow.

The volume is 70% full (12.84TB used out of 18.14TB total).

The box has 4GB of RAM, of which only 750MB is used.

 

3 Replies

  • StephenB's avatar
    StephenB
    Guru - Experienced User

    Are you seeing any errors in the log zip file?

    • fastfwd's avatar
      fastfwd
      Virtuoso

      Thanks for the suggestion, StephenB.  The only errors I saw when I grepped the log files for "error" were a bunch of mail-server authentication errors in msmtp.log.  But those aren't being generated unusually frequently or anything.

       

      Root partition was 82% full because I'd thoughtlessly cloned a big git repo there.  But I just deleted that directory and brought the usage back down to 38%.  No change in btrfs scrub's behavior.

       

      • fastfwd's avatar
        fastfwd
        Virtuoso

        Oh wait a minute.  Looks like the "82% full" messages I saw when I grepped for "warn" were the problem!

         

        Scrub is now writing at 56 MB/sec, up from a few KB every few seconds.

         

        Woohoo!  Thanks, StephenB !

         

         

        root@NAS3:~# btrfs scrub status /data
        scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
                scrub started at Sat Dec 28 23:37:38 2024, running for 640:56:16
                total bytes scrubbed: 975.44GiB with 0 errors
        root@NAS3:~# btrfs scrub status /data
        scrub status for 67868afa-d33f-4eaa-a77a-b7825ffc66e7
                scrub started at Sat Dec 28 23:37:38 2024, running for 640:58:31
                total bytes scrubbed: 982.71GiB with 0 errors

         

NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology! 

Join Us!

ProSupport for Business

Comprehensive support plans for maximum network uptime and business peace of mind.

 

Learn More