× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

RAID 1 resync incredibly slow

RealityDev
Aspirant

Re: RAID 1 resync incredibly slow

Turned off X-RAID and made the setting changes without shutting down but now I notice something interesting. If I change speed_limit_min it changes it only temporarily. If you immediately check the setting after changing you can see the min_speed_limit has indeed changed however after about a minute or so the setting reverts back.

Message 26 of 38
Sandshark
Sensei

Re: RAID 1 resync incredibly slow

When I was running some expansion experiments, I, too, found RAID1 sync to be incredibly slow.  It takes less time for a sync of a RAID5 than for a RAID1 with drives of the same size.  I speculated it had something to do with read/write collisions on a single drive, but I didn't have a lot to go on in making that speculation. 

Message 27 of 38
StephenB
Guru

Re: RAID 1 resync incredibly slow


@Sandshark wrote:

It takes less time for a sync of a RAID5 than for a RAID1 with drives of the same size. 


I am in process of going from 3x6TB+10TB -> 2x6TB+2x10TB. (WD60EFRX + WD100EFAX drives)

  • The 4x6TB RAID group synced in about 50 hours - about 35 MB/sec per drive.
  • The new 2x4TB RAID group looks like it will take about 40 hours - about 30 MB/sec per drive.

While this is certainly slower, it's not "incredibly slower" - though I'll know for sure tomorrow, when the second sync finishes. The throttle seems to be dev.raid.speed_limit_min (which is 30 MiB/sec per drive).  It would be nice to have a turbo mode, where you could boost the sync speed (though you'd likely get very poor performance if you accessed the NAS with this engaged).  

 

@RealityDev is seeing speeds closer to 5 MB/sec per drive - much slower than what I am seeing.

Message 28 of 38
Sandshark
Sensei

Re: RAID 1 resync incredibly slow

While it was on an older Ultra4, I recall speeds of around 7k/sec for a RAID1 sync.

Message 29 of 38
StephenB
Guru

Re: RAID 1 resync incredibly slow


@Sandshark wrote:

While it was on an older Ultra4, I recall speeds of around 7k/sec for a RAID1 sync.


7K/sec?  Or 7 MB/sec

Message 30 of 38
Sandshark
Sensei

Re: RAID 1 resync incredibly slow

Oops, 7M.  MDSTAT reporting in the 7000K/sec range.

Message 31 of 38
MOW3
Aspirant

Re: RAID 1 resync incredibly slow

Ok, here is what I found to solve my RAID rebuilding slow speed:

 

(Old ReadyNAS Pro 2 with 2 WD 1TB drives in RAID 1)

My NAS told me at the beginning: 109 hours  for 2*1TB drives. Thats the standard GUI info.

That's forever.

I thought that updating the OS to 6.10.2 would help, but no.

 

Then I started digging around and followed the instructions listed here. And immediately got better results!

Yes, blockdev --report reported the raise to 65536, but still it should take 40 hours.
(drives are fine btw)
cat /proc/mdstat :
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid1 sdb3[1] sda3[0]
971911808 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.6% (6612160/971911808) finish=2342.3min speed=6868K/sec

That's it?

No!

After I activated the write cache on the drives, the rebuild speed finally got to expected levels:

hdparm -W 1 /dev/sdb

and

hdparm -W 1 /dev/sda

 

cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid1 sdb3[1] sda3[0]
971911808 blocks super 1.2 [2/2] [UU]
[>....................] resync = 3.1% (30783488/971911808) finish=198.9min speed=78828K/sec

 

Well, that means a dramatic drop in the expected finishing time: Just around 3 hours.

No hassling with NCQ, no need to change dev.raid.speed_limit_min or dev.raid.speed_limit_max

And yes, I will remove the WriteCache Setting for the drives after it finished rebuilding, just in case. Smiley Wink

 

Model: RNDP2000|ReadyNAS Pro 2 Chassis only
Message 32 of 38
MOW3
Aspirant

Re: RAID 1 resync incredibly slow

I have to correct my last post a little bit, you can set:

echo 50000 > /proc/sys/dev/raid/speed_limit_min

echo 300000 > /proc/sys/dev/raid/speed_limit_max

 

This should also provide better performance when rebuilding

My second ReadyNAS is rebuilding 2x500GB RAID1 in expected time less than 2 hours.

Message 33 of 38
Sandshark
Sensei

Re: RAID 1 resync incredibly slow


@MOW3 wrote:

I have to correct my last post a little bit, you can set:

echo 50000 > /proc/sys/dev/raid/speed_limit_min

echo 300000 > /proc/sys/dev/raid/speed_limit_max

 

This should also provide better performance when rebuilding

My second ReadyNAS is rebuilding 2x500GB RAID1 in expected time less than 2 hours.


Done by itself, changing the speed limits had negligible effect when I was doing the sync on the Ultra4.  But, WOW, what a difference turning on the write cache had.  I wonder why it is turned off during resync.  hdparm indicates it's on normally on my systems.

/etc/default/hdparm does have some comment text that makes it seem like turning caching on and off during resync is a bad idea, but also contains a work-around (that is not enabled).

Message 34 of 38
MOW3
Aspirant

Re: RAID 1 resync incredibly slow

Well that maybe a concern, but if so, you could cancel the running rebuild, activate write cache and restart rebuilding process I guess.

We just have to keep in mind what a power failure may cause with active write cache.

Since I had no data on the disks, I was not worried.

Maybe a sticky / KB on this would help other frustrated users give them better exerience with Netgear NAS?

Message 35 of 38
Sandshark
Sensei

Re: RAID 1 resync incredibly slow

OK, so I just lost a drive on a Pro2 running OS6, which is on an UPS and normally has write caching on.  The re-sync said it was going to take a huge amount of time, and I checked and found drive caching off.  I turned drive caching on, and the time droped from nearly 40 hours to around 5 (actual time was about 5.5hrs).  It was early in the process, so some of the time reduction reduction may have happened on it's own -- but not that much.

 

I recently had a RAID6 volume to re-sync, and drive caching was not turned off, so I don't know what';s going on with RAID1 resync and drive caching.  Maybe it's something that happens because it's down to only one drive?

 

FYI, @StephenB, something is wrong with your math.

 

3 x 6TB = 24TB in 50 hours is 0.48TB/Hr for the RAID5 and 2 x 4TB = 8TB in 40 hours is 0.20TB/Hr. for the RAID1  So, yes, I would say that 2.4x slower is incredibly slower.  If you use only the usable space (18TB vs. 4TB), it's 3.6X slower for the RAID1.  These numbers are in the same ballpark as my observations.

Message 36 of 38
StephenB
Guru

Re: RAID 1 resync incredibly slow


@Sandshark wrote:

 

FYI, @StephenB, something is wrong with your math.

 


It was a pretty old post - what exactly are you thinking was wrong?

 

The logs have rotated, so I can't see how long the 2x4TB RAID sync actually took.

 

Recreating the first RAID group was 4x6TB, not 3x6TB

 

In any event, you were computing the throughput for the volume, while I was looking at the throughput per drive in the volume (since I was thinking the issue was the drive limit  "dev.raid.speed_limit_min (which is 30 MiB/sec per drive)"  So the factor I was looking at was 1.2, not 2.4.

 

Obviously turning off caching has a huge speed impact, not sure what Netgear was thinking there.  Perhaps they were thinking it would be more robust if the power failed in the middle of the resync?

Message 37 of 38
Sandshark
Sensei

Re: RAID 1 resync incredibly slow


@StephenB wrote:

 

Obviously turning off caching has a huge speed impact, not sure what Netgear was thinking there.  Perhaps they were thinking it would be more robust if the power failed in the middle of the resync?


Yeah, that was my thought at first, but why would a non-redundant RAID1 be any more at risk than a non-redundant RAID5? Obviously, my RAID6 was not completely without protection, so not necessarilly a good comparison, but the increased speed of the RAID5 sync seems to indicate that caching is enabled.

 

So, I'm thinking maybe it's a bug, because I don't remember it always being so much slower.  And maybe it has something to do with having just one good drive.  Cache is turned off because of that (for reasons unknown at this time) and not turned back on when re-sync starts.  I see more experiments on the horizon.  I did not think to check caching status before I replaced the failed drive.

 

As noted above, there are warnings about changing cache status in mid re-sync, but it worked for me as well as it did for the OP.

Message 38 of 38
Top Contributors
Discussion stats
  • 37 replies
  • 6244 views
  • 2 kudos
  • 5 in conversation
Announcements