RAID 1 resync incredibly slow

Tried raising the min to 200000 but still no dice on that, it actually says about 130 hours now. I also notice the setting doesn't stick when you restart so I'm not sure changing the settings is actually really changing them.

I think it does change them, but that the system restores the original settings on reboot.

You could potentially remove a disk, change the setting, and reinsert the disk (and likely re-add it to the array).  That of course starts the process over, and might not help.

FWIW, my own NAS is expanding at the moment (and also happens to be syncing a 2x4TB RAID group).  It's slower than it should be - around 25 MB/sec.  So it appears to be applying the min speed. Still, that should take about 45 hours - not 140.

Are the disks hot swappable? What I've been doing is shutting the thing down before I change a disk and then when it starts up it automatically starts syncing. If I remember right I didn't even get presented with any options to create a RAID to begin with, it just decided to make a mirrored RAID on its own.

---

RealityDev wrote:

Are the disks hot swappable? 

---

They are - you don't need to power down (and my normal recommendation is not to).

---

RealityDev wrote:

If I remember right I didn't even get presented with any options to create a RAID to begin with, it just decided to make a mirrored RAID on its own.

---

Yes, that is default behavior with XRAID.  Though if the disk is formatted, the system won't add it until you select the disk and chose "format" again on the volume tab.

With FlexRAID you explicitly add the disk to the array (and select the RAID mode).  You shift to FlexRAID by clicking on the XRAID control on the volume tab.  If a green stripe shows on the control, you are in XRAID.  If there is no stripe, you are in FlexRAID.

Turned off X-RAID and made the setting changes without shutting down but now I notice something interesting. If I change speed_limit_min it changes it only temporarily. If you immediately check the setting after changing you can see the min_speed_limit has indeed changed however after about a minute or so the setting reverts back.

When I was running some expansion experiments, I, too, found RAID1 sync to be incredibly slow.  It takes less time for a sync of a RAID5 than for a RAID1 with drives of the same size.  I speculated it had something to do with read/write collisions on a single drive, but I didn't have a lot to go on in making that speculation. 

---

Sandshark wrote:

It takes less time for a sync of a RAID5 than for a RAID1 with drives of the same size. 

---

I am in process of going from 3x6TB+10TB -> 2x6TB+2x10TB. (WD60EFRX + WD100EFAX drives)

• The 4x6TB RAID group synced in about 50 hours - about 35 MB/sec per drive.
• The new 2x4TB RAID group looks like it will take about 40 hours - about 30 MB/sec per drive.

While this is certainly slower, it's not "incredibly slower" - though I'll know for sure tomorrow, when the second sync finishes. The throttle seems to be dev.raid.speed_limit_min (which is 30 MiB/sec per drive).  It would be nice to have a turbo mode, where you could boost the sync speed (though you'd likely get very poor performance if you accessed the NAS with this engaged).  

RealityDev is seeing speeds closer to 5 MB/sec per drive - much slower than what I am seeing.

While it was on an older Ultra4, I recall speeds of around 7k/sec for a RAID1 sync.

---

Sandshark wrote:

While it was on an older Ultra4, I recall speeds of around 7k/sec for a RAID1 sync.

---

7K/sec?  Or 7 MB/sec

Oops, 7M.  MDSTAT reporting in the 7000K/sec range.

Ok, here is what I found to solve my RAID rebuilding slow speed:

(Old ReadyNAS Pro 2 with 2 WD 1TB drives in RAID 1)

My NAS told me at the beginning: 109 hours  for 2*1TB drives. Thats the standard GUI info.

That's forever.

I thought that updating the OS to 6.10.2 would help, but no.

Then I started digging around and followed the instructions listed here. And immediately got better results!

Yes, blockdev --report reported the raise to 65536, but still it should take 40 hours.
(drives are fine btw)
cat /proc/mdstat :
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid1 sdb3[1] sda3[0]
971911808 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.6% (6612160/971911808) finish=2342.3min speed=6868K/sec

That's it?

No!

After I activated the write cache on the drives, the rebuild speed finally got to expected levels:

hdparm -W 1 /dev/sdb

and

hdparm -W 1 /dev/sda

cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid1 sdb3[1] sda3[0]
971911808 blocks super 1.2 [2/2] [UU]
[>....................] resync = 3.1% (30783488/971911808) finish=198.9min speed=78828K/sec

Well, that means a dramatic drop in the expected finishing time: Just around 3 hours.

No hassling with NCQ, no need to change dev.raid.speed_limit_min or dev.raid.speed_limit_max

And yes, I will remove the WriteCache Setting for the drives after it finished rebuilding, just in case. :smileywink:

I have to correct my last post a little bit, you can set:

echo 50000 > /proc/sys/dev/raid/speed_limit_min

echo 300000 > /proc/sys/dev/raid/speed_limit_max

This should also provide better performance when rebuilding

My second ReadyNAS is rebuilding 2x500GB RAID1 in expected time less than 2 hours.

---

MOW3 wrote:

I have to correct my last post a little bit, you can set:

echo 50000 > /proc/sys/dev/raid/speed_limit_min

echo 300000 > /proc/sys/dev/raid/speed_limit_max

 

This should also provide better performance when rebuilding

My second ReadyNAS is rebuilding 2x500GB RAID1 in expected time less than 2 hours.

---

Done by itself, changing the speed limits had negligible effect when I was doing the sync on the Ultra4.  But, WOW, what a difference turning on the write cache had.  I wonder why it is turned off during resync.  hdparm indicates it's on normally on my systems.

/etc/default/hdparm does have some comment text that makes it seem like turning caching on and off during resync is a bad idea, but also contains a work-around (that is not enabled).

Well that maybe a concern, but if so, you could cancel the running rebuild, activate write cache and restart rebuilding process I guess.

We just have to keep in mind what a power failure may cause with active write cache.

Since I had no data on the disks, I was not worried.

Maybe a sticky / KB on this would help other frustrated users give them better exerience with Netgear NAS?

OK, so I just lost a drive on a Pro2 running OS6, which is on an UPS and normally has write caching on.  The re-sync said it was going to take a huge amount of time, and I checked and found drive caching off.  I turned drive caching on, and the time droped from nearly 40 hours to around 5 (actual time was about 5.5hrs).  It was early in the process, so some of the time reduction reduction may have happened on it's own -- but not that much.

I recently had a RAID6 volume to re-sync, and drive caching was not turned off, so I don't know what';s going on with RAID1 resync and drive caching.  Maybe it's something that happens because it's down to only one drive?

FYI, StephenB, something is wrong with your math.

3 x 6TB = 24TB in 50 hours is 0.48TB/Hr for the RAID5 and 2 x 4TB = 8TB in 40 hours is 0.20TB/Hr. for the RAID1  So, yes, I would say that 2.4x slower is incredibly slower.  If you use only the usable space (18TB vs. 4TB), it's 3.6X slower for the RAID1.  These numbers are in the same ballpark as my observations.

---

Sandshark wrote:

 

FYI, StephenB, something is wrong with your math.

 

---

It was a pretty old post - what exactly are you thinking was wrong?

The logs have rotated, so I can't see how long the 2x4TB RAID sync actually took.

Recreating the first RAID group was 4x6TB, not 3x6TB

In any event, you were computing the throughput for the volume, while I was looking at the throughput per drive in the volume (since I was thinking the issue was the drive limit  "dev.raid.speed_limit_min (which is 30 MiB/sec per drive)"  So the factor I was looking at was 1.2, not 2.4.

Obviously turning off caching has a huge speed impact, not sure what Netgear was thinking there.  Perhaps they were thinking it would be more robust if the power failed in the middle of the resync?

---

StephenB wrote:

---

 

Obviously turning off caching has a huge speed impact, not sure what Netgear was thinking there.  Perhaps they were thinking it would be more robust if the power failed in the middle of the resync?

---

Yeah, that was my thought at first, but why would a non-redundant RAID1 be any more at risk than a non-redundant RAID5? Obviously, my RAID6 was not completely without protection, so not necessarilly a good comparison, but the increased speed of the RAID5 sync seems to indicate that caching is enabled.

So, I'm thinking maybe it's a bug, because I don't remember it always being so much slower.  And maybe it has something to do with having just one good drive.  Cache is turned off because of that (for reasons unknown at this time) and not turned back on when re-sync starts.  I see more experiments on the horizon.  I did not think to check caching status before I replaced the failed drive.

As noted above, there are warnings about changing cache status in mid re-sync, but it worked for me as well as it did for the OP.