- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- « Previous
-
- 1
- 2
- Next »
RAID 1 resync incredibly slow
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
Turned off X-RAID and made the setting changes without shutting down but now I notice something interesting. If I change speed_limit_min it changes it only temporarily. If you immediately check the setting after changing you can see the min_speed_limit has indeed changed however after about a minute or so the setting reverts back.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
When I was running some expansion experiments, I, too, found RAID1 sync to be incredibly slow. It takes less time for a sync of a RAID5 than for a RAID1 with drives of the same size. I speculated it had something to do with read/write collisions on a single drive, but I didn't have a lot to go on in making that speculation.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
@Sandshark wrote:
It takes less time for a sync of a RAID5 than for a RAID1 with drives of the same size.
I am in process of going from 3x6TB+10TB -> 2x6TB+2x10TB. (WD60EFRX + WD100EFAX drives)
- The 4x6TB RAID group synced in about 50 hours - about 35 MB/sec per drive.
- The new 2x4TB RAID group looks like it will take about 40 hours - about 30 MB/sec per drive.
While this is certainly slower, it's not "incredibly slower" - though I'll know for sure tomorrow, when the second sync finishes. The throttle seems to be dev.raid.speed_limit_min (which is 30 MiB/sec per drive). It would be nice to have a turbo mode, where you could boost the sync speed (though you'd likely get very poor performance if you accessed the NAS with this engaged).
@RealityDev is seeing speeds closer to 5 MB/sec per drive - much slower than what I am seeing.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
While it was on an older Ultra4, I recall speeds of around 7k/sec for a RAID1 sync.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
@Sandshark wrote:
While it was on an older Ultra4, I recall speeds of around 7k/sec for a RAID1 sync.
7K/sec? Or 7 MB/sec
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
Oops, 7M. MDSTAT reporting in the 7000K/sec range.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
Ok, here is what I found to solve my RAID rebuilding slow speed:
(Old ReadyNAS Pro 2 with 2 WD 1TB drives in RAID 1)
My NAS told me at the beginning: 109 hours for 2*1TB drives. Thats the standard GUI info.
That's forever.
I thought that updating the OS to 6.10.2 would help, but no.
Then I started digging around and followed the instructions listed here. And immediately got better results!
Yes, blockdev --report reported the raise to 65536, but still it should take 40 hours.
(drives are fine btw)
cat /proc/mdstat :
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid1 sdb3[1] sda3[0]
971911808 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.6% (6612160/971911808) finish=2342.3min speed=6868K/sec
That's it?
No!
After I activated the write cache on the drives, the rebuild speed finally got to expected levels:
hdparm -W 1 /dev/sdb
and
hdparm -W 1 /dev/sda
cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid1 sdb3[1] sda3[0]
971911808 blocks super 1.2 [2/2] [UU]
[>....................] resync = 3.1% (30783488/971911808) finish=198.9min speed=78828K/sec
Well, that means a dramatic drop in the expected finishing time: Just around 3 hours.
No hassling with NCQ, no need to change dev.raid.speed_limit_min or dev.raid.speed_limit_max
And yes, I will remove the WriteCache Setting for the drives after it finished rebuilding, just in case.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
I have to correct my last post a little bit, you can set:
echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 300000 > /proc/sys/dev/raid/speed_limit_max
This should also provide better performance when rebuilding
My second ReadyNAS is rebuilding 2x500GB RAID1 in expected time less than 2 hours.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
@MOW3 wrote:I have to correct my last post a little bit, you can set:
echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 300000 > /proc/sys/dev/raid/speed_limit_max
This should also provide better performance when rebuilding
My second ReadyNAS is rebuilding 2x500GB RAID1 in expected time less than 2 hours.
Done by itself, changing the speed limits had negligible effect when I was doing the sync on the Ultra4. But, WOW, what a difference turning on the write cache had. I wonder why it is turned off during resync. hdparm indicates it's on normally on my systems.
/etc/default/hdparm does have some comment text that makes it seem like turning caching on and off during resync is a bad idea, but also contains a work-around (that is not enabled).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
Well that maybe a concern, but if so, you could cancel the running rebuild, activate write cache and restart rebuilding process I guess.
We just have to keep in mind what a power failure may cause with active write cache.
Since I had no data on the disks, I was not worried.
Maybe a sticky / KB on this would help other frustrated users give them better exerience with Netgear NAS?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
OK, so I just lost a drive on a Pro2 running OS6, which is on an UPS and normally has write caching on. The re-sync said it was going to take a huge amount of time, and I checked and found drive caching off. I turned drive caching on, and the time droped from nearly 40 hours to around 5 (actual time was about 5.5hrs). It was early in the process, so some of the time reduction reduction may have happened on it's own -- but not that much.
I recently had a RAID6 volume to re-sync, and drive caching was not turned off, so I don't know what';s going on with RAID1 resync and drive caching. Maybe it's something that happens because it's down to only one drive?
FYI, @StephenB, something is wrong with your math.
3 x 6TB = 24TB in 50 hours is 0.48TB/Hr for the RAID5 and 2 x 4TB = 8TB in 40 hours is 0.20TB/Hr. for the RAID1 So, yes, I would say that 2.4x slower is incredibly slower. If you use only the usable space (18TB vs. 4TB), it's 3.6X slower for the RAID1. These numbers are in the same ballpark as my observations.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
@Sandshark wrote:
FYI, @StephenB, something is wrong with your math.
It was a pretty old post - what exactly are you thinking was wrong?
The logs have rotated, so I can't see how long the 2x4TB RAID sync actually took.
Recreating the first RAID group was 4x6TB, not 3x6TB
In any event, you were computing the throughput for the volume, while I was looking at the throughput per drive in the volume (since I was thinking the issue was the drive limit "dev.raid.speed_limit_min (which is 30 MiB/sec per drive)" So the factor I was looking at was 1.2, not 2.4.
Obviously turning off caching has a huge speed impact, not sure what Netgear was thinking there. Perhaps they were thinking it would be more robust if the power failed in the middle of the resync?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RAID 1 resync incredibly slow
@StephenB wrote:
Obviously turning off caching has a huge speed impact, not sure what Netgear was thinking there. Perhaps they were thinking it would be more robust if the power failed in the middle of the resync?
Yeah, that was my thought at first, but why would a non-redundant RAID1 be any more at risk than a non-redundant RAID5? Obviously, my RAID6 was not completely without protection, so not necessarilly a good comparison, but the increased speed of the RAID5 sync seems to indicate that caching is enabled.
So, I'm thinking maybe it's a bug, because I don't remember it always being so much slower. And maybe it has something to do with having just one good drive. Cache is turned off because of that (for reasons unknown at this time) and not turned back on when re-sync starts. I see more experiments on the horizon. I did not think to check caching status before I replaced the failed drive.
As noted above, there are warnings about changing cache status in mid re-sync, but it worked for me as well as it did for the OP.
- « Previous
-
- 1
- 2
- Next »