Of course using the tips there is at your own risk.
I wouldn't disable NCQ myself. All the posts I'm seeing that suggest that are quite old, and seem to target hardware controllers (not software RAID).
I don't have any experience in tuning read-ahead - so if I changed that during sync, I'd change it back to the original value when it is done. You can see that value with blockdev --getra /dev/md127
The firmware version is 6.10.0 and logging in as root allowed me to change the settings. However it doesn't seem like the settings make any difference, it's still ultra slow at over 140 hours to sync a 4TB mirrored RAID. I can't believe Netgear does not optimize the device to sync faster than this, people are okay with taking a week or more to do something that should take less than a day?
I can't believe Netgear does not optimize the device to sync faster than this, people are okay with taking a week or more to do something that should take less than a day?
It shouldn't take a week for 2x4TB so I think something is wrong.
What disks are you using (manufacturer and model)?
Have you looked at the smart stats for both disks?
# smartctl -x /dev/sda # smartctl -x /dev/sdb
The -x will give you some data on recent failed commands, which I think is relevant here.
root@nas1:~# smartctl -x /dev/sda smartctl 6.6 2017-11-05 r4594 [armv7l-linux-4.4.163.alpine.1] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION === Model Family: Western Digital Re Device Model: WDC WD4000FYYZ-05UL1B0 Serial Number: WD-WMC130D3C11E LU WWN Device Id: 5 0014ee 0ae925986 Firmware Version: 00.0NS05 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue May 21 09:13:42 2019 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM level is: 254 (maximum performance) Rd look-ahead is: Enabled Write cache is: Disabled DSN feature is: Unavailable ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (46080) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 497) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported.
SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -41/85 Celsius Temperature History Size (Index): 478 (126)
root@nas1:~# smartctl -x /dev/sdb smartctl 6.6 2017-11-05 r4594 [armv7l-linux-4.4.163.alpine.1] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION === Model Family: Western Digital Re Device Model: WDC WD4000FYYZ-05UL1B0 Serial Number: WD-WCC13JSVRE5D LU WWN Device Id: 5 0014ee 2b54a011d Firmware Version: 00.0NS05 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue May 21 09:14:53 2019 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM level is: 254 (maximum performance) Rd look-ahead is: Enabled Write cache is: Disabled DSN feature is: Unavailable ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (46800) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 506) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported.
General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 5 Comprehensive SMART error log 0x03 GPL R/O 6 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 NCQ Command Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x24 GPL R/O 1 Current Device Internal Status Data log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log 0xa8-0xb1 GPL,SL VS 1 Device vendor specific log 0xb2 GPL VS 65535 Device vendor specific log 0xb2 SL VS 255 Device vendor specific log 0xb3-0xb7 GPL,SL VS 1 Device vendor specific log 0xbd GPL,SL VS 1 Device vendor specific log 0xc0 GPL,SL VS 1 Device vendor specific log 0xc1 GPL VS 24 Device vendor specific log 0xd0 GPL VS 1 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors) Device Error Count: 310 (device log contains only the most recent 24 errors) CR = Command Register FEATR = Features Register COUNT = Count (was: Sector Count) Register LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8 LH = LBA High (was: Cylinder High) Register ] LBA LM = LBA Mid (was: Cylinder Low) Register ] Register LL = LBA Low (was: Sector Number) Register ] DV = Device (was: Device/Head) Register DC = Device Control Register ER = Error register ST = Status register Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 310 [21] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cc 10 5f 40 00 Error: UNC at LBA = 0x02cc105f = 46927967
Error 309 [20] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb d0 e0 40 00 Error: UNC at LBA = 0x02cbd0e0 = 46911712
Error 308 [19] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb d0 e0 40 00 Error: UNC at LBA = 0x02cbd0e0 = 46911712
Error 307 [18] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb c6 64 40 00 Error: UNC at LBA = 0x02cbc664 = 46909028
Error 306 [17] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb bb e8 40 00 Error: UNC at LBA = 0x02cbbbe8 = 46906344
Error 305 [16] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb bb e7 40 00 Error: UNC at LBA = 0x02cbbbe7 = 46906343
Error 304 [15] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb c6 64 40 00 Error: UNC at LBA = 0x02cbc664 = 46909028
Error 303 [14] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb b1 6b 40 00 Error: UNC at LBA = 0x02cbb16b = 46903659
SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -41/85 Celsius Temperature History Size (Index): 478 (202)
There is an automated spam filter that might be kicking in. Mods should be looking in there periodically to release false positives - but if there is a flood of spam that isn't always practical. You can PM me if you have this problem, and I will release your post as soon as I can.
The ~300 errors on sdb are concerning. The ones in the disk cache are all UNCs (uncorrectable errors). That likely is enough to explain the slow sync time.
It isn't clear why they aren't showing up in the SMART stats - though I have run into this before on one of my own disks.
Anyway, I think this disk needs to be replaced - and that if you test it with Lifeguard (both the long read test and the full write-zeros test) you will find that is fails. At least that was the outcome with my own drive.
Error 310 [21] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cc 10 5f 40 00 Error: UNC at LBA = 0x02cc105f = 46927967
Error 309 [20] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb d0 e0 40 00 Error: UNC at LBA = 0x02cbd0e0 = 46911712
Error 308 [19] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb d0 e0 40 00 Error: UNC at LBA = 0x02cbd0e0 = 46911712
Error 307 [18] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb c6 64 40 00 Error: UNC at LBA = 0x02cbc664 = 46909028
Error 306 [17] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb bb e8 40 00 Error: UNC at LBA = 0x02cbbbe8 = 46906344
Error 305 [16] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb bb e7 40 00 Error: UNC at LBA = 0x02cbbbe7 = 46906343
Error 304 [15] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb c6 64 40 00 Error: UNC at LBA = 0x02cbc664 = 46909028
Error 303 [14] occurred at disk power-on lifetime: 179 hours (7 days + 11 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 51 00 00 00 00 02 cb b1 6b 40 00 Error: UNC at LBA = 0x02cbb16b = 46903659
Okay, running the extended test now with Lifeguard. I'm assuming this would give me some kind of report I could give to WD to get a replacement for a defective drive?
Definitely a bad drive, the extended test stopped with the following. Funny thing is it's a 4TB drive but says it's only 1801.76 GB.
Test Option: EXTENDED TEST Model Number: WDC WD40 00FYYZ-05UL1B0 Unit Serial Number: Firmware Number: NS05 Capacity: 1801.76 GB SMART Status: Not Available Test Result: FAIL Test Error Code: 08-Too many bad sectors detected. Test Time: 11:22:47, May 21, 2019
I'm assuming this would give me some kind of report I could give to WD to get a replacement for a defective drive?
WD will give you a recertified replacement (1 year warranty). So return it to the seller if you can - that way you'll get a new drive with the full warranty.
As far as documentation goes, just telling WD that it failed lifeguard is enough. They of course reserve the right to test the returned disk themselves - but neither Seagate nor WD have ever challenged one of my returns.
So I replaced the bad drive, did an extended test with Lifeguard to confirm that it was good. I also did an extended test on the other original good drive just to make sure and it passed the extended test as well. No dice, still just as slow. I then made the following setting changes as outlined in the original post I referenced:
blockdev --setra 65536 /dev/md127
for drive in sd{a..l} do echo 1 > /sys/block/$drive/device/queue_depth done
Still no dice, just as slow. I guess I'm out of options at this point, wish we could go back and get another NAS. Also, I notice when I restart the NAS the setting changes do not stick and are reset to the original values, not sure if this could have something to do with it.
So I replaced the bad drive, did an extended test with Lifeguard to confirm that it was good. I also did an extended test on the other original good drive just to make sure and it passed the extended test as well. No dice, still just as slow.
Tried raising the min to 200000 but still no dice on that, it actually says about 130 hours now. I also notice the setting doesn't stick when you restart so I'm not sure changing the settings is actually really changing them.
I think it does change them, but that the system restores the original settings on reboot.
You could potentially remove a disk, change the setting, and reinsert the disk (and likely re-add it to the array). That of course starts the process over, and might not help.
FWIW, my own NAS is expanding at the moment (and also happens to be syncing a 2x4TB RAID group). It's slower than it should be - around 25 MB/sec. So it appears to be applying the min speed. Still, that should take about 45 hours - not 140.
Are the disks hot swappable? What I've been doing is shutting the thing down before I change a disk and then when it starts up it automatically starts syncing. If I remember right I didn't even get presented with any options to create a RAID to begin with, it just decided to make a mirrored RAID on its own.
If I remember right I didn't even get presented with any options to create a RAID to begin with, it just decided to make a mirrored RAID on its own.
Yes, that is default behavior with XRAID. Though if the disk is formatted, the system won't add it until you select the disk and chose "format" again on the volume tab.
With FlexRAID you explicitly add the disk to the array (and select the RAID mode). You shift to FlexRAID by clicking on the XRAID control on the volume tab. If a green stripe shows on the control, you are in XRAID. If there is no stripe, you are in FlexRAID.
