× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

dlavineway
Aspirant

ReadyNAS Ultra 4 - The array is still in degraded mode

Hello just added two new 4 TB drive to my Ultra4; first drive loaded and sync correctly however after installing, test and sync the 2nd I'm now in Unprotected mode with the following messge:

"RAID sync finished on volume C. The array is still in degraded mode, however. This can be caused by a disk sync failure or failed disks in a multi-parity disk array."

 

Current config is as follows

Ch 1 : WDC WD40EFRX-68N32N0 [3726 GB] 3721 GB allocated (disk failured)
Ch 2 : WDC WD40EFRX-68N32N0 [3726 GB] 3721 GB allocated
Ch 3 : WDC WD30EFRX-68EUZN0 [2794 GB] 2790 GB allocated
Ch 4 : WDC WD30EFRX-68EUZN0 [2794 GB] 2790 GB allocated

 

I've removed the failed drive formated it and scanned it for any faults, everything came back clean; tried installing it again to the same result.

 

I can provide logs to anyone who needs them. I'd appreciate any help that you could provide.

Model: RNDU4000|ReadyNAS Ultra 4 Chassis only
Message 1 of 19

Accepted Solutions
StephenB
Guru

Re: ReadyNAS Ultra 4 - The array is still in degraded mode


@dlavineway wrote:

I've tried posting the mdstat.log contents multiple time unfortunatly the post keeps disappearing, trying it as an attachment


Likely just caught in the spam filter. 

 

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]

md4 : active raid5 sda5[0] sdb5[1]
976750784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [2/2] [UU]

md3 : active raid5 sda4[4] sdc4[3] sdd4[2] sdb4[5]
5860511616 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md2 : active raid5 sdd3[7] sdb3[5] sdc3[4]
2916119040 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]

md1 : active raid1 sda2[6] sdb2[5] sdc2[4] sdd2[7]
524276 blocks super 1.2 [4/4] [UUUU]

md0 : active raid1 sda1[6] sdb1[5] sdc1[4] sdd1[7]
4193268 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

It looks like the problem is with md2 (and that you started off with a 4x1 TB array).

 

You could try

  • removing the disk
  • power down the NAS
  • disconnect from the power main for a few minutes (maybe 15)
  • reboot the NAS
  • reinsert the disk.

Part of the rationale here is to clear out any bios state associated with the disk.  Sometime the NAS drivers will declare a SATA bay down, and this should make sure it's up again.

 

 

View solution in original post

Message 16 of 19

All Replies
StephenB
Guru

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

What RAID mode are you running?

 

One thing you can try - connect the disk to a Windows PC and test it with WD's lifeguard program.  Use the "advanced" write zeros test this time - it sometimes picks up issues that the non-destructive test misses.

 

Also, have you looked at the smart statistics in the logs?

Message 2 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

Hi StephenB


Raid Mode: RAID Level X-RAID2, 4 disks

Smart status: says Dead

When I tested the drive orginially I used AOMEI partition assistant from: https://www.disk-partition.com/ and used the quick scan

Message 3 of 19
StephenB
Guru

Re: ReadyNAS Ultra 4 - The array is still in degraded mode


@dlavineway wrote:

 

When I tested the drive orginially I used AOMEI partition assistant from: https://www.disk-partition.com/ and used the quick scan


Then I suggest retesting with Lifeguard.  Do the long non-destructive test first - if it passes, move on to the full write-zeros test.  I suspect one of those two tests will fail.

 


@dlavineway wrote:

 

Smart status: says Dead

 


You are likely looking at the web ui.  You should see a lot more in disk_smart.log (in the log zip).  That would include reallocated and current pending sectors, uncorrectable errors, crc error counts, etc.

 

Message 4 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

attaching smart log

Message 5 of 19
StephenB
Guru

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

Can you post again, showing the "RAW_VALUE" field at the end.

 

FWIW, you can also copy/paste the stats for disk 1 into your reply.  Click on the "insert code" tool and then paste the stats into the pop-up.

Message 6 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:    WD-WCC7K0AUHZJH
LU WWN Device Id: 5 0014ee 2113048c3
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ACS-3 (unknown minor revision code: 0x006d)
Local Time is:    Thu Aug  1 05:59:28 2019 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(44820) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 255) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   166   166   021    Pre-fail  Always       -       6683
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       11
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       125
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       6
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       3
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       12
194 Temperature_Celsius     0x0022   120   111   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       111         -
# 2  Short offline       Completed without error       00%        95         -
# 3  Short offline       Completed without error       00%        42         -
# 4  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Message 7 of 19
StephenB
Guru

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

Thx.  So this drive doesn't have any SMART errors logged.  Perhaps check the other disks for errors (looking at the RAW_VALUE field).

 

Can you also post mdstat.log?

Message 8 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

Currently on the last test for lifeguard so far the drive is passing test 😞

 

Here is the mdstat.log

 

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md4 : active raid5 sda5[0] sdb5[1]
      976750784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [2/2] [UU]
      
md3 : active raid5 sda4[4] sdc4[3] sdd4[2] sdb4[5]
      5860511616 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
md2 : active raid5 sdd3[7] sdb3[5] sdc3[4]
      2916119040 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
      
md1 : active raid1 sda2[6] sdb2[5] sdc2[4] sdd2[7]
      524276 blocks super 1.2 [4/4] [UUUU]
      
md0 : active raid1 sda1[6] sdb1[5] sdc1[4] sdd1[7]
      4193268 blocks super 1.2 [4/4] [UUUU]
      
unused devices: <none>
Message 9 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

disk_smart.log (disk2)

***** smartctl output for sdb *****

smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.37.6.RNx86_64.2.4] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:    WD-WCC7K2TTF449
LU WWN Device Id: 5 0014ee 2bbdb519b
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ACS-3 (unknown minor revision code: 0x006d)
Local Time is:    Thu Aug  1 05:59:29 2019 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(44700) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 255) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       1
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       118
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1
194 Temperature_Celsius     0x0022   118   110   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Message 10 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

On the last test of lifeguard, so far the drive is passing the test.

 

mdstat.log

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md4 : active raid5 sda5[0] sdb5[1]
      976750784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [2/2] [UU]
      
md3 : active raid5 sda4[4] sdc4[3] sdd4[2] sdb4[5]
      5860511616 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
md2 : active raid5 sdd3[7] sdb3[5] sdc3[4]
      2916119040 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
      
md1 : active raid1 sda2[6] sdb2[5] sdc2[4] sdd2[7]
      524276 blocks super 1.2 [4/4] [UUUU]
      
md0 : active raid1 sda1[6] sdb1[5] sdc1[4] sdd1[7]
      4193268 blocks super 1.2 [4/4] [UUUU]
      
unused devices: <none>

 

Message 11 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

On the last test of lifeguard, so far the drive is passing the test.

mdstat.log

 

 

 

Message 12 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

On the last test of lifeguard, disk has passed all test so far 😞

Message 13 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

On the last test of lifeguard, so far the disk has passed all other tests

Message 14 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

I've tried posting the mdstat.log contents multiple time unfortunatly the post keeps disappearing, trying it as an attachment

 

Message 15 of 19
StephenB
Guru

Re: ReadyNAS Ultra 4 - The array is still in degraded mode


@dlavineway wrote:

I've tried posting the mdstat.log contents multiple time unfortunatly the post keeps disappearing, trying it as an attachment


Likely just caught in the spam filter. 

 

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]

md4 : active raid5 sda5[0] sdb5[1]
976750784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [2/2] [UU]

md3 : active raid5 sda4[4] sdc4[3] sdd4[2] sdb4[5]
5860511616 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md2 : active raid5 sdd3[7] sdb3[5] sdc3[4]
2916119040 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]

md1 : active raid1 sda2[6] sdb2[5] sdc2[4] sdd2[7]
524276 blocks super 1.2 [4/4] [UUUU]

md0 : active raid1 sda1[6] sdb1[5] sdc1[4] sdd1[7]
4193268 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

It looks like the problem is with md2 (and that you started off with a 4x1 TB array).

 

You could try

  • removing the disk
  • power down the NAS
  • disconnect from the power main for a few minutes (maybe 15)
  • reboot the NAS
  • reinsert the disk.

Part of the rationale here is to clear out any bios state associated with the disk.  Sometime the NAS drivers will declare a SATA bay down, and this should make sure it's up again.

 

 

Message 16 of 19
Sandshark
Sensei

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

It doesn't look like a dead bay issue, because all the other partitions of Drive 1 (sda) are included in their appropriate RAID groups.

 

I think you are going to find a bad sector on the drive.  But why it wouldn't just re-allocate it and go on is a mystery.

 

If it still doesn't sync after you've done all @StephenB has advised, I can give you SSH command for it to re-try adding the partition to the RAID.  But it's probably unlikely that would be any more successful.

Message 17 of 19
dlavineway
Aspirant

Re: ReadyNAS Ultra 4 - The array is still in degraded mode

StephenB, your solution work; thank you very much

Message 18 of 19
StephenB
Guru

Re: ReadyNAS Ultra 4 - The array is still in degraded mode


@Sandshark wrote:

It doesn't look like a dead bay issue, because all the other partitions of Drive 1 (sda) are included in their appropriate RAID groups.

It didn't seem that likely to me either.  But in the first post he said the drive was identified in web ui as dead - and it wasn't clear as to why.  So I suggested the longer procedure.

Message 19 of 19
Top Contributors
Discussion stats
  • 18 replies
  • 3660 views
  • 1 kudo
  • 3 in conversation
Announcements