Ultra 4 with 6.4.0: Volume data is Degraded

[F_L_]

I got an email from my NAS saying that Volume data is Degraded.

Fri Dec 11 2015 1:00:33	
Volume: Volume data is Degraded.
Fri Dec 11 2015 0:12:32	
snapshot: Smart Snapshot successfully deleted snapshot 2015_11_13__00_00_41 from share or LUN Pictures.
Fri Dec 11 2015 0:12:19	
snapshot: Smart Snapshot successfully deleted snapshot 2015_11_13__00_00_40 from share or LUN Documents.
Fri Dec 11 2015 0:12:14	
snapshot: Smart Snapshot successfully deleted snapshot 2015_11_13__00_00_38 from share or LUN Backup.
Fri Dec 11 2015 0:00:39	
Miscellaneous: Snapshot c_1449788433 was successfully created for share or LUN Pictures.
Fri Dec 11 2015 0:00:32	
Miscellaneous: Snapshot c_1449788432 was successfully created for share or LUN Documents.
Fri Dec 11 2015 0:00:30	
Miscellaneous: Snapshot c_1449788429 was successfully created for share or LUN Backup.
Thu Dec 10 2015 19:55:14	
Volume: Volume data health changed from Redundant to Degraded.

But looking at my drives all 4 show healthy with green icon.

State ONLINE and Error Count is 0 on all of them.

These are all "old" HDDs so I have not added any new before getting this message.

If one drive is failing, how can I know which one?

What actions can I take? (after backup)

[StephenB · Accepted solution]

---

@F_L_ wrote:

Best way is to hot-replace the old one right?

---

Yes.  

Though if the kernel has disabled the sata interface for the slot, you might need to reboot.  So if the system doesn't see the drive insertion, try that.

[F_L_]

Disk info logs:

Device:             sda
Controller:         0
Channel:            0
Model:              WDC WD20EARS-00J99B0
Serial:             WD-WCAWZ1395124
Firmware:           80.00A80
Class:              SATA
Sectors:            3907029168
Pool:               data
PoolType:           RAID 5
PoolState:          3
PoolHostId:         5e26b5f2
Health data ;
  ATA Error Count:                0
  Reallocated Sectors:            0
  Reallocation Events:            0
  Spin Retry Count:               0
  Current Pending Sector Count:   0
  Uncorrectable Sector Count:     0
  Temperature:                    28
  Start/Stop Count:               61
  Power-On Hours:                 36891
  Power Cycle Count:              59
  Load Cycle Count:               32004

Device:             sdb
Controller:         0
Channel:            1
Model:              WDC WD20EARS-00J99B0
Serial:             WD-WCAWZ1499595
Firmware:           80.00A80
Class:              SATA
Sectors:            3907029168
Pool:               data
PoolType:           RAID 5
PoolState:          3
PoolHostId:         5e26b5f2
Health data ;
  ATA Error Count:                0
  Reallocated Sectors:            0
  Reallocation Events:            0
  Spin Retry Count:               0
  Current Pending Sector Count:   1
  Uncorrectable Sector Count:     0
  Temperature:                    28
  Start/Stop Count:               57
  Power-On Hours:                 36864
  Power Cycle Count:              55
  Load Cycle Count:               29812

Device:             sdc
Controller:         0
Channel:            2
Model:              WDC WD30EFRX-68AX9N0
Serial:             WD-WMC1T1298310
Firmware:           80.00A80
Class:              SATA
Sectors:            5860533168
Pool:               data
PoolType:           RAID 5
PoolState:          3
PoolHostId:         5e26b5f2
Health data ;
  ATA Error Count:                0
  Reallocated Sectors:            0
  Reallocation Events:            0
  Spin Retry Count:               0
  Current Pending Sector Count:   0
  Uncorrectable Sector Count:     0
  Temperature:                    24
  Start/Stop Count:               36
  Power-On Hours:                 23934
  Power Cycle Count:              36
  Load Cycle Count:               17

Device:             sdd
Controller:         0
Channel:            3
Model:              WDC WD30EFRX-68AX9N0
Serial:             WD-WCC1T1604088
Firmware:           80.00A80
Class:              SATA
Sectors:            5860533168
Pool:               data
PoolType:           RAID 5
PoolState:          3
PoolHostId:         5e26b5f2
Health data ;
  ATA Error Count:                0
  Reallocated Sectors:            0
  Reallocation Events:            0
  Spin Retry Count:               0
  Current Pending Sector Count:   0
  Uncorrectable Sector Count:     0
  Temperature:                    21
  Start/Stop Count:               27
  Power-On Hours:                 16715
  Power Cycle Count:              27
  Load Cycle Count:               14

[F_L_]

Think I found it in the Kernel.log:

Dec 10 19:54:28 Ultra kernel: sd 1:0:0:0: [sdb] tag#20 Add. Sense: Unrecovered read error - auto reallocate failed
Dec 10 19:54:28 Ultra kernel: sd 1:0:0:0: [sdb] tag#20 CDB: Write(10) 2a 00 a9 bf e7 c0 00 00 40 00
Dec 10 19:54:28 Ultra kernel: blk_update_request: I/O error, dev sdb, sector 2847926208
Dec 10 19:54:28 Ultra kernel: md/raid:md127: Disk failure on sdb3, disabling device.
Dec 10 19:54:28 Ultra kernel: md/raid:md127: Operation continuing on 3 devices.
Dec 10 19:54:28 Ultra kernel: sd 1:0:0:0: [sdb] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 10 19:54:28 Ultra kernel: sd 1:0:0:0: [sdb] tag#21 Sense Key : Medium Error [current] [descriptor] 
Dec 10 19:54:28 Ultra kernel: sd 1:0:0:0: [sdb] tag#21 Add. Sense: Unrecovered read error - auto reallocate failed
Dec 10 19:54:28 Ultra kernel: sd 1:0:0:0: [sdb] tag#21 CDB: Read(10) 28 00 6c 69 ad 00 00 00 40 00
Dec 10 19:54:28 Ultra kernel: blk_update_request: I/O error, dev sdb, sector 1818864896
Dec 10 19:54:28 Ultra kernel: ata2: EH complete
Dec 10 19:54:30 Ultra kernel: ata2.00: exception Emask 0x0 SAct 0x8000040 SErr 0x0 action 0x0
Dec 10 19:54:30 Ultra kernel: ata2.00: irq_stat 0x40000008
Dec 10 19:54:30 Ultra kernel: ata2.00: failed command: READ FPDMA QUEUED
Dec 10 19:54:30 Ultra kernel: ata2.00: cmd 60/08:d8:38:e8:bf/00:00:a9:00:00/40 tag 27 ncq 4096 in
Dec 10 19:54:30 Ultra kernel:          res 41/40:00:38:e8:bf/00:00:a9:00:00/40 Emask 0x409 (media error) <F>
Dec 10 19:54:30 Ultra kernel: ata2.00: status: { DRDY ERR }
Dec 10 19:54:30 Ultra kernel: ata2.00: error: { UNC }
Dec 10 19:54:30 Ultra kernel: ata2.00: configured for UDMA/133
Dec 10 19:54:30 Ultra kernel: sd 1:0:0:0: [sdb] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 10 19:54:30 Ultra kernel: sd 1:0:0:0: [sdb] tag#27 Sense Key : Medium Error [current] [descriptor] 
Dec 10 19:54:30 Ultra kernel: sd 1:0:0:0: [sdb] tag#27 Add. Sense: Unrecovered read error - auto reallocate failed
Dec 10 19:54:30 Ultra kernel: sd 1:0:0:0: [sdb] tag#27 CDB: Read(10) 28 00 a9 bf e8 38 00 00 08 00
Dec 10 19:54:30 Ultra kernel: blk_update_request: I/O error, dev sdb, sector 2847926328
Dec 10 19:54:30 Ultra kernel: md/raid:md127: read error not correctable (sector 2838489080 on sdb3).
Dec 10 19:54:30 Ultra kernel: RAID conf printout:
Dec 10 19:54:30 Ultra kernel:  --- level:5 rd:4 wd:3
Dec 10 19:54:30 Ultra kernel:  disk 0, o:1, dev:sda3
Dec 10 19:54:30 Ultra kernel:  disk 1, o:0, dev:sdb3
Dec 10 19:54:30 Ultra kernel:  disk 2, o:1, dev:sdc3
Dec 10 19:54:30 Ultra kernel: ata2: EH complete
Dec 10 19:54:30 Ultra kernel:  disk 3, o:1, dev:sdd3
Dec 10 19:54:30 Ultra kernel: RAID conf printout:
Dec 10 19:54:30 Ultra kernel:  --- level:5 rd:4 wd:3
Dec 10 19:54:30 Ultra kernel:  disk 0, o:1, dev:sda3
Dec 10 19:54:30 Ultra kernel:  disk 2, o:1, dev:sdc3
Dec 10 19:54:30 Ultra kernel:  disk 3, o:1, dev:sdd3

Seems like the second drive is failing. Strange I couldn't see this from the GUI.

Or am I missing something @mdgm-ntgr, @StephenB?

Thanks !

[StephenB]

---

@F_L_ wrote:

Seems like the second drive is failing. Strange I couldn't see this from the GUI.

Or am I missing something @mdgm-ntgr, @StephenB?

---

 That's the single "current sector pending count" shown in the SMART stats.

RAID should have recovered the data from the parity block automatically, and it should not have degraded the array.  So I'm not sure why it took the drive out of service (and of course the web ui should have told you the drive had failed).  Though SMART stats don't tell you everything.

You could power down the system and test the drive with lifeguard in a PC.  Or just replace it with a WD30EFRX.

[F_L_]

Thank you for explaining  @StephenB!

I got myself a new WD30EFRX.

Best way is to hot-replace the old one right?

[StephenB · Accepted solution]

---

@F_L_ wrote:

Best way is to hot-replace the old one right?

---

Yes.  

Though if the kernel has disabled the sata interface for the slot, you might need to reboot.  So if the system doesn't see the drive insertion, try that.

[F_L_]

Hot-plug worked. System up and running. Thank you

[BrianL2]

Hi F_L_,

Thanks for your feedback. You may now tag this thread as resolved by clicking the "Accept as solution" button in one of the responses that you received.

Kind regards,

BrianL
NETGEAR Community Team