Hard drive failure - Multiple New drives failing SMART Test?
Hi,
I've got a v strange issue with a ReadyNas Pro 6 which has been ticking along nicely. Basically disk 2 has failed (funny this is the second drive failure in a few years in the same slot.. what are the chances?)
ok, so I tried to replace with a couple of brand new 1TB Western Digital drives which were on the HCL. Both drives came up as failing the SMART test.
So I grabbed two of close to the exact same Seagate Barracuda drives (Seagate ST31000524AS vs existing Seagate ST31000524NS) and replaced... same issue failed SMART test.. so thats 4 drives that just failed SMART??... wha?
Currently the device is saying vol c is unprotected of course.. I have backups of all the data but it is a MASSIVE pain if i have to go through a complete restore obviously.
Any ideas on the issue? shall i give it a reboot in case its a SMART glitch? i'm just paranoid it may not boot again with the failed drive.. or another may fail during spin down/spin up.
Currently running Raidar 4.2.24
excerpt from system log below when trying to swap the disk...
First disk replacement attempt:
Mar 3 16:35:06 BOWNAS kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen Mar 3 16:35:06 BOWNAS kernel: ata2: irq_stat 0x00400000, PHY RDY changed Mar 3 16:35:06 BOWNAS kernel: ata2: SError: { PHYRdyChg 10B8B } Mar 3 16:35:06 BOWNAS kernel: ata2: hard resetting link Mar 3 16:35:07 BOWNAS kernel: ata2: SATA link down (SStatus 0 SControl 300) Mar 3 16:35:12 BOWNAS kernel: ata2: hard resetting link Mar 3 16:35:12 BOWNAS kernel: ata2: SATA link down (SStatus 0 SControl 300) Mar 3 16:35:12 BOWNAS kernel: ata2: limiting SATA link speed to 1.5 Gbps Mar 3 16:35:17 BOWNAS kernel: ata2: hard resetting link Mar 3 16:35:17 BOWNAS kernel: ata2: SATA link down (SStatus 0 SControl 310) Mar 3 16:35:17 BOWNAS kernel: ata2.00: disabled Mar 3 16:35:17 BOWNAS kernel: ata2: EH complete Mar 3 16:35:17 BOWNAS kernel: ata2.00: detaching (SCSI 1:0:0:0) Mar 3 16:35:17 BOWNAS kernel: sd 1:0:0:0: [sdb] Synchronizing SCSI cache Mar 3 16:35:17 BOWNAS kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Mar 3 16:35:17 BOWNAS kernel: sd 1:0:0:0: [sdb] Stopping disk Mar 3 16:35:17 BOWNAS kernel: sd 1:0:0:0: [sdb] START_STOP FAILED Mar 3 16:35:17 BOWNAS kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Mar 3 16:35:17 BOWNAS kernel: scsi: killing requests for dead queue Mar 3 16:35:19 BOWNAS RAIDiator: Disk removal detected. [Disk 2] Mar 3 16:35:19 BOWNAS RAIDiator: A disk was removed from the ReadyNAS. One or more RAID volumes are currently unprotected, and an additional disk failure or removal may result in data loss. Please add a replacement disk as soon as possible. Mar 3 16:36:33 BOWNAS kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen Mar 3 16:36:33 BOWNAS kernel: ata2: irq_stat 0x00000040, connection status changed Mar 3 16:36:33 BOWNAS kernel: ata2: SError: { CommWake DevExch } Mar 3 16:36:33 BOWNAS kernel: ata2: hard resetting link Mar 3 16:36:38 BOWNAS kernel: ata2: link is slow to respond, please be patient (ready=0) Mar 3 16:36:41 BOWNAS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Mar 3 16:36:41 BOWNAS kernel: ata2.00: ATA-8: ST31000528AS, CC38, max UDMA/133 Mar 3 16:36:41 BOWNAS kernel: ata2.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 3 16:36:41 BOWNAS kernel: ata2.00: configured for UDMA/133 Mar 3 16:36:41 BOWNAS kernel: ata2: EH complete Mar 3 16:36:41 BOWNAS kernel: scsi 1:0:0:0: Direct-Access ATA ST31000528AS CC38 PQ: 0 ANSI: 5 Mar 3 16:36:41 BOWNAS kernel: sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 3 16:36:41 BOWNAS kernel: sd 1:0:0:0: [sdb] Write Protect is off Mar 3 16:36:41 BOWNAS kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Mar 3 16:36:41 BOWNAS kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 3 16:36:41 BOWNAS kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0 Mar 3 16:36:41 BOWNAS kernel: sdb: sdb1 sdb2 Mar 3 16:36:41 BOWNAS kernel: sd 1:0:0:0: [sdb] Attached SCSI disk Mar 3 16:36:59 BOWNAS RAIDiator: Disk removal detected. [Disk 2] (BOWNAS) : A disk was removed from the ReadyNAS. One or more RAID volumes are currently unprotected, and an additional disk failure or removal may result in data loss. Please add a replacement disk as soon as possible. Mar 3 16:37:14 BOWNAS RAIDiator: New disk detected. If multiple disks have been added, they will be processed one at a time. Please do not remove any added disk(s) during this time. [Disk 2] Mar 3 16:37:16 BOWNAS kernel: sdb: unknown partition table Mar 3 16:37:19 BOWNAS kernel: sdb: sdb1 sdb2 Mar 3 16:37:54 BOWNAS RAIDiator: New disk detected. [Disk 2] (BOWNAS) : A new disk was added to the ReadyNAS. If multiple disks have been added, they will be processed one at a time. Please do not remove any added disk(s) during this time. Mar 3 16:39:37 BOWNAS kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen Mar 3 16:39:37 BOWNAS kernel: ata2: irq_stat 0x00400000, PHY RDY changed Mar 3 16:39:37 BOWNAS kernel: ata2: SError: { PHYRdyChg } Mar 3 16:39:37 BOWNAS kernel: ata2: hard resetting link Mar 3 16:39:37 BOWNAS kernel: ata2: SATA link down (SStatus 0 SControl 300) Mar 3 16:39:42 BOWNAS kernel: ata2: hard resetting link Mar 3 16:39:43 BOWNAS kernel: ata2: SATA link down (SStatus 0 SControl 300) Mar 3 16:39:43 BOWNAS kernel: ata2: limiting SATA link speed to 1.5 Gbps Mar 3 16:39:48 BOWNAS kernel: ata2: hard resetting link Mar 3 16:39:48 BOWNAS kernel: ata2: SATA link down (SStatus 0 SControl 310) Mar 3 16:39:48 BOWNAS kernel: ata2.00: disabled Mar 3 16:39:48 BOWNAS kernel: ata2: EH complete Mar 3 16:39:48 BOWNAS kernel: ata2.00: detaching (SCSI 1:0:0:0) Mar 3 16:39:48 BOWNAS kernel: sd 1:0:0:0: [sdb] Synchronizing SCSI cache Mar 3 16:39:48 BOWNAS kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Mar 3 16:39:48 BOWNAS kernel: sd 1:0:0:0: [sdb] Stopping disk Mar 3 16:39:48 BOWNAS kernel: sd 1:0:0:0: [sdb] START_STOP FAILED Mar 3 16:39:48 BOWNAS kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Mar 3 16:39:48 BOWNAS kernel: scsi: killing requests for dead queue Mar 3 16:39:48 BOWNAS RAIDiator: Disk removal detected. [Disk 2] Mar 3 16:39:48 BOWNAS RAIDiator: A disk was removed from the ReadyNAS. One or more RAID volumes are currently unprotected, and an additional disk failure or removal may result in data loss. Please add a replacement disk as soon as possible. Mar 3 16:39:49 BOWNAS RAIDiator: Disk removal detected. [Disk 2] (BOWNAS) : A disk was removed from the ReadyNAS. One or more RAID volumes are currently unprotected, and an additional disk failure or removal may result in data loss. Please add a replacement disk as soon as possible.
Second attempt - Different Disk (Same Model):
Mar 3 16:44:27 BOWNAS kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen Mar 3 16:44:27 BOWNAS kernel: ata2: irq_stat 0x00000040, connection status changed Mar 3 16:44:27 BOWNAS kernel: ata2: SError: { CommWake DevExch } Mar 3 16:44:27 BOWNAS kernel: ata2: hard resetting link Mar 3 16:44:33 BOWNAS kernel: ata2: link is slow to respond, please be patient (ready=0) Mar 3 16:44:36 BOWNAS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Mar 3 16:44:36 BOWNAS kernel: ata2.00: ATA-8: ST31000528AS, CC38, max UDMA/133 Mar 3 16:44:36 BOWNAS kernel: ata2.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 3 16:44:36 BOWNAS kernel: ata2.00: configured for UDMA/133 Mar 3 16:44:36 BOWNAS kernel: ata2: EH complete Mar 3 16:44:36 BOWNAS kernel: scsi 1:0:0:0: Direct-Access ATA ST31000528AS CC38 PQ: 0 ANSI: 5 Mar 3 16:44:36 BOWNAS kernel: sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 3 16:44:36 BOWNAS kernel: sd 1:0:0:0: [sdb] Write Protect is off Mar 3 16:44:36 BOWNAS kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Mar 3 16:44:36 BOWNAS kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 3 16:44:36 BOWNAS kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0 Mar 3 16:44:36 BOWNAS kernel: sdb: sdb1 sdb2 Mar 3 16:44:36 BOWNAS kernel: sd 1:0:0:0: [sdb] Attached SCSI disk Mar 3 16:44:46 BOWNAS RAIDiator: New disk detected. If multiple disks have been added, they will be processed one at a time. Please do not remove any added disk(s) during this time. [Disk 2] Mar 3 16:44:49 BOWNAS kernel: sdb: unknown partition table Mar 3 16:44:53 BOWNAS kernel: sdb: sdb1 sdb2 Mar 3 16:44:53 BOWNAS RAIDiator: New disk detected. [Disk 2] (BOWNAS) : A new disk was added to the ReadyNAS. If multiple disks have been added, they will be processed one at a time. Please do not remove any added disk(s) during this time.
This had the same result but the logs are truncated at this point so I dont have the rest of the failure.. Same deal though, frontview suggests it is dead and unprotected...
