NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Disk Failure Detected...
I've recently purchased a ReadyNAS Ultra 6 along with 6 2 Tb Seagate ST2000DL003 disks (which are on the HCL).
I've set up the NAS in a dual redundancy X-RAID2 configuration and have starting copying all the data over the network from my old ReadyNAS NV to the new ultra 6...
About half way through copying (on 6th March), I got a disk failure detected (on channel 4). I powered down the NAS took the disk out and reinserted it, assuming there might be some kind of connection problem... When I powered back up it detected the disk, tested it and started to resync (which takes about 24 hours)... I left it alone while it did that and then it seemed to be ok, so I started copying the rest of my data across. There is nothing in the SMART+ log for disk 4 which would indicate that there was ever a problem with that disk.
A few minutes ago, I just got another disk failure (this time on channel 2). Exactly the same story... powered down and then back up again, the disk comes back to life and the NAS starts testing it and resyncing it... again, there is nothing in the SMART+ log for disk 2 which indicates (to me at least) that there was ever a problem.
After both occasions, I've downloaded the system logs from the NAS, but I'm not sure what to do with them. Is there something in the log which would show what exactly failed?
Any idea what's going on here? Have I got a couple of dud disks which need to be sent back, or is there something else going on? If they are dud, I'd need to be able to prove to the retailer that they were... the only indication I have of a problem is that the ReadyNAS ultra 6 _said_ that they had failed... but they both seem to be working fine now.
Thanks,
Ash.
P.S. Here's the SMART+ report from disk 2:
This looks like the appropriate section of system.log for the failure which occurred today:
and here is what looks like the relevant part of the log from the failure on 6th March:
I've set up the NAS in a dual redundancy X-RAID2 configuration and have starting copying all the data over the network from my old ReadyNAS NV to the new ultra 6...
About half way through copying (on 6th March), I got a disk failure detected (on channel 4). I powered down the NAS took the disk out and reinserted it, assuming there might be some kind of connection problem... When I powered back up it detected the disk, tested it and started to resync (which takes about 24 hours)... I left it alone while it did that and then it seemed to be ok, so I started copying the rest of my data across. There is nothing in the SMART+ log for disk 4 which would indicate that there was ever a problem with that disk.
A few minutes ago, I just got another disk failure (this time on channel 2). Exactly the same story... powered down and then back up again, the disk comes back to life and the NAS starts testing it and resyncing it... again, there is nothing in the SMART+ log for disk 2 which indicates (to me at least) that there was ever a problem.
After both occasions, I've downloaded the system logs from the NAS, but I'm not sure what to do with them. Is there something in the log which would show what exactly failed?
Any idea what's going on here? Have I got a couple of dud disks which need to be sent back, or is there something else going on? If they are dud, I'd need to be able to prove to the retailer that they were... the only indication I have of a problem is that the ReadyNAS ultra 6 _said_ that they had failed... but they both seem to be working fine now.
Thanks,
Ash.
P.S. Here's the SMART+ report from disk 2:
SMART Information for Disk 2
Model: ST2000DL003-9VT166
Serial: 5YD2196G
Firmware: CC32
SMART Attribute
Spin Up Time 0
Start Stop Count 12
Reallocated Sector Count 0
Power On Hours 151
Spin Retry Count 0
Power Cycle Count 12
Reported Uncorrect 0
High Fly Writes 0
Airflow Temperature Cel 42
G-Sense Error Rate 0
Power-Off Retract Count 6
Load Cycle Count 12
Temperature Celsius 42
Current Pending Sector 0
Offline Uncorrectable 0
UDMA CRC Error Count 0
Head Flying Hours 221474283585687
ATA Error Count 0
This looks like the appropriate section of system.log for the failure which occurred today:
Mar 13 20:00:09 ultranas ntpdate[11162]: step time server 194.238.48.3 offset 0.310812 sec
Mar 13 20:16:27 ultranas kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Mar 13 20:16:27 ultranas kernel: ata2.00: failed command: FLUSH CACHE EXT
Mar 13 20:16:27 ultranas kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Mar 13 20:16:27 ultranas kernel: res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Mar 13 20:16:27 ultranas kernel: ata2.00: status: { DRDY }
Mar 13 20:16:27 ultranas kernel: ata2: hard resetting link
Mar 13 20:16:33 ultranas kernel: ata2: link is slow to respond, please be patient (ready=0)
Mar 13 20:16:37 ultranas kernel: ata2: COMRESET failed (errno=-16)
Mar 13 20:16:37 ultranas kernel: ata2: hard resetting link
Mar 13 20:16:43 ultranas kernel: ata2: link is slow to respond, please be patient (ready=0)
Mar 13 20:16:47 ultranas kernel: ata2: COMRESET failed (errno=-16)
Mar 13 20:16:47 ultranas kernel: ata2: hard resetting link
Mar 13 20:16:53 ultranas kernel: ata2: link is slow to respond, please be patient (ready=0)
Mar 13 20:17:23 ultranas kernel: ata2: COMRESET failed (errno=-16)
Mar 13 20:17:23 ultranas kernel: ata2: limiting SATA link speed to 1.5 Gbps
Mar 13 20:17:23 ultranas kernel: ata2: hard resetting link
Mar 13 20:17:28 ultranas kernel: ata2: COMRESET failed (errno=-16)
Mar 13 20:17:28 ultranas kernel: ata2: reset failed, giving up
Mar 13 20:17:28 ultranas kernel: ata2.00: disabled
Mar 13 20:17:28 ultranas kernel: ata2.00: device reported invalid CHS sector 0
Mar 13 20:17:28 ultranas kernel: ata2: EH complete
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 0
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 90 00 50 00 00 02 00
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 9437264
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 9437264
Mar 13 20:17:28 ultranas kernel: **************** super written barrier kludge on md2: error==IO 0xfffffffb
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 72
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 72
Mar 13 20:17:28 ultranas kernel: **************** super written barrier kludge on md0: error==IO 0xfffffffb
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 51 8f 30 00 00 28 00
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 5345072
Mar 13 20:17:28 ultranas kernel: raid1: sdb1: rescheduling sector 5342960
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 90 00 50 00 00 02 00
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 9437264
Mar 13 20:17:28 ultranas kernel: md: super_written gets error=-5, uptodate=0
Mar 13 20:17:28 ultranas kernel: raid5: Disk failure on sdb5, disabling device.
Mar 13 20:17:28 ultranas kernel: raid5: Operation continuing on 5 devices.
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 13 20:17:28 ultranas kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 13 20:17:28 ultranas kernel: end_request: I/O error, dev sdb, sector 72
Mar 13 20:17:28 ultranas kernel: md: super_written gets error=-5, uptodate=0
Mar 13 20:17:28 ultranas kernel: raid1: Disk failure on sdb1, disabling device.
Mar 13 20:17:28 ultranas kernel: raid1: Operation continuing on 5 devices.
Mar 13 20:17:28 ultranas kernel: RAID5 conf printout:
Mar 13 20:17:28 ultranas kernel: --- rd:6 wd:5
Mar 13 20:17:28 ultranas kernel: disk 0, o:1, dev:sda5
Mar 13 20:17:28 ultranas kernel: disk 1, o:0, dev:sdb5
Mar 13 20:17:28 ultranas kernel: disk 2, o:1, dev:sdc5
Mar 13 20:17:28 ultranas kernel: disk 3, o:1, dev:sdd5
Mar 13 20:17:28 ultranas kernel: disk 4, o:1, dev:sde5
Mar 13 20:17:28 ultranas kernel: disk 5, o:1, dev:sdf5
Mar 13 20:17:28 ultranas kernel: RAID5 conf printout:
Mar 13 20:17:28 ultranas kernel: --- rd:6 wd:5
Mar 13 20:17:28 ultranas kernel: disk 0, o:1, dev:sda5
Mar 13 20:17:28 ultranas kernel: disk 2, o:1, dev:sdc5
Mar 13 20:17:28 ultranas kernel: disk 3, o:1, dev:sdd5
Mar 13 20:17:28 ultranas kernel: disk 4, o:1, dev:sde5
Mar 13 20:17:28 ultranas kernel: disk 5, o:1, dev:sdf5
Mar 13 20:17:28 ultranas kernel: RAID1 conf printout:
Mar 13 20:17:28 ultranas kernel: --- wd:5 rd:6
Mar 13 20:17:28 ultranas kernel: disk 0, wo:0, o:1, dev:sda1
Mar 13 20:17:28 ultranas kernel: disk 1, wo:1, o:0, dev:sdb1
Mar 13 20:17:28 ultranas kernel: disk 2, wo:0, o:1, dev:sdc1
Mar 13 20:17:28 ultranas kernel: disk 3, wo:0, o:1, dev:sdd1
Mar 13 20:17:28 ultranas kernel: disk 4, wo:0, o:1, dev:sde1
Mar 13 20:17:28 ultranas kernel: disk 5, wo:0, o:1, dev:sdf1
Mar 13 20:17:28 ultranas kernel: RAID1 conf printout:
Mar 13 20:17:28 ultranas kernel: --- wd:5 rd:6
Mar 13 20:17:28 ultranas kernel: disk 0, wo:0, o:1, dev:sda1
Mar 13 20:17:28 ultranas kernel: disk 2, wo:0, o:1, dev:sdc1
Mar 13 20:17:28 ultranas kernel: disk 3, wo:0, o:1, dev:sdd1
Mar 13 20:17:28 ultranas kernel: disk 4, wo:0, o:1, dev:sde1
Mar 13 20:17:28 ultranas kernel: disk 5, wo:0, o:1, dev:sdf1
Mar 13 20:17:28 ultranas kernel: raid1: sdf1: redirecting sector 5342960 to another mirror
Mar 13 20:17:32 ultranas RAIDiator: Disk failure detected.\n\nIf the failed disk is used in a RAID level 1, 5, or X-RAID volume, please note that volume is now unprotected, and an additional disk failure may render that volume dead. If this disk is a part of a RAID 6 volume, your volume is still protected if this is your first failure. A 2nd disk failure will make your volume unprotected. It is recommended that you replace the failed disk as soon as possible to maintain optimal protection of your volume.\n\n[Sun Mar 13 20:17:29 WET 2011]
Mar 13 20:20:24 ultranas kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
and here is what looks like the relevant part of the log from the failure on 6th March:
Mar 6 16:00:07 nas-EA-A6-42 ntpdate[12452]: step time server 62.84.188.34 offset -0.103568 sec
Mar 6 18:48:21 nas-EA-A6-42 kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Mar 6 18:48:22 nas-EA-A6-42 kernel: ata4.00: failed command: FLUSH CACHE EXT
Mar 6 18:48:22 nas-EA-A6-42 kernel: ata4.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Mar 6 18:48:22 nas-EA-A6-42 kernel: res 40/00:00:b8:f7:0e/00:00:00:00:00/40 Emask 0x4 (timeout)
Mar 6 18:48:22 nas-EA-A6-42 kernel: ata4.00: status: { DRDY }
Mar 6 18:48:22 nas-EA-A6-42 kernel: ata4: hard resetting link
Mar 6 18:48:27 nas-EA-A6-42 kernel: ata4: link is slow to respond, please be patient (ready=0)
Mar 6 18:48:32 nas-EA-A6-42 kernel: ata4: COMRESET failed (errno=-16)
Mar 6 18:48:32 nas-EA-A6-42 kernel: ata4: hard resetting link
Mar 6 18:48:37 nas-EA-A6-42 kernel: ata4: link is slow to respond, please be patient (ready=0)
Mar 6 18:48:42 nas-EA-A6-42 kernel: ata4: COMRESET failed (errno=-16)
Mar 6 18:48:42 nas-EA-A6-42 kernel: ata4: hard resetting link
Mar 6 18:48:47 nas-EA-A6-42 kernel: ata4: link is slow to respond, please be patient (ready=0)
Mar 6 18:49:17 nas-EA-A6-42 kernel: ata4: COMRESET failed (errno=-16)
Mar 6 18:49:17 nas-EA-A6-42 kernel: ata4: limiting SATA link speed to 1.5 Gbps
Mar 6 18:49:17 nas-EA-A6-42 kernel: ata4: hard resetting link
Mar 6 18:49:22 nas-EA-A6-42 kernel: ata4: COMRESET failed (errno=-16)
Mar 6 18:49:22 nas-EA-A6-42 kernel: ata4: reset failed, giving up
Mar 6 18:49:22 nas-EA-A6-42 kernel: ata4.00: disabled
Mar 6 18:49:22 nas-EA-A6-42 kernel: ata4: EH complete
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 72
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 72
Mar 6 18:49:22 nas-EA-A6-42 kernel: **************** super written barrier kludge on md0: error==IO 0xfffffffb
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Write(10): 2a 00 00 93 9e 80 00 00 08 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 9674368
Mar 6 18:49:22 nas-EA-A6-42 kernel: raid5: Disk failure on sdd5, disabling device.
Mar 6 18:49:22 nas-EA-A6-42 kernel: raid5: Operation continuing on 5 devices.
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Write(10): 2a 00 34 c5 68 48 00 00 80 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 885352520
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Write(10): 2a 00 34 c6 f0 c8 00 00 50 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 885453000
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 91 28 c8 00 00 38 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 9513160
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 91 29 10 00 00 10 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 9513232
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 91 29 28 00 00 10 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 9513256
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 91 29 40 00 00 08 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 9513280
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 93 88 48 00 00 08 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 9668680
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 93 a1 90 00 00 10 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 9675152
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 34 c5 38 48 00 00 08 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 885340232
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 34 c5 64 48 00 00 80 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 885351496
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 34 c6 f1 18 00 00 30 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 885453080
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Write(10): 2a 00 00 80 00 48 00 00 02 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 8388680
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 8388680
Mar 6 18:49:22 nas-EA-A6-42 kernel: **************** super written barrier kludge on md1: error==IO 0xfffffffb
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 31 8d 58 00 00 28 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 3247448
Mar 6 18:49:22 nas-EA-A6-42 kernel: raid1: sdd1: rescheduling sector 3245336
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB:
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Unhandled error code
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 6 18:49:22 nas-EA-A6-42 kernel: sd 3:0:0:0: [sdd] CDB: Write(10)Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 72
Mar 6 18:49:22 nas-EA-A6-42 kernel: :md: super_written gets error=-5, uptodate=0
Mar 6 18:49:22 nas-EA-A6-42 kernel: 2a
Mar 6 18:49:22 nas-EA-A6-42 kernel: raid1: Disk failure on sdd1, disabling device.
Mar 6 18:49:22 nas-EA-A6-42 kernel: raid1: Operation continuing on 5 devices.
Mar 6 18:49:22 nas-EA-A6-42 kernel: 00 00 80 00 48 00 00 02 00
Mar 6 18:49:22 nas-EA-A6-42 kernel: end_request: I/O error, dev sdd, sector 8388680
Mar 6 18:49:22 nas-EA-A6-42 kernel: md: super_written gets error=-5, uptodate=0
Mar 6 18:49:22 nas-EA-A6-42 kernel: raid5: Disk failure on sdd2, disabling device.
Mar 6 18:49:22 nas-EA-A6-42 kernel: raid5: Operation continuing on 5 devices.
Mar 6 18:49:23 nas-EA-A6-42 kernel: RAID1 conf printout:
Mar 6 18:49:23 nas-EA-A6-42 kernel: --- wd:5 rd:6
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 0, wo:0, o:1, dev:sda1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 1, wo:0, o:1, dev:sdb1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 2, wo:0, o:1, dev:sdc1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 3, wo:1, o:0, dev:sdd1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 4, wo:0, o:1, dev:sde1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 5, wo:0, o:1, dev:sdf1
Mar 6 18:49:23 nas-EA-A6-42 kernel: RAID1 conf printout:
Mar 6 18:49:23 nas-EA-A6-42 kernel: --- wd:5 rd:6
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 0, wo:0, o:1, dev:sda1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 1, wo:0, o:1, dev:sdb1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 2, wo:0, o:1, dev:sdc1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 4, wo:0, o:1, dev:sde1
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 5, wo:0, o:1, dev:sdf1
Mar 6 18:49:23 nas-EA-A6-42 kernel: RAID5 conf printout:
Mar 6 18:49:23 nas-EA-A6-42 kernel: --- rd:6 wd:5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 0, o:1, dev:sda5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 1, o:1, dev:sdb5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 2, o:1, dev:sdc5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 3, o:0, dev:sdd5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 4, o:1, dev:sde5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 5, o:1, dev:sdf5
Mar 6 18:49:23 nas-EA-A6-42 kernel: RAID5 conf printout:
Mar 6 18:49:23 nas-EA-A6-42 kernel: --- rd:6 wd:5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 0, o:1, dev:sda5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 1, o:1, dev:sdb5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 2, o:1, dev:sdc5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 4, o:1, dev:sde5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 5, o:1, dev:sdf5
Mar 6 18:49:23 nas-EA-A6-42 kernel: RAID5 conf printout:
Mar 6 18:49:23 nas-EA-A6-42 kernel: --- rd:6 wd:5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 0, o:1, dev:sda2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 1, o:1, dev:sdb2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 2, o:1, dev:sdc2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 3, o:0, dev:sdd2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 4, o:1, dev:sde2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 5, o:1, dev:sdf2
Mar 6 18:49:23 nas-EA-A6-42 kernel: raid1: sdb1: redirecting sector 3245336 to another mirror
Mar 6 18:49:23 nas-EA-A6-42 kernel: RAID5 conf printout:
Mar 6 18:49:23 nas-EA-A6-42 kernel: --- rd:6 wd:5
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 0, o:1, dev:sda2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 1, o:1, dev:sdb2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 2, o:1, dev:sdc2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 4, o:1, dev:sde2
Mar 6 18:49:23 nas-EA-A6-42 kernel: disk 5, o:1, dev:sdf2
Mar 6 18:49:53 nas-EA-A6-42 RAIDiator: Disk failure detected.\n\nIf the failed disk is used in a RAID level 1, 5, or X-RAID volume, please note that volume is now unprotected, and an additional disk failure may render that volume dead. If this disk is a part of a RAID 6 volume, your volume is still protected if this is your first failure. A 2nd disk failure will make your volume unprotected. It is recommended that you replace the failed disk as soon as possible to maintain optimal protection of your volume.\n\n[Sun Mar 6 18:49:51 WET 2011]
144 Replies
Replies have been turned off for this discussion
- On the Mac OS X question: I will eventually be using my new ReadyNAS with a Mac or two, but not until a clean RAID-6 volume is built...
- More evidence that there may be something funky going on here, firmware wise.
Have a Pro Pioneer and after updating it this evening to the latest firmware, suddenly channels 2, 3, 4, and 5 report as dead. Haven't even had a SMART error up until now. Everything was fine until I did the firmware update.
If that isn't odd, here's what happened immediately after the firmware update:
1. RAIDar shows all drives as green.
2. Frontview simultaneously shows channels 2-5 as DEAD.
3. I can browse the shares and successfully both loaded files off the drives and moves files around.
About ten minutes later RAIDar also reported the drives had failed, as did the front panel on the ReadyNAS Pro Pioneer. The shares were no longer accessible.
Then I rebooted the device. After the reboot the same situation as above happens again. Good in RAIDar, DEAD in FrontView, shares accessible.
Ran a volume scan and it completed with no errors. Yet the same conditions apply: RAIDar thinks thinks are fine, FrontView thinks I'm screwed. My confidence is quite low at the moment...
-greg - Greg you probably should start a new thread, open an Online tech support case and post your case number there.
- Reading back through this, there seems to be a commonality in our disks being erroneously reported as dead and the use of OSX.
Summary: (user: details)
bluewombie: Ultra 6 (not plus), OSX Snowleopard, AFP
imlucid: Pro Pioneer, OSX Snow Leopard and Leopard, Time Machine
citizenplain (me): Ultra 6 Plus, OSX Snow Leopard, AFP file transfer, Time Machine
evanhatesspam: Ultra 6 Plus
None of us have had abnormal results from any disk checks. All of us are having trouble with the Seagate ST2000DL003-9VT166 drives.
Didn't catch if user bluewombie has Time Machine enabled. Not sure how evanhatesspam is accessing his NAS.
I updated to the new firmware (RAIDiator-x86 4.2.16) recently, but haven't had a need to do a large file transfer since updating. The updates described in the change log for this version didn't sound like they specifically address this problem. We'll see if it makes a difference.
Anything I'm missing here or any other details we could add to this? Bluewombie, did you ever get anywhere with tech support? - CitizenPlain: Yes, that looks like a good summary. I am also using Time Machine.
My conversations with tech support weren't enormously helpful... the guy reccomended:
1. Running the ReadyNAS extended disk test (passed)
2. Running SeaTools long drive test on each disk (they all passed)
3. Reinstalling the OS (done)
Since I followed these steps, I haven't _yet_ had any further problems... though I'm reluctant to say that the issue is fixed because it seems to me that none of the above steps _ought_ to fix the problem.
Since then I have also installed the new firmware update (RAIDiator-x86 4.2.16).
I have tried to do some fairly large file transfers, but haven't yet managed to make the problem reappear. - Disk 3 reported "failed" a week ago, now disk 4...
Still no reported failures on my mirrored rsync unit...
Kevin - Hmm, interesting new developments...
So after drive reported failure I shutdown, pulled drive, replaced and booted.
Drive still shows missing so I do the same steps again waiting a bit longer and on boot it starts resyncing the drive. So far so good.
After it was around 1.6% done or so, I checked to see if I could mount cifs via finder, no problem.
I tried to load the frontview web admin and at first it wouldn't load, but after a couple of tries it loads but doesn't fill in any text. Then the drive's fans go on full blast and what appears to be something like a progress bar on the LED on the front of the NAS starts flashing from left to right a few times and the NAS then reboots...
Interesting.
So I do the same steps again and once again right when I'm trying to load the web UI, the server reboots again with same symptoms...
I'm now going to leave it alone as it is trying to resync again (3rd time a charm?).
Kevin - I have just experienced the same issue with my Ultra4 and these seagate harddrives.
I have performed a factory reset and this has fixed it for now but there has to be a bug somewhere. - Just had the same problem with Ultra4 Plus.
Also running all OSX in the house. The NAS has been up for weeks now. I've copied all my media on there. "today" was the 1st time I've really had a chance to use it. Besides copying several multi-gig isos around everything seemed to be ok. I enabled ReadyNAS backup today and kicked off a backup. It was about 70gig into a 250gig backup and I got a failure email from disk3.
I have not done any of the steps yet in this thread, but something weird going on and I specifically bought off the HCL having had issues with NV/NV+ and WD EARS drives awhile go.
from DMESG:
ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
ata5.00: failed command: FLUSH CACHE EXT
ata5.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
ata5.00: status: { DRDY }
ata5: hard resetting link
ata5: link is slow to respond, please be patient (ready=0)
ata5: COMRESET failed (errno=-16)
ata5: hard resetting link
ata5: link is slow to respond, please be patient (ready=0)
ata5: COMRESET failed (errno=-16)
ata5: hard resetting link
ata5: link is slow to respond, please be patient (ready=0)
ata5: COMRESET failed (errno=-16)
ata5: limiting SATA link speed to 1.5 Gbps
ata5: hard resetting link
ata5: COMRESET failed (errno=-16)
ata5: reset failed, giving up
ata5.00: disabled
ata5.00: device reported invalid CHS sector 0
ata5: EH complete
sd 4:0:0:0: [sdc] Unhandled error code
sd 4:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 4:0:0:0: [sdc] CDB: Read(10): 28 00 a9 e8 08 00 00 00 08 00
end_request: I/O error, dev sdc, sector 2850555904
sd 4:0:0:0: [sdc] Unhandled error code
sd 4:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 4:0:0:0: [sdc] CDB: Read(10): 28 00 00 12 89 80 00 00 08 00
end_request: I/O error, dev sdc, sector 1214848
md/raid1:md0: sdc1: rescheduling sector 1212736
sd 4:0:0:0: [sdc] Unhandled error code
sd 4:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 4:0:0:0: [sdc] CDB: Write(10): 2a 00 00 90 00 50 00 00 02 00
end_request: I/O error, dev sdc, sector 9437264
end_request: I/O error, dev sdc, sector 9437264
md: super_written gets error=-5, uptodate=0
md/raid:md2: Disk failure on sdc5, disabling device.
<1>md/raid:md2: Operation continuing on 3 devices.
sd 4:0:0:0: [sdc] Unhandled error code
sd 4:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 4:0:0:0: [sdc] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
end_request: I/O error, dev sdc, sector 72
end_request: I/O error, dev sdc, sector 72
md: super_written gets error=-5, uptodate=0
md/raid1:md0: Disk failure on sdc1, disabling device.
<1>md/raid1:md0: Operation continuing on 3 devices.
RAID conf printout:
--- level:5 rd:4 wd:3
disk 0, o:1, dev:sda5
disk 1, o:1, dev:sdb5
disk 2, o:0, dev:sdc5
disk 3, o:1, dev:sdd5
RAID conf printout:
--- level:5 rd:4 wd:3
disk 0, o:1, dev:sda5
disk 1, o:1, dev:sdb5
disk 3, o:1, dev:sdd5
RAID1 conf printout:
--- wd:3 rd:4
disk 0, wo:0, o:1, dev:sda1
disk 1, wo:0, o:1, dev:sdb1
disk 2, wo:1, o:0, dev:sdc1
disk 3, wo:0, o:1, dev:sdd1
RAID1 conf printout:
--- wd:3 rd:4
disk 0, wo:0, o:1, dev:sda1
disk 1, wo:0, o:1, dev:sdb1
disk 3, wo:0, o:1, dev:sdd1
md/raid1:md0: redirecting sector 1212736 to other mirror: sdb1
This sux :( It's not cheap to put this stuff together. - Disk 2 has just done the same thing.
It's got to be something with the backplane/raid, firmware and/or drive incompatibility. Maybe they the drives should be marked as such in the Compatibility guide.
Note. I plugged in my USB drive and was moving files from that to the raid disk locally. CP jobs were started from the console. Large files +/-2gig each.
My drive 3 had been stable since my previous post. I pray for a firmware upgrade...
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
