Disk Failure Detected...

It looks like those disks 2 and 4 might be duds. I'd run a disk test on the NAS just to make sure (it'll run an extended SMART test on the drives, rather than the short test that's run usually).

Power off the NAS, boot with the reset button in the back depressed until the LCD shows boot menu then release, press the backup button until "Disk Test" shows, then depress the reset button one more time and it'll start the test.

I ran the extended disk test as you suggested... it ran overnight (and then I guess it finished it's resync)... it hasn't reported any problems. Can I see the results of the test in any of the logs?

Everything seems to be working fine now.

Should I ignore the problems I had with disk 2 and 4, or are there other tests I should run?

bluewomble wrote:
I ran the extended disk test as you suggested... it ran overnight (and then I guess it finished it's resync)... it hasn't reported any problems. Can I see the results of the test in any of the logs? Everything seems to be working fine now. Should I ignore the problems I had with disk 2 and 4, or are there other tests I should run?

Out of interest, how did you run this extended disk test and is it available on the 1100 model?

Thanks.. Karl

I just followed siigna's instructions from above:

Power off the NAS, boot with the reset button in the back depressed until the LCD shows boot menu then release, press the backup button until "Disk Test" shows, then depress the reset button one more time and it'll start the test.

I'm not sure what models it's available on...

bluewomble wrote:
I ran the extended disk test as you suggested... it ran overnight (and then I guess it finished it's resync)... it hasn't reported any problems. Can I see the results of the test in any of the logs? Everything seems to be working fine now. Should I ignore the problems I had with disk 2 and 4, or are there other tests I should run?

If there were any major problems in the disk test the NAS would have let you known that the disk was bad on the LCD, you should be able to pull the SMART logs under Status > Health and see if anything's out of the blue. Take a look at the reallocated sector counts, CRC errors and ATA errors, they should be at or close to 0. If everything's looking OK then keep an eye on the NAS at least for the next week or two and make sure nothing odd happens, after that I'd say you're good to go.

dataactive wrote:
Out of interest, how did you run this extended disk test and is it available on the 1100 model? Thanks.. Karl

Unfortunately the sparc based units don't have this boot option. If you have SSH access to the NAS you can always run "smartctl -t long /dev/hdX". hdc would be disk 1, hde is disk 2, hdg is disk 3, hdi is disk 4.

Otherwise you can test the disks using the manufacturer's testing tools (see the FAQ here).

Thanks, I'll give it a go.

I'm also having problems with disk 4. When I try fdisk -l /dev/hdi nothing happens. Is there another command that I can use to list all of the devices? Perhaps it's not listed as hdi?

dataactive wrote:
I'm also having problems with disk 4. When I try fdisk -l /dev/hdi nothing happens. Is there another command that I can use to list all of the devices? Perhaps it's not listed as hdi?

Yes, just run fdisk -l with no arguments.

bluewomble wrote:
Everything seems to be working fine now. Should I ignore the problems I had with disk 2 and 4, or are there other tests I should run?

Have you experienced any more issues with this setup or have you been running smoothly since your initial two failures? My ReadyNAS Ultra 6 Plus with 6x ST2000DL003-9VT166 (CC32) experienced a similar failure. /dev/sdb disappeared on hour 42 of operation. Health listed the Status as Dead (not Failed) with the SMART+ button grayed out, and Volume Settings had a missing entry with no Locate button.

Pulling and reinserting the drive did nothing. Inserting a fresh drive did nothing. The disk interface was just dead. I cold rebooted the ReadyNAS, the disk happily resynced, and SMART reported no errors.

Everything has been running fine since the 16th March... until today -- this time disk 5 was reported as failed.

Just like you, the health status was listed as 'dead' (not failed) with 'SMART+' greyed out. I have exactly the same disks ( ST2000DL003-9VT166) and exactly the same firmware (CC32).

I powered down the NAS, pulled the disk out, switched it back on and reinserted the same disk and the disk came back to life and started to resync. It's interesting that you found that even a hot swap of a new drive didn't work... which would (presumably) suggest a problem with the disk interface on the NAS rather than the disks?

Surely I can't have 3 failed disks out of 6? Once the volume has resynced, I'll do another extended disk test, but the first one came back fine after disks 2 and 4 had failed.

What's the official advice here? I can't really send back 3 disks which to all appearances work perfectly well and have no errors (and pass the extended test)... but I don't want to risk having multiple disks 'fail' in this manner and loosing my data. Is there some kind of incompatibility between the CC32 formware and the NAS? What version of the disk / firmware was tested when the disk was put onto the HCL?

Thanks,
Ash.

I just hit this myself.

I've had my Pro Pioneer for a few weeks and running smoothly for the last of those (after all the bring up etc.)

I've got 4 of those same seagates and two Hitachi HDS722020ALA330's but it was one of the ST2000DL003-9VT166's that reported as failed.

I did the shut down, pulled the drive, pushed back in and booted and it was recognized and the ReadyNAS started resyncing. The SMART+ info showed no issues whatsoever...

Kevin

Just had what sounds exactly like the problem the original poster describes happen twice here in the last two days. Two different drives listed as dead, then rescying and acting fine.

- ReadyNAS Ultra 6 Plus, just set up 5 days ago.
- with 6 Seagate ST2000DL003-9VT166 drives (on HCL, ordered from Newegg) also with the same firmware - CC32.
- latest firmware: RAIDiator-x86 4.2.15
- dual redundancy

The NAS has been up and running for a couple days and on a UPS. I'd enabled Time Machine backup on my Macbook Pro (~200 GB worth of data) and let that get up to date over the course of a day or so. Everything seemed fine except a minor chirping noise coming from it's fan.

Yesterday I began copying data from my old RAID to the Ultra 6+. Halfway through the file transfer via the AFP share, I received alerts that drive 5 was dead. As others mentioned, no SMART info available, just greyed-out "dead" listed for that drive. I canceled the file copy, shut down the RAID, pulled the drive and reinserted it (also suspecting a connection problem as drive 5 was the only one I had any difficulty seating in the enclosure initially). The RAID booted back up, ran a scan with no errors, then re-synced over the course of 8 or 10 hours. All seemed well.

Then, today, the same thing happened to drive 2. In the middle of a trying to recopy the data (roughly half a terabyte), I was alerted that drive 2 was dead. My file copy froze, I noticed that Time Machine was attempting backup activity, I stopped Time Machine and the file transfer immediately restarted while the array was degraded. Again, I've shut down, pulled the drive, re-inserted, and it ran a check on drive 2 and is currently re-syncing.

SMART info for the drives doesn't show any errors.

Is there any input from Netgear on what to do here? Should I be submitting a report to Netgear support? I also don't want to have to worry about this continuing to be a thing any time I do a big file copy -- my data is too precious. I wonder if there are compatibility problems with these drives? It doesn't make sense for one to be listed as dead and the boot right back up, resync, and be off to the races.

Like Bluewombie, I downloaded the logs but I'm not sure what to do with them.

Logs from the Frontview:

	Sun Mar 27 12:15:53 PDT 2011	If the failed disk is used in a RAID level 1, 5, or X-RAID volume, please note that volume is now unprotected, and an additional disk failure may render that volume dead. If this disk is a part of a RAID 6 volume, your volume is still protected if this is your first failure. A 2nd disk failure will make your volume unprotected. It is recommended that you replace the failed disk as soon as possible to maintain optimal protection of your volume.
	Sun Mar 27 12:15:53 PDT 2011	Disk failure detected.
	Sat Mar 26 23:53:59 PDT 2011	RAID sync finished on volume C.
	Sat Mar 26 12:40:26 PDT 2011	System is up.
	Sat Mar 26 12:40:10 PDT 2011	Volume scan found no errors.
	Sat Mar 26 12:37:13 PDT 2011	Rebooting device...
	Sat Mar 26 12:37:13 PDT 2011	Please close this browser session and use RAIDar to reconnect to the device. System rebooting...
	Sat Mar 26 12:28:06 PDT 2011	If the failed disk is used in a RAID level 1, 5, or X-RAID volume, please note that volume is now unprotected, and an additional disk failure may render that volume dead. If this disk is a part of a RAID 6 volume, your volume is still protected if this is your first failure. A 2nd disk failure will make your volume unprotected. It is recommended that you replace the failed disk as soon as possible to maintain optimal protection of your volume.
	Sat Mar 26 12:28:06 PDT 2011	Disk failure detected.
	Fri Mar 25 16:54:19 PDT 2011	System is up.

System logs for the first dead drive (Disk 5):

Mar 26 12:00:06 harrison ntpdate[20094]: no server suitable for synchronization found
Mar 26 12:24:39 harrison cnid_dbd[20697]: Set syslog logging to level: LOG_NOTE
Mar 26 12:27:39 harrison kernel: ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Mar 26 12:27:39 harrison kernel: ata5.00: failed command: FLUSH CACHE EXT
Mar 26 12:27:39 harrison kernel: ata5.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Mar 26 12:27:39 harrison kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Mar 26 12:27:39 harrison kernel: ata5.00: status: { DRDY }
Mar 26 12:27:39 harrison kernel: ata5: hard resetting link
Mar 26 12:27:39 harrison kernel: ata5: link is slow to respond, please be patient (ready=0)
Mar 26 12:27:39 harrison kernel: ata5: COMRESET failed (errno=-16)
Mar 26 12:27:39 harrison kernel: ata5: hard resetting link
Mar 26 12:27:39 harrison kernel: ata5: link is slow to respond, please be patient (ready=0)
Mar 26 12:27:39 harrison kernel: ata5: COMRESET failed (errno=-16)
Mar 26 12:27:39 harrison kernel: ata5: hard resetting link
Mar 26 12:27:39 harrison kernel: ata5: link is slow to respond, please be patient (ready=0)
Mar 26 12:27:39 harrison cnid_dbd[15520]: error reading message header: Connection reset by peer
Mar 26 12:27:39 harrison kernel: ata5: COMRESET failed (errno=-16)
Mar 26 12:27:39 harrison kernel: ata5: limiting SATA link speed to 1.5 Gbps
Mar 26 12:27:39 harrison kernel: ata5: hard resetting link
Mar 26 12:27:39 harrison kernel: ata5: COMRESET failed (errno=-16)
Mar 26 12:27:39 harrison kernel: ata5: reset failed, giving up
Mar 26 12:27:39 harrison kernel: ata5.00: disabled
Mar 26 12:27:39 harrison kernel: ata5.00: device reported invalid CHS sector 0
Mar 26 12:27:39 harrison kernel: ata5: EH complete
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 0
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Unhandled error code
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] CDB: Write(10): 2a 00 00 90 00 50 00 00 02 00
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 9437264
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 9437264
Mar 26 12:27:39 harrison kernel:  **************** super written barrier kludge on md2: error==IO 0xfffffffb
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Unhandled error code
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 72
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 72
Mar 26 12:27:39 harrison kernel:  **************** super written barrier kludge on md0: error==IO 0xfffffffb
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Unhandled error code
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] CDB: Read(10): 28 00 00 3d 19 20 00 00 30 00
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 4004128
Mar 26 12:27:39 harrison kernel: raid1: sde1: rescheduling sector 4002016
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Unhandled error code
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] CDB: Write(10): 2a 00 00 90 00 50 00 00 02 00
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 9437264
Mar 26 12:27:39 harrison kernel: md: super_written gets error=-5, uptodate=0
Mar 26 12:27:39 harrison kernel: raid5: Disk failure on sde5, disabling device.
Mar 26 12:27:39 harrison kernel: raid5: Operation continuing on 5 devices.
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Unhandled error code
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 72
Mar 26 12:27:39 harrison kernel: md: super_written gets error=-5, uptodate=0
Mar 26 12:27:39 harrison kernel: raid1: Disk failure on sde1, disabling device.
Mar 26 12:27:39 harrison kernel: raid1: Operation continuing on 5 devices.
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Unhandled error code
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 26 12:27:39 harrison kernel: sd 4:0:0:0: [sde] CDB: Read(10): 28 00 00 5d 0f a8 00 00 08 00
Mar 26 12:27:39 harrison kernel: end_request: I/O error, dev sde, sector 6098856
Mar 26 12:27:39 harrison kernel: raid1: sde1: rescheduling sector 6096744
Mar 26 12:27:39 harrison kernel: RAID5 conf printout:
Mar 26 12:27:39 harrison kernel:  --- rd:6 wd:5
Mar 26 12:27:39 harrison kernel:  disk 0, o:1, dev:sda5
Mar 26 12:27:39 harrison kernel:  disk 1, o:1, dev:sdb5
Mar 26 12:27:39 harrison kernel:  disk 2, o:1, dev:sdc5
Mar 26 12:27:39 harrison kernel:  disk 3, o:1, dev:sdd5
Mar 26 12:27:39 harrison kernel:  disk 4, o:0, dev:sde5
Mar 26 12:27:39 harrison kernel:  disk 5, o:1, dev:sdf5
Mar 26 12:27:39 harrison kernel: RAID5 conf printout:
Mar 26 12:27:39 harrison kernel:  --- rd:6 wd:5
Mar 26 12:27:39 harrison kernel:  disk 0, o:1, dev:sda5
Mar 26 12:27:39 harrison kernel:  disk 1, o:1, dev:sdb5
Mar 26 12:27:39 harrison kernel:  disk 2, o:1, dev:sdc5
Mar 26 12:27:39 harrison kernel:  disk 3, o:1, dev:sdd5
Mar 26 12:27:39 harrison kernel:  disk 5, o:1, dev:sdf5
Mar 26 12:27:39 harrison kernel: RAID1 conf printout:
Mar 26 12:27:39 harrison kernel:  --- wd:5 rd:6
Mar 26 12:27:39 harrison kernel:  disk 0, wo:0, o:1, dev:sda1
Mar 26 12:27:39 harrison kernel:  disk 1, wo:0, o:1, dev:sdb1
Mar 26 12:27:39 harrison kernel:  disk 2, wo:0, o:1, dev:sdc1
Mar 26 12:27:39 harrison kernel:  disk 3, wo:0, o:1, dev:sdd1
Mar 26 12:27:39 harrison kernel:  disk 4, wo:1, o:0, dev:sde1
Mar 26 12:27:39 harrison kernel:  disk 5, wo:0, o:1, dev:sdf1
Mar 26 12:27:39 harrison kernel: RAID1 conf printout:
Mar 26 12:27:39 harrison kernel:  --- wd:5 rd:6
Mar 26 12:27:39 harrison kernel:  disk 0, wo:0, o:1, dev:sda1
Mar 26 12:27:39 harrison kernel:  disk 1, wo:0, o:1, dev:sdb1
Mar 26 12:27:39 harrison kernel:  disk 2, wo:0, o:1, dev:sdc1
Mar 26 12:27:39 harrison kernel:  disk 3, wo:0, o:1, dev:sdd1
Mar 26 12:27:39 harrison kernel:  disk 5, wo:0, o:1, dev:sdf1
Mar 26 12:27:39 harrison kernel: raid1: sdc1: redirecting sector 4002016 to another mirror
Mar 26 12:27:39 harrison kernel: raid1: sdb1: redirecting sector 6096744 to another mirror
Mar 26 12:28:11 harrison RAIDiator: Disk failure detected.\n\nIf the failed disk is used in a RAID level 1, 5, or X-RAID volume, please note that volume is now unprotected, and an additional disk failure may render that volume dead.  If this disk is a part of a RAID 6 volume, your volume is still protected if this is your first failure.  A 2nd disk failure will make your volume unprotected.  It is recommended that you replace the failed disk as soon as possible to maintain optimal protection of your volume.\n\n[Sat Mar 26 12:28:06 PDT 2011]
Mar 26 12:37:18 harrison shutdown[21227]: shutting down for system reboot
Mar 26 12:37:19 harrison init: Switching to runlevel: 6
Mar 26 12:37:20 harrison kernel: ata1.00: configured for UDMA/133
Mar 26 12:37:20 harrison kernel: ata1: EH complete
Mar 26 12:37:20 harrison kernel: sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Mar 26 12:37:20 harrison kernel: ata2.00: configured for UDMA/133
Mar 26 12:37:20 harrison kernel: ata2: EH complete
Mar 26 12:37:20 harrison kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Mar 26 12:37:21 harrison kernel: ata3.00: configured for UDMA/133
Mar 26 12:37:21 harrison kernel: ata3: EH complete
Mar 26 12:37:21 harrison kernel: sd 2:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Mar 26 12:37:21 harrison kernel: ata4.00: configured for UDMA/133
Mar 26 12:37:21 harrison kernel: ata4: EH complete
Mar 26 12:37:21 harrison kernel: sd 3:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Mar 26 12:37:21 harrison kernel: ata6.00: configured for UDMA/133
Mar 26 12:37:21 harrison kernel: ata6: EH complete
Mar 26 12:37:21 harrison kernel: sd 5:0:0:0: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Mar 26 12:37:24 harrison exiting on signal 15
Mar 26 12:40:15 harrison syslogd 1.4.1#18: restart.
Mar 26 12:40:15 harrison kernel: klogd 1.4.1#18, log source = /proc/kmsg started.
Mar 26 12:40:15 harrison kernel: Linux version 2.6.33.7.RNx86_64.2.2 

System.log for the second dead drive (Disk 2):

Mar 27 12:00:06 harrison ntpdate[26530]: no server suitable for synchronization found
Mar 27 12:11:34 harrison cnid_dbd[26669]: Set syslog logging to level: LOG_NOTE
Mar 27 12:14:50 harrison kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Mar 27 12:15:52 harrison kernel: ata2.00: failed command: FLUSH CACHE EXT
Mar 27 12:15:52 harrison kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Mar 27 12:15:52 harrison kernel:          res 40/00:20:98:40:39/00:00:01:00:00/40 Emask 0x4 (timeout)
Mar 27 12:15:52 harrison kernel: ata2.00: status: { DRDY }
Mar 27 12:15:52 harrison kernel: ata2: hard resetting link
Mar 27 12:15:52 harrison kernel: ata2: link is slow to respond, please be patient (ready=0)
Mar 27 12:15:52 harrison kernel: ata2: COMRESET failed (errno=-16)
Mar 27 12:15:52 harrison kernel: ata2: hard resetting link
Mar 27 12:15:52 harrison kernel: ata2: link is slow to respond, please be patient (ready=0)
Mar 27 12:15:52 harrison kernel: ata2: COMRESET failed (errno=-16)
Mar 27 12:15:52 harrison kernel: ata2: hard resetting link
Mar 27 12:15:52 harrison kernel: ata2: link is slow to respond, please be patient (ready=0)
Mar 27 12:15:52 harrison kernel: ata2: COMRESET failed (errno=-16)
Mar 27 12:15:52 harrison kernel: ata2: limiting SATA link speed to 1.5 Gbps
Mar 27 12:15:52 harrison kernel: ata2: hard resetting link
Mar 27 12:15:52 harrison kernel: ata2: COMRESET failed (errno=-16)
Mar 27 12:15:52 harrison kernel: ata2: reset failed, giving up
Mar 27 12:15:52 harrison kernel: ata2.00: disabled
Mar 27 12:15:52 harrison kernel: ata2: EH complete
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 0
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 90 00 50 00 00 02 00
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 9437264
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 9437264
Mar 27 12:15:52 harrison kernel:  **************** super written barrier kludge on md2: error==IO 0xfffffffb
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 72
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 72
Mar 27 12:15:52 harrison kernel:  **************** super written barrier kludge on md0: error==IO 0xfffffffb
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 2b 89 b8 00 00 08 00
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 2853304
Mar 27 12:15:52 harrison kernel: raid1: sdb1: rescheduling sector 2851192
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 90 00 50 00 00 02 00
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 9437264
Mar 27 12:15:52 harrison kernel: md: super_written gets error=-5, uptodate=0
Mar 27 12:15:52 harrison kernel: raid5: Disk failure on sdb5, disabling device.
Mar 27 12:15:52 harrison kernel: raid5: Operation continuing on 5 devices.
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:15:52 harrison kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 00 00 48 00 00 02 00
Mar 27 12:15:52 harrison kernel: end_request: I/O error, dev sdb, sector 72
Mar 27 12:15:52 harrison kernel: md: super_written gets error=-5, uptodate=0
Mar 27 12:15:52 harrison kernel: raid1: Disk failure on sdb1, disabling device.
Mar 27 12:15:52 harrison kernel: raid1: Operation continuing on 5 devices.
Mar 27 12:15:52 harrison kernel: RAID5 conf printout:
Mar 27 12:15:52 harrison kernel:  --- rd:6 wd:5
Mar 27 12:15:52 harrison kernel:  disk 0, o:1, dev:sda5
Mar 27 12:15:52 harrison kernel:  disk 1, o:0, dev:sdb5
Mar 27 12:15:52 harrison kernel:  disk 2, o:1, dev:sdc5
Mar 27 12:15:52 harrison kernel:  disk 3, o:1, dev:sdd5
Mar 27 12:15:52 harrison kernel:  disk 4, o:1, dev:sde5
Mar 27 12:15:52 harrison kernel:  disk 5, o:1, dev:sdf5
Mar 27 12:15:52 harrison kernel: RAID5 conf printout:
Mar 27 12:15:52 harrison kernel:  --- rd:6 wd:5
Mar 27 12:15:52 harrison kernel:  disk 0, o:1, dev:sda5
Mar 27 12:15:52 harrison kernel:  disk 2, o:1, dev:sdc5
Mar 27 12:15:52 harrison kernel:  disk 3, o:1, dev:sdd5
Mar 27 12:15:52 harrison kernel:  disk 4, o:1, dev:sde5
Mar 27 12:15:52 harrison kernel:  disk 5, o:1, dev:sdf5
Mar 27 12:15:52 harrison kernel: RAID1 conf printout:
Mar 27 12:15:52 harrison kernel:  --- wd:5 rd:6
Mar 27 12:15:52 harrison kernel:  disk 0, wo:0, o:1, dev:sda1
Mar 27 12:15:52 harrison kernel:  disk 1, wo:1, o:0, dev:sdb1
Mar 27 12:15:52 harrison kernel:  disk 2, wo:0, o:1, dev:sdc1
Mar 27 12:15:52 harrison kernel:  disk 3, wo:0, o:1, dev:sdd1
Mar 27 12:15:52 harrison kernel:  disk 4, wo:0, o:1, dev:sde1
Mar 27 12:15:52 harrison kernel:  disk 5, wo:0, o:1, dev:sdf1
Mar 27 12:15:52 harrison kernel: RAID1 conf printout:
Mar 27 12:15:52 harrison kernel:  --- wd:5 rd:6
Mar 27 12:15:52 harrison kernel:  disk 0, wo:0, o:1, dev:sda1
Mar 27 12:15:52 harrison kernel:  disk 2, wo:0, o:1, dev:sdc1
Mar 27 12:15:52 harrison kernel:  disk 3, wo:0, o:1, dev:sdd1
Mar 27 12:15:52 harrison kernel:  disk 4, wo:0, o:1, dev:sde1
Mar 27 12:15:52 harrison kernel:  disk 5, wo:0, o:1, dev:sdf1
Mar 27 12:15:52 harrison kernel: raid1: sda1: redirecting sector 2851192 to another mirror
Mar 27 12:16:00 harrison RAIDiator: Disk failure detected.\n\nIf the failed disk is used in a RAID level 1, 5, or X-RAID volume, please note that volume is now unprotected, and an additional disk failure may render that volume dead.  If this disk is a part of a RAID 6 volume, your volume is still protected if this is your first failure.  A 2nd disk failure will make your volume unprotected.  It is recommended that you replace the failed disk as soon as possible to maintain optimal protection of your volume.\n\n[Sun Mar 27 12:15:53 PDT 2011]
Mar 27 12:16:45 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:16:45 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:16:45 harrison kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 80 00 48 00 00 02 00
Mar 27 12:16:45 harrison kernel: end_request: I/O error, dev sdb, sector 8388680
Mar 27 12:16:45 harrison kernel: end_request: I/O error, dev sdb, sector 8388680
Mar 27 12:16:45 harrison kernel:  **************** super written barrier kludge on md1: error==IO 0xfffffffb
Mar 27 12:16:45 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:16:45 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:16:45 harrison kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 80 00 48 00 00 02 00
Mar 27 12:16:45 harrison kernel: end_request: I/O error, dev sdb, sector 8388680
Mar 27 12:16:45 harrison kernel: md: super_written gets error=-5, uptodate=0
Mar 27 12:16:45 harrison kernel: raid5: Disk failure on sdb2, disabling device.
Mar 27 12:16:45 harrison kernel: raid5: Operation continuing on 5 devices.
Mar 27 12:16:45 harrison kernel: RAID5 conf printout:
Mar 27 12:16:45 harrison kernel:  --- rd:6 wd:5
Mar 27 12:16:45 harrison kernel:  disk 0, o:1, dev:sda2
Mar 27 12:16:45 harrison kernel:  disk 1, o:0, dev:sdb2
Mar 27 12:16:45 harrison kernel:  disk 2, o:1, dev:sdc2
Mar 27 12:16:45 harrison kernel:  disk 3, o:1, dev:sdd2
Mar 27 12:16:45 harrison kernel:  disk 4, o:1, dev:sde2
Mar 27 12:16:45 harrison kernel:  disk 5, o:1, dev:sdf2
Mar 27 12:16:45 harrison kernel: RAID5 conf printout:
Mar 27 12:16:45 harrison kernel:  --- rd:6 wd:5
Mar 27 12:16:45 harrison kernel:  disk 0, o:1, dev:sda2
Mar 27 12:16:45 harrison kernel:  disk 2, o:1, dev:sdc2
Mar 27 12:16:45 harrison kernel:  disk 3, o:1, dev:sdd2
Mar 27 12:16:45 harrison kernel:  disk 4, o:1, dev:sde2
Mar 27 12:16:45 harrison kernel:  disk 5, o:1, dev:sdf2
Mar 27 12:25:51 harrison kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Mar 27 12:26:04 harrison kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 00 00 00 20 00
Mar 27 12:26:09 harrison kernel: end_request: I/O error, dev sdb, sector 0
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 0
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 1
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 2
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 3
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 27 12:26:09 harrison kernel: end_request: I/O error, dev sdb, sector 0
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 0
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 e8 e0 88 a8 00 00 08 00
Mar 27 12:26:09 harrison kernel: end_request: I/O error, dev sdb, sector 3907029160
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 488378645
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 e8 e0 88 a8 00 00 08 00
Mar 27 12:26:09 harrison kernel: end_request: I/O error, dev sdb, sector 3907029160
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 488378645
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Unhandled error code
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Mar 27 12:26:09 harrison kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 27 12:26:09 harrison kernel: end_request: I/O error, dev sdb, sector 0
Mar 27 12:26:09 harrison kernel: Buffer I/O error on device sdb, logical block 0

Thanks for any input someone might be able to offer. This certainly seems like a pattern...

Best,
- CitizenPlain