One drive initially marked as failed, Resync now at 6.3% after 12 hours. What next? ReadyNAS Ultra 4

[Bobby_V]

Decided to do some housekeeping and was deleting files from the Ultra 4 when I checked Frontview and found that one of the drives had been marked as having failed. I immediately shut down the NAS using the Frontview command. I then unplugged it from the mains and ethernet. 

When I turned it back on to back up important files it performed a file system check and then imediately started a Resync. Disk 4, previously marked as Failed, is now marked as "Resync" under the status, and checking the SMART status for all of the drives shows 0 reallocated sectors, and no other errors. The SMART status of disk 4 is nearly identical to that of the other three. This is all with the original four disks, I had not yet swapped out disk 4.

I began transferring some of the important files over to other external backup drives, but the transfer speed is incredibly slow (estimated to take 3 days to copy 250GB), and I suspect this is due to the active Resync being performed. I have 4x 2TB drives in the NAS, and the Resync is at 6.3 % complete after more than 12 hours (would take ~8 days to complete at this point). I have turned off all the active services to speed things up as much as possible, and have two replacement disks ready to swap in. The unit and all original disks are from 2010, with about 45,000 power-on hours.

My main goal is to protect the data on the NAS. Much of the data, but not all, is also stored in other places, but not enough redundancy that I would be comfortable with a factory reset.

My questions:

 - Do you think Disk 4 has actually failed? If the drive has failed, then is there any point in letting the Resync process continue?

- Is there a risk to letting Resync continue, finding bad sectors across multiple disks, and having the entire array fail?

- What would be the best way to proceed with backing up the data with the active Resync process?

 - Should I let Resync continue, or somehow try to stop it to perform a quicker backup? (And how would I stop it?)

Even if the Resync process completes and says all is good, and the SMART status for Disk 4 shows green, I'm leaning toward replacing it, which would mean it would have to go through the whole Resync process all over again.

[StephenB]

What stats are you seeing on disk 4 - particularly pending sectors, reallocated sectors, and ATA timeouts? 

Are stats getting worse for any drive while the resync is proceeding?

[Bobby_V]

Thank you for your quick response!

Here are the stats from Drive 4 (very similar to the others)

SMART Attribute
Spin Up Time	0
Start Stop Count	3453
Reallocated Sector Count	0
Power On Hours	45055
Spin Retry Count	0
Power Cycle Count	315
Runtime Bad Block	0
End-to-End Error	0
Reported Uncorrect	0
Command Timeout	3
High Fly Writes	0
Airflow Temperature Cel	31
G-Sense Error Rate	0
Power-Off Retract Count	11
Load Cycle Count	3454
Temperature Celsius	31
Current Pending Sector	0
Offline Uncorrectable	0
UDMA CRC Error Count	0
Head Flying Hours	64265595669968
Total LBAs Written	412058933
Total LBAs Read	3157320595
ATA Error Count	0

No real changes that I see occuring for any of the disks. 

Thanks again.

[StephenB]

The only possible concern in the stats would be the command timeouts - and if they aren't increasing they aren't happening now.

Completion estimates for resync on the ultra/pro are often way off, but it should be farther along than 6% by now.

I think I'd let it complete if you can stand the poor performance.  

You could of course pull drive 4, and the resync would stop. But I'd be more inclined to do that if we were seeing clear evidence that the drive was getting worse - and we aren't.

[Bobby_V]

Ok. I will keep an eye on the SMARt status for the drives and check it again in a few hours. Hopefully the Resync speeds up a bit. 

Thanks again for your advice!

[Bobby_V]

Resync just finished. It seemed to go much faster after the files finished backing up. The SMART status of Disk 4 is now "OK" and there are no additional errors. 

Really makes me wonder what may have caused the drive to be read as "failed." At this point do I just continue on as if nothing happened or replace the drive? Would there be any point in mounting the disk externally and running DiskWarrior to check the file system (would this be dangerous to the RAID arcitecture)?

[StephenB]

You can't check the filesystem of a single drive in a RAID array.  The data is striped across all the drives.

Maybe schedule a scrub.  Resyncing the array wrote to every sector of the drive that's in the data array.  A scrub would read all those sectors and verify RAID parity.

[Bobby_V]

Just wanted to follow-up on the situation and where things are at for anyone else that may find this helpful. 

I scheduled a scrub of the disks (then somehow accidentally wound up doing a second one), both completed without error, and all of the drives are still read as in good working order. 

Thank you very much for your help with this issue, your time is deeply appreciated.