I have a big problem! No backup (yes, I know), and my RNP seems to be dead in the water. I suspect I've lost all the data, but I'm hoping there might be some way to recover… The data consists of several terabytes of videos that I've recorded over the years. Not a tragic loss, but I'd really, really like to get them back.
I have an RNP Pioneer, running RAIDiator 4.2.26 firmware. The NAS is configured as X-RAID2, unfortunately NOT using dual redundancy.
So here's what I saw (it was 2 days ago, so I may have confused some of the details):
1. ReadyNas wend offline, so I rebooted it. 2. I opened Frontview to look at the status. 3. It indicated that Disk 4 was accumulating errors, and the disk might be failing. 4. I dug out a brand new drive to replace Disk 4. 4. I restarted the NAS from Frontview 5. Surprisingly, the NAS started to resync and rebuild Disk 4. 6. I let it run, and the next day the status showed:
Disk 4 as "Spare" Disk 1 as "Dead"
This is the current state. I didn't expect the NAS to resync (I had not yet replaced any drives) and I did not expect a second drive failure.
So this is a poster-child case for A. keeping backups, and B. using dual redundancy. *sigh*
Anyway, is my data toast, or *IF* I can use a utility like SpinRite to recover either Disk 1 or Disk 4, might I be able to convince the RNP to recognize the original array, and salvage the data?
Once a drive has been marked as "DEAD", can anything be done to un-dead it?
We have paid data recovery options available from support to look into issues like this.
Can you send me the logs (see link in my sig) from your unit?
Yes, I would recommend dual-redundancy.
The errors on disk 4 in this case lead to a resync being necessary. Unfortunately disk 1 was also failing. A resync outs heavy stress on all disks and can finish a failing disk off.
I just captured a new set of logs and emailed them.
To summarize my problem:
1. Disk 4 was having issues, I was about to replace it 2. NAS performed an unexpected Resync 3. After the Resync, Disk 4 marked as "Spare" and Disk 1 marked as "Dead"
) 2. Re-install both disks 3. *** Prod the NAS into recognizing at least one of the two repaired disks, and show me my data once again *** 4. Replace both disks (one at a time)
Thanks, vandermerwe. The videos aren't important enough to spend big bucks to recover.
I'm hoping a ReadyNAS-knowledgable person can help with the do-it-yourself aspect, and illuminate me re. can the drives be marked "good", and will the data - or even some of it - be readable by the NAS. That, or point me to a resource that will help.
By the way, by all accounts, SpinRite is definitely NOT a generic tool. It appears to be best-in-class at repairing marginal sectors on a failing or failed drive.
Thanks again, vandermerwe, that was an interesting thread, for sure. There was mention of the kernel having marked the drive faulty, which happened, I think, in my case. How does one "unmark" such a drive?
I should add that I have email alerts set up on all three of my RNP units. The two "RNP Business" units send me email alerts just fine. But the (now failed) RNP Pioneer unit, for whatever reason, does not send email alerts.
Well looking at your logs all of your four disks appear to have issues. This is a perilous situation where any attempt to bring the array back online needs to be done very carefully. It's possible cloning of disks may be needed.
Thanks, mgdm. I have a hard time interpreting the "disk_smart.log" files. Any pointers to specific things I should be watching?
I would still like to better understand what's happening when the NAS marks a drive as "Dead" and "Spare". Does it set a flag on the drive, or does it record the drive serial number in some data structure on the NAS itself? And how do I undo that? I mean, if I clone the "dead" drive, wouldn't it still think it's Dead?
ATA errors or reallocated sectors with a count over 50 indicates that a disk is failing.
There are other things to look out for such as a non-zero current pending sector count.
A disk would be marked as dead when it fails to respond in a reasonable amount of time repeatedly, I think. As spare when it marked as such by md raid, I think.
If you clone a dead drive you might be able to to make use of it again. But that is generally something to try if forcing it online doesn't work first.
With the number of disks you have failing any resync could seriously affect the chances of data recovery so one does need to be careful how the array is brought back online.
