This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
ReadyNAS Pro 6 - Volume scan failed to run properly
Hi -- Long time ReadyNAS user with very few problems to report. Just ran into one, sadly.
I have a ReadyNAS Pro Business Edition [X-RAID2] with 6 disks running RAIDiator 4.2.28.
FrontView reported increasing SMART errors on Disk 4, so I quickly ordered a replacement.
With the new drive in hand, I accidentially pulled Drive 1. Quickly realizing my mistake (30-45 seconds) I reseated Drive 1 and pulled Drive 4 and swapped it for the new drive. Yes, Google now tells me that was not the best course! Unfortunately, the new drive was DOA (a first for me!). Left it to resync overnight, not realizing the issue with the new drive.
Next day, saw there were still errors so pulled the bad "new" Drive 4 and reinstalled the original, hoping this would at least return the NAS to working order while I replaced the faulty drive.
Through multiple reboots, I receive the error: "Volume scan failed to run properly." and "The paths for the shares listed below could not be found. Typically, this occurs when the ReadyNAS is unable to access the data volume. media-u6 XFER backup-u6"
So now all of the shares are inaccessible. The user share will mount, but shows no files.
I've opened a support ticket #26094210, but perhaps the community can help as they have in the past. mdgm solved an issue with too many logs on my NV+--I'm still grateful for that assist!
Re: ReadyNAS Pro 6 - Volume scan failed to run properly
A follow-up : I did indeed need support, but they were able to restore the volume with 5 good disks out of the six. They had to escalate to level 3 and remote in, but were able to get everything working again. The NAS was well past warranty, but the $75 per-incident support was money well spent.
And for the record, Netgear support was stellar. Every person I spoke to was competent, knowledgeable and polite. They called back when they said they would and didn't miss a beat, regardless of the handoff between four different people (or more, for all I know). Great job, ReadyNAS Support Team!
Re: ReadyNAS Pro 6 - Volume scan failed to run properly
As you likely realize now, when you reseated drive 1 the NAS treated it as a drive replacement, and started rebuilding the array. During the rebuild period, you don't have RAID redundancy. Since you pulled drive 4 before the rebuild finished, you lost the volume.
Re: ReadyNAS Pro 6 - Volume scan failed to run properly
A follow-up : I did indeed need support, but they were able to restore the volume with 5 good disks out of the six. They had to escalate to level 3 and remote in, but were able to get everything working again. The NAS was well past warranty, but the $75 per-incident support was money well spent.
And for the record, Netgear support was stellar. Every person I spoke to was competent, knowledgeable and polite. They called back when they said they would and didn't miss a beat, regardless of the handoff between four different people (or more, for all I know). Great job, ReadyNAS Support Team!
Re: ReadyNAS Pro 6 - Volume scan failed to run properly
We have this same problem with our Pro 6 but we NEVER pulled a drive while the array was rebuilding.
Here's the chronology of events:
1) drive 5 reported errors and eventually died altogether
2) drive 6 started reporting errors but wasn't dead when we pulled 5 and replaced it
3) the array started rebuilding but eventually failed with an error and drive 6 was marked dead and noticed 5 was marked as a hot spare and the volume went into read only life support mode (very puzzling as 4 drives are plenty for at least RAID5 and we're using FLEX RAID2 or something to it doesn't make sense)
4) we don't have another drive yet so we reseated 5 and booted back up and it began to rebuild and we were able to work again for about 3 hours, drive 6 was just marked dead
5) the rebuild failed again and yet again the unit went into life support mode with 5 as a hot spare so we shut down and this time reseated both 5 and 6 and booted back up
6) the unit failed a volume check now (and has ever since) and again put the volume into life support mode with 5 as a hot spare
7) we shut down and pulled drive 6 and reseated 5 and booted back up and had the same problem
😎 we shut down again and reseated 5 and put 6 back in and booted back up and now the volume isn't being recognized at all (and hasn't been since)
We still have the old failed drive 5. I'm going to try to put that back in to see if that makes a damn bit of difference but am not hopeful. The total data footprint here is laughable. It's like 500GB TOPS and we have 6 2TB drives in this sucker.
So we NEVER pulled a drive while a rebuild was happening ever. Now the unit is totally down and tech support is totally pessimistic of being able to help because 2 drives are failed. We now need level 3 support at an additional $200 (and we already added a support contract for ~$300 to get back in the graces of tech support help) for a "maybe but not likely able to help". They suggested we call the number when we made the decision to go forward with level 3 support and that the tech helping us was going home for the weekend and when I called, the automated number hangs up when we tell it we're calling about a ReadyNAS storage product and it says to file a support ticket online. SO we have NO HELP THIS ENTIRE WEEKEND and this is the repository of all data for 2 companies. We are dead in the water but do have backups that we're looking into now, but does anyone have any suggestion on how to move forward? If this were a regular server I was maintaining, Linux or Windows with a RAID 6 array with failed drives I'd be doing some Linux scripting with dd and attempting to rebuild the data given the amount we need is SO SMALL ANY 1 DRIVE SHOULD CONTAIN A COMPLETE FRIGGING COPY WITH OVER 70% FREE SPACE!
the array started rebuilding but eventually failed with an error and drive 6 was marked dead and noticed 5 was marked as a hot spare and the volume went into read only life support mode (very puzzling as 4 drives are plenty for at least RAID5 and we're using FLEX RAID2 or something to it doesn't make sense)
So you had xraid with single redundancy - which is the same as RAID-5 in your case. You pulled a drive, and immediately lost the redundancy as a result. Then you installed a new drive, and the resync started.
So far, so good. But now you had a second drive failure during the resync. The resync hadn't completed, so redundancy hadn't been restored. So the second failure destroyed the array.
Here's the chronology of events: If this were a regular server I was maintaining, Linux or Windows with a RAID 6 array with failed drives I'd be doing some Linux scripting with dd and attempting to rebuild the data given the amount we need is SO SMALL ANY 1 DRIVE SHOULD CONTAIN A COMPLETE FRIGGING COPY WITH OVER 70% FREE SPACE!
Well, RAID doesn't work that way. The data is spread across all the drives, including the two that failed. Data recovery basically does try to recover the data from the existing drives, and put the pieces back together. As you say it is labor intensive.
...this is the repository of all data for 2 companies. We are dead in the water but do have backups that we're looking into now, but does anyone have any suggestion on how to move forward?
If you have full backups, then the fastest way back is to do a factory reset with the remaining 4 healthy drives in place, and then reconfigure the NAS and restore the data from backup.
If there is data that hasn't been backed up that is worth the expensive of recoverying, you could get two new 2TB drives. Remove all the existing drives, marking them by slot (and keeping the original failed drive).
Then put in the two 2TB drives, do the factory reset, and restore the backup data. That keeps the option of data recovery open.
Re: ReadyNAS Pro 6 - Volume scan failed to run properly
I never intended on that hot spare. We have XRAID2 and for whatever reason (this never has happened before when one drive failed, was replaced, while another was giving trouble) AFTER the rebuild failed it set the new good drive 5 was set as a hot spare for no frigging reason. The 4 drives were showing all the data in the volume just fine, but it was read only due to life support mode. Why wouldn't it rebuild with 5 I have no idea. We still have the old drive 5, but in theory, we'd only need one new drive to replace the dead drive 6.
There has GOT to be a way to recover this because we DID HAVE RAID 6 YESTERDAY! How can the entire volume be lost with RAID 6 and/or why the hell would it go from RAID 6 to RAID 5 with a hot spare just because drive 6 just died?
There has GOT to be a way to recover this because we DID HAVE RAID 6 YESTERDAY!
You didn't say anything about RAID-6 in your earlier post. You are saying you had set up XRAID2 with dual redundancy? By default it is only single redundancy.
Netgear Support can tell you exactly what raid mode you were in from your logs.
Understood. My explanation (which assumed single redundancy btw) is that the resync failed because drive-6 failed while it was running. So drive 5 was never added to the array. Of course I only have the information in your post to go on.
Again, Netgear support should be able to tell you what happened.
Re: ReadyNAS Pro 6 - Volume scan failed to run properly
It is XRAID2 and it did say RAID6 (which I thought defaultly WAS dual redundancy).
Here's some output from dmesg that leads me to believe it's RAID6 still (from the logs we sent to support who are completely pessimistic on being able to help and decided to go home for the weekend):
md/raid:md1: device sda2 operational as raid disk 0 md/raid:md1: device sdd2 operational as raid disk 3 md/raid:md1: device sdc2 operational as raid disk 2 md/raid:md1: device sdb2 operational as raid disk 1 md/raid:md1: allocated 4272kB md/raid:md1: raid level 6 active with 4 out of 4 devices, algorithm 2 RAID conf printout: --- level:6 rd:4 wd:4 disk 0, o:1, dev:sda2 disk 1, o:1, dev:sdb2 disk 2, o:1, dev:sdc2 disk 3, o:1, dev:sdd2 md1: detected capacity change from 0 to 1073610752
I just put in the old bad drive 5 to see if that'd help and it didn't (volume scan still fails and no volume found at all), tho now Frontview seems to show all 6 drives as healthy/available for use. We do have backups of most of our key data but I'm still FLOORED that this happened with 4 perfectly healthy drives in a RAID6 configuration.
It is XRAID2 and it did say RAID6 (which I thought defaultly WAS dual redundancy).
XRAID2 by default uses single redundancy (RAID5 if you have > 2 disks). You can choose to add a disk for extra redundancy if you wish - telling it to use RAID6. That's a key piece of information, that was missing in your first post. My original explanation assumed the default single redundancy setup.
Re: ReadyNAS Pro 6 - Volume scan failed to run properly
Well for what it's worth arrays md0 and md1 (the core array for the partition housing the main /etc and other folders for the OS, and the array for the swap partitions respectively) both show as RAID 6 arrays. But somehow md2 is RAID 5 (which is and was the problem). Because of the chronology of events, it's clear to me that it WASN'T RAID 5 Thursday morning when we pulled the dead drive 5 out and installed the new one and a resync began. That morning, BEFORE we pulled drive 5 we saw drive 6 show up as dead as well (or dying, I can't remember at this point). Nonetheless, we're in the predicament that we're in and we are faced with attempting to clone the now dead drive 6 to a good drive and put it in and see what we can do about reconstructing the RAID 5 array without drive 5.
In short, this shouldn't have happened with 4 perfectly working drives in a RAID 6 array, I should've been getting life support errors and that's only until it realized a perfectly good drive was inserted ready for it to just be 1 drive down. This has happened to us before and wasn't an issue at all.
Never once have I complained that RAID should be our backup, by the way, the need for offsite backup is NOT lost on me whatsoever and is why we are able to have SOMETHING in place at this point.
Thanks for the help, and for what it's worth Netgear support is quite useful when they're open and actively responding.
