× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Data Volume lost after disk replacement on Readynas Pro 6

Hombibi
Guide

Data Volume lost after disk replacement on Readynas Pro 6

Hi, I just ran into a serious problem, and I hope someone can help me.

 

Currently my Readynas Pro 6 does not see any volume on the 6 disk X-Raid Array anymore.

 

I walked past my NAS a couple of hours ago and saw the display flashing with the message that the volume was degraded. A bit strange as I had not gotten any warning on disk degradation before. I identified the disk (#2) and replaced it with a spare which I had ready for such occasions. After the rebuild was finished the readynas displayed the message that the volume was still degraded: some 5 to 10 minutes later the readynas identified disk 5 to be replaced. Two disks at the same time? I did not have another spare disk so I thought to start a scrub to see if that would make any difference. During the scrub my NAS switched off on the timer. When I restarted it it had lost its X-Raid array.

 

Would anyone have any idea if I can fix this? The nas pro6 runs 6.2.4., the admin interface works fine, I have access to the logfiles.

 

Erik

Message 1 of 21

Accepted Solutions
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Yes, required for other than windows filesystems.

Obviously much cheaper than any professional recovery service.

 

View solution in original post

Message 21 of 21

All Replies
StephenB
Guru

Re: Data Volume lost after disk replacement on Readynas Pro 6

Ouch.  Normally I'd suggest contacting Netgear support - but of course OS 6 is not supported on the Pro.  Still they might be willing to take a data recovery contract (not sure).

 

Multiple disk failures do happen, and rebuilding the array when you replace a disk stresses all the others.

 

BTW (for future reference) it is a very bad idea to run a scrub on a compromised volume.

Message 2 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Thanks StephenB,

If I understand X-Raid2 well enough (since yesterday..) than it is essentially a raid 5 configuration. With my limited knowledge I would say that as only one disk failed at any moment it should be possible to recover if I have not damaged anything seriously with the scrubbing. Unfortunately I have no idea where to start and obviously I am done experimenting at this stage.
I'll be happy to contact Netgear to see if they can help me, but I'll be just as happy if anyone from the community is able to show me what needs to be done. I hope it can be fixed.

Message 3 of 21
vandermerwe
Master

Re: Data Volume lost after disk replacement on Readynas Pro 6

Although you received notification of the second disk failure after being notified that the resync had completed, it is possible that the disk actually failed during the resync.  I've certainly seen many users experience this pattern of events.

If you have no backup then you should try support as StephenB has suggested.  There isn't anything you can do which isn't going to make data recovery more difficult. 

If support won't help then I guess you can look for other data recovery services.

 

 

Message 4 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Thanks vandermerwe,

I'll contact Netgear, I truly hope they can help me out

 

Your message made me think however: Since my problem I have been thinking to setup a second 6 bay system as hot standbye or as a live backup system.

But if it is common that two disks fail at the same time, and resync is often not succesfull, and if the chances for a simple recovery with a raid5 solution are slim therefore, wouldn't it be much safer to buy a simple dual disk NAS system and set it up only to mirror the other disk? In that case the chance for losing all data would be the chance that two disks out of two fail at the same time. In my 6 bay X-raid/Raid5 configuration the chances that two disks fail at the same time is 15 times higher (if my chance calc is right). I had never thought about it, but to me that sounds too risky (now). Maybe it doesn't justify the capacity loss of a simple mirrored setup with only two disks or the additional investment in more expensive 6 bay systems. But moreover, doesn't the much increased risk of data loss put X-Raid aside as a reliable solution for data storage?

 

 

 

Message 5 of 21
vandermerwe
Master

Re: Data Volume lost after disk replacement on Readynas Pro 6

With higher numbers of disks Xraid with dual redundancy (or RAID 6) is what most here would recommend.

A lot of us have a dual nas setup, using replicate or other synchronisation

Message 6 of 21
StephenB
Guru

Re: Data Volume lost after disk replacement on Readynas Pro 6


@Hombibi wrote:

 

Since my problem I have been thinking to setup a second 6 bay system as hot standbye or as a live backup system.

But if it is common that two disks fail at the same time, and resync is often not succesfull, and if the chances for a simple recovery with a raid5 solution are slim therefore, wouldn't it be much safer to buy a simple dual disk NAS system and set it up only to mirror the other disk? In that case the chance for losing all data would be the chance that two disks out of two fail at the same time. In my 6 bay X-raid/Raid5 configuration the chances that two disks fail at the same time is 15 times higher (if my chance calc is right). I had never thought about it, but to me that sounds too risky (now). Maybe it doesn't justify the capacity loss of a simple mirrored setup with only two disks or the additional investment in more expensive 6 bay systems. But moreover, doesn't the much increased risk of data loss put X-Raid aside as a reliable solution for data storage?


First of all, though two disks failing at around the same time does happen, I wouldn't call it "common".  Also, the chances are the same if you have a 6-bay NAS running xraid or if it is running jbod.  What changes is the impact of those failures.  If you were running jbod, then one failure would lost 1/6 of your data, and two failures would have lost 1/3.  Running xraid, 1 failure loses nothing, but 2 or more lose everthing.

 

Math aside, recovery of a RAID array can be difficult.  One option is to lower the odds, and run RAID-6 (or maybe RAID-10, or multiple RAID-1 volumes) as vandermerwe suggests.  You can also run jbod on the backup NAS, which makes it less likely that you'd lose everything.  I've taken the latter strategy (I have 3 copies of all my data, and my backup NAS(s) are using jbod to simplify recovery.  The main NAS uses xraid/RAID-5.

 

So there's more than one approach here. However, I think that if most of the experienced users here had to choose between a single NAS with RAID-6, or a main NAS with RAID-5 + a backup, they'd vote for RAID-5 + backup.

 

Message 7 of 21
mdgm-ntgr
NETGEAR Employee Retired

Re: Data Volume lost after disk replacement on Readynas Pro 6

Please send me the logs (see the Sending Logs link in my sig).

Message 8 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

mdgm, done.

Thanks for looking into my problem.

 

Message 9 of 21
mdgm-ntgr
NETGEAR Employee Retired

Re: Data Volume lost after disk replacement on Readynas Pro 6

Definitely a data recovery situation. Since this is OS6 on a legacy ReadyNAS you would need to consider 3rd party data recovery. Note that data recovery may be unsuccessful.

Message 10 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Thanks mdgm.

Obviously I am a very disappointed that by replacing a disk to prevent a bigger problem I actually caused it.

I'll go the data recovery route.

Thanks everyone.

Message 11 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Got a quote for data recovery: 500 Euro for investigation, and over 2500 for recovery. Smiley Sad Not only is the expensive 6 bay system more vulnerable than a simple two disk mirrored system, it is also much more expensive to recover, if at all possible.

 

I probably will have to try to recover my data myself. Does anyone have experience with r-studio, http://www.diydatarecovery.nl, or with the products by http://www.reclaime.com/? Or does anyone know of any good manuals out there that could guide me to investigate and recover?

Message 12 of 21
vandermerwe
Master

Re: Data Volume lost after disk replacement on Readynas Pro 6

I don't I'm afraid.  You are however asking for information that you have established could be worth 3000 euros.  

Is that the only quote you have found, it does sound a little steep?

Message 13 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

I have decided to give it go myself.

 

Read a lot, found that raid/parity setups are not really good solutions anymore in these days where disks are so much larger and so different from those when these systems were created (1995-ish?). Raid 10 seems to be the current preferred setup by professionals. (no complicated expensive parity calculation, disk space is "cheap"). I further found some real good insight on recovery chances: according to one: in 85% of cases it can be solved with software, in 10% it is the circuit board, which you can even hotswap and I have 5 spare.., in 4% it is the heads or the platters, and in 1% it is the motor.

So  I reckon I have a larger than 95% chance to recover my data. Worth a try.

 

First port of call is to make bit by bit images of the drives so that I don't have to use them for the actualy inspection and recovery of the data. I use the free linux version of r-data to make them, so that I have some idea of the chances for succes before I buy the software. I could also have used ddrescue, both these tools  read the disk sequentially (as an old fashioned record) which is the least demanding for the (now fragile) HDDs.

 

Bought a esata hot swap cradle, filled the nas pro with 6 new disks (JBOD) and started with r-data to make images. The original disk2 (which triggered the problem) can't be read. That is a candiate for another PCB, and I could eventually replace the heads if necessary.  I am currently imaging nr5, the second one that give up. Apart from a few strange noises in the early stage, the imaging process is proceeding without errors. I think that if this one works and the sixth images fine I should be able to restor the array.

 

I am not a pro as I stated above, any advice is welcome, to be continued.

 

 

 

 

Message 14 of 21
StephenB
Guru

Re: Data Volume lost after disk replacement on Readynas Pro 6

A lot of the analysis you'll see on line about the "death of raid" is flawed.  One observation is that most of the statistical analyses are grounded in the manufacturer specs, and not on real-world data.  

 

It is true that a lot of enterprises and data centers use RAID-10.  They also are often hosting storage for local VM servers, and aren't constrained by 1 gigabit ethernet network speeds. RAID-10 is faster than RAID-5 or RAID-6 - and if you aren't network-limited that performance improvement benefit can overcome the cost of the extra disks needed.

 

I'm sticking with RAID-5 for my main NAS myself.  I am shifting to jbod for my backups, because recovery (if needed) is simpler.

 

Good luck with your recovery efforts - please followup (either way) on the results.

Message 15 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Disk 5 and 6 imaged successfully, according to r-Linux.

 

 

R-linux can scan the images, I see lots of files but they are all numbered. Hell of a task to open each one and rename.....

 

 

So next is trying to recover the array: Downloaded and installed reclaime free raid recovery on a Windows 8.1 laptop and started the raid 5 analysis. Reclaime seems the be very good in finding back the original raid configuration, which can than be used by r-studio to recover the files with original names and file structure.

 

After I started the raid 5 scan, reclaime quickly found disk two to be empty. This means that the sync process had failed. If the other images are ok than I think this should be enough. Otherwise I might still have to try to recover the original disk2.

 

The analysis can take a few days, more news if and when.

Message 16 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

In the mean time I am setting up an NV+ v1 as dedicated backup machine (boot, check integrity, backup, shutdown).

Does anyone know how to setup an NV+ to JBOD?

Message 17 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Never mind, found it.

Message 18 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Succes! The data is back!

 

Most discussions I recently read on raid 5 recovery were not very encouraging. And XRaid-2 got even less reactions, let alone positive ones. Not very promising: After I had made the images of the six drives I tried first to repair the faulty disk that started the disaster.  Not immediately succesfull I turned  to software recovery from the images. I started with Reclaime's free raid recovery, but that did not work for me.The raid array was not recognised. During the scans I searched for the raid parameters of a XRaid-2 config. Turns out they do not exist, as it is a dynamic raid setup. Then I found a first digestable insight in nas partitions, with referral to readynas XRaid-2 on Youtube: https://youtu.be/kGvBZdKFdFc. After 20 minutes on btrfs, lvm and dynamic raid arrays, recovering dynamic raid arrays did not sound very promising anymore..

 

I tried RTT's software, and I contacted a few software vendors, who let me know that as XRaid-2 is a proprietary raid setup they could not detect, analyse or repair it. Encouragement levels dropping ...

I then contacted Reclaime with the question if their "free raid recovery" software could (should be able to) detect and rebuild the XRaid-2 array from a six disk array with one probably missing (the failed sync on disk 2). I got a message back advising me to use "Reclaime File Recovery". This did not immediately work, but it did provide some encouraging results recognising the size of the array and the original number of drives. After a few mails and an update the software recognised all the images and the full original array. Within minutes the original filesystem was available after which I started copying out files to my JBOD NV+. Within three hours I had recoverd all my critical data.

At the moment I am still copying less important files, but I am not entirely sure if I will recover all the data. It might be that I have lost something due to the two drive crashes and subsequent sync and scrub actions. The software is still analysing the disc-images, repiecing files that might have been unreadable due to the disk crashes or deleted due to the failed sync process or disk scrub. I understand that btrfs is good for that, lets see.

 

Me however, I am amazed, and as you may image very, very happy that I found a solution. One of the things that impressed me is Reclaime's software, their knowledge or raid arrays, HDD drives and Nas system,  and their support: Exemplary. Turns out that the video I found was theirs, and they have a whole library of youtube video's on the subjects, higly recommended.

 

So, what's next: setup my readynas pro again as a X-Raid2 system? Well.. with solutions from Recliame I might just give it a go again....

 

But I probaly won't: If I am correct the key reason for Raid was to offer multiple fysical drives as one logical volume, thus offering capacity and speed.  (in 1995, a 1GB HDD became commonplace according to http://www.relativelyinteresting.com/comparing-todays-computers-to-1995s/)  This introduced interdependency between disks that was subsequently adressed with parity or redundancy, or both.

 

Today's 2TB HDD's are common, fast, and can store so much data that I really wonder if there is a need to offer multiple 2GB drives in a raid/parity setup as one logical drive for a home situation, or a small office. And as we still need to rely on a backups I believe there is a good case for increasing capacity eliminating parity, and reducing dependency between disks, thus avoiding Raid completely: A weekly backup may do, and I will never lose more than the actual disks that fail, if the backup fails too. Much less risk, and what is more, much less hassle, especially in recovery...

 

 

 

 

 

 

 

 

Message 19 of 21
vandermerwe
Master

Re: Data Volume lost after disk replacement on Readynas Pro 6

That is fantastic, well done!

You presumably needed Reclaime Ultimate?

Message 20 of 21
Hombibi
Guide

Re: Data Volume lost after disk replacement on Readynas Pro 6

Yes, required for other than windows filesystems.

Obviously much cheaper than any professional recovery service.

 

Message 21 of 21
Top Contributors
Discussion stats
  • 20 replies
  • 10170 views
  • 0 kudos
  • 4 in conversation
Announcements