ReadyNAS 314 - a year of hell and now lost 10years of data!

Reasonable comments.

However, I bought the camera and disks under the guidance of the company who manufacture the device that they were to supposed to work with. There's no one else better positioned to advise is there?
My gripe with this is that once I discovered the device was pretty much useless for my requirements for the first 6 months, through no fault of my own, I was stuck with waiting and trying to get it to work because of the additional items I had bought to try to make it work wouldn't necessarily work with another (probably much better) solution.

I understand disks do fail (but considering I have over 40 discs running at work and not one of them has failed *finds some wood to touch* it seems strange that this ReadyNas has destroyed 2 in under a year>
My device was used as large file storage, written to more often than read from, and maybe only really active once a day - Not some highly intensive read/write application or SQL server (like at my work).

I think it is the Surveillance add-on that is overworking them as this records my camera 24/7 - If I turn this add-on off, the disk errors stopp completely! Not a single one in a whole week that I tested it. As soon as it was turned back on again - disk errors on all 4 disks!

So considering the device was (and still will) throw false positives about any disks put in it suggesting they need replacing, how is an end-user supposed to identify when one actually DOES need replacing?
If I had gone with the information provided by the device I would be replacing all four disks at least ten times a day - Not going to happen, stupid idea, and would soon use up any savings I have!
If I go with the Netgear support's advice to not worry about them, then I end up with 2 disks failed and a loss of my data!
I don't see who else is qualified to advise (without the end result being Netgear saying, we didn't give you that advise so tough!)
It's a no win situation...

Easy for all parties involved to say, not our fault, these things happen?

As a home consumer, I cannot justify spending twice as much as I already had on having a 2nd ReadyNas to back up to (or something else that can store approx. 7TB of data?)

I will get a couple of new disks and see if I can scrape anything off the 1 bad disk that still responds.
Thank you for your advice on dd_rescue. Already downloaded and BootCD just booted so I understand how to work it.

Regarding to the camera, I cannot argue. If it does not work, you should get back to Netgear, they cannot advertise something that does not work.

Re. drives: they do fail. If your box of HDDs was dropped on the floor by the courier :) they would fail within months. No surprise. Your NAS hasn't broken them. It happens. Have them replaced.
Now, you say the device is giving you false warning: have you checked the disks on a PC, with the seagate tool? If the disks are fine, I would ask Netgear to replace the box or, if the issue is a software one, to exchange it with another one which does not have this problem. Or a refund. That sounds reasonable to me.

How broken are the disks? Two dropped disks from an array is no problem, you can restore data from that, as long as the disks aren't physically broken, meaning you can actually see the broken disk and read some data from it. Disks can be dropped for lots of reasons.

The bigger problem is of course that BTRFS is the file system on the array. Few recovery tools support BTRFS. But I would try assembling the existing disks in an array in a Linux box, or hire a nerd to do it for you. :)

I would be interested to know how you restore data from a raid 5 array with 2 dead disks. simplified way on how Raid 5 works is it stores data like this 1+2+3=6 and recovers it doing 1+X+3=6 so 6-(3+1)=X=2.
Are you counting on the fact that the whole disk is not dead and that you can get back part of the data and hope that the two dead only partially failed so that you may recover one number of that sum and rebuild part of the raid using the two dead disk but recovering some data because they didn't fail on blocks concerning the same data ?

As for the initial problem, I assume that everybody read the Netgear Manual (I personally read only what interested me, not a good example for that ;) ) and I assume it clearly states that the device could fail and that the data wouldn't be recoverable that this is up to you to provide sufficient backup capabilities. So for the backup, your fault, your responsibility.
For the camera, you had some time to send back the product that didn't meet the requirements (Two weeks in Europe at least). But I agree this should have worked out of the box.

IF (and I stress the if) you used the software of the camera that was provided by Netgear and you had drives certified for that particular use (otherwise Netgear will say your disks weren't able to handle it) in a NAS AND certified by Netgear for the NAS, then Netgear ought to have provided you with adequate support and they didn't if they said to ignore the warning. They should have updated the software promptly to avoid those warnings. They also should have warned you that they may be false positives or not and to take action accordingly. They also should have told you to send the disks back to check if this was done with new ones or at least to check your disk status with an utility. If disks had come clean, this would have let them with only two options : their software was failing, their hardware was failing. Either case you should have had a decent solution from Netgear.
Now the problem is that we don't have a full log of what was said and done and we can't judge it (you noticed that what I said earlier is conditional). I know that when I call tech support I am a pain in the ass like some people would say, I don't give up until I think the support provided me the best advices they could (and I do check everything they say and try to find a solution before calling), I always stay polite though but have no fear to escalate the claim if I deem it necessary. This means that when someone tells me "you got a warning but that's nothing", I reply "then why is there a warning ?" and if they don't answer I ask to be put in relation with someone who can explain. That said I would also have tried to do the disk replacing for a day or two just to get more intel for the tech support to work on, but I would definitely have refused that as a solving solution either and would have continued to call tech support once or twice a week till solved (automated mail if needed) hoping to get someone more qualified to answer my claim.

My understanding (from a PM) is that one disk is totally dead, and that the second one may have some read errors. I suggested cloning the second disk.

It's hard to understand why Netgear gave the advice they did without seeing the case notes.

My general view is that protecting my data is my responsibility, not theirs, and that it is not protected if there is only a single copy (with or without RAID). That said, it is regrettable that the data is possibly lost, and I hope the cloning (or other intervention) gets it back.

xeltros wrote:

I would be interested to know how you restore data from a raid 5 array with 2 dead disks. simplified way on how Raid 5 works is it stores data like this 1+2+3=6 and recovers it doing 1+X+3=6 so 6-(3+1)=X=2. Are you counting on the fact that the whole disk is not dead and that you can get back part of the data and hope that the two dead only partially failed so that you may recover one number of that sum and rebuild part of the raid using the two dead disk but recovering some data because they didn't fail on blocks concerning the same data ?

Yes. A dropped raid doesn't necessarily mean that the data is done for, or even corrupt. It only means that the RAID array controller cannot guarantee data integrity.

Now if two disks are physically dead -- let's say ground to pieces -- in a single redundancy system (RAID5) data is indeed impossible to get back. But the typical "disk fail" error is a read or write error, and that can often be reconstructed. How it's handled depends on how the array controller (including software raid) handles a broken array, but even there, you can use commercial RAID reconstruction tools to get data back.

I have personally reconstructed two broken RAID5 arrays with two "dropped" disks each, one on a QNAP nas and one on a Dell PERC6 (LSI) hardware RAID, and these undertakings have made me view RAID as something much less magical and much more mundane than I thought it was before. :)

StephenB wrote:
My understanding (from a PM) is that one disk is totally dead, and that the second one may have some read errors. I suggested cloning the second disk.

This is excellent advice, if it's possible to do.

And in this case: it's possible (theoretically) to reconstruct such an array, because RAID5 can handle the one completely broken disk. For the disk with read errors, so what, so some stripes of data are lost. The tricky part is finding a "controller" that doesn't say "OMG a broken stripe set I MUST DROP THIS DISK FROM THE ARRAY IMMEDIATELY, OH NOES ME DED NAO". And essentially this is what RAID reconstruction software does, it reads the "data chunks" from the disks as is, combining them, accepting all read errors, rereading broken sectors multiple times and carrying on anyway. For working online RAID sets time and "write integrity" are critical factors, you can't hang the array to make multi-hour recovery operations and you must drop the array to make sure nothing is written to a faulty array, but recovery software have all the time in the world and won't make any writes.

In this non-theoretical case it's much harder, of course, because of the legendarily stupid decision to use BTRFS in the new products. One of my main arguments against BTRFS was always that no software supports unfinished beta file systems, a fact which unfortunately now will bite OP in the behind. The right approach here would probably be to try reconstructing the array and read it out as a disk image, storing it on another storage area of equal size, and then mounting that giant image as BTRFS. It's icky as this method doesn't support recovery of single files until the very last step, and I don't know how BTRFS will handle multiple blocks of randomly corrupt data (the stripe sets with read errors). Even then, if the resulting image can't even be mounted as a file system, recognizable unfragmented data could perhaps be restored anyway, using data recover tools that looks for known file headers (testdisk/PhotoRec could work directly with that image without even looking at the file system).

But my shortcut approach to the problem as a whole until I tried the procedure above would be:
0) Check that the problem is not due to usage of known faulty drives. Some models are simply bad. If that is the case, I would clone ALL the disks in next step, weighing the cost for all this against the value of my data.
1) Clone the working faulty disk; this will give you an image with read errors.
2) Give that faulty disk a workout with non-destructive reconstruction software such as Spinrite (I know of the hyperbole and crap the author spouts about this software but it's good for some veeerrry specific cases, such as this). If you are very very VERY lucky, data will be recovered to spare sectors and then you can force-mount the original array in Linux.

If this doesn't work, try recovering the array with the image-approach above.

Don't forget marking the disks when you take them out of the unit.

All this will take several days of work, maybe weeks.

I missed this part:

AndyBee1 wrote:
and a 3rd party want best part of £1000 to recover it for me!

Pal, that's quite seriously a *very* *good* price. I'm not sure that party will actually honor this offer (try telling them that the disks are BTRFS on top of MDRAID, meaning, not native BTRFS RAID [which they would certainly not support], and see if they still are interested in doing it) but I can say that after earlier experiences, I couldn't be bought that cheaply. Never ever. The last reconstruction I made (the PERC6) cost the customer three~four times that much, sure that included getting a working system up and rebuilding the storage area with new disks as a RAID6, but anyways.

This is expert work. Be prepared to pay accordingly, or learn to do it yourself.

In other word you do what I did many times when a CD didn't play well anymore, you use a player that is less picky and you burn it again to a new CD, except that here you use two bad CD to make one good.

Raid is under the BTRFS level so it doesn't matter if they can rebuild the raid and clone it to 4 good disks that would mount on a linux box it could do the trick without any problem. BTRFS is a mainstream filesystem now.That said if they try to recover at filesystem level like easy recovery pro and that kind of stuff there may be some problems but that's not something that costs 1000£, they ought to do better than that.
1000£ for 16Tb I agree is not that much but still it depends of the data and context, for a consumer it's a great deal, for an enterprise that's nearly nothing. That said 4 or 5 years ago I heard someone say that recovery in white room (translated form french so not sure of the word but a sterilised room where they tear the disks appart to read data) it was 100€/Gb.
Rebuilding a ray6 array with new disks and installing the system is merely 1 hour work (and many hours waiting), that's worth less than 200€, most will do it for 100€. I believe your price was already friendly. I think I read somewhere than 2 out of 3 enterprises that lost their data closed within two years, even 10K€ that's cheap in that case. That said for a normal person that just not possible to pay that much.

Yes, I understand that if I had the time and willpower to continuously be on the phone to Netgear for a product that should have worked out of the box I may have got a solution earlier, and maybe even averted this disaster - but as with everything I wish to do urgently, life gets in the way as I have a family and a job.
The camera could not be returned, as it took Netgear 6 months to provide me with software that showed it didn't really work. Can you imagine a 3rd party (Panasonic) saying "Yeah no problem that Netgear didn't actually make the software for the product you bought, it's only been 6 months, just remove the camera from it's permanent location on your outside wall and take the waterproofing off it - we'll happily refund you"?
Basically I was stuck in the middle from day 1 with this, and it has been a headache ever since it arrived.
I know I won't get a refund from Netgear, I also will never get a refund for the camera, the 4 discs which apparently were pointless for the application, or for the tens of hours of my time trying pointless things and reinstalling non-working software to test Netgears virgin product!
I'm just going to keep airing my gripes though, so other people who may find this forum and are in the same position as me (not complete tech wizards who have to rely on the information they get form the manufacturers and their support departments) can make an educated decision to look at other options.

mangrove & StephenB - I will try the approach you suggested.
I may be back on for some advice later on in the week when the new disks arrive (bought 2 the same for now just to try to rebuild the array)
Maybe I will then but an Enterprise disk for the camera part of the setup (if it ever works!!) and just have 3 disks for my builk storage needs.
You are right in what you say, if this was my companies data they'd pay big money to recover it, but as I am just storing stuff that would be a pain in the a$$ to get again, and maybe lost, I don't have the budget to spend £1000 on recovering it.