NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
Disk fail in X-RAID2, after sync half my files are gone!
Hi, I'm on a ReadyNAS NV+ v2, with RAIDiator 5.3.11.
I had it configured for X-RAID2 with 3x 2GB drives and one older 512GB drive. A few days ago the 512GB drive failed, leaving the array unprotected. No big deal - I stuck a 2GB drive in instead of the failed drive, and left it to sync up.
It did leave me with the ominous message:
RAID sync finished on volume C. The array is still in degraded mode, however. This can be caused by a disk sync failure or failed disks in a multi-parity disk array.
But I figured that wasn't too big of a deal - all the hard disks were still lit up green in raidiator and everything else seemed ok. However this morning when I looked half my files were gone! No warning emails or anything.
Before the sync, all my files were there, as I checked and managed to back a few of them up. Looking at the logs I now see pages of:
Apr 23 04:22:13 nv kernel: EXT4-fs error (device dm-0): ext4_find_entry: reading directory #3969027 offset 0
Apr 23 04:22:13 nv kernel: EXT4-fs error (device dm-0): __ext4_get_inode_loc: unable to read inode block - inode=3048192, block=48758863
Apr 23 04:22:13 nv kernel: EXT4-fs error (device dm-0): __ext4_get_inode_loc: unable to read inode block - inode=1051475, block=16777429
Apr 23 04:22:13 nv kernel: EXT4-fs error (device dm-0): __ext4_get_inode_loc: unable to read inode block - inode=1057490, block=16777805
Apr 23 04:22:13 nv kernel: EXT4-fs error (device dm-0): __ext4_get_inode_loc: unable to read inode block - inode=1057491, block=16777805
in system.log.
What can I do? Is there a way to recover these files? I thought this was the whole point of having a NAS.
rabidh wrote:
It seems particularly unlucky that the replacement drive I put in was faulty. Having just read into it a bit, I wasn't aware that in most RAID systems if one copy of the data becomes corrupt then even though it is duplicated ...
In your case your NAS is using RAID-5. RAID-5 doesn't duplicate your data. Rather it uses parity blocks that allow it to reconstruct data when something is missing.
Putting this in mathematical terms: Imagine a 4-disk RAID-5 array. If disks 1,2, and 3 have A, B, and C data blocks at sector N, then the fourth disk would have P=A+B+C in that sector. (It doesn't use normal addition, but does something else that has the same effect). Then if the disk 3 is replaced, the NAS reconstructs C using P-A-B.
This only works if the remaining disks can all be read (and when all have the correct data). If a disk can't be read duiring reconstruction, then the reconstruction fails (and the NAS knows that). If a disk is read, but gives the wrong data, then the reconstruction gives the wrong result (and the NAS has no way to detect that). Similarly, if the wrong data was somehow written to one of the disks in the first place (or if a disk write was lost), then the reconstruction will fail (and there is no way to detect that).
rabidh wrote:
it'll probably still cause corruption ... and probably the more high end systems have options in place to work around that.
Once corruption happens, then there is risk of data loss - that's just as true in high-end enterprise/cloud systems as it is in home NAS.
High-end systems have some features which can reduce the chance of corruption happening in the first place. For instance
- Error-correcting RAM
- Dual Power Supplies to help ensure that a PSU failure doesn't result in lost writes.
- UPS protection
BTW, UPS protection is something I always recommend (for all NAS). Often data corruption occurs with unexpected power loss.
Also if you have more disks in the NAS, there are some advanced RAID modes that can handle more than one failed disk. There is a price for that (both reduction in capacity and lower performance). And they don't help if the wrong data is on one or more disks. They only help if the disk can't be read.
Newer OS-6 ReadyNAS (at all price points) do have some features that are relevant here. They have more scheduled maintenance functions, that can detect issues sooner. They also use a newer file system called BTRFS, which supports built-in checksums that can detect corruption. That also gives those NAS some more sophisticated options for reconstruction.
But for all storage (enterprise and home) the primary defense against data loss/corruption is to have independent backups - full copies of the data on other devices,
17 Replies
Replies have been turned off for this discussion
RAID makes data loss less likely, and it simplifies expansion of storage (without loss of availability).
However, it is not enough to keep your data safe. For that you need backups on other devices.
You are facing data recovery (either using RAID recovery software like ReclaiMe, or using a data recovery service like Netgear's).
But perhaps start by looking at the SMART stats of the remaining disks, and checking for disk errors (reallocated sectors, etc).
It looks like there were some smart errors on the disk that I put in as a repacement, so I pulled that, restarted with a filesystem check, and some of my files are back (not all). Is there anything else I can do short of sending the nas off, or pulling drives and spending $200 on ReclaiMe like you suggested?
Luckily I do have some of the more important files backed up elsewhere because I lost some confidence in my ReadyNAS a while back, but I have still lost data that I needed.
Is it Netgear's official position that you can lose data each time you swap a drive on a ReadyNAS? If so it seems like I should probably reconsider my storage choices.
rabidh wrote:
Is it Netgear's official position that you can lose data each time you swap a drive on a ReadyNAS? If so it seems like I should probably reconsider my storage choices.
Netgear's OS 6 software manual (page 231 of http://www.downloads.netgear.com/files/GDC/READYNAS-100/READYNAS_OS_6_SM_EN.pdf) says
If your data is important enough to store, it is important enough to back up. Data can be lost due to a number of events, including natural disaster (for example, fire or flood), theft, improper data deletion, and hard drive failure. If you regularly back up your data, you can recover your data if any of these situations occur.
If someone else tells you something different, and you actually believe them ... then I have a bridge to sell you :smileywink:
Data that is not backed up is always at risk. And that risk is higher when the RAID array is being resynced due to a disk replacement.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
