NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
NVX, share content missing after vol chk, case 20484647
Just filed the issue with NTGR and figured I'd best start documenting now.
System was running and data was accessible however there appeared to be a permissions error on a share I could not correct through Frontview. I thought I'd best work with known sane filesystem so I (stupidly) thought running a FSCK (through Frontview) would be a good place to start. My bad. Rebooted the NVX BE and selected 'Perform volume scan on next boot". After the unit came back up all data in each share appears missing. Volume still shows a non-trivial percentage in use. Logging in with SSH I see the shares but nothing in them. No add-ons in use, only Bonnie installed. Only CIFS, HTTP and RSYNC enabled.
I'm a little worried as this issue (or one that seems somewhat similar) has been seen on the forum before:
http://www.readynas.com/forum/viewtopic.php?f=20&t=68890&p=384524
http://www.readynas.com/forum/viewtopic.php?f=50&t=69247&p=384290
http://www.readynas.com/forum/viewtopic.php?f=20&t=67688&p=379780
http://www.readynas.com/forum/viewtopic.php?f=50&t=67051&p=375481
It appears in some cases the volume can be accessed after being mounted read-only (though I believe mine is already mounted) and in others repair to the volume was enough - while some found their data gone for good. In any case - suggestions?
System was running and data was accessible however there appeared to be a permissions error on a share I could not correct through Frontview. I thought I'd best work with known sane filesystem so I (stupidly) thought running a FSCK (through Frontview) would be a good place to start. My bad. Rebooted the NVX BE and selected 'Perform volume scan on next boot". After the unit came back up all data in each share appears missing. Volume still shows a non-trivial percentage in use. Logging in with SSH I see the shares but nothing in them. No add-ons in use, only Bonnie installed. Only CIFS, HTTP and RSYNC enabled.
I'm a little worried as this issue (or one that seems somewhat similar) has been seen on the forum before:
http://www.readynas.com/forum/viewtopic.php?f=20&t=68890&p=384524
http://www.readynas.com/forum/viewtopic.php?f=50&t=69247&p=384290
http://www.readynas.com/forum/viewtopic.php?f=20&t=67688&p=379780
http://www.readynas.com/forum/viewtopic.php?f=50&t=67051&p=375481
It appears in some cases the volume can be accessed after being mounted read-only (though I believe mine is already mounted) and in others repair to the volume was enough - while some found their data gone for good. In any case - suggestions?
23 Replies
Replies have been turned off for this discussion
- Thanks again, I actually do not have access to the server that 'pushes' the data down (but I do know it isn't a NAS) so I'll file a request with my organization to have the job modified to calculate checksums as opposed to mod-time or size (if it isn't already).
JohnnyB wrote: However, even after the factory default, expansion does not work properly! After factory default with 2x2TB and 2x1TB drives installed and resyncing the initially automatically created 3TB volume, online expansion to 4TB took place, and resyncing again (~10h in total). I rebooted and, then, again but with the manual file system check: There are plenty of errors while there are only 60 files. Some sound very promising like "Inode x has compression flag set on filesystem without compression support."
Update: I rebooted again with a manual file system check and since there were no more errors, I restored the NAS from my backup the last days.
Because I feared that the FrontView online consistency check would force me to do an offline check again, I did a "fsck.ext4 -n" via SSH. And, there are no errors despite those that happen AFAIK if you fsck a mounted file system: wrong free block and inode counts.- In my case I would note there have been several volume expansions. I believe I started with two 1.5 TB drives, added two 2 TB drives and then replaced all four with 3 TB drives.
Given what you have seen in your testing - is an NVX FS safe if no volume expansions have been performed? Would it be safe after a factory default with four identical drives in place? What i'm trying to get at - is it the volume expansion that corrupts the file system such that the next run of FSCK causes the loss of data or will FSCK cause this regardless of whether or not the filesystem had been expanded? If it's the former it almost sounds like an X-RAID2 issue as I would assume vanilla EXT4 + FSCK should not equal loss of data in an otherwise good environment.
Obviously NTGR would do well to weigh in on this issue. ahpsi wrote: In my case I would note there have been several volume expansions. I believe I started with two 1.5 TB drives, added two 2 TB drives and then replaced all four with 3 TB drives.
I did one expansion two years ago, from 2x1TB to 3x1TB, without any problems.ahpsi wrote: Given what you have seen in your testing - is an NVX FS safe if no volume expansions have been performed? Would it be safe after a factory default with four identical drives in place?
It seems to be. The fsck was good on the 4x1TB volume across my 2x2TB and 2x1TB drives initially created by factory default.ahpsi wrote: What i'm trying to get at - is it the volume expansion that corrupts the file system such that the next run of FSCK causes the loss of data or will FSCK cause this regardless of whether or not the filesystem had been expanded? If it's the former it almost sounds like an X-RAID2 issue as I would assume vanilla EXT4 + FSCK should not equal loss of data in an otherwise good environment.
IMHO fsck is a very well tested tool. If it says that there are errors, there are errors. Thus, I think the expansion kills the volume, see the thread you linked in your original post http://www.readynas.com/forum/viewtopic.php?t=68890: "After the initial 4T volume was built [...] I decided to force an FSCK on reboot. That one came back clean. After the additional 1TB of space was added to the volume [...] I forced an FSCK on reboot and rebooted the NVX. Now it's come back with errors and the data volume is offline." Same thing happened to me, see my thread http://www.readynas.com/forum/viewtopic.php?t=69366 for details.- OK, to summarize, filesystem expansion followed by an FSCK may result in the loss of data stored on a ReadyNAS NVX running FW 4.2.22. This has been reported or discussed by four users in the past seven days:
me, Case # 20484647 -> this thread
johnnyb, multiple posts and links to earlier occurrences -> http://www.readynas.com/forum/viewtopic.php?f=66&t=69366&p=385056#p385056
farohk, Case # 20451479 -> http://www.readynas.com/forum/viewtopic.php?f=50&t=69247
horshack, Case # 20304759-> http://www.readynas.com/forum/viewtopic.php?f=20&t=68890
horshack, Case # 19449493 -> http://www.readynas.com/forum/viewtopic.php?f=66&t=66586
While everyone talks about the importance of backups not everyone will have them. If the NVX running the latest RAIDiator is not a safe place to put your data shouldn't there be a warning posted? We have already seen (in my case) that tech support will most probably not be able to help you get your data back, aren't the NVX's sitting out there just time bombs waiting for an expansion and reboot with volume scan checked? Although advised by mdgm not to run FSCK how would the user not reading the forum know this? Not through the online documentation. Not through the RAIDiator release notes . Not through a warning posted on the site.
What is the official recommendation from NTGR? - It would seem I have become a victim of this "issue" as well. After replacing one of my drives and expanding the volume two days ago, fsck reported problems during last night's disk scan.
Is there a way to at least access the data before doing a factory reset? I'd prefer to attempt to copy anything that isn't corrupt onto a local disk instead of having to recover everything over a very small pipe.
Dave - Well, I was lucky in that the bulk of my data was 'recovered' to /c/lost+found and I was able to copy it off and then back to the unit without too much hassle. As johnnyb mentioned there is no guarantee the data is not corrupted however mine has been validating 'good' against known good copies (so far).
If you wanted to try this the way I did it you would need the enable root SSH add-on -> http://www.readynas.com/download/addons/x86/4.2/EnableRootSSH_1.0-x86.bin and your choice of file access utilities (I prefer WinSCP -> http://winscp.net/eng/index.php). The usual boilerplate applies (Netgear may deny support, you run the rick of doing something the NAS won't like etc).
Netgear support emailed me today and asked if I had thought about purchasing the $200 service plan. :)
Still waiting for a Jedi to comment - this is a rather serious issue and full disclosure early on would greatly improve community relations. - My /c drive was toast and couldn't be mounted even in read only mode - kept throwing a "sale NFS handle" error for some reason. I gave up and reset it to factory default this morning. I'm currently finishing up a reboot with volume check to make sure it's stable. Fortunately I can recover everything from a second Readynas on the other end of the country, although it's going to take a while.
Going to get a cloud backup system set up to make recoveries faster - Symform or Bitcasa both offer cheap backup for a large data set.
Hopefully Netgear are able to get this sorted out before people who don't maintain a proper backup lose important data. - Make sure you force an FSCK check. I haven't been able to get 4.2.22 to factory reset to a clean volume. If you have the same problem 4.2.19 might work (it did for me)
- I've gone through one expansion, a forced fsck with 4.2.19 and haven't hadn't any problems so far. I'll be doing it twice more with two more drives being added.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
