× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

NVX, share content missing after vol chk, case 20484647

ahpsi1
Tutor

NVX, share content missing after vol chk, case 20484647

Just filed the issue with NTGR and figured I'd best start documenting now.

System was running and data was accessible however there appeared to be a permissions error on a share I could not correct through Frontview. I thought I'd best work with known sane filesystem so I (stupidly) thought running a FSCK (through Frontview) would be a good place to start. My bad. Rebooted the NVX BE and selected 'Perform volume scan on next boot". After the unit came back up all data in each share appears missing. Volume still shows a non-trivial percentage in use. Logging in with SSH I see the shares but nothing in them. No add-ons in use, only Bonnie installed. Only CIFS, HTTP and RSYNC enabled.

I'm a little worried as this issue (or one that seems somewhat similar) has been seen on the forum before:
http://www.readynas.com/forum/viewtopic.php?f=20&t=68890&p=384524
http://www.readynas.com/forum/viewtopic.php?f=50&t=69247&p=384290
http://www.readynas.com/forum/viewtopic.php?f=20&t=67688&p=379780
http://www.readynas.com/forum/viewtopic.php?f=50&t=67051&p=375481

It appears in some cases the volume can be accessed after being mounted read-only (though I believe mine is already mounted) and in others repair to the volume was enough - while some found their data gone for good. In any case - suggestions?
Message 1 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

I suppose I should also add fs_check.log reports a huge number of inode issues corrected (file size is 282k covering two runs) however I do NOT see any out of memory entries nor anything indicating fsck failed.
Message 2 of 24
JohnnyB11
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

Here too. Everything is lost after a manual triggered file system check on boot after expansion of a an ext3 volume.
It ran twice and I got a huge email (2.2MB) with countless inode issues corrected. Looks like fsck moved everything to lost+found and the volume usage did not drop.
I am factory resetting at the moment. I will test whether this happens again when it is finished.
Message 3 of 24
mdgm-ntgr
NETGEAR Employee Retired

Re: NVX, share content missing after vol chk, case 20484647

A factory reset is a drastic step to take as it wipes the disks of all data. If you do so though you would get a clean setup on the firmware you are running and filesystem improvements that can only be obtained via a factory default. Since 4.2.12 the NVX has had native EXT4 with huge file support (requires factory reset if upgrading from previous firmware).
Message 4 of 24
JohnnyB11
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

Yes. I know.

However, even after the factory default, expansion does not work properly! After factory default with 2x2TB and 2x1TB drives installed and resyncing the initially automatically created 3TB volume, online expansion to 4TB took place, and resyncing again (~10h in total). I rebooted and, then, again but with the manual file system check: There are plenty of errors while there are only 60 files. Some sound very promising like "Inode x has compression flag set on filesystem without compression support."

There are no SMART errors and the drives are IMHO on the HCL: 2xWD20EFRX and 2xWD1002FBYS.
Message 5 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

Just got off the phone with tech support. Spoke with an L2 who told me what I had feared would be the case - I had deleted all my data when I ran FSCK through the volume scan checkbox. I asked if there was any way to recover the data and was told that, although there are no guarantees if I have software support there is a possibility the data can be recovered. Unfortunately I would need to purchase the support contract @ ~ $200.

So, NVX is OK, I run a checkdisk, hose my data and support can't help without a software agreement. I can't complain, I was helped quickly and told the truth as far as I can tell. The Netgear representative was polite and friendly. I had good communication via email and over the phone.

My data is mostly backups of offsite data so the only thing I would have lost would be the time and bandwidth it would take to pull a couple hundred GB over a tiny pipe - EXCEPT for JohnnyB reminding me my data was still safe in lost+found - saving me a ton of work.

Netgear gets points for helping quickly, JohnnyB gets points for actually getting my data back. I suppose I owe him $200?

This situation completely underscores what has been discussed on the forum over the past few months. The forum saved me - without it I would have most probably replaced the NVX tomorrow with another vendors product. Now, I can factory default, restore config, restore data and be on my way. My only fear is I''ll see this again because I do not understand what caused this in the first place.
Message 6 of 24
mdgm-ntgr
NETGEAR Employee Retired

Re: NVX, share content missing after vol chk, case 20484647

In future don't run the volume scan boot option unless support requests you do so and you have an up to date backup.
Message 7 of 24
JohnnyB11
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

ahpsi wrote:
Netgear gets points for helping quickly, JohnnyB gets points for actually getting my data back. I suppose I owe him $200?


No way, because: Please be careful, the files in lost+found might be corrupted!I would calculate some MD5 or SHA1 checksums directly on the offsite backup system and compare them to these calculated on my local files.
Message 8 of 24
JohnnyB11
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

mdgm wrote:
In future don't run the volume scan boot option unless support requests you do so.


Could you elaborate on your statement?
I know that you do not work for Netgear. And I appreciate your posts and blog very much. So, with all due respect: IMHO, there must be something terribly wrong with the NVX if a fsck of a clean file system kills it! How should I trust it anymore if I cannot verify the file system's integrity?
Message 9 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

RSYNC runs one-way with the NVX as the destination. So far all files appear intact but if not the jobs will run tonight and do that for me. Thanks for the heads up though.

As far as not running a volume scan unless tech support tells me to - Netgear should be clear on this - their documentation doesn't agree with that statement -> http://kb.netgear.com/app/answers/detail/a_id/21296/~/how-do-i-shut-down-my-readynas-ultra-series,-u....
Message 10 of 24
JohnnyB11
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

ahpsi wrote:
RSYNC runs one-way with the NVX as the destination. So far all files appear intact but if not the jobs will run tonight and do that for me. Thanks for the heads up though.


Good! However, just to be sure, you are using the "-c" aka "--checksum" option for rsync? Because otherwise it just compares mod-time & size that would not be helpful in this case.
Message 11 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

Thanks again, I actually do not have access to the server that 'pushes' the data down (but I do know it isn't a NAS) so I'll file a request with my organization to have the job modified to calculate checksums as opposed to mod-time or size (if it isn't already).
Message 12 of 24
JohnnyB11
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

JohnnyB wrote:
However, even after the factory default, expansion does not work properly! After factory default with 2x2TB and 2x1TB drives installed and resyncing the initially automatically created 3TB volume, online expansion to 4TB took place, and resyncing again (~10h in total). I rebooted and, then, again but with the manual file system check: There are plenty of errors while there are only 60 files. Some sound very promising like "Inode x has compression flag set on filesystem without compression support."


Update: I rebooted again with a manual file system check and since there were no more errors, I restored the NAS from my backup the last days.
Because I feared that the FrontView online consistency check would force me to do an offline check again, I did a "fsck.ext4 -n" via SSH. And, there are no errors despite those that happen AFAIK if you fsck a mounted file system: wrong free block and inode counts.
Message 13 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

In my case I would note there have been several volume expansions. I believe I started with two 1.5 TB drives, added two 2 TB drives and then replaced all four with 3 TB drives.

Given what you have seen in your testing - is an NVX FS safe if no volume expansions have been performed? Would it be safe after a factory default with four identical drives in place? What i'm trying to get at - is it the volume expansion that corrupts the file system such that the next run of FSCK causes the loss of data or will FSCK cause this regardless of whether or not the filesystem had been expanded? If it's the former it almost sounds like an X-RAID2 issue as I would assume vanilla EXT4 + FSCK should not equal loss of data in an otherwise good environment.

Obviously NTGR would do well to weigh in on this issue.
Message 14 of 24
JohnnyB11
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

ahpsi wrote:
In my case I would note there have been several volume expansions. I believe I started with two 1.5 TB drives, added two 2 TB drives and then replaced all four with 3 TB drives.


I did one expansion two years ago, from 2x1TB to 3x1TB, without any problems.

ahpsi wrote:
Given what you have seen in your testing - is an NVX FS safe if no volume expansions have been performed? Would it be safe after a factory default with four identical drives in place?


It seems to be. The fsck was good on the 4x1TB volume across my 2x2TB and 2x1TB drives initially created by factory default.

ahpsi wrote:
What i'm trying to get at - is it the volume expansion that corrupts the file system such that the next run of FSCK causes the loss of data or will FSCK cause this regardless of whether or not the filesystem had been expanded? If it's the former it almost sounds like an X-RAID2 issue as I would assume vanilla EXT4 + FSCK should not equal loss of data in an otherwise good environment.


IMHO fsck is a very well tested tool. If it says that there are errors, there are errors. Thus, I think the expansion kills the volume, see the thread you linked in your original post http://www.readynas.com/forum/viewtopic.php?t=68890: "After the initial 4T volume was built [...] I decided to force an FSCK on reboot. That one came back clean. After the additional 1TB of space was added to the volume [...] I forced an FSCK on reboot and rebooted the NVX. Now it's come back with errors and the data volume is offline." Same thing happened to me, see my thread http://www.readynas.com/forum/viewtopic.php?t=69366 for details.
Message 15 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

OK, to summarize, filesystem expansion followed by an FSCK may result in the loss of data stored on a ReadyNAS NVX running FW 4.2.22. This has been reported or discussed by four users in the past seven days:
me, Case # 20484647 -> this thread
johnnyb, multiple posts and links to earlier occurrences -> http://www.readynas.com/forum/viewtopic.php?f=66&t=69366&p=385056#p385056
farohk, Case # 20451479 -> http://www.readynas.com/forum/viewtopic.php?f=50&t=69247
horshack, Case # 20304759-> http://www.readynas.com/forum/viewtopic.php?f=20&t=68890
horshack, Case # 19449493 -> http://www.readynas.com/forum/viewtopic.php?f=66&t=66586

While everyone talks about the importance of backups not everyone will have them. If the NVX running the latest RAIDiator is not a safe place to put your data shouldn't there be a warning posted? We have already seen (in my case) that tech support will most probably not be able to help you get your data back, aren't the NVX's sitting out there just time bombs waiting for an expansion and reboot with volume scan checked? Although advised by mdgm not to run FSCK how would the user not reading the forum know this? Not through the online documentation. Not through the RAIDiator release notes . Not through a warning posted on the site.

What is the official recommendation from NTGR?
Message 16 of 24
dcpck
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

It would seem I have become a victim of this "issue" as well. After replacing one of my drives and expanding the volume two days ago, fsck reported problems during last night's disk scan.

Is there a way to at least access the data before doing a factory reset? I'd prefer to attempt to copy anything that isn't corrupt onto a local disk instead of having to recover everything over a very small pipe.

Dave
Message 17 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

Well, I was lucky in that the bulk of my data was 'recovered' to /c/lost+found and I was able to copy it off and then back to the unit without too much hassle. As johnnyb mentioned there is no guarantee the data is not corrupted however mine has been validating 'good' against known good copies (so far).

If you wanted to try this the way I did it you would need the enable root SSH add-on -> http://www.readynas.com/download/addons/x86/4.2/EnableRootSSH_1.0-x86.bin and your choice of file access utilities (I prefer WinSCP -> http://winscp.net/eng/index.php). The usual boilerplate applies (Netgear may deny support, you run the rick of doing something the NAS won't like etc).

Netgear support emailed me today and asked if I had thought about purchasing the $200 service plan. 🙂

Still waiting for a Jedi to comment - this is a rather serious issue and full disclosure early on would greatly improve community relations.
Message 18 of 24
dcpck
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

My /c drive was toast and couldn't be mounted even in read only mode - kept throwing a "sale NFS handle" error for some reason. I gave up and reset it to factory default this morning. I'm currently finishing up a reboot with volume check to make sure it's stable. Fortunately I can recover everything from a second Readynas on the other end of the country, although it's going to take a while.

Going to get a cloud backup system set up to make recoveries faster - Symform or Bitcasa both offer cheap backup for a large data set.

Hopefully Netgear are able to get this sorted out before people who don't maintain a proper backup lose important data.
Message 19 of 24
horshack
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

Make sure you force an FSCK check. I haven't been able to get 4.2.22 to factory reset to a clean volume. If you have the same problem 4.2.19 might work (it did for me)
Message 20 of 24
dcpck
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

I've gone through one expansion, a forced fsck with 4.2.19 and haven't hadn't any problems so far. I'll be doing it twice more with two more drives being added.
Message 21 of 24
horshack
Aspirant

Re: NVX, share content missing after vol chk, case 20484647

Yeah, my NVX is completely functional and clean. I built/expanded completely on 4.2.19 and then upgraded to 4.2.22. In 4.2.22 everything is still working and clean (RAID Scrub, Online FSCK, etc...)
Message 22 of 24
Leia
NETGEAR Employee Retired

Re: NVX, share content missing after vol chk, case 20484647

Hi guys, we upgrade the ext2/ext3/ext4 file system utilities - e2fsprogs in 4.2.23 T build, so it's fixed now. Please do not expand volume in 4.2.22. Just wait 4.2.23.
And the bug is only in NVX, 32 bits system. Ultra & Pro serials, 64 bits system have no problem.
Thanks.
-Leia
Message 23 of 24
ahpsi1
Tutor

Re: NVX, share content missing after vol chk, case 20484647

Nice!

You have my gratitude for addressing this issue. :thumbsup:
Message 24 of 24
Top Contributors
Discussion stats
  • 23 replies
  • 3103 views
  • 0 kudos
  • 6 in conversation
Announcements