This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
ReadyNAS Pro 6 volume inaccessible, shows 100% full
Bear with me--there are a few steps to this story....
My NAS started its life with 6x2TB desktop drives (I know) in XRAID-2, and I have gradually replaced the 2TB drives with 4TB NAS HDDs as they have failed. As of this morning, I had only one 2TB drive left, installed in bay 2. And I suspect it's at the root of my troubles.
As far as I know, everything was working fine(-ish) on my NAS this morning. I was messing w/ Plex, and I saw some of my server content available--i.e., the NAS seemed to be functioning and my data intact. I also discovered that the log was full of warnings about high numbers of failures on Disk 2.
Shortly thereafter I walked into the room with my NAS to find it stuck in some kind of reboot loop. It would boot, start a resync, then power down (?) / reset. Rinse, repeat. I could no longer access it through the web app. Thinking this was some kind of problem with Disk 2, I powered down the NAS and swapped that disk with my last spare 4TB drive. Unfortunately, that also resulted in a restart shortly after trying to resync.
Next, I removed Disk 2 entirely. Although I could now boot the NAS and view it through the web interface. the volume shows as 100% full (which it's not, or not supposed to be--it should be at ~51% of 7.37TB or so). The data volume is inaccesible to the NAS and none of my shares are visible. There was a log message about a "difficult to resolve error."
I used the boot menu to run a memory check, which passed, and a disk check, which is still running after ~8 hours or so. I have not tried an OS reset, but wonder if that might be a next step.
I do NOT have a good recent backup and stand to lose a great deal of data, so pretty nervous about what to do. I do still have the old flaky 2TB drive and the new 4TB drive (which is still out of the system for the disk check). Really hoping this is some kind of configuration snafu that can be cleared up.
Any ideas? Potential diagnoses based on symptoms? Suggestions for what to do (or what not to do) next? Is an OS reset a good or terrible idea?
Since you have the log zip file, I suggesting sending a private message to one of the mods ( @JohnCM_S or @Marc_V ), and ask them to analyze them. They can likely give you some idea on what's going on, and if data recovery is needed.
Put the zip file into cloud storage (dropbox, google drive, etc), and include a downloadable link to the log zip. You send a PM using the envelope icon on the upper right of the forum page. Please don't post the download link publicly.
Re: ReadyNAS Pro 6 volume inaccessible, shows 100% full
I wouldn't do the OS reinstall just yet. If the OS partition is also full, then that could make things worse.
What firmware is your NAS running?
Is the Plex server running on the NAS itself?
Is ssh enabled on the NAS?
What do you mean when you say "the data volume is inaccessible to the NAS"? That sounds inconsistent with saying that the NAS is showing the volume as 100% full.
Re: ReadyNAS Pro 6 volume inaccessible, shows 100% full
Thanks so much for the reply! First, a quick status update:
The disk scan completed overnight. I do not see signs of any (new) errors. Interestingly, Frontview reports Disk 2 as "dead," even though it is physically missing (removed). Otherwise, the Frontview logs show the 3 same messages I got before when the NAS came up:
1. Volume scan found errors that it could not easily correct. Please ensure that you have current backups of all valuable data before performing a full volume scan by rebooting the NAS through Frontview with the volume scan option enabled.
2. The paths for the shares listed below could not be found. Typically, this occurs when the ReadyNAS is unable to access the data volume....
3. System is up.
I am able to download full logs, but not sure yet what to look for that might be revealing. I am scanning them now, will report anything that stands out.
--
To answer your questions:
The NAS is running RAIDiator 4.2.31.
Yes, the Plex server is running directly on the NAS, using locally stored media. This is the standard plug-in version of Plex installed through Frontview. It's very old, almost never used, and I don't need the server itself for anything--just the media.
I do not believe SSH is enabled. I previously tried to log in via SSH (Cygwin), but was unable to do so. I also do not see the SSH plugin listed anywhere by FrontView. I do see it available for downloand and could try installing it--the fact that the NAS comes up and is responsive to Frontview seems encouraging.
The comment about "100% full volume" comes from Frontview when looking at Volume Settings. It reports "7334 GB (100%) of 7334 GB used." The comment about it being inaccessible comes from the Frontview status log message #2 above. Actual data usage is ~51%. The 100% message is clearly in error or could be indicative of the problem.
--
I'll hold off on OS reinstall. Should I attempt to install the SSH plugin? In the meantime, I'll be looking through the logs (preliminary skim seems to show disk scan passed).
Since you have the log zip file, I suggesting sending a private message to one of the mods ( @JohnCM_S or @Marc_V ), and ask them to analyze them. They can likely give you some idea on what's going on, and if data recovery is needed.
Put the zip file into cloud storage (dropbox, google drive, etc), and include a downloadable link to the log zip. You send a PM using the envelope icon on the upper right of the forum page. Please don't post the download link publicly.
Re: ReadyNAS Pro 6 volume inaccessible, shows 100% full
Will do, thanks. I did run across this is a "fs_check" log that has me feeling a bit deflated:
Block #538969100 (502146803) causes directory to be too big. CLEARED. Illegal block #538969101 (3823912954) in inode 18750446. CLEARED. Error storing directory block information (inode=18750446, block=0, num=93062815): Memory allocation failed
/dev/c/c: ***** FILE SYSTEM WAS MODIFIED ***** e2fsck: aborted
/dev/c/c: ***** FILE SYSTEM WAS MODIFIED *****
***** File system check performed at Sat Nov 23 23:41:28 MST 2019 ***** fsck 1.42.12 (29-Aug-2014) /dev/c/c: Note: if several inode or block bitmap blocks or part of the inode table require relocation, you may wish to try running e2fsck with the '-b 32768' option first. The problem may lie only with the primary block group descriptors, and the backup block group descriptors may be OK.
/dev/c/c: Block bitmap for group 960 is not in group. (block 2101739520)
/dev/c/c: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options)
--
The "memory allocation failed" message is worrisome, as are the fairly long string of notes about modifications. I didn't realize a file system check had been run, but it looks fairly descructive. 😞
I'll get the full log set online and PM the link. And start combing through the backups I do have to see what I may have permanently lost.
Re: ReadyNAS Pro 6 volume inaccessible, shows 100% full
Coming back to report on the fallout, since it's frustrating to search for old forum discussions and then guess how things ended up.
First, the volume was indeed corrupted. It's still not clear how that happened, but I'm guessing that drive errors may have triggered the system reboot, and the reboot may have occurred in the middle of a sensitive operation. Pulling the drive probably did not help (though I couldn't reach a stable boot without doing so), and whatever triggered the fscheck really did not help, since that attempted to move data around. In any case, the web interface reporting 100% volume usage was a clear red flag, and fscheck logs essentially confirmed file system corruption.
Now, the good news: not touching the NAS once things looked weird apparently limited the damage done. As recommended here I paid for a per-incident consultation, which was enough to not only confirm the file system corruption, but also to perform extensive (basic) recovery. I had to purchase a sufficiently large USB external drive (Seagate 8TB USB3 backup "hub" worked great), then with the aid of Netgear support we dumped recovered files to that drive.
Upon performing an inventory of the recovered data I confirmed a very small amount of data loss (3 folders worth of music among thousands) vs a recent backup. I have no doubt that mucking with the NAS at all could have turned that into extensive, permanent loss.
It's still possible that I may have permanently lost some important files, but I have recovered so much that anything missing is not immediately obvious. All in all, that's a fair trade for a valuable life lesson--have and implement a better backup plan.
Thanks to everyone who quickly responded to my inquiry here regarding my now very old NAS, and thanks to Netgear support for salvaging my data!
