NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Langzik_NAS
Jul 27, 2018Guide
ReadyNAS couldn't update Antivirus, then System volume 'root' usage was 85 %, now I can't boot...
Hi all,
I have 2 identical ReadyNAS 516 systems (with only different storage capacities and content); one is working fine, and the other one seems to have gone crazy. I am 99% certain that BOTH...
Langzik_NAS
Jul 27, 2018Guide
Thanks for the heads up Stephen.
1. I'll mentally shelve that OS reinstall option then! (lol) :smileyhappy:
2. It looks like I'm not covered by those criterion (I bought my NAS in Feb of 2014, and the second one in Dec of 2017); story of my life, I always fall between these cracks (lol). I'm super strapped for cash, so I likely won't be able to afford extra support at the moment unfortunately (unless a miracle happens). :smileyindifferent:
3. I'm basically a newb when it comes to Linux; I've tinkered with BASH in a course, but I definitely consider myself far from proficiently competent. Also, contextually speaking, I'm not feeling particularly confident given that this system's seemingly self-destructing a little bit more every time I make any move at all. Having said that, [depending on the level of complexity] I'm willing to try the command-line approach if there's no other option (only because I'm desperate), but my main concern is data loss, given that we're talking about almost 18TB, which would definitely feel like a crushing catastrophy to lose... I'd probably have a very hard time even looking at the unit for a few months if I were to lose that data [to be honest]. :smileyfrustrated:
StephenB
Jul 28, 2018Guru - Experienced User
Cleaning the root partition does require some linux skills, since you can do more damage if you make a mistake. Also, the procedure isn't cook-book - you need to first determine where the space is going, and the remediation depends on that analysis. Sometimes the full OS partition results in some corruption to the configuration files on the NAS, and in those cases they need to be manually repaired.
Someone from Netgear here might offer to help, otherwise I suggest powering down the NAS for now, and engaging paid support when you can afford it.
Longer term, you should put a backup plan in place since RAID isn't enough to keep your data safe. (I realize that costs, but unfortunately there is no other way to protect your data from loss).
- Langzik_NASJul 30, 2018Guide
Thanks again for your insight Stephen;
For the sake of clarification, my interpretation of what you're saying is that there isn't a clear "one-size-fits-all" solution, and thus the remedy requires a contextual finess of sorts, otherwise I would likely do more harm than good.
I'll power it down for now (after a reboot yesterday, it's stuck at 99% today), and brush up on my Linux skills in the event that my only opportunity to fix this requires me to take that risk. Given that I can't fully boot, how would I even get the chance to interact with my unit? Physically connect a monitor/TV via HDMI and plug a keyboard directly into the back? And would I need to boot a portable version of Linux from a usb to navigate through the ReadyNAS, or is Linux itself booted/available with the ReadyNAS OS being a separate layer/wrapper of sorts? :manfrustrated:
&& you're 100% right about having a backup solution of course; my situation is entirely circumstancial, it's never been my intention to live vicariously (illness wiped out 'my life', and it's been a super long and ongoing road to recovery). By backup plan, I'm assuming you mean either an offsite [online] backup solution or a separate file system specifically set up for file redundancy (like one NAS periodically making a differential backup of the other).
Thanks again for your help Stephen. :manhappy:
[I thought I hit send 3 days ago, apparently I didn't; my NAS has been turned off since I read your feedback]- StephenBJul 31, 2018Guru - Experienced User
Langzik_NAS wrote:
I'll power it down for now (after a reboot yesterday, it's stuck at 99% today), and brush up on my Linux skills in the event that my only opportunity to fix this requires me to take that risk. Given that I can't fully boot, how would I even get the chance to interact with my unit? Physically connect a monitor/TV via HDMI and plug a keyboard directly into the back? And would I need to boot a portable version of Linux from a usb to navigate through the ReadyNAS, or is Linux itself booted/available with the ReadyNAS OS being a separate layer/wrapper of sorts? :manfrustrated:
Normally you'd boot up the NAS in tech support mode, and access it with ssh/putty. Then you can manually mount the OS partition on the disks and examine it - truncating any oversize logs for instance.
So you don't need to use the console output (though that can also be done).
Langzik_NAS wrote:
By backup plan, I'm assuming you mean either an offsite [online] backup solution or a separate file system specifically set up for file redundancy (like one NAS periodically making a differential backup of the other).
Yes. Or USB drives if you like.
Personally I do back up my NAS to other NAS, and also use Crashplan for offsite/disaster recovery. I run Crashplan on a PC (mapping the NAS data volume to a PC drive letter).
- Langzik_NASAug 20, 2018Guide
Nice; sounds like a sweet setup StephenB [SUPERUSER Guru StephenB]. Ultimately my endeavour will be to have a rack-mounter server; I don't think a PC will be enough to give me what I'm looking for [given my insatiable appetite for data storage], but then... $$$. :smileyhappy:
Anyway, now that I've briefly brushed up on some BASH [command-line] study material; let's start from the beginning (before I somehow mess this all up)...
RECAP: At around the same time as my anti-virus suddenly started stating that it was unable to update, I began getting:
Volume: System volume 'root' usage is 85 %. This condition should not occur in normal conditions. Please contact technical support.
Downloading the logs became impossible, attempting to clear them seemed equally futile and unhelpful. After enabling SSH and unsuccessfully attempting to log in, I got locked out of Frontview as my system became unable to fully boot. Access to Technical Support [as advised/instructed in the error] also appears out of reach.
So, the SOLUTION?...
1. Given that I can't fully boot my NAS normally anymore after enabling SSH on my ReadyNAS 516 via Frontview, I need to boot via tech support mode (given that I've effectively been locked out using all other means interacting with the NAS) using the instructions here:
https://kb.netgear.com/23005/How-do-I-access-the-boot-menu-on-my-ReadyNAS-316-516-716(hold down the reset button at the back of the ReadyNAS while booting up until the Boot Menu pops up, at which point one should use the arrows to select Tech Support mode).
[According to the LCD panel at the front of my device] This brings me into the "Debugging Mode", along with a number of some sort (a unique identifier perhaps?) in square brackets and the local IP Address.
2. Next, I use Putty to TELNET into my NAS using the following:
IP ADDRESS (now displayed on the LCD of the ailing ReadyNAS on my network)
PORT 23 (SSH would be Port 22 if one had access to that alternative)3. Next I need to log in [using 'Netgear Tech staff' account credentials(?) based on instructions found here]
root / infr8ntdebug
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- this is where I stopped so far because I'm worried about bricking my device -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
4. At this point, I need to mount the OS using a shell command (based on the same source above)
echo DEVICE partitions > /etc/mdadm.conf mdadm --examine --scan >> /etc/mdadm.conf mdadm --assemble --scan mount /dev/md0 /mnt
OR (based on a different source)
start_raid.sh mount /dev/md0 /sysroot
Not sure which option to run with here...
NOTE: My understing is that the "md0" portion of the code above implies that I'm utilizing X-Raid, which I'm likely using given that the RAID 5 configuration I enabled on OS 6.9.3 (x86) is said to somehow relate to this setup parameter).
5. OPTIONAL: To view the logs,
# View 5000 lines of system logs via journalctl
journalctl -a --no-pager --lines 5000 _SYSTEMD_UNIT=readynasd.serviceanother possible place to look would be:
#--------------------------------------------------#
#\\\\\\\\\\\\| CHECKING LOG FILES |////////////#
#\\\\\\\\\\\\| [for beginners] |////////////#
#--------------------------------------------------#
# STEP 1: Using the 'cd' command, navigate to the following absolute path address: cd /var/log
# STEP 2: LIST ALL files & folders [including hidden ones] within that directory in 'HUMAN-READABLE' format SORTED in REVERSE order.
ls -al -h --sort=size -r
# STEP 3: View a Log File directly in the commandline window
cat "<Exact_filename.extension>"
#OR if it's too big, use 'less' to view it in a scrollable page-based format#
less "<Exact_filename.extension>"
# STEP 4: Consider repeating the steps above for the following locations:
cd /usr
cd /etc6. Given the original error/warning indicating that I should reach out to Tech Support, either:
A) The OS partition is being filled up (typically by a huge log file), in which case the offending file(s) would be copied onto the data volume (just in case), and removed from the OS partition.
#--------------------------------------------------#
#\\\\\\\\\\\\\\\| WORKFLOW |//////////////#
#\\\\\\\\\\\\\\\| [for beginners] |//////////////#
#--------------------------------------------------#
# NOTE FOR BEGINNERS: Don't forget that you can add the --help flag on most commands to get an overview of what it does
#- - - - - - - - - - - - - - - - - - - - - - - - -
#-[ PART 1 ]- All items in this section will be non-invasive and non-destructive, so relax, these commands can't mess anything up.
#- - - - - - - - - - - - - - - - - - - - - - - - -
# STEP 0: [OPTIONAL] Display the 'DISK FILE-SYSTEM' information in 'HUMAN-READABLE' format [for reference]
df -h
#- - - - - - - - - - - - - - - - - - - - - - - - -
# STEP 1: Display 'DISK USAGE' of ALL Files & Folders in 'HUMAN-READABLE' format, with 1 LEVEL of folder display DEPTH
## NOTE: The location '/' is the root starting point
du -ah --max-depth=1 / | sort -h
# STEP 1.5: Likely areas of investigation will likely be [var, usr, etc]:
## NOTE: Adjust 'max-depth' to allow the computer to do more work for you (at the expense of processing power & [potentially] time)
du -ah --max-depth=3 /var/log | sort -h
du -ah --max-depth=3 /usr | sort -h
du -ah --max-depth=3 /etc | sort -h
#- - - - - - - - - - - - - - - - - - - - - - - - -
# STEP 2: Based on the outcome(s), navigate to abnormally large looking folders (where <location> refers to the address) and repeat STEP 1
## NOTE: Addresses starting in '/' are ABSOLUTE addresses, whereas those starting in './' are RELATIVE to the current location
cd "<location>"
# STEP 2.5: Alternately, one could plug the questionable folder <location> directly into the code in Step 1
du -ah --max-depth=3 "<location>" | sort -h
#- - - - - - - - - - - - - - - - - - - - - - - - -
# STEP 3: [OPTIONAL] After identifying a potentially misbehaving file, view its properties via:
## OPTION A: "ls" to LIST ALL files & folders in current directory; SORTED based on SIZE in REVERSE order (largest files last)
## NOTE: Includes properties by default, & the '-h' flag makes it 'Human-Readable'
# To view the details of the file in relation to the others in the current folder
ls -al -h --sort=size -r
#OR [same as previous] if the file is in another folder#
ls -al -h "<location>" --sort=size -r
#OR to get the details of ONLY that specific file#
ls -l -h "<location>/<Exact_filename.extension>"
## OPTION B: "stat" to get more STATUS-related details of a particular file
stat "<location>/<Exact_filename.extension>"
#- - - - - - - - - - - - - - - - - - - - - - - - -
#-[ PART 2 ]- All items in this section will be invasive and/or destructive; be very careful from here-on in.
#- - - - - - - - - - - - - - - - - - - - - - - - -
# STEP 4: COPY the suspected offending file/folder to a new location [not on the root volume] by FORCE & use VERBOSE mode for feedback
## OPTION A [if it's a file]
cp -f -v "<original location>/<Exact_filename.extension>" "<new location>/<Exact_filename.extension>"
## OPTION B [if it's a folder]
cp -f -v "<original location>" "<new location>"
# STEP 4.5: Check to confirm that the file has in fact been copied
ls -al -h "<new location>"
#- - - - - - - - - - - - - - - - - - - - - - - - -
# STEP 5: REMOVE the suspected offending file by FORCE from the conflicted location & use VERBOSE mode for feedback
## NOTE: You can string together multiple file in the same line
cd "<original_location>"
rm -f -v "<Exact_filename.extension>"
#OR#
rm -f -v "<original_location>/<Exact_filename.extension>"
#OTHERWISE, if it's an entire directory#
cd "<original_location>"
rmdir -f -v "<directory to delete>"
#OR#
rmdir -f -v "<original_location>/<directory to delete>"OR
B) The inodes were all being used up, in which case, a different command would be needed [to seek out a huge number of small files on the OS partition].
# Display information of the 'Disk File-System' in 'Human-Readable' format, listing inode info instead of block usage df -i -h
[I have no idea where I would go from here here if it's an inode-related issue]
7. At this point I need to properly unmount the OS somehow (?), restart the NAS, and evaluate my success based on whether I can boot correctly, access Frontview, and if the logs don't display any errors (?). Otherwise, I should consider reversing the previous changes if conditions remain the same [or deteriorate further], and consult with the community to gauge which file(s)/folder(s) may be a more likely culprit of my issue(s).
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE 1: In my particular case, given that I have the ReadyNAS 516, the OS partition is only ~4GB in size, so that's something I'll need to keep in mind while looking at the files [for the sake of having a contextual reference point].
NOTE 2: I'm more of a computer artist than a programmer, so I wrote this for absolute beginners [like me] having these issues.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feedback? [please, & thanks again for any help/support anyone can provide] :smileyhappy:
- Tommy Langzik
Related Content
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!