× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Failing Readynas Ultra 6

alexofindy1
Guide

Failing Readynas Ultra 6

Hi,

 

I have an old Readynas ultra 6 running the old 4.2.x firmware, which is starting to fail.  I'd like to know if I can salvage it.  It has 4 disks in a raid configuration, runs OS 4.2.x, with shares that I access via SMB and AFP, and backup to a ReadyNas 316 via rsync. the Smart status of the drives is all healthy.

 

The device was working well, with the shares readily accessible.  Today I tried to shut it down via frontview;  frontview wouldn't power off the device.   I looked at the log page, and the device stated, mysteriously, that there were no logs.  It did power off gracefully via the front panel power button, but on reboot boot the shares would not mount on client machines via SMB.  They do seem accessible to rsync backups running from my Readynas 316, so I think my data is OK (and it is backed up).

 

So the onboard OS  or firmware is likely corrupted.

 

Can someone point me at documentation or postings on how I might try to reinstall the OS without destroying the data?  If something goes wrong it won't be a crisis, but I'd like to give non-destructive recovery a try.

 

Thanks!! 

 

 

 

 

 

Model: RNDP600U (ReadyNAS Ultra 6 Plus)|READYNAS ULTRA PLUS 6 (DISKLESS)|EOL
Message 1 of 22

Accepted Solutions
StephenB
Guru

Re: Failing Readynas Ultra 6

Do you have ssh enabled on the Ultra?

Do the shares still show up in Frontview?

 

I'm wondering if the OS partition is filling.

View solution in original post

Message 2 of 22

All Replies
StephenB
Guru

Re: Failing Readynas Ultra 6

Do you have ssh enabled on the Ultra?

Do the shares still show up in Frontview?

 

I'm wondering if the OS partition is filling.

Message 2 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Thanks very, very much for your help!!

 

I'm not at home now, so I can't check if the data  shares show up in Frontview, but I'll bet they do.  When I ran rsync incremental  backups from my backup 316 Readynas (the jobs executed on the 316), all the backups ran to completion.  So, I think the data is intact.

 

I don't have SSH installed; I've installed the plug-in SSH toggle before, but I always run it a second time to disable SSH when I'm through on general security principles, so it's disabled now.  I can try running the toggle again, and then checking the OS partition to see if it's full.  That sure sounds like the sort of thing that could cause my symptoms.  Maybe the logs files got too big.

 

But maybe I don't need to bother.  Is there a simple way of cleaning the OS directory?  Will reinstalling the firmware (assuming I can do this from front view with my system in its current state) fix it?

 

If not, I'll try running the SSH toggle, and logging in.   What files should I delete to make the necessary space - do you know?  For example, maybe I should clear  the log directory?  

 

Message 3 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Also, I seem to remember there's a "clear logs" button in Frontview.  That's the first thing I'll try!

Message 4 of 22
StephenB
Guru

Re: Failing Readynas Ultra 6


@alexofindy1 wrote:

Also, I seem to remember there's a "clear logs" button in Frontview.  That's the first thing I'll try!


One challenge here is that if the OS partition gets very full, then installing the ssh plug-in or downloading the log zip can fail.  If that were to happen, you'd need to go in tech support mode, manually mount the OS partition and clean it.

 

Try clearing the logs, and then downloading the zip. Hopefully that will free up enough space to avoid filling the OS completely,

 

Once downloaded, look in disk_info.log.  The OS partition is the first entry ( /dev/md0 ). Normally it is about 25% full.  If you use ReadyDLNA, then there is a small chance that the OS has run out of inodes - but I believe the inode utilization isn't in the log zip file.  You can install ssh, and then use df -i to check for that.

 

Message 5 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

I don't run readyDLNA.  I'll certainly try the easy stuff first, like clearing the logs, or if that fails, trying the ssh toggle.  But if both of those approaches fail, and I have to go in to "tech support mode" are there instructions anywhere on how I do that?

 

My linux skills are not great, but I would assume from your email that once I have access, it's just something like

 

cd  /

mkdir temp1

mount /dev/md0  /temp1

 

and the erase the log files from the OS directory tree to free up space.

 

Thanks again!

Message 6 of 22
StephenB
Guru

Re: Failing Readynas Ultra 6


@alexofindy1 wrote:

 But if both of those approaches fail, and I have to go in to "tech support mode" are there instructions anywhere on how I do that?

 


I'll post them here if it were to come to that.

 


@alexofindy1 wrote:

 

cd  /

mkdir temp1

mount /dev/md0  /temp1

 

and the erase the log files from the OS directory tree to free up space.

 


One thing I want to clarify - the OS partition might be filling.  We don't know that for certain.  I think it's worth looking at before we look for other things.

 

If the NAS is booted normally, I usually suggest using /mnt (which already exists)

mount --bind / /mnt

 

It's not necessarily log files, it could be something else.  But basically you'd need to look for things that don't belong (files stored "under" mount points, oversize logs).  Add-ons often are to blame, so knowing what add-ons you have installed would be helpful.

 

Message 7 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Thanks once again.  I don't have access to the system  now, it will be a week before I can actually check out your suggestion, and try to examine the OS partition to see if it's full.  But, from the symptoms I have (normal smart status, system boots, data is accessible to an rsync backup job, but data is not accessible to client computers via  SMB) a "crippled" OS due to a full partition is a definite possibility.  Since the system is EOL'ed, I haven't done a firmware update in years.  The logs files should prune themselves automatically to prevent this sort of thing, but who knows.  And, as you say, it might be other "junk"  files.  I don't have any add-ons installed, BTW.

 

So, I will try your suggestions in a few days, and post the outcome, and ask about "tech support mode" if it looks like I need it.  Clearing the log files and trying to load the SSH toggle so I can log in are unlikely to makes things worse, regardless of what the actual problem is.

 

Do you happen to know if forcing a firmware uninstall from frontview, if indeed this function works, cleans the OS partition?  I would be reinstalling the same firmware I am running now, since it is the latest available.

 

And, at the risk of being repetitive, thanks again very much for your help!

Message 8 of 22
StephenB
Guru

Re: Failing Readynas Ultra 6


@alexofindy1 wrote:

 

Do you happen to know if forcing a firmware uninstall from frontview, if indeed this function works, cleans the OS partition?  I would be reinstalling the same firmware I am running now, since it is the latest available.

 


You can't uninstall firmware, though you can over-install it.  That won't clean the OS partition though (and the process will use some temporary space on it).

Message 9 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Well, I'm back futzing with my failed Ultra 6+ now.

 

Good news is, my backup server, a 316 device, which was rsync'ed nightly with the Ultra 6+ that failed, seems OK, and I am now using it as my "production" device.  It backs up to an external drive.

 

Bad news is my Ultra 6+ is going from bad to worse; Frontview no longer shows any shares (though the data does appear to be there, as it seems to be accessible through rsync.  I could not install the SSH toggle, it just said that the add-on was incompatible with my system, even though I was using the correct add-in.

 

So....

 

I think I'll wait a few days, and be sure all my data is safe  on my backup 316.  If it is I'll use the boot options on my failed Ultra 6 to check the drives and the memory; if it's a bad drive, I'll just replace it. If the memory and the drives check out, I'll use the the boot menu  to reinstall the OS.  If that fails, I'll do a factory default.  I don't think I'll replace bad memory, though - sounds like too much effort.

 

If my Ultra 6 cannot be fixed, time for a new box, as a backup. Anything thoughts on a Readynas 424 or 426?  I should be able to reuse the drives from my Ultra 6+  (losing the data, of course) from the failed Ultra 6 to save a few dollars.

 

One last question: is it possible to use rsync to backup an OS 6 Readynas (i.e., my 316) to or from a non-netgear NAS?  (may not be a fair questions for this forum....)

 

Thanks all, and in particular Stephen B for his help a week ago!!

 

 

 

 

Message 10 of 22
Sandshark
Sensei

Re: Failing Readynas Ultra 6

It is actually quite easy to replace the memory in an Ultra6+ if that turns out to be the issue, it's just standard DDR2 DIMMs.

 

While power supply problems are common on aging legacy products, this doesn't really have symptoms typical of that.

 

You should be able to get RSYNC working with any Linux based NAS.  It's a standard Linux backup process.  I've not had a lot of luck with Windows based RSYNC products, though.

 

If the data is still there and you can't get to it with SSH, the recovery mode is still a possibility.  It uses FTP.  You can check Google (the forum search is abominable, you'll do better on Goodle) on how ot go about doing that.

Message 11 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Thanks very much, Sandshark!

Message 12 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Well, I have not been able to fix it yet.

 

I have used the boot menu to check the disks and the memory, both seem fine.

 

I then used the boot menu to reinstall the OS.  The OS did reinstall, as far as I can tell; the network settings were reset and the admin password changed to the default, which should indicate a successful reinstall of the OS, but the other problems remain - the shares are listed, but won't mount on clients, the log files are not visible, etc.  I cannot even change the admin password to something other than the system default.

 

I cannot install the SSH toggle plugin; the system reports it is not compatible with the firmware, and I believe I am using the correct version oof the SSH toggle, for OS 4.2.x.

 

So, assuming the problem, as suggested by Stephen B, is a full OS partition, reinstalling the OS did not fix it.  It would be great if there were a means for me to access the system from a command line shell (ssh or telnet).  Can this be done via tech support mode from the boot menu, and, if so, is there any documentation out there?

 

If not, I'll probably try a factory default.  There'd be a bit of data I'd lose, but my important stuff seems alive and well on my backup unit, a newer 316 box.  If the factory default fails, I'll be buying a new box.  (I'm backing up the 316 to external hard drives, but these would not work as production shares should the 316 get in to trouble).

 

Thanks again!!

 

 

Message 13 of 22
StephenB
Guru

Re: Failing Readynas Ultra 6

Message 14 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

That's the one I tried.  It won't load, telling it is not compatible with my firmware.

 

Is there another way to get shell/command line access, for example, the "tech support mode" of the boot menu?

 

Thank again!


@StephenB wrote:

The ssh add-on you need is here: http://www.readynas.com/download/addons/x86/4.2/EnableRootSSH_1.0-x86.bin


 

Message 15 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

OK.  More help needed!  I'm at the limit of my knowledge of linux.

 

I can now access the device by using "tech support" mode in the boot menu, and login as root using the password posted elsewhere.

 

But, I can't figure out how to mount the OS partition.  It is not called /dev/md0, and it's not /dev/hdc1

 

How do I figure out which device corresponds to the OS partition?  Do I need to start the RAID service first, and if so how do I do this?

 

I can post any commend output needed, just tell me which command to run.

Message 16 of 22
StephenB
Guru

Re: Failing Readynas Ultra 6


@alexofindy1 wrote:

 

I can now access the device by using "tech support" mode in the boot menu, and login as root using the password posted elsewhere.

 Begin by entering 

# start_raid.sh

On the Ultra,the OS partition is /dev/md0:

# mount /dev/md0 /sysroot

Then 

# mount --bind /proc /sysroot/proc
# mount --bind /dev /sysroot/dev
# mount --bind /dev/pts /sysroot/dev/pts
# mount --bind /sys /sysroot/sys
# chroot /sysroot /bin/bash
Message 17 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Thanks again very much, StephenB.

 

Though I haven't yet proven it, a full OS partition seems like it may be the cause of my problem.  I will proceed to mount the OS partition from tech support mode in the next few days, using the commands you suggested.

 

Thinking about it, I do one thing with my Readynas that's sort of non standard; I use the device to archive TV shows from my Tivo.  This is a native function of the 4.2.x OS, but is probably not a common practice.  This creates a large index database file of some sort; if this file resides in the OS partition, it wouldn't surprise me if it got quite large.  I will be looking for such a file.  Tivo archiving is not the principle job of my ReadyNas, of course; if I have to disable this function, no big deal.

 

In any case, I know what to do next!

 

 

Message 18 of 22
Sandshark
Sensei

Re: Failing Readynas Ultra 6

If the TIVO archiving turns out to be the issue, another possible option is to replace the folder containing the database with a link to one in the apps share and move everything in that folder to the linked one.  That's the way every app on a ReadyNAS should do it to keep from filling the OS partition..

Message 19 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Well, I am getting there.

 

Once I starrted the raid service and mounted the OS filesystem, but before I did the binds and chroot, the output of df is

 

Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 10240 0 10240 0% /dev
/dev/md0 4184756 4162260 0 100% /sysroot

 

So, I am at 100% usage of the OS file system, which presumably explains why my box doesn't work. I tried the bind and chroot suggested by stephenb, but has a bit of trouble due to my limited familiarity with linux (running df from the chroot shell indicated no filesystems; maybe this is is what it should say, but for now I'm just using the basic tech support mode shell, and not giving the bind and chroot commands).

 

Now I have to find what's taking up so much space. The tivo databases seem to live in

/sysroot/var/lib/readytivo

This takes up only about 2,900 blocks in the du output, so it is probably not the culprit.

The log directory /sysroot/var/log  is only about 71,000 blocks, so that is probably not the problem.

 

The thing that catches my eye are the USB directories. I use USB drives for backup.  The first part of the output of the ls command is

 

drwxr-xr-x 2 4096 Mar 12 2008 USB
drwxr-xr-x 3 4096 Dec 1 05:05 USB_HDD_1_1
lrwxrwxrwx 1 16 Dec 16 2016 USB_HDD_3_2 -> /USB/USB_HDD_3_2
lrwxrwxrwx 1 16 Dec 16 2016 USB_HDD_3_3 -> /USB/USB_HDD_3_3

 

The first line USB directory is empty, which is what I would expect in tech support mode.  The 3rd and 4th lines, USB_HDD_3_2 and USB_HDD_3_3 are therefore links presently pointing to nothing; again this is likely appropriate for tech support mode.

 

But the second line, USB_HDD_1_1 is a real directory, which, according to du, is 3478624 blocks.  It contains one directory, fileserver, which is the name of one of my shares which I do backup to external harddrives.

 

Could something have gone wrong, such that the system backed-up my "fileserver" share to the OS partition, rather than an external drive?  Perhaps the external drive could have disconnected during the backup job; this is actually something I have observed, not infrequently. Usually I just reconnect the backup drive the next morning, and think nothing of it.

 

If this is what happened, then

 

cd /sysroot

rm -fr USB_HDD_1_1

 

should fix my problem.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Message 20 of 22
Sandshark
Sensei

Re: Failing Readynas Ultra 6

There have been others that have reported that a mounting issue with a USB drive caused data to actually be written to the root directory instead of a mountpoint there pointing to the drive.  It sounds like that's your issue.  Note that there could be files there that you think are on your USB drive, but aren't.  So take a look at them before you delete them.

Message 21 of 22
alexofindy1
Guide

Re: Failing Readynas Ultra 6

Thanks very much to those who responded, especially StephenB, who correctly diagnosed my problem and provided me witth the technical details to solve it.  For the record, in case anyone else has the same issue:

 

Problem: My ReadyNas Ultra 6+ is backed up to external disk drives, and to a newer Readynas 316.  A few weeks ago, it suddently failed in a peculiar manner. It would not serve shares to client computers.  Frontview worked, but was broken: it showed no logs on the log page, no shares on the share page, it would not allow me to install add-ons such as the SSH toggle.

 

The diagnosis: StephenB nailed it - my OS partition was at 100%, and thus full

 

The Solution: confirm the diagnosis by accessing the system through tech support mode; to get to tech support mode follow the instructions in the Readynas Harware Manual, and then telnet to the Readynas; I used putty on a Windows PC.  You have to log in with user root and PW  infr8ntdebug (this password is easy to find in this forum and on the internet, so I see no harm in posting it here).

You then have to start the raid service (type the command start_raid.sh) , and mount the OS file system (mount /dev/md0 /sysroot); the device may be different on other Readynas systems.

 

Tech Support mode provides a Linux shell with limited commands; earlier in this thread StephenB describes how to use mount --bind and chroot to invoke the OS system itself, which has access to more commands.  I did not find this necessary, the tech support mode shell was all I needed.

 

I was able to confirm StephenB's diagnosis with the df command, which showed my OS partittion at 100% usage.  I then had to use variations of ls and du to find the culprit; the Linux find command may be useful, though I didn't use it.  Helps to know a bit of Linux (I know a bit, but I'm far from a guru myself)

 

In my case, the problem was a directory tree under /sysroot called USB_HDD_1_1  This directory tree should not have been there; it should have been a link to an external drive; instead the OS partition contained part of a backup of one of my shares.  The backup should have been saved on an external USB drive, but something went wrong and it was saved in the OS partition, thereby filling it up, and causing my system to fail.  Most likely my USB drive disconnected due to a hardware glitch while my backup job was running. I deleted the directory tree using the command line with the appropriate rm command, rebooted, and all seems well.

 

One other note: there is an option in the Readynas boot menu to "reinstall OS."  This will not fix the problem; this option will reinstall the OS, but it doesn't remove errant files, and won't clear out an over-full OS partition.  StephenB points this out earlier in the thread.  To fix the problem, you need to use tech support mode, telnet, and know the basics of linux commands.

 

 

Message 22 of 22
Top Contributors
Discussion stats
  • 21 replies
  • 2783 views
  • 5 kudos
  • 3 in conversation
Announcements