readynas pro 6 stuck in boot/check root FS/check FS loop

[Number6]

Help: readynas pro 6 stuck in boot/check root FS/check FS loop

I had a power failure. Rebooting this ReadyNAS it's stuck in a loop of booting... checking root fs... checking fs rinse-and-repeat.

no clue what OS. The 18TB is almost full and I need the data.

it's behind a nat box, not world accessible.

any suggestions what I can try / do please?

 Thanks 

[mdgm]

The volume check is to be expected after an unsafe poweroff.

Perhaps try an OS Re-install: https://kb.netgear.com/21107/How-do-I-access-the-boot-menu-on-my-ReadyNAS-Ultra-6-Ultra-6-Plus-Pro-6...

I don't think the Skip volume check would be the way forward here as you wouldn't expect to have a reboot loop if that was what was needed.

[Number6]

Thanks VERY much!!!

Yep, any unexpected / unsafe shutdown does a volume check etc. But, this never finishes the fscheck--- it just stays on the "screen" for a few seconds and reboots.

(1) I had found your suggestion, but the KB "OS reinstall boot mode also resets some settings" isn't clear what's REALLY going to be reset. If It's just the root pwd and IP address, I can deal with that. But not knowing what else makes it seem scary.

(2) I was thinking of doing the skip volume check to have it reboot and then ssh into it where I can manually do an fsck (I did have the ssh plug-in installed). 

Based on this little bit of extra info, do you still suggest reinstalling the OS reinstall boot menu option?

[mdgm]

Re-installing the OS resets the admin password and resets some network settings back to default e.g. pickup an I.P. address via DHCP. It leaves most settings intact. It just resets settings which if forgotten or not set correctly are likely to render the device inaccessible. It also re-installs the OS over the top of itself. So if an app you use stops working you may need to re-install the app over the top of itself to fix it.

I suspect the reboots are being triggered by a shutdown script left behind in which case if you try running the skip volume check it should still enter a reboot loop. If this is the case and if the OS Re-install doesn't fix it the script would need to be removed manually or the firmware updated then another OS Re-install performed.

I would try the OS Re-install.

[Number6]

The OS reinstall didn't do it.

now, it powers on; shows "......"; shows 6 squares w/ "booting", then fills in 1, 2, 3... and power cycles. It might have done other stuff right after the 1st is reinstall, but I didn't record that.

so, should I try skip volume check?

if there's a script there, how do I manually remove it?

[Number6]

I went back and looked... now it's showing the 6 squares w/ booting, but none are filled in... it then just power cycles.

soi tried skip volume check, and "....."; 6 squares w/ readynas then booting, none filled in: then power cy.

I can't lose all that data!!

What does tech support option do?

hellllllp....

[StephenB]

It's possible that a power surge might have damaged the chassis and/or some of the disks.

Try removing all the disks (labeling by slot, so you can put them back properly).  Then power up the NAS and confirm that it shows a "no disks" status on the LCD.

You can then test the disks in a Windows PC using vendor tools (Seatools for Seagate, Lifeguard for Western Digital). Disks can be connected with either SATA or a USB adapter/dock.

If you have a spare disk (not part of the array), you can also try doing a factory install on that disk.  If that works, and boots properly, then you can try booting the NAS with that disk in each slot - that will confirm that the all slots are working properly.  Of course power down the NAS before shifting the disk to another slot.

[Number6]

Thanks, I'll try... to get some data.

a surge is unlikely since it was on a UPS whose battery died and didn't maintain power (one thing after another).

[Number6]

re: a spare disk

Does it have to be the same size as the six in there now, or can it be just ANY disk?

[mdgm]

It could be any disk. Just needs to not be from the array.

If you are on old firmware I would update the firmware using the spare/scratch disk.

Once you’ve verified it’s come up fine on latest firmware (4.2.31) after the reboot, I would power down, remove the scratch disk and put your disks in (same order as before)

If the problem is a shutdown script left behind then an OS Re-install on the latest firmware should most likely resolve the problem.

[Number6]

a bit confusing... I think because it's not clear to me where the settings are kept.

I take out all six disks; try to reboot seeing if it shows "no disks".

THEN

I put in a spare disk of any size ---- do I then just reboot like that or must I do a factory default install onto that spare disk? And then, when it's rebooted somehow on that spare disk, can I THEN install the most recent firmware? (I don't recall what firmware was on the box)... ???

and then, when I take that spare disk out and put the original six back in, will it try to reboot THOSE disks with the settings that are stored on THOSE disks so that I don't lose all of my share settings etc.

[mdgm]

The settings are stored on the disks.

Changes made on the scratch disk won't affect settings on your array that's removed from the NAS.

When you update the firmware the firmware is updated on the internal flash then on the disks.

OS Re-installs and Factory Resets use the firmware on the internal flash.

Firmware updates include fixes where possible for some known problems.

Say, for example the UPS told your NAS to shutdown but the battery died cutting power during the shutdown process a script may have been left behind causing the NAS to power down before completing the boot process. 

On the latest firmware an OS Re-install should remove such a script but on old firmware that isn't the case.

If the scratch disk is used you'd need to do a factory default to wipe it. As only the scratch disk is installed it wouldn't do anything to your array with your data on it.

If you use the scratch disk to update the firmware, when you power down, remove the scratch disk and put your disks back in the NAS will attempt to update the firmware on the disks to correct the mismatch with the newer firmware on the internal flash.

[Number6]

took all six drives out; put in a spare; boot mode 'factory reset'. it came all the way up. within web interface I updated the firmware; it completed; rebooted; came all the way back up again; raidar shows 'healthy' and web intfc shows healthy.

told web interface to shotdown / power down;

removed that one spare drive; carefully re-inserted all six original drives in the right places;

power up: it shows "....." then six squares; then 'booting'; then it power cycles; rinse-and-repeat. never shows checking fs.

at some point during one of the booting/power-cycles, raidar showed it 'starting up' (with the previously saved name and ip address).

so, I took the six drives out; put that one spare back in; rebooted it just fine; set to volume scan next boot; did shutdown and power down. put the six drives back in...

and... no difference: .......; six squares; booting; power-cycle rinse-and-repeat.

any more ideas? 

[StephenB]

---

@Number6 wrote:

and... no difference: .......; six squares; booting; power-cycle rinse-and-repeat.

any more ideas? 

---

First, I think the power failure damaged the filesystem somehow (likely on the OS partition itself) - maybe not the script bug @mdgm is thinking. You could start by trying an OS reinstall from the boot menu though.

If that doesn't help, then try using the boot menu to skip the volume check on boot.  It is a long shot, but easy to try, and won't do any damage. If that does work, then you should back up your data before rebooting again.

If that fails, then (as suggested earlier), test the disks in a PC with vendor tools (lifeguard for Western Digital, seatools for for Seagate).  The system normally boots from disk 1, so at least test that one.  If that's not possible, you could try booting the system with disk 1 removed.

If you can boot up in tech support mode, you can engage paid Netgear support.  They can rebuild the OS partition (and if more is needed, Netgear does offer a data recovery service).

If you can connect all the disks to a PC, you could also use RAID recovery software to offload the data.

[Number6]

Thanks... I don't think I have a windows machine but I'll see... will try other things first.

of course, due to a water main break, trying to fix my Readynas will have to wait a day.

[Number6]

a bit of info...

As I noted, I removed all six drives and put in a spare. That DID work (and I reinstalled the firmware), so it means the box itself and drive 0 are working.

Then tried all six drives in, rebooting with re-install os. it then reboots. each time it's a wee bit different. Sometimes it's six empty squares "booting..." and a few times it's six squares that get filled in "1,2,3,4,5,6" and "checking root fs" but then it power cycles. (once it only filled in 1,2).

my next try is to reboot with skip volume check... wish me luck.

[Number6]

Before trying skip check, I plugged it into a known-good ups. It booted... did checking root fs, then checking fs 2%... then power cycled. I then tried to skip check and... it didn't fill in any squares, but power cycled... then checking fs to 10%, then power cycled.

hen 

so it didn't seem to skip volume check. weird weird

I just checked on it and it's recycled... this time just did checking root fs, then checking fs and immediately power cycled.

Is there some "trick" to get it to actually skip volume check?

[StephenB]

---

@Number6 wrote:

Is there some "trick" to get it to actually skip volume check?

---

Just the boot menu.

If it boots up in tech support mode, then you might want to engage paid Netgear support.  I'm suspecting that its a corrupted OS partition, and they can fix that remotely,

[Number6]

I'll definitely aim to do that Monday afternoon.

If it were that, though, why would (a) it not boot removing drive 0? or (b) boot and start fsck'ing, but then reboot? Purely gut feeling that something else is going on... but, YOU'RE the true expert here! If you can think of anything else to try before Monday, please let me know.

When I try booting into tech support mode, what should I see that'll show me it "worked"?

Since it's behind a firewall and a NAT box, what port(s) will I need to open and map to the DHCP'd address please? (or a URL if that's easy)/

thanks again

[StephenB]

---

@Number6 wrote:

Since it's behind a firewall and a NAT box, what port(s) will I need to open and map to the DHCP'd address please? (or a URL if that's easy)/

---

Booting up in tech support mode allows Netgear to access your NAS with no need to open ports, etc.

I'm suggesting booting in tech support mode first, just to make sure you can.  After that you'd engage paid support using my.netgear.com.

If you are skilled with linux, you can also connect to the NAS locally in tech support mode using telnet.  But if you don't have those skills already, now is not the time to learn them.

---

@Number6 wrote:

If it were that, though, why would (a) it not boot removing drive 0? or (b) boot and start fsck'ing, but then reboot? Purely gut feeling that something else is going on...

---

The fact that it is booting up on the spare disk says that it's not likely to be the chassis.

That says it's something with the disks.  Given that this is happening early in the boot process, and booting w/o the volume check also fails suggests that it's a problem with the OS partition itself.  It might be something else, but if it is something with disk-related then Netgear paid support should be able to sort it out.

---

@Number6 wrote:

If you can think of anything else to try before Monday, please let me know.

---

You could try testing the disks in a Windows PC using vendor tools (lifeguard for Western Digital; seatools for Seagate).  You can connect them with either SATA or a USB adapter/dock.

Normally I'd suggest running the full non-destructive test, but you wouldn't have time to run that on all the disks by Monday afternoon.  So maybe start by running the quick test for each disk (normally takes about 2 minutes), and then circle back and start running the full test (at least on disk 1).  The full test will take 6-7 hours for a 3 TB drive.

[Number6]

@StephenB 

Sorry, got pulled away with lots of IRL hassles... don't ask!

Ok, it does boot into tech support mode.

The userid / password defaults listed in the docs don't work, however. What are they? (you can PM me if needed)

Having used and admin'ed unix systems for decades, I'd like to just see if fsck etc. might work... but if I don't try that........

(1) I'll have to open telnet port on my firewall

and

(2) Set up a port forward for netgear tech support to the ReadyNAS

However, I've looked and looked and couldn't find how to set up a tech support request (nor know the pricing ahead of time -- I sure can't afford a blank check). Is there a URL to get me to the place to open this request (if a simple fsck doesn't solve it)???

Thanks much

[Number6]

p.s.

All I had found was an extended warranty service, but this is more than 90 days since purchase. I had found nothing close to buying a single service support ticket. I'll be happy to sign up for a year of pro-home support if that's the right thing... Just have to figure out where the serial number is. (is it echoed out on the tech support telnet login?)

[Sandshark]

Tech support username is root and the password is infr8ntdebug.  Since you are logged in as root, there is no need for the use of sudo as you would in most Linux systems.

I can't help with the Netgear support except to say that it's not covered by Gearhead support if that's the contract you are looking at and that they do offer per-incident support, though data recovery is extra if needed.  I've not bought as new ReadyNAS since an NV+ V1, so I can't get support.

The serial number is physically on the NAS and if you have a saved log .zip, it's in boot_info.log.

[Number6]

thanks much!!

able to log in in tech support mode. just doing an fsck won't be that simple - 

Any specific command(s) you can suggest ala fsck (dmesg looks like there's raid hardware and THAT I'm not familiar with))?

Is there anything in the dmesg that might help diagnose what's wrong (before I try to get support)? (and, is the serial number in the dmesg since I don't think I've got a log.zip around)

[StephenB]

The Pro-6 uses software RAID, not hardware.

Try to mount the OS partition:

# start_raid
# mount /dev/md0 /sysroot

if you also want to try chroot you can then:

# mount --bind /proc /sysroot/proc
# mount --bind /dev /sysroot/dev
# mount --bind /dev/pts/sysroot/dev/pts
# mount --bind /sys /sysroot/sys
# chroot /sysroot /bin/bash