Re: readynas pro 6 stuck in boot/check root FS/check FS loop
My suspicion is that there may be a spool script to shutdown the NAS left behind in /var/spool/frontview on the sysroot. If this is the case the system will get stuck in a reboot loop until the script is removed.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
thanks...I think...
Here's what happened:
# start_raid.sh mdadm: /dev/md/0 has been started with 5 drives (out of 6). mdadm: /dev/md/1 has been started with 6 drives. mdadm: /dev/md/2 has been started with 6 drives. mdadm: /dev/md/0 already active, cannot restart it! mdadm: /dev/md/0 needed for /dev/sda1... # Connection closed by foreign host.
and now the Pro 6 is no longer visible by Raidair.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
One of the disks has dropped out of the RAID array for the root volume.
You could check the state of the root RAID array by doing e.g.
# mdadm --examine /dev/sd[a-z]1 | egrep -i 'sd|event|uuid|recovery|super|role|update'
# for i in /dev/sd[a-z]1; do echo $i; mdadm -E $i | egrep "Creat|Update|State|Raid"; done;
You may also wish to check the SMART stats of the disks to see if there is a problem disk
# for i in a b c d e f g h i j k l m n; do smartctl -a /dev/sd${i} | egrep -i "Device Model|Serial Number|Reallocated_sec|ATA Er|offline_uncorrect|current_pending_sector|Power_on"; done
# for i in /dev/sd[a-z]; do echo $i; smartctl -a $i | egrep "Model|Serial|Firm|Sector|Hours|Error|Uncorr"; done;
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
(yes, I know I'll have to reboot into tech support mode "each time")
Thanks. here's lots of results... still trying to figure out what to do:
(1) # mdadm --examine /dev/sd[a-z]1 | egrep -i 'sd|event|uuid|recovery|super|role|update' /dev/sda1: Array UUID : c8a57984:570f900c:6fbe8815:6febea3f Super Offset : 8 sectors Device UUID : 5d34be00:4ca6ea2d:064fec68:bfeffdee Update Time : Sun Feb 7 23:19:05 2021 Events : 2864 /dev/sdb1: Array UUID : c8a57984:570f900c:6fbe8815:6febea3f Super Offset : 8 sectors Device UUID : a0429c31:96db2c16:990aeabe:47a89640 Update Time : Sun Feb 7 23:19:05 2021 Events : 2864 /dev/sdc1: Array UUID : c8a57984:570f900c:6fbe8815:6febea3f Super Offset : 8 sectors Device UUID : 0a00bff0:2340dc15:8529ebd1:f0e0ed9d Update Time : Sun Feb 7 23:19:05 2021 Events : 2864 /dev/sdd1: Array UUID : c8a57984:570f900c:6fbe8815:6febea3f Super Offset : 8 sectors Device UUID : c74b6a70:98908d4c:fca771fd:0ac7b738 Update Time : Sun Feb 7 23:19:05 2021 Events : 2864 /dev/sde1: Array UUID : c8a57984:570f900c:6fbe8815:6febea3f Super Offset : 8 sectors Device UUID : 6742adae:a354e3be:75b11fcf:c220e183 Update Time : Sun Feb 7 23:19:05 2021 Events : 2864 /dev/sdf1: Array UUID : c8a57984:570f900c:6fbe8815:6febea3f Super Offset : 8 sectors Device UUID : 5c201efc:7c3cc971:3b14d652:f52c1d3b Update Time : Sun Feb 7 23:19:05 2021 Events : 2864 mdadm: No md superblock detected on /dev/sdg1.
(2) # for i in /dev/sd[a-z]1; do echo $i; mdadm -E $i | egrep "Creat|Update|State|Raid"; done; /dev/sda1 Creation Time : Mon Dec 12 17:51:52 2011 Raid Level : raid1 Raid Devices : 6 State : active Update Time : Sun Feb 7 23:19:05 2021 Array State : Uuuuuu 378 failed /dev/sdb1 Creation Time : Mon Dec 12 17:51:52 2011 Raid Level : raid1 Raid Devices : 6 State : active Update Time : Sun Feb 7 23:19:05 2021 Array State : uUuuuu 378 failed /dev/sdc1 Creation Time : Mon Dec 12 17:51:52 2011 Raid Level : raid1 Raid Devices : 6 State : active Update Time : Sun Feb 7 23:19:05 2021 Array State : uuUuuu 378 failed /dev/sdd1 Creation Time : Mon Dec 12 17:51:52 2011 Raid Level : raid1 Raid Devices : 6 State : active Update Time : Sun Feb 7 23:19:05 2021 Array State : uuuUuu 378 failed /dev/sde1 Creation Time : Mon Dec 12 17:51:52 2011 Raid Level : raid1 Raid Devices : 6 State : active Update Time : Sun Feb 7 23:19:05 2021 Array State : uuuuUu 378 failed /dev/sdf1 Creation Time : Mon Dec 12 17:51:52 2011 Raid Level : raid1 Raid Devices : 6 State : active Update Time : Sun Feb 7 23:19:05 2021 Array State : uuuuuU 378 failed /dev/sdg1 mdadm: No md superblock detected on /dev/sdg1.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
That's interesting. They all show the same update time. The drives appear to be healthy (although there are some ATA errors they were all a very long time ago).
(10) # mdadm -A -f -v -v --scan --homehost=001F33EADAB1 --auto-update-homehost --auto=md mdadm: looking for devices for further assembly mdadm: no recogniseable superblock on /dev/sda mdadm: no recogniseable superblock on /dev/sdb mdadm: no recogniseable superblock on /dev/sdc mdadm: no recogniseable superblock on /dev/sdd mdadm: no recogniseable superblock on /dev/sde mdadm: no recogniseable superblock on /dev/sdf mdadm: no recogniseable superblock on /dev/sdg mdadm: no RAID superblock on /dev/sdg1 mdadm: /dev/sda1 is identified as a member of /dev/md/0, slot 0. mdadm: /dev/sdb1 is identified as a member of /dev/md/0, slot 1. mdadm: /dev/sdc1 is identified as a member of /dev/md/0, slot 2. mdadm: /dev/sdd1 is identified as a member of /dev/md/0, slot 3. mdadm: /dev/sde1 is identified as a member of /dev/md/0, slot 4. mdadm: /dev/sdf1 is identified as a member of /dev/md/0, slot 5. mdadm: added /dev/sdb1 to /dev/md/0 as 1 mdadm: added /dev/sdc1 to /dev/md/0 as 2 mdadm: added /dev/sdd1 to /dev/md/0 as 3 mdadm: added /dev/sde1 to /dev/md/0 as 4 mdadm: added /dev/sdf1 to /dev/md/0 as 5 mdadm: added /dev/sda1 to /dev/md/0 as 0 mdadm: /dev/md/0 has been started with 6 drives. mdadm: looking for devices for further assembly mdadm: no recogniseable superblock on /dev/sdg1 mdadm: /dev/sda2 is identified as a member of /dev/md/1, slot 0. mdadm: /dev/sdb2 is identified as a member of /dev/md/1, slot 1. mdadm: /dev/sdc2 is identified as a member of /dev/md/1, slot 2. mdadm: /dev/sdd2 is identified as a member of /dev/md/1, slot 3. mdadm: /dev/sde2 is identified as a member of /dev/md/1, slot 4. mdadm: /dev/sdf2 is identified as a member of /dev/md/1, slot 5. mdadm: added /dev/sdb2 to /dev/md/1 as 1 mdadm: added /dev/sdc2 to /dev/md/1 as 2 mdadm: added /dev/sdd2 to /dev/md/1 as 3 mdadm: added /dev/sde2 to /dev/md/1 as 4 mdadm: added /dev/sdf2 to /dev/md/1 as 5 mdadm: added /dev/sda2 to /dev/md/1 as 0 mdadm: /dev/md/1 has been started with 6 drives. mdadm: looking for devices for further assembly mdadm: /dev/sda3 is identified as a member of /dev/md/2, slot 0. mdadm: /dev/sdb3 is identified as a member of /dev/md/2, slot 1. mdadm: /dev/sdc3 is identified as a member of /dev/md/2, slot 2. mdadm: /dev/sdd3 is identified as a member of /dev/md/2, slot 3. mdadm: /dev/sde3 is identified as a member of /dev/md/2, slot 4. mdadm: /dev/sdf3 is identified as a member of /dev/md/2, slot 5. mdadm: added /dev/sdb3 to /dev/md/2 as 1 mdadm: added /dev/sdc3 to /dev/md/2 as 2 mdadm: added /dev/sdd3 to /dev/md/2 as 3 mdadm: added /dev/sde3 to /dev/md/2 as 4 mdadm: added /dev/sdf3 to /dev/md/2 as 5 mdadm: added /dev/sda3 to /dev/md/2 as 0 mdadm: /dev/md/2 has been started with 6 drives. mdadm: looking for devices for further assembly mdadm: looking for devices for further assembly mdadm: no recogniseable superblock on /dev/md/2 mdadm: no recogniseable superblock on /dev/md/1 mdadm: no recogniseable superblock on /dev/md/0 mdadm: cannot open device /dev/sda: Device or resource busy mdadm: cannot open device /dev/sdb: Device or resource busy mdadm: cannot open device /dev/sdc: Device or resource busy mdadm: cannot open device /dev/sdd: Device or resource busy mdadm: cannot open device /dev/sde: Device or resource busy mdadm: cannot open device /dev/sdf: Device or resource busy mdadm: no recogniseable superblock on /dev/sdg mdadm: cannot open device /dev/sda1: Device or resource busy mdadm: cannot open device /dev/sdb1: Device or resource busy mdadm: cannot open device /dev/sdc1: Device or resource busy mdadm: cannot open device /dev/sdd1: Device or resource busy mdadm: cannot open device /dev/sde1: Device or resource busy mdadm: cannot open device /dev/sdf1: Device or resource busy mdadm: no recogniseable superblock on /dev/sdg1 mdadm: cannot open device /dev/sda2: Device or resource busy mdadm: cannot open device /dev/sdb2: Device or resource busy mdadm: cannot open device /dev/sdc2: Device or resource busy mdadm: cannot open device /dev/sdd2: Device or resource busy mdadm: cannot open device /dev/sde2: Device or resource busy mdadm: cannot open device /dev/sdf2: Device or resource busy mdadm: cannot open device /dev/sda3: Device or resource busy mdadm: cannot open device /dev/sdb3: Device or resource busy mdadm: cannot open device /dev/sdc3: Device or resource busy mdadm: cannot open device /dev/sdd3: Device or resource busy mdadm: cannot open device /dev/sde3: Device or resource busy mdadm: cannot open device /dev/sdf3: Device or resource busy # Connection closed by foreign host.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
Try assembing just md0 and see if the connection remains up doing that.
It's strange that you are losing connection when assembling the RAID. One wonders if the crashing could be due to some kind of hardware problem, whether a disk or some other hardware e.g. the PSU.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
Thanks very much!
(1) The connection drops because it reboots into the boot loop.
(2) How would I assemble just md0?
(3) going back to previous trouble shooting... Taking out all drives and installing a "new" (ok, old from somewhere) drive) it boots aok (wipes that drive and fully boots to factory defaults up and all running. So, that rules out the main hardware and drive 0 hardware (not the drive itself, though). BUT, it boots into that loop when I install only five drives leaving drive 0 removed.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
That would seem to suggest that the SATA backplane is probably fine.
If the disks are fine, then I guess it would point to a problem on the disks, or looking at doing a memory test of the RAM (if not done already) or perhaps a problem with the PSU.
Try starting just the 4GB root volume. I think from memory it's something like this:
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
If the system crashes doing a simple thing like starting the RAID then paying for software service is going to be a waste of time unless it's just that one of the disks is bad and needs to be removed or cloned or something like that.
The problem is either:
failing PSU
bad disk
some other hardware issue e.g. RAM etc.
Have you run the memory test boot menu option for at least a few passes?
Have you tried the disk test boot menu option and/or done e.g.
# smartctl -t long /dev/sda
Seeing your system is crashing I would wait for each disk's test to finish before trying the next one if you do this.
If you are confident the disks are fine and don't want to go down the route of trying replacing the PSU and/or RAM, then getting a new NAS e.g. RN426, RN526X etc. and moving the disks across is probably what should be tried next.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
Hadn't run the boot mode tests yet. Ran smartctl smarttest on all six seemingly ok. starting a long test on sda now.
If it's a bad disk, then shouldn't I be able to try booting with one disk removed at a time? Thinking it might be disk 0, I did try that but it didn't work. If that's a good test for a bad drive, then I'll do it and if it works with just one removed, then I can replace that drive.
I've hunted on the bay for RN426 OR RN526X and didn't find any. Are those the only 6 bay units that are upward compatible with my Pro 6? -- i.e. that I can just plug my six drives in and they should just work? Going that way sounds attractive since I could upgrade to 4-8GB drives (the Pro is stuck at 3GB drives max).
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
The latest models are RN426, RN428, RN526X, RN528X, RN626X, RN628X
Older OS6 models are RN316, RN516 and RN716X
If the disks, array and OS are fine it will boot with the volume read-only so you can copy your data off, then do a factory reset (wipes all data, settings, everything) and restore data from backup.
I don't recommend getting RAIDiator-x86 units (e.g. Ultra, Ultra Plus, Pro) unless you really love buying old hardware. They would all have been manufactured over 8 years ago and thus even if in working condition now, are more likely to fail sooner.
Your Pro is not limited to 3TB disks, however upgrading to higher capacity disks whilst staying on RAIDiator-x86 would run into expansion limitations.
If it's a bad disk, then shouldn't I be able to try booting with one disk removed at a time? Thinking it might be disk 0, I did try that but it didn't work. If that's a good test for a bad drive, then I'll do it and if it works with just one removed, then I can replace that drive.
You could, however I don't like to do it that way. If the disks become out of sync it could force a resync leading to further problems.
It sounds to me like most likely it's a PSU issue with spinning up all the disks drawing too much power from a failing PSU. One way to test this theory would be to put 6 scratch disks in and see if it enters the reboot loop. If it doesn't enter the reboot loop then it would suggest a bad disk.
If you remove the PSU you can inspect it to see if it's damaged. PSUs are one of the most likely components to fail in a PC and they have failed in a number of Pro units. It's hardly surprising that PSUs are failing well after the warranty has expired.
I've hunted on the bay for RN426 OR RN526X and didn't find any. Are those the only 6 bay units that are upward compatible with my Pro 6? -- i.e. that I can just plug my six drives in and they should just work?
No OS6 unit will have that level of compatibility unless the Pro6 was converted to OS6 (and I can see yours wasn't). You can put the drives from your unit in one in order to recover data in a read-only mode, but it won't "just work" as a production NAS.
If you have an ATX power supply lying around, you could connect it externally in place of the internal one and see if that resolves the problem, or even run it that way long enough to recover the data. If you do plan to do that and run it a long time, use masking tape to block the gap the cable goes through and the internal power supply fan outlet so the chassis fan properly draws air around the drives and motherboard. If that works, then you can get more life from your NAS by replacing the internal supply with an SFX format one.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
I'm running the long tests on each disk now (on second right now). each takes 4 hours. After I run all six, THEN I'll try six spare drives to see what happens.
I don't think I have a spare PS (the only spare I had was already put into use).
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
If you can temporarily "borrow" an ATX supply from a working machine long enough to determine if the supply is the issue, that would help. If that is it, we can guide you in replacing the internal one with a standard SFX one. One issue may be the number of Molex 4-pin connections, as you need three. But SATA to 4-pin adapters and 4-pin splitters can be employed. Otherwise, it just needs the standard ATXv12 24-pin main and 4-pin CPU connections. It should also be at least 450W to insure it has sufficient 12V power.
For data recovery only, if you have a way to mount the drives in a PC internnaly and/or externally using SATA to USB or eSATA connections, you can use R-Linux for Windows or a native Linux system to recover the data. I'm not expert on that, but others can guide you.
Re: readynas pro 6 stuck in boot/check root FS/check FS loop
Thanks very much to @Sandshark I was able to replace the entire box and ... it all worked just fine!!! Woo Hoo !!!!!!!
So, it *might* have been the PSU (or something else?).
Now that I have a ReadyNas Pro 6 that *might* just need a working PSU, I'd like to consider finding and trying to replace the PSU to see if I can get it working. It seems that I can find a power supply that will probably fit (kinda) or it might need a 24pin cable extender on ebay for about $30 --- or I see one there that's supposed to all fit right for $65. My experience over the years is that if something's not going to fit, I'll be the one that experiences it and ends up having trouble finding the right extra pieces.
Any specific suggestions (the cheaper the better) for a PSU that will "just fit" gladly accepted and appreciated.
