Readynas Pro 6 stuck in power off - reboot power off cycle.

[ferg1]

Hello. I've got a Readynas Pro 6, that I updated to the 6.0 firmware some time ago. It's been running flawlessly for a few years now.\

The unit is used for backup (Time Machine and rsync from Linux boxes) and serving media.

Earlier today I noticed that the device had rebooted. A few mnintes later it's rebooted again.

From the LED screen at power on I see:

- checks each drive (6 x 3TB).

- Connects network

- Checks boot drive, get's to about 50% then powers down.

- After ten seconds or so it powers back up.

At first I suspected overheating, so I opened it up cleaned all the fans. But that's not the issue.

I guess my next guess is the power supply.

Any tips as to what this issue is?

Many thanks and Happy New Year.

Cheers

Ferg

- 

[StephenB]

It could be the disks.  Perhaps try testing disk 1 in a PC with vendor tools (Seatools for Seagate; Lifeguard for Western Digital).

[Sandshark]

If you have (or can borrow from a PC) a standard ATX power supply, you can hook that up externally just to see if it is the supply.  If it ius, the replacement is an SFX form factor one.  If you are going to run it any length of time with the external supply, cover the gaps where the cable comes out and the internal power supply fan opening with masking tape to keep air flowing properly over the drives.

[ferg1]

Thanks. That is certainly not the type of job to really be doing with a New year's Day hangover....! Those molex are pretty tight. I hooked up a spare ATX PSU. Now the unit briefly turns on then back off again. So I guess the issue is more serious. Oh well.

[ferg1]

Weird...! I reconnected the old PSU and it booted up fine. It's now resyncing...! Either it was the UPS unit it was previously plugged into, or I did remove and reseat the DIMM when reconnecting the PSU cable. i wonder if that had worked itself loose? Seems unlikely. Anyway fingers crossed it resyncs fully, Thank you!

[ferg1]

Weirder and weirder. I just powered off the unit to return it to its normal place under my desk. Once in place I powered it up again and the same issue reoccurs. It starts to boot then fails. This is after running for a few days without issue. I guess I'm now back at square one!

[Sandshark]

Did you re-boot multiple time before this?  If not, it could still be a failing power supply and you just got lucky to get through the initial current demand once.

Otherwise, perhaps a loose connection somewhere?  Try re-seating the drives.

[ferg1]

Thanks. But no go. 

However, I managed to get it booted. When I did so I looked the logs and found out many Command Timeout messages for two disks....! Turns out I'd changed the password for my SMTP server a few months ago but forgotten to change it on the ReadyNAS. So the trusty ReadyNAS had been trying to warn me that disk 1 and 6 had many Command Timeouts for a month now. But I never noticed. 

Two impending disk failures on a six disk RAID5 array...!

I bought two replacements then tried to figure which disk was failing. I chose disk6, but that refused to boot too. So it was actually Disk1 that was the culprit. By booting a degraded Raid5 array with two failed units out of six disks I've now messed up that array.

I am experienced enough to repare this situation if it was plain old RAID 5 using MDADM, but I have no idea if I can restore this array on a ReadyNAS. RAIDOS6 uses ZFS now right?  Any tips for recovering my array? 

(It's not critical as all is backed up, but it's a pain to reconfigure!).

Thanks!

[StephenB]

OS6 in fact uses mdadm for RAID, and btrfs for the file system.

Whether it is purely RAID-5 or not depends on your expansion history.  If all the disks were always the same size as you have installed now, then it should be purely RAID-5.  If you've vertically expanded (or have unequal size disks), then there are multiple RAID groups concatenated into a single volume.

[ferg1]

So an update on this issue. As mentioned previously I'd got it booted up but with two drives marked as failed on a six drive RAID5 array was not looking too good...! After 24 hrs it died again and powercycled on subsequent boots. So on a whim I replaced the PSU*. Now it boots fine. The array was repaired that took 12 hours, but the unit is now up and running fine.

I used a SilverStone SST-ST45SF v3. It fits nicely albeit with many redundant cabling. I needed a ATX24 pin extension. It was about 100mm short! I also needed a Molex splitter as the PSU comes with only two Molex plugs (many SATA though!).

Now let's see if I still get COMMAND TIMEOUTS on the two drives as before. I have got two replacements ready to pop in if so. I wonder (without any evidence to back this up) if the failing PSU supplied low power to the two drives and caused the errors.. Probably now 🙂

Thanks for all your suggestions.

Stay safe and healthy!

[Sandshark]

A command time-out can be caused by either side, so low voltage to the drive or to the electronics controlling it absolutely could cause those errors.  I suspect that your "dead" drives were just not spinning up in time, too.  You just got lucky once or twice and they made it.  Are those two drives different, maybe requiring a bit more power?  That's not a problem for a healthy NAS, but could be a part of the puzzel.

Good that you identified the failing power supply before it really did take your volume with it.  If the Molex connectors are both on one wire bundle, I recommend a SATA to Molex adapter coming from another wire bundle instead of splitting off a Molex.  If there are only two bundles, both containing both SATA and Molex, it's a don't care.

[ferg1]

Thanks. That's helpful. The drives are indeed both different to the others in the NAS. Others are ST2000M2001, but the error prone pair are different (I cannot actually remember the drives).