ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
I have a ReadyNAS Pro 6 (RNDP6350) that I've only had online for about 1 year (was a backup unit on a shelf with no drives for ages). I put 6x New 4TB Seagates (ST4000VN000) in the NAS less than 1 year ago and updated the NAS to OS6. It is currently running v6.6.1 with a Flex-RAID6 config.
For the most part the NAS is terribly stable and performs great. But on a couple occasions (and again this evening) it will effectively go offline. No web interface, no response from apps, can't SSH in. I will get about 30% return on ICMP pings, but with some ridiculously high response time (greater than 5-10 seconds!). The primary NIC is configured via static IPv4 (which is what it normally runs). I leave the second NIC on DHCP, and will try to plug it in when it goes in to tihs failure state. I do not see it pick up an IP, nor can I force the second NIC to respond to ARP requests. Frontview says everything is happy, but effectively the NAS appears offline. I try to do a soft-shutdown via the front button, it initiates, but then hangs on the shutting down screen for hours. So I cycle it with the front button.
When I try to power the device back up, I end up with an error along the lines of "No Disks Detected.". I try power cycling several times, same result.
At this point, I usually unplug the UPS, Ethernet, and Power from the NAS and move it from the closet it lives in onto my desk. I power it back up, and every time (this being the third time), it comes right back up. There is no mention in the logs of anything. I see my snapshot from earlier in the morning, then the startup stamp from when I booted it on my desk. SMART is entirely clean (not one error).
This has happened on 3 seperate occasions. Same errors, same "fix". The last time this happened, I reseated every drive. That was ~4 months ago.
My theory now is that one of two things is happening:
1) The chassis is requiring the physical removal of power (I haven't tried turning off the power supply or unplugging it without moving it yet), and not just a soft reset;
-or-
2) When I physcially relocated it, something that was (or wasn't) making contact gets jostled, and now isn't (or is) making contact. This seems unlikely as I'm pretty damn delicate when I move it, but _shrug_.
I'm somewhat stumped. Has anyone seen anything like this before? I feel like I have my own special version of the Magic / More Magic switch. 😕
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
Thank you for the reply Jenn, much appreciated!
The LED and LED Display stay on, and appear to be functioning properly. I'm even able to initiate a soft-shutdown using the power button and confirming on the display. However, that soft-shutdown never completes (at least not after waiting >6h).
I could certainly test the NAS not plugged in to the UPS, however it takes months for this problem to reappear. And given the weather we've been having in CA lately, our power hasn't been as predictable as it usually is.
Certainly welcome other thougths on this. This is a weird one.
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
Are the LEDs on the NIC lighted when this happens? Does your router/switch show it has a connection?
The NIC and power control functions are powered by the 5V standby power from the power supply, which is constantly on even when the unit is off. Turning the switch on the back off and on or unplugging and plugging in will re-start the standby power regulator. It may take some time for the capacitors to dischage for a full reset, too -- whihc you get when you physically move it. So, the problem may be in the power supply. Not really enough evidence yet to suggest you replace it, but keep it in mind in case it gets worse. Next time, try just switching it off from the back, waiting a couple minutes, and re-powering and observe what the NIC LED's do during the process.
Your thread has made me aware to do some extra checks on the unit when it enters this limbo state (check if the NIC is still on, if there is traffic, or if Raidar shows anything at all)
Just as added information in regards to your situation and it having similarities, but not 100% identical;
- in my situation I am not required to turn the power completely off (ie disconnect from the outlet). I just press the power button for 4+ seconds and then turn it on again, it boots normally
- I haven't let it try and shutdown the machine when pressing the power button twice (saying that it will shutdown). The reason for this is that I noticed that it seems that the daemon controlling the display is running, but doesnt require to do something else (ie: write to disk or something like that). So when you press it will try to do so, and then hangs because it can't access the disk. I think this might be true due to the fact that when im verociously keep repeating pressing the button results in it going back and forward between normal behavior and shutdown message. So I think the controls aren't working at all
- I noticed an extreme minority of the situations (seen it only once) that the NAS wants to check (fschk) the disks. So it seems there is no apparent disk activity going on and the forced shutdown is relative clean.
- Lastly I too noticed an increase in ping responses. But havn't been around the nas right at the moment of it happening. My monitoring tool has timeouts on ping checks of a few seconds, so the graphs in my thread don't accurately reflect this.
Are the LEDs on the NIC lighted when this happens? Does your router/switch show it has a connection?
This is an excellent question Sandshark. I do not know the answer specifically to the LEDs on the NICs themselves. I will check this the next time it fails.
However, IIRC when I've plugged in the second NIC (after the NAS has gone into this failure state), I get a link light on the switch. However, I never see a DHCP request come through for that NIC. I'm 80% sure that I get the link on the switch.
I believe that the primary NIC still shows a link light on the switch as well (however, I'm only 30% sure of this).
Next time it fails, I'll check the LEDs on the NICs themselves.
The NIC and power control functions are powered by the 5V standby power from the power supply, which is constantly on even when the unit is off. Turning the switch on the back off and on or unplugging and plugging in will re-start the standby power regulator. It may take some time for the capacitors to dischage for a full reset, too -- whihc you get when you physically move it. So, the problem may be in the power supply. Not really enough evidence yet to suggest you replace it, but keep it in mind in case it gets worse. Next time, try just switching it off from the back, waiting a couple minutes, and re-powering and observe what the NIC LED's do during the process.
This is an excellent point. Great thought. I'm guessing that replacing the PSU on one of these is probably not terribly difficult (and likely easier than finding the offending caps or other component).
Lastly, it sounds like MaxxMark and I are having very similar problems. I believe I've tried to let it sit in a soft-off state (PSU on, soft-switch on the front powered off) for several minutes in the past, and still been unable to get it to boot. However, in addition to the NIC LED check I mentioned above, I will explicitly try this the next time I have an issue.
Thanks everyone! I'm gaining optimism that we can find a path through this! Yay!
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
Ok, I had the non-responsive state reoccur today (well, I've been out-of-town for 10 days, so it happened sometime in the past 10 days). Here's what I noticed:
The front display was dark and unresponsive this time. Pressing the power button did not make anything appear on the screen.
The lights on the primary NIC were lit (link was solid, and activity was blinking as if it was a-ok).
The lights on the secondary NIC performed as expected when I plugged it in (solid link, blinking activity).
There was NO response on either IP. Could not get an ARP reply, nor ICMP or HTTP(S) traffic.
I was able to reboot the system to operations by holding the front-panel power button for ~4s, then immediately restarting the unit. No moving, no power supply switch cycling necessary.
So I was able to verify that the NIC lights were on, but unable to verify that switching off the power supply would fix the issues (as a soft-power cycle was adequate).
Does the NIC light state lead more or less credence to the swapping the PSU being a potential solution?
I've had some UPS brownout issues (but nothing in the logs since the last time I cycled the power), that could be a culprit. Also, when I first saw this problem, I uninstalled a couple services off the ReadyNAS, and recently reinstalled Plex. I don't think that would have any relation, but figure it is worth mentioning as potentially correlated issues.
Suggestions appreciated. I'd hate to abandon this hardware; am quite a big fan.
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
Status of the NIC LEDs seems to indicate the +5V standby power is OK. But if you can't even press and hold the power button to power down, that still points to that being a possibility. If the problem happened more often, doding a temporary replacement with a standard ATX supply externally would be a next good step, but I don't really like to recommend you do that and let it run for 2 weeks to see if it's fixed.
At this pint, its a crapshoot. If you get another supply and that's not the issue, you likely won't replace the unit with one that uses a similar supply where you would at least have a spare.
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
I think you misread; CJLittle described that holding the power-button DID reboot the system.
His description of this incident describes accurately what I experience as well. When the NAS becomes unresponsive, at first you are able to press the powerbutton which will cycle through the various messages. If you wait a longer amount of time, the display becomes unresponsive as well.
In addition I started monitoring the voltages (which are reported) as well, just in case they might provide some insight. However, at the time of a recent outage, there was nothing out of the ordinary there:
As described there I want to try two more things; 1 is running sar (which is like top, only with the ability to redirect it to a file) which can run along. When the machine halts, the file should contain everything up untill the moment the system could not write to filesystem anymore (which hopefully is later than when the network gives up and might give a clue).
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
I just wanna chime in with the exact same problem.
I have aquired an old ReadyNAS Pro (6 bay unit) and I have upgraded it to OS 6.6.1 succesfully. Everything was working fine untill this morning when I noticed it was offiline. No response to pings, nothing. I didn't even get the chance to reboot it yet..
It's a very old model : RNDP6350-100NAS
I think this one only has 1GB of RAM and i'm strongly considering upgrading it..
Have you guys figured out what causes these units to die like that ? or any clues as to what triggers it ? I really like OS6 over the old 4.x so I much rather keep it as is..
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
I have more than one 6-bay Pro runing OS6.6.1, and they all have 2GB or better of RAM. But my Pro2 has but 1GB and works fine, so I'm not sure if it makes a difference. But putting in some more RAM could be worth a try.
Re: ReadyNAS Pro becomes unresponsive; will not reboot until physically moved
I am in a similar situation. I have an Ultra 6 running 6.6.1. I run CrashPlan and after a few days I would get a note that my backup had not completed. At that point I would have complete access to the NAS. I would SSH into and found that DNS would not resolve (See Other Thread). At that point I could reboot from the GUI or the front power button and everything would be fine for a few days. What I found was that if I left the NAS after I saw that DNS would not resolve it would hang completly sometime in the next 12 - 24 hours. I also have the Anti-Virus disabled.
