ReadyNAS Pro 6 has become unstable and regularly hangs

[budda1]

My ReadyNAS Pro has started hanging after running fine for many years. 

Currently running firmware 6.9.2 but have had the same problem with the last few versions released.

I've tried disabling the four apps I have running, as well as turning off features such as the scheduled snapshotting.

But still, since around November 2017, I've found my ReadyNAS will just freeze and needs the power button holding down to shut it off and then press again to start it up.

What sort of things can cause this hanging in the ReadyNAS Pro and what logs can I check to look for any clues?

[StephenB]

Start by downloading the log zip file, and look in volume.log.

Look at the SMART stats for the disks and make sure those look healthy.

Then scroll down and look at the space usage in /dev/md0.  That is normally around 20%.  Let us know what you are seeing there.

[budda1]

Thanks for the pointers Stephen.

volume.log shows all disk SMART info as passed/healthy, and no errors or retry counts for any of the disks. I don't claim to understand what most of the info in the volume.log is about.

I'm using RAID5 if that is of importance, using 4 disks.

=== df -h ===
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  717M  3.0G  20% /

[StephenB]

What services do you have enabled?

[budda1]

• Running (green light) services are SMB, AFP, UPnP, SSH, HTTPS, HTTP

Other info may be of use...

• Apps i have Plex, CouchPotato, Sickbeard, uTorrent. I've also turned them all off to rule them out, and the ReadyNAS has still done the same freezing after a few days/week.

• I have no backups configured.
• Bonded network carts on round-robin.
• All shares have snapshot turned off as I first thought that might be triggering it.
• CPU running at an average 43 C

[StephenB]

---

@budda1 wrote:

• Bonded network carts on round-robin.

---

Do you have a static LAG set up on the switch?  Round-robin requires that.

[budda1]

No, I've not got LAG configured on the switch ports.

Thinking about it -- around the time this ReadyNAS issue probably started happening Netgear factory reset the GS748Tv5 switch whilst we were trying to diagnose a different problem last year.

So I've added the two switch ports the ReadyNAS is plugged in to on the LAG table:

Could this be a reason for the ReadyNAS locking up randomly after a few days?

[mdgm-ntgr]

---

@budda1 wrote:

Could this be a reason for the ReadyNAS locking up randomly after a few days?

---

Quite possible. I guess we'll know more in a few days time.

[StephenB]

---

@budda1 wrote:

No, I've not got LAG configured on the switch ports.

Thinking about it -- around the time this ReadyNAS issue probably started happening Netgear factory reset the GS748Tv5 switch whilst we were trying to diagnose a different problem last year.

So I've added the two switch ports the ReadyNAS is plugged in to on the LAG table:

Could this be a reason for the ReadyNAS locking up randomly after a few days?

---

It could be.  Round-robin really needs one IP address to work properly.  If that isn't used, the IP connections might sometimes time out, which could look like a lockup.  Please do report back if the problem happens with the LAGs in place (or if it runs a week or two without any issues).

FWIW, I think Netgear should change the name of that mode to include "static LAG".

[budda1]

When my ReadyNAS is hung even tapping the power button to wake up the display of disk usage and installed disks doesn't work.

[StephenB]

---

@budda1 wrote:

When my ReadyNAS is hung even tapping the power button to wake up the display of disk usage and installed disks doesn't work.

---

That sounds like more than a networking issue, but maybe give it a couple of days just to confirm.

[budda1]

It has been a few days, and the ReadyNAS stayed up longer than expected. But it's died again. I've not been up into the loft to reboot it yet so not sure of the exact date and time it was last writing to logs.

[budda1]

i've upgraded to firmware 6.9.3 in the off chance to helps stability in some way.

[budda1]

Really still struggling with the stability of the ReadyNAS. It still regularly falls offline and needs a physical power cycle to bring it back online - sometimes for as short as two days!

Is there any other advice/suggestions on what I could try to fathom what a cause could be? I'm really stumped on what to check. Happy to SSH on to the box to gather data.

[StephenB]

Maybe ask @mdgm-ntgr if he is willing to analyze your logs?

[budda1]

Today I've un-bonded my two ethernet cards and disabled the second one to see if that helps narrow down the cause of the box becoming unresponsive.

I also disabled UPnP service from the admin settings page.

[mdgm-ntgr]

You can send in your logs (see the Sending Logs link in my sig)

Have you looked at e.g. systemd-journal.log and kernel.log around when you boot it up again after it hangs (these logs have timestamped entries)? Do errors show in the logs?

[budda1]

I've sent the logs by email as per the described process.

systemd-journal.log

The last stuff in the log before the "-- Reboot --" is a cron task

Jun 09 03:17:01 nas-01-ng CRON[15009]: pam_unix(cron:session): session opened for user root by (uid=0)
Jun 09 03:17:01 nas-01-ng CRON[15013]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jun 09 03:17:01 nas-01-ng CRON[15009]: pam_unix(cron:session): session closed for user root

I have nothing in the /etc/cron.hourly/ directory.

kernel.log

I couldn't see any errors just before the "--Reboot--" message.

[mdgm-ntgr]

Not sure what's causing this.

Have you tried running the memory test boot menu option?

[budda1]

I have not tried the memory test, I didn't know it was possible - but have just read the KB article on how to do it so will give it a go.

[budda1]

How long does a memory test usually take? I've had it running for a number of days now. I took a quick screenshot of the ReadyNAS LCD yesterday.

Do I just leave it to crack on until I get to 100% prog or does it loop around and start again?

[mdgm-ntgr]

On the Pro it'll keep on running additional passes till you stop it. You could have stopped it after running just two or three passes, I think.

[budda1]

Ok, so it passed okay enough times then. I'll stop it in the morning as the ReadyNAS lives in the loft out of the way.

Where does that leave me with regards to debugging the regular crashing?

[850t5]

As I have the exact same issue since some months, but my /dev/md0 shows more than 20%  I wonder if that is giving a hint and possible action to try?

Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        4.0G  1.3G  2.4G  36% /

[budda1]

What’s special about the md0 mount? I’m not familiar with it’s relevance and the 20% ?