Troubleshooting instability/unresponsive NAS

[alaeth]

I need some help in troubleshooting an access/instability problem I've started having.

Up until recently, my Readytnas Pro (running 6.6.1) has been perfectly stable.  But I've had to manually power it down twice in the past 24 hours.

Each time the symptoms are the same:

* unable to launch Plex (we "cut the cord" about a year ago, so 100% on Plex and Netflix)\

* unable to ssh in

* RAIDar cannot detect it

* no errors or indicators on the OLED display

(holding the power button forces it to power off).

* apon reboot, tail of the journalctl shows nothing... except a huge gap from several hours before until the message "-- Reboot --"

Any help on where to look closer would be greatly apprecialted.

The last entries in the journalctl look perfectly normal to me:

Mar 03 01:41:20 readynas01 proftpd[28379]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - Login timeout exceeded, disconnected
Mar 03 01:41:20 readynas01 proftpd[28379]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - FTP session closed.
Mar 03 01:46:20 readynas01 proftpd[28765]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - FTP session opened.
Mar 03 01:47:24 readynas01 proftpd[28765]: pam_unix(ftp:session): session opened for user tomato by (uid=0)
Mar 03 01:47:24 readynas01 proftpd[28765]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - USER tomato: Login successful.
Mar 03 01:56:26 readynas01 proftpd[29159]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - FTP session opened.
Mar 03 01:57:25 readynas01 proftpd[28765]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - Passive data transfer failed, possibly due to network issues
Mar 03 01:57:25 readynas01 proftpd[28765]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - Check your PassivePorts and MasqueradeAddress settings,
Mar 03 01:57:25 readynas01 proftpd[28765]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - and any router, NAT, and firewall rules in the network path.
Mar 03 01:57:25 readynas01 proftpd[28765]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - FTP no transfer timeout, disconnected
Mar 03 01:57:25 readynas01 proftpd[28765]: pam_unix(ftp:session): session closed for user tomato
Mar 03 01:57:25 readynas01 proftpd[28765]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - FTP session closed.
Mar 03 02:01:27 readynas01 proftpd[29159]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - Login timeout exceeded, disconnected
Mar 03 02:01:27 readynas01 proftpd[29159]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - FTP session closed.
Mar 03 02:06:26 readynas01 proftpd[29527]: 127.0.0.2 (192.168.0.1[192.168.0.1]) - FTP session opened.

[mdgm-ntgr]

Perhaps try ReadyNASOS 6.7.0-T180 (Beta 3)

[Jose00]

Similar issue here, went away for a few days and came back the NAS essentially wasn't running. Rebooted twice and finally got to the admin screen and uploaded the new Beta 3 firmware but its been sitting here installing and waiting to reboot for 30 old mins. Will advise if Beta 3 works (when it installs).

[mdgm-ntgr]

30 minutes? It shouldn't take that long. You may wish to reboot it.

[Jose00]

Yeah something has gone wrong, essentially its stuck at around 96%-99%, I left it overnight and tried several times today. Now attempting USB recovery and will try boot menu OS reinstall next as doesn't look like the USB recovery has worked either.

[Jose00]

Sorry for hijacking this thread, I finally got the NAS314 booting. I removed 3 of the 4 drives and it came back to life with Beta 3 installed. I'm guessing that the USB recovery may have done something, the Boot Menu would only let me get to 'Boot Menu' on the LCD but I couldn't do anything further (pressing backup wouldn't select any other options).

However I have now lost my Volume. I had 4x 4TB drives in there setup as JBOD. I haven't done anything but have I lost all my data?

[mdgm-ntgr]

So your backup is not up to date?

[Jose00]

I decided to not have a backup as I wanted the space.. in theory it is data that can be replaced.

Looks like the volume has gone and I've got disks that are showing up as GPT and raw disks. Currently running EaseUS Data Recovery on a disk as that is actually finding files but will be a mess to clean up. 

Unless you can recommend another way to get access to the data or rebuild the volume without losing everything? I've tried a Debian image in a VMware player but that didn't really work, I've only got Windows PCs so linux is a bit over my head but can get up to speed if needed.

[alaeth]

That sucks Jose00...

Part of the reason I posted is I wanted to try less intrusive troubleshooting means first is I didn't want to risk 6 x 3TB data since It's annoying to have a second on-site backup (and cloud backups take forever to resync).

[YeZ]

Jose00, will PM you to see if we can take a look at your NAS to re-mount the drive. 

[atz6975]

Hi,

same problem here.

I couldn't uninstall transmission app  (nastools version and it wouldn't work anymore) and I read in forum that 6.7 had a fix.

Indeed I removed the app, and reinstalled the one from poussin on my 6.7 T180.

Unfortunately, the Nas is now unresponsive and no USB boot is possible (tried 6 keys), Boot Menu is stuck at "Boot Menu"....

SSH seams to work but after a while all hangs and system is stuck at 96% (or close ).

I don't want to loose the volume...

Thanks for any recommandation.

[atz6975]

I post a quick reply to say that I'm running this on a Readynas314.

I have a spare unit Pro4 running 661.

Thanks for any help.

[Skywalker]

@atz6975, how long does SSH work before you get disconnected? Try running `journalctl -af` from your SSH terminal and see if it ouputs anything interesting.

[Jose00]

Hi atz6975, I found I could boot once I removed one of the 4 drives I have in my NAS (the 4th drive). I was getting stuck around 96-99%. For some reason that was stopping the boot process. Netgear are currently looking at my volume for me, I'd recommend trying to remove a drive and if you can boot in they may be able to fix your volume if its gone.

[mdgm-ntgr]

I don't think removing a drives is a good step to take without expert advice. Of course it all depends on your appetite for risk and how confident you are you have sufficient backups.

[YeZ]

We suspect that there is a SATA port issue on your unit, please RMA the unit if it's still under warranty. Thank you.

[alaeth]

Bit of a side-stream discussion going on here.  heh.

What I've discovered so far on my issue:

* setting the power down settings daily (effectively rebooting the NAS every night at 2am) has eliminated the problem

So I've disabled that again, and installed the Splunk Universal Forwarder.  Configuring all the logs that might be useful:

/var/log
/data/applications/SickRage/Logs
/data/applications/CouchPotato/Logs
/apps/plexmediaserver/MediaLibrary/Plex Media Server/Logs

My hope is to capture "something" relevant/useful immediately before it becomes unresponsive.

Question for the experts:

How do I force the chart data to write to disk (and reload on boot)?  The graphs only seem to have data as old as the last reboot.

[atz6975]

Hi Yez,

thank you for your input.

I just created support case #28178428.

So far, no RMA is planed because there is no evidence supporting it.

While the support was very friendly, it seems that the fact that I used a beta is a show stopper for any RMA (Netgear France is very RMA resistant, I've experienced it many times over).

As for other suggestions, I have no intention to remove a drive, because I have no backup of my data.

The unit can be sshd into for 30min after that it is less responsive.

My "prime suspect" is that Transmission module ( I used the one from Poussin) doesn't startup correctly.

Maybe we can "remove" it through SSH? Is there a way to clean the 6.7 Init via SSH?

As for the RMA process, maybe you can get in touch with French support to explain the reasons?

Thank you very much.

[Skywalker]

@atz6975, Please try running `journalctl -af` from your SSH terminal and see if it ouputs anything interesting.