Readynas instability - power button does nothing.. weirdness

[netmask]

Greetings,

I've been using my NV+ for a couple of years now and often participate in the betas. I came back from vacation today and the Readynas is being very strange.

I can't access the services on it (file shares, web interface, etc).. The TCp socket opens to them, but can't ever actually pass data.

When I ping the unit, every 4th ping spikes to 500ms+, the other 3 being sub-1ms. can't SSH into it, can't web into it.

What's more strange, the power button on the front isn't doing anything either.. usually holding it down would cause a shut down. If I tap it or hold it, there's no response.. the panel doesn't light up, nothing.

When I hard power cycle it, I'm able to SSH in for a little while.

It did recently start complaining of ATA errors on disk 4, but in the 2 years i've had it.. i've replaced 4 seagate 1TB drives already via RMA, never while a drive was failed did that cause the OS to be unstable.

Not really sure what else to do, it seems like the OS itself is partially puking.. to a point the TCp stack is partially up, but perhaps the applications are locking up.

I did notice it was running at 100mb on this Netgear switch, i swapped the cable and it's at gig now.. but nothing else changed. (I recently moved and don't have the wiring finished the house, so I'm using crappy netgear equipment until I can get the new cat6 run to the cabinet that has my 3750s in it)

64 bytes from 192.168.1.200: icmp_seq=5180 ttl=64 time=493 ms
64 bytes from 192.168.1.200: icmp_seq=5181 ttl=64 time=0.274 ms
64 bytes from 192.168.1.200: icmp_seq=5182 ttl=64 time=0.342 ms
64 bytes from 192.168.1.200: icmp_seq=5183 ttl=64 time=0.277 ms
64 bytes from 192.168.1.200: icmp_seq=5184 ttl=64 time=305 ms

Any ideas what could cause the power button to stop working, but you can get ping respones, but every 4th packet spikes it latency.. and tcp services are only 1/2 functional?

I don't know exactly what version it's running, but I checked for latest update 2 weeks ago, so I imagine it's current.

[netmask]

also to note.. after a hard power cycle.. pings are normal.. always sub 0.5ms. then after about an hour, goes back to being wonky.

Raidiator 4.1.7

speed/duplex auto
mtu 1500, jumbo frames off

(just lowered mtu to 1400 for the hell of it.. which it backgrounded as a change, took it a few minutes before it actually made the change, i'd guess behind the raid sync)

[netmask]

was able to at least get a process list off of it this time before SSH locked up on it:

internet:/var/log/frontview# ps auxww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.3 2000 880 ? Ss 20:31 0:05 init [3]
root 2 0.0 0.0 0 0 ? SN 20:31 0:00 [ksoftirqd/0]
root 3 0.0 0.0 0 0 ? S< 20:31 0:00 [events/0]
root 4 0.0 0.0 0 0 ? S< 20:31 0:00 [khelper]
root 5 0.0 0.0 0 0 ? S< 20:31 0:00 [kthread]
root 10 0.0 0.0 0 0 ? S< 20:31 0:00 [kblockd/0]
root 13 0.0 0.0 0 0 ? S< 20:31 0:00 [khubd]
root 41 0.0 0.0 0 0 ? S 20:31 0:00 [pdflush]
root 42 0.0 0.0 0 0 ? S 20:31 0:00 [pdflush]
root 43 0.0 0.0 0 0 ? S 20:31 0:01 [kswapd0]
root 44 0.0 0.0 0 0 ? S< 20:31 0:00 [aio/0]
root 45 0.0 0.0 0 0 ? S< 20:31 0:00 [cifsoplockd]
root 46 0.0 0.0 0 0 ? S< 20:31 0:00 [cifsdnotifyd]
root 92 0.0 0.0 0 0 ? S< 20:31 0:00 [kvblade]
root 93 0.0 0.0 0 0 ? S 20:31 0:00 [mtdblockd]
root 106 0.0 0.0 0 0 ? S 20:31 0:00 [hotplug-sata]
root 131 0.0 0.0 0 0 ? S 20:32 0:01 [djsyncd]
root 132 0.0 0.0 0 0 ? S 20:32 0:00 [djcheckd]
root 135 0.0 0.0 0 0 ? S 20:32 0:00 [hotplug-gmac]
root 367 0.0 0.0 0 0 ? S< 20:32 0:00 [kjournald]
root 523 0.0 0.7 4992 1760 ? Ss 20:32 0:00 /usr/sbin/sshd
root 668 0.0 0.0 0 0 ? S< 20:32 0:00 [kjournald]
root 846 0.7 7.2 21520 16448 ? Ss 20:33 0:23 /usr/sbin/apache-ssl -f /etc/frontview/apache/httpd.conf
daemon 1351 0.0 0.2 2240 624 ? Ss 20:38 0:00 /sbin/portmap
root 1356 0.0 0.4 2128 960 ? Ds 20:38 0:00 /sbin/syslogd -m 0
root 1369 0.0 0.3 2016 720 ? Ss 20:38 0:00 /sbin/klogd -x -c 3
daemon 1371 0.0 0.3 2480 864 ? Ss 20:38 0:00 /usr/sbin/atd
root 1373 0.0 0.3 2000 704 ? Ss 20:38 0:00 /usr/sbin/inetd
root 1389 0.0 0.5 2656 1264 ? Ss 20:38 0:00 /usr/sbin/cron
root 1397 0.0 0.7 4672 1792 ? Ds 20:38 0:00 /usr/sbin/cupsd
admin 1404 0.0 0.8 3648 1840 ? Ss 20:38 0:01 avahi-daemon: running [internet.local]
nobody 1427 0.0 1.8 10016 4272 ? SLs 20:38 0:00 proftpd: (accepting connections)
root 1621 0.0 1.9 10160 4368 ? SNs 20:39 0:00 /usr/sbin/smbd -D
root 1638 0.0 1.4 8512 3328 ? Ss 20:39 0:02 sshd: root@pts/0
root 1652 0.0 1.5 10160 3456 ? SN 20:39 0:00 /usr/sbin/smbd -D
root 1711 0.0 0.4 2240 912 ? S 20:39 0:00 /usr/sbin/cnid_metad
root 1720 0.0 1.0 7296 2448 ? D 20:39 0:00 /usr/sbin/afpd -U uams_dhx.so,uams_clrtxt.so,uams_guest.so,uams_guest.so -c 50 -n internet
root 1726 0.0 1.1 5712 2496 pts/0 Ss 20:39 0:01 -bash
root 1743 0.3 2.1 10496 4896 ? SN 20:39 0:10 /usr/sbin/smbd -D
root 1828 0.0 0.4 2144 960 ? Ss 20:39 0:00 /usr/sbin/raidard
root 1853 0.0 0.2 2080 672 ? S 20:39 0:00 /usr/sbin/lld2d eth0
root 1854 0.0 0.9 16464 2224 ? Ss 20:39 0:00 /usr/sbin/minidlna
root 1855 0.0 0.9 16464 2224 ? S 20:39 0:00 /usr/sbin/minidlna
root 1867 0.0 0.3 2000 864 ? Ss 20:39 0:00 /usr/sbin/upnpd eth0
root 1872 0.0 0.9 16464 2224 ? SN 20:39 0:00 /usr/sbin/minidlna
root 1896 0.0 1.1 6880 2544 ? Ds 20:39 0:00 nmbd -D
root 1897 0.0 1.0 7008 2320 ? S 20:39 0:00 nmbd -D
admin 1908 0.5 9.6 27472 21744 ? S 20:39 0:15 /usr/sbin/apache-ssl -f /etc/frontview/apache/httpd.conf
admin 1944 0.2 9.1 26592 20816 ? S 20:40 0:06 /usr/sbin/apache-ssl -f /etc/frontview/apache/httpd.conf
root 2005 0.0 0.5 2400 1184 ? Ss 20:40 0:02 /frontview/bin/monitor_enclosure
root 2128 0.0 0.3 2016 896 ttyS1 Ss+ 20:40 0:00 /sbin/getty -L ttyS1 9600 vt100
ReadyNAS 2547 91.5 1.6 12864 3840 ? R 21:18 5:49 /usr/sbin/afpd -U uams_dhx.so,uams_clrtxt.so,uams_guest.so,uams_guest.so -c 50 -n internet
root 2548 0.3 1.5 15296 3456 ? S 21:18 0:01 /usr/sbin/cnid_dbd /c/.timemachine/.AppleDB 4 1 default log_note
root 2553 0.0 0.9 3904 2128 ? D 21:19 0:00 /bin/bash /etc/hotplug/sata.agent
root 2568 0.0 0.6 6880 1456 ? D 21:19 0:00 nmbd -D
root 2572 0.0 0.9 3904 2128 ? D 21:19 0:00 /bin/bash /etc/hotplug/sata.agent
root 2590 0.0 0.9 3904 2128 ? D 21:19 0:00 /bin/bash /etc/hotplug/sata.agent
root 2605 0.0 0.6 6880 1456 ? D 21:19 0:00 nmbd -D
root 2606 0.0 0.4 7296 1104 ? D 21:19 0:00 /usr/sbin/afpd -U uams_dhx.so,uams_clrtxt.so,uams_guest.so,uams_guest.so -c 50 -n internet
root 2610 0.1 0.9 3904 2128 ? D 21:19 0:00 /bin/bash /etc/hotplug/sata.agent
root 2625 0.0 0.6 6880 1440 ? D 21:19 0:00 nmbd -D
root 2626 0.0 1.8 10464 4112 ? DN 21:20 0:00 /usr/sbin/smbd -D
root 2627 0.0 0.6 6880 1456 ? D 21:20 0:00 nmbd -D
root 2628 0.0 0.6 6880 1440 ? D 21:20 0:00 nmbd -D
root 2629 0.0 1.8 10464 4176 ? DN 21:20 0:00 /usr/sbin/smbd -D
root 2630 0.0 0.6 6880 1456 ? D 21:20 0:00 nmbd -D
root 2631 0.0 1.8 10464 4176 ? DN 21:20 0:00 /usr/sbin/smbd -D
root 2632 0.0 1.8 10464 4112 ? DN 21:20 0:00 /usr/sbin/smbd -D
root 2634 0.0 0.4 7296 1104 ? D 21:21 0:00 /usr/sbin/afpd -U uams_dhx.so,uams_clrtxt.so,uams_guest.so,uams_guest.so -c 50 -n internet
root 2635 0.0 0.6 6880 1456 ? D 21:21 0:00 nmbd -D
root 2636 0.0 1.8 10464 4112 ? DN 21:21 0:00 /usr/sbin/smbd -D
root 2637 0.0 0.8 3488 1920 ? S 21:22 0:00 sh -c grep -i trapcommunity /etc/snmp/snmpd.conf | awk '{ print $2 }' > /var/log/frontview/trapComm.info
root 2639 0.0 0.4 3488 1072 ? D 21:22 0:00 sh -c grep -i trapcommunity /etc/snmp/snmpd.conf | awk '{ print $2 }' > /var/log/frontview/trapComm.info
root 2640 0.0 1.8 10464 4112 ? DN 21:22 0:00 /usr/sbin/smbd -D
root 2641 0.0 0.5 3120 1312 ? S 21:23 0:00 /USR/SBIN/CRON
root 2642 0.1 1.6 10448 3728 ? SN 21:23 0:00 /usr/sbin/smbd -D
root 2643 0.2 1.0 4672 2384 ? Ss 21:24 0:00 /usr/sbin/sshd -R
root 2644 0.2 1.6 10448 3728 ? SN 21:24 0:00 /usr/sbin/smbd -D
root 2658 0.0 0.6 3360 1456 pts/0 R+ 21:25 0:00 ps auxww


I see 11 smbd processes, 9 nmbd.. smbd after about 40 minutes of running started spawning more and more processes.

4 afpd running..

(I only have 5 shares on this host)

maybe the multiple processes isn't strange, though the start times make them look strange.

i tried to run 'uptime' to check the load, and at that point SSH locked up with no response.

whatever happened, ping went back to the low pings to the every 4th ping being high.

front button is unresponsive again.

this sucks, any ideas?

[mdgm-ntgr]

Have you opened a tech support case?

As you're not using your normal setup perhaps the problem is with some of your other equipment.

Do you still have issues if you do a direct-connection?: http://sphardy.com/web/directconnect

[netmask]

as soon as I popped out 'drive 4' that has the failing errors.. pings immediately went back to normal, and the shutdown i had tried to start from the power button immediately initiated.

How does a failing drive cause the Readynas to have a service outage? thats never happened before.

[netmask]

mdgm wrote:
Have you opened a tech support case? As you're not using your normal setup perhaps the problem is with some of your other equipment. Do you still have issues if you do a direct-connection?: http://sphardy.com/web/directconnect

I hadn't opened a case.. nothing abnormal about my setup really, it appears now that since disk 4 was failing, it somehow was causing the OS to have issues.. as the second i popped out disk 4 it returned to normal, pings went immediately back to low.. i was able to SSH in, and the shutdown i had tried to do from the front panel minutes prior immediately kicked in. (So my SSH that succeeded only lasted about 4 seconds :P).

I've had several disk failures over the years and never had this kind of behavior from the ReadyNAS, very strange.

[mdgm-ntgr]

Drives fail in a variety of different ways. Normally this should be handled gracefully, but occasionally a disk failure can cause problems like this.

Your data should be fine, but if you're not already, I'd suggest you start doing regular backups. There are problems that RAID cannot protect you against. Take a look at Preventing Catastrophic Data Loss.

[netmask]

mdgm wrote:
Drives fail in a variety of different ways. Normally this should be handled gracefully, but occasionally a disk failure can cause problems like this. Your data should be fine, but if you're not already, I'd suggest you start doing regular backups. There are problems that RAID cannot protect you against. Take a look at Preventing Catastrophic Data Loss.

yeah i have daily backups and always have a new drive on standby to be swapped in.. so never worried about data loss, just the downtime to the data.. which has never occurred with any disk failure in the readynas before until this one that seems to have made the system unstable.