ReadyNAS104 - Device is Offline - v6.2.4

[Stoffen]

Hi.
I need some assistance in troubleshooting my RN104.
For some time now, the webgui has been unavailable for all purposes.
It states the RN104 is Offline.
RaidAR for windows finds the device, and I do have access to the apache2 sites that are configured.
I can also access it with SSH.

Output from /var/log/readynas-agent.log

Error processing nml DiskEnclosure request
Error processing nml volume request
Warning: Failed to connect to the agentx master agent ([NIL]): 
readynas-agent is up and running.
NET-SNMP version 5.4.3 AgentX subagent connected
Error processing nml DiskEnclosure request
Error processing nml volume request
Warning: Failed to connect to the agentx master agent ([NIL]): 
readynas-agent is up and running.
NET-SNMP version 5.4.3 AgentX subagent connected
readynas-agent is quitting on signal 15.

Error processing nml DiskEnclosure request
Error processing nml volume request
Warning: Failed to connect to the agentx master agent ([NIL]): 
readynas-agent is up and running.
NET-SNMP version 5.4.3 AgentX subagent connected
broken pipe
AgentX master agent failed to respond to ping.  Attempting to re-register.
broken pipe
NET-SNMP version 5.4.3 AgentX subagent connected
broken pipe
AgentX master agent failed to respond to ping.  Attempting to re-register.
broken pipe
NET-SNMP version 5.4.3 AgentX subagent connected
broken pipe
AgentX master agent failed to respond to ping.  Attempting to re-register.
broken pipe
NET-SNMP version 5.4.3 AgentX subagent connected
broken pipe
AgentX master agent failed to respond to ping.  Attempting to re-register.
broken pipe
NET-SNMP version 5.4.3 AgentX subagent connected

At some point, I had a firewall sending syslog to the NAS, and I thought that could be part of the problem, as it filled the /var/log/messages file.
I stopped the syslogging on the device, and removed the 'messages' file. I then created a new messages file and gave it the same permissions as the old one.
But not a single log event has been written to that file....

I have of course booted the device, and done the apt-get update / upgrade.
Anyone with any good tips on how to troubleshoot this?

Thanks!

--Christoffer

[mdgm-ntgr]

We don't recommend doing an

# apt-get upgrade

while we do try to pin packages you may update something that you shouldn't.

What is the output of the following?:

# btrfs fi show
# df -h | grep "Filesystem\|md0"
# df -i | grep "Filesystem\|md0"

[Stoffen]

Hi.
Here are the output from the requested commands:

root@NAS01:~# btrfs fi show
Label: '0e351bf8:data'  uuid: 79550d93-8e67-4f2f-9cb9-f3f3d0048b30
        Total devices 1 FS bytes used 715.54GiB
        devid    1 size 3.63TiB used 726.04GiB path /dev/md127

Btrfs v3.17.3
root@NAS01:~# df -h | grep "Filesystem\|md0"
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        4.0G  2.2G  1.7G  57% /

root@NAS01:~# df -i | grep "Filesystem\|md0"
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/md0        65536 26634 38902   41% /

[mdgm-ntgr]

Do you have a backup of your data?

What's the output of the following?:

# journalctl -a | grep readynasd | tail -n 20
# journalctl -a | grep apache2 | tail -n 20

[Stoffen]

Hi.
Here are the output.

I have backup of the most important data, but if possible I would prefer to fix the issue without factory defaults / dataloss...
If possible...

I am in a moving phase as well at the moment, so the backup is stored in a location that will not be available for a few weeks.

root@NAS01:~# journalctl -a | grep readynasd | tail -n 20
Jun 13 16:00:51 NAS01 readynasd[1917]: Snapshot 'c_1434204051' is successfully created for share or LUN 'istat'.
Jun 13 16:00:52 NAS01 readynasd[1917]: Snapshot 'c_1434204052' is successfully created for share or LUN 'vpnserver'.
Jun 13 16:37:36 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__16_00_04' from share or LUN 'ddclient'.
Jun 13 16:38:02 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__16_00_05' from share or LUN 'istat'.
Jun 13 16:38:41 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__16_00_06' from share or LUN 'vpnserver'.
Jun 13 17:00:55 NAS01 readynasd[1917]: Snapshot 'c_1434207654' is successfully created for share or LUN 'ddclient'.
Jun 13 17:00:56 NAS01 readynasd[1917]: Snapshot 'c_1434207656' is successfully created for share or LUN 'istat'.
Jun 13 17:00:57 NAS01 readynasd[1917]: Snapshot 'c_1434207657' is successfully created for share or LUN 'vpnserver'.
Jun 13 17:39:08 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__17_00_09' from share or LUN 'ddclient'.
Jun 13 17:39:33 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__17_00_10' from share or LUN 'istat'.
Jun 13 17:40:14 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__17_00_11' from share or LUN 'vpnserver'.
Jun 13 18:01:00 NAS01 readynasd[1917]: Snapshot 'c_1434211259' is successfully created for share or LUN 'ddclient'.
Jun 13 18:01:01 NAS01 readynasd[1917]: Snapshot 'c_1434211261' is successfully created for share or LUN 'istat'.
Jun 13 18:01:02 NAS01 readynasd[1917]: Snapshot 'c_1434211262' is successfully created for share or LUN 'vpnserver'.
Jun 13 18:40:42 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__18_00_14' from share or LUN 'ddclient'.
Jun 13 18:41:07 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__18_00_15' from share or LUN 'istat'.
Jun 13 18:41:46 NAS01 readynasd[1917]: Smart Snapshot successfully delete snapshot '2015_06_11__18_00_16' from share or LUN 'vpnserver'.
Jun 13 19:00:05 NAS01 readynasd[1917]: Snapshot 'c_1434214804' is successfully created for share or LUN 'ddclient'.
Jun 13 19:00:06 NAS01 readynasd[1917]: Snapshot 'c_1434214806' is successfully created for share or LUN 'istat'.
Jun 13 19:00:07 NAS01 readynasd[1917]: Snapshot 'c_1434214807' is successfully created for share or LUN 'vpnserver'.
root@NAS01:~# journalctl -a | grep apache2 | tail -n 20
root@NAS01:~# 

[mdgm-ntgr]

# systemctl status -a apache2

[Stoffen]

Here it is

root@NAS01:~# systemctl status -a apache2
apache2.service - The Apache HTTP Server
          Loaded: loaded (/lib/systemd/system/apache2.service; enabled)
          Active: active (running) since Sun, 14 Jun 2015 15:46:20 +0200; 1 weeks and 0 days ago
         Process: 6271 ExecReload=/bin/kill -USR1 $MAINPID (code=exited, status=0/SUCCESS)
         Process: 6257 ExecReload=/usr/sbin/apache2 $OPTIONS -t (code=exited, status=0/SUCCESS)
        Main PID: 1857 (apache2)
          CGroup: name=systemd:/system/apache2.service
                  ├   609 /usr/sbin/apache2 -k start
                  ├  1857 /usr/sbin/apache2 -k start
                  ├  6319 apache_log
                  ├  6339 /usr/sbin/apache2 -k start
                  ├  6342 /usr/sbin/apache2 -k start
                  ├ 11383 /usr/sbin/apache2 -k start
                  ├ 11384 /usr/sbin/apache2 -k start
                  ├ 11389 /usr/sbin/apache2 -k start
                  ├ 11418 /usr/sbin/apache2 -k start
                  ├ 12488 /usr/sbin/apache2 -k start
                  ├ 13092 /usr/sbin/apache2 -k start
                  ├ 22975 /usr/sbin/apache2 -k start
                  â”” 23742 /usr/sbin/apache2 -k start

Jun 19 13:18:59 NAS01 apache_access[6319]: Suppressed 1113 duplicate messages
Jun 19 13:18:59 NAS01 apache_access[6319]: 192.168.1.73 "POST /dbbroker HTTP/1.1" 200
Jun 19 14:19:00 NAS01 apache_access[6319]: Suppressed 1113 duplicate messages
Jun 19 14:19:00 NAS01 apache_access[6319]: 192.168.1.73 "POST /dbbroker HTTP/1.1" 200
Jun 19 15:19:00 NAS01 apache_access[6319]: Suppressed 1114 duplicate messages
Jun 19 15:19:00 NAS01 apache_access[6319]: 192.168.1.73 "POST /dbbroker HTTP/1.1" 200
Jun 21 11:00:51 NAS01 apache_access[6319]: 192.168.1.73 "POST /dbbroker HTTP/1.1" 200
Jun 21 21:04:11 NAS01 apache_access[6319]: Suppressed 208 duplicate messages
Jun 21 21:04:11 NAS01 apache_access[6319]: 192.168.1.73 "POST /dbbroker HTTP/1.1" 200
root@NAS01:~#