Re: ReadyNAS NV (Sparc) OS Partition Full

rickwookie · ‎2016-09-17

Hi

I've been doing a lot of googling and think I'm nearly at the point I can get access back to my NV Frontview.

Basically, NFS started to fail, then CIFS went down, then I got all sorts of odd things happen in frontview. Anyway, I first did the reinstall OS boot option thinking that would help, but then I got corrupt Frontview (did get CIFS access back though), so now I've managed to get in via telnet (I never have enablled rootSSH access), mounted the OS partition like so:

# /bin/start_raid.sh
# mount /dev/hdc1 /sysroot

# /sysroot/bin/df -h
Filesystem            Size Used Avail Use% Mounted on
rootfs                7.7M 6.5M 1.3M 83% /
/dev/root             7.7M 6.5M 1.3M 83% /
tmpfs                  16k     0   16k   0% /dev/shm
/dev/hdc1             1.9G 2.0G   10M 100% /sysroot

# /sysroot/bin/df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
rootfs                  2048     248    1800   13% /
/dev/root               2048     248    1800   13% /
tmpfs                  31583       1   31582    1% /dev/shm
/dev/hdc1             128000    7789 120211    7% /sysroot

# /sysroot/usr/bin/du -csh /sysroot/var/log/*
0       /sysroot/var/log/LeafP2P.log
0       /sysroot/var/log/LeafP2P.log.old
0       /sysroot/var/log/LeafP2P.log.old.old
0       /sysroot/var/log/LeafP2P.log.old.old.old
32k     /sysroot/var/log/LeafP2P.log.old.old.old.old
64k     /sysroot/var/log/auth.log
64k     /sysroot/var/log/cron.log
64k     /sysroot/var/log/cups
80k     /sysroot/var/log/daemon.log
16k     /sysroot/var/log/ddns
32k     /sysroot/var/log/debug
16k     /sysroot/var/log/dmesg
0       /sysroot/var/log/dmesg.old
32k     /sysroot/var/log/exim
720k    /sysroot/var/log/frontview
32k     /sysroot/var/log/fsck.log
3.1M    /sysroot/var/log/kern.log
16k     /sysroot/var/log/ksymoops
0       /sysroot/var/log/lpr.log
0       /sysroot/var/log/mail.log
3.1M    /sysroot/var/log/messages
16k     /sysroot/var/log/netatalk.log
0       /sysroot/var/log/netatalk.log.old
0       /sysroot/var/log/netatalk.log.old.old
0       /sysroot/var/log/netatalk.log.old.old.old
16k     /sysroot/var/log/ntpstats
0       /sysroot/var/log/proftpd.log
0       /sysroot/var/log/proftpd.log.old
0       /sysroot/var/log/proftpd.log.old.old
0       /sysroot/var/log/proftpd.log.old.old.old
0       /sysroot/var/log/proftpd.log.old.old.old.old
0       /sysroot/var/log/proftpd.log.old.old.old.old.old
0       /sysroot/var/log/proftpd.log.old.old.old.old.old.old
0       /sysroot/var/log/proftpd.log.old.old.old.old.old.old.old
0       /sysroot/var/log/proftpd.log.old.old.old.old.old.old.old.old
32k     /sysroot/var/log/proftpd.log.old.old.old.old.old.old.old.old.old
16k     /sysroot/var/log/raidiator_version
0       /sysroot/var/log/raidiator_version.old
192k    /sysroot/var/log/samba
1.7G    /sysroot/var/log/syslog
16k     /sysroot/var/log/user.log
1.7G    total

So, can I safely clean /sysroot/var/log/syslog with the follwing?

echo "" > /sysroot/var/log/syslog

if I should first copy it to the data partition, how to do first mount that?

Many thanks in advance

mdgm-ntgr · ‎2016-09-17

That's the main log. Best to copy it in case it contains something important in case there's something else wrong.

If you're familiar with LVM2 commands such as vgscan and vgchange it's pretty straightforward to mount the data volume (at your own risk of course and assuming the data volume is fine).

StephenB · ‎2016-09-17

The only reason to copy it to the data volume is to archive it for later analysis. It's good to keep it of course, but if you aren't familar with the commands you could get into trouble.

You could just empty it and get the NAS running again. Then regularly download the logs from the web ui over the next days/weeks, and keep an eye on the sizes and also the contents of syslog.

rickwookie · ‎2016-09-17

That's what I thought, but unfortunately I don't have a scooby when it comes to LVM2 commands such as vgscan and vgchange.

I'm pretty sure my data volume is fine.

Any chance you could let me know the commands I would need to mount it?

Thanks again.

rickwookie · ‎2016-09-17

Ok, a bit more googling gets me:

start_raid.sh
mount /dev/hdc1 /sysroot
chroot /sysroot
mount proc
vgscan
vgchange -ay c
mount -a

Obviously I've done the first two lines to get me where I am now, but is the rest of that sequence of commands correct? I'd rather copy the syslog if I can.

rickwookie · ‎2016-09-17

ok I jumped in a did it anyway, all looks good, copied all 1.7 GB with:

cp /var/log/syslog /c/backup/

then did:

echo "" > /var/log/syslog

So last question, how to I cleanly unmount everything before rebooting (and do I need to initiate the reboot from the telnet prompt)?

Thanks.

mdgm-ntgr · ‎2016-09-17

To reboot once you've unmounted everything and run

# vgchange -a n

to deactivate the LVM group you then just need do

# reboot -f

rickwookie · ‎2016-09-17

So what does the

# touch /.os_update

# sync

do?

Is this just to force a resync? Do I need to do this if so, because it will take days if previous experience is anything to go by?

rickwookie · ‎2016-09-17

Hang on mgdm, I've lost your other reply, did you edit/delete it? How to I unmout everything?

rickwookie · ‎2016-09-17

Never mind, figured it out (basically just umount everything previously mounted right?)

So after the reboot, I'm still getting this:

Is this to be expected? Is this because when I did the OS re-install the OS partition was already too full to complete the re-install and now I just need to do it again?

mdgm-ntgr · ‎2016-09-17

Did you have a corrupt /etc/default/services file? If so, when you copied the default one across did you then remember to add the line

APACHESSL=1

i.e.

# echo "APACHESSL=1" >> /sysroot/etc/default/services

Note that it is two > not just one of those. >> appends to the end of a file.

Also have you closed the web admin interface, cleared your web browser cache, then reopened the web admin page again?

rickwookie · ‎2016-09-17

Tbh I have no idea if the /etc/default/services file was corrupt. CIFS certainly came back up, but NFS didn't, so presumably this was using the default. I threrefore didn't do # echo "APACHESSL=1" >> /sysroot/etc/default/services

Anyway, I did the OS re-install again, and this time Frontview has come up fine. Only things I needed to change was password, IP address and the share settings for FTP had been lost. Everything else was as I had it before the initial problem.

Is the 4.1.14 software that I'm now running the latest version for the Sparc units? I thought I was using a higher version previously, though I may be wrong.

StephenB · ‎2016-09-17

The current version is 4.1.15 - I suspect you were running 4.1.14 before.

rickwookie · ‎2016-09-17

http://kb.netgear.com/app/answers/detail/a_id/20684/~/readynas-downloads?cid=wmt_netgear_organic

This shows me 4.1.14, as does clicking "Check for Update".

Where do I go for the latest version (it used to be so much easier on readynas.com)?

StephenB · ‎2016-09-18

Googling "ReadyNAS 4.1.15" works. http://kb.netgear.com/app/answers/detail/a_id/30914/~/raidiator-version-4.1.15-(sparc)

It should be showing up on the update server though.

rickwookie · ‎2016-09-20

Aaaaaaaagh!

I thought I now how a fully working system again, but it appears that I can't remotely write files via FTP.

I think I've narrowed this down to the FTP server on the ReadyNAS ignoring the values I've input for the passive port range, so when an FTP client issues the PASV command, the response if more often than not a port well outside the range I've specified (and opened in firewalls at each location), in this case ports 1024-1074.

This explains why for the last two days most of the files that have been written (they come from a remote IP camera periodically saving still jpegs to the ReadyNAS) have zero byte length while the odd file is sucessfully written (when the random passive port gets lucky and falls in the range opened in the firewalls.

Can anyone help me get the FTP server on the ReadyNAS to correctly use the range specified in Frontview once more? I did download the logs and in services.conf it matches what I have in frontview:

FTP_MODE=user
FTP_PASSIVE_END=1074
FTP_PASSIVE_START=1024
FTP_PORT=21
FTP_UPLOAD_RESUME=1

There are also loads of entries in daemon.log like:

Sep 20 16:29:47 MyNAS proftpd[26292]: MyNAS ((*clientexternalip*)[*clientexternalip*]) - Refused PORT 192,168,254,2,234,158 (address mismatch)

which I'm assuming is only because the FTP client on the IP camera is trying active mode since passive mode is failing?

Is my only option now to use EnableRootSSH and manually configure the ProFTPd config? (I suppose worrying about voiding the warranty on a NAS this old is a little pointless anyway).

StephenB · ‎2016-09-20

Did the IP camera ever work?

@rickwookie wrote:

Sep 20 16:29:47 MyNAS proftpd[26292]: MyNAS ((*clientexternalip*)[*clientexternalip*]) - Refused PORT 192,168,254,2,234,158 (address mismatch)

.

This is port 60062 (234*256+158). It could be an active connection.

Did the NAS IP address change? Can you give any details on the camera's FTP configuration, and/or a more complete connection log?

rickwookie · ‎2016-09-21

It's been working every weekday for the last two months, right up until I did the OS re-install that corrupted Frontview this last weekend. No IP addresses or hostnames have changed at either end and my firewall router configuration is the same.

I've actually gone ahead now and enabled the SSH access, since I figured there was no way this was going to get resolved using Frontview alone, and Telnet-ing in using the support mode was a real pain tbh.

FTP tranfers are now compleating successfully, since I've added

PassivePorts 1024 1074

to /etc/proftpd.conf

The entire file consisted of just:

Include /etc/frontview/proftpd/ftps.conf
Include /etc/frontview/proftpd/User.conf

and neither of those files, nor /etc/frontview/proftpd/Shares.conf that User.conf 'Include's make any reference to PassivePorts or FTP_PASSIVE_START or anything like it.

I would love to know where the link is from the frontview settings to the proftpd config files to make it all work as it should again, but for now at least I've got the FTP server configured to work as I need it.

I strongly suspect that there will be other settings from the file services page that are not making it through to the associated configs, but maybe they are working as expect with the default settings. Certainly Master Browser is suspect since the ReadyNAS doesn't seem to show up under Network in Windows Explorer anymore, but I can browser it fine if I type the hostname or IP address into the address bar.

mdgm-ntgr · ‎2016-09-21

@rickwookie wrote:

Telnet-ing in using the support mode was a real pain tbh.

Well it is called tech support mode for a reason. It's intended for support and it is a low-level diagnostic mode useful for diagnosing much more serious problems than a full OS partition.

rickwookie · ‎2016-09-22

@mdgm wrote:
@rickwookie wrote:
Telnet-ing in using the support mode was a real pain tbh.
Well it is called tech support mode for a reason. It's intended for support and it is a low-level diagnostic mode useful for diagnosing much more serious problems than a full OS partition.

Um, thanks for the heads up?

In the mean time I'll continue to find my own solutions to the problem, although I will confess I'm struggling a little here.

I'd really appreciate it if anyone could direct me where to look for the missing link that I currently have between the Frontview settings for ftp and the active proftpd configuration.

So far I've found what appears to be the default config in /etc/default/ but I still can work out why changes in the Frontview web UI are being ignored.

StephenB · ‎2016-09-22

@rickwookie wrote:

The entire file consisted of just:

Include /etc/frontview/proftpd/ftps.conf
Include /etc/frontview/proftpd/User.conf

and neither of those files, nor /etc/frontview/proftpd/Shares.conf that User.conf 'Include's make any reference to PassivePorts or FTP_PASSIVE_START or anything like it.

That is quite odd, because the proftpd.conf files on my v1 NAS aren't that simple and do include PassivePorts. I'm copying one below for reference (with the control port and passive port range I use for that machine XXXXXd out).

I think there is still damage from the original full OS partition issue. You could simply try to fix things as you find problems, but it might be more prudent (and simpler) to back up your data, do a factory reset, rebuild/configure the NAS and restore the data from backup..

NVNAS:/etc# cat proftpd.conf
# This is a basic ProFTPD configuration file (rename it to
# 'proftpd.conf' for actual use. It establishes a single server
# and a single anonymous login. It assumes that you have a user/group
# "nobody/nogroup" and "ftp" for normal operation and anon.
ServerName "NETGEAR ReadyNAS"
ServerType standalone
DeferWelcome off

MultilineRFC2228 on
DefaultServer on
ShowSymlinks on
AllowOverwrite on
AllowStoreRestart on
UseReverseDNS off
IdentLookups off
TimesGMT off
WtmpLog off
ServerLog /var/log/proftpd.log
SystemLog /var/log/proftpd.log
TransferLog /var/log/proftpd.log
BanLog /var/log/proftpd.log
MaxLoginAttempts 2
BanEngine on
BanTable /ramfs/ftp_ban.tbl
BanOnEvent MaxLoginAttempts 5/00:10:00 00:15:00

TimeoutNoTransfer 600
TimeoutStalled 600
TimeoutIdle 1200

DisplayLogin welcome.msg
DisplayChdir .message

DenyFilter \*.*/

# Uncomment this if you are using NIS or LDAP to retrieve passwords:
PersistentPasswd on

# Uncomment this if you would use quota module:
#Quotas on

# Port 21 is the standard FTP port.
Port XXXXXX
MasqueradeAddress NVNAS
DynMasqRefresh 7200
PassivePorts XXXXX XXXXX
AllowForeignAddress on

# To prevent DoS attacks, set the maximum number of child processes
# to 30. If you need to allow more than 30 concurrent connections
# at once, simply increase this value. Note that this ONLY works
# in standalone mode, in inetd mode you should use an inetd server
# that allows you to limit maximum number of processes per service
# (such as xinetd)
MaxInstances 30

# Set the user and group that the server normally runs at.
User nobody
Group nogroup

# Normally, we want files to be overwriteable.
<Directory /*>
# Umask 022 is a good standard umask to prevent new files and dirs
# (second parm) from being group and world writable.
Umask 022 022

AllowOverwrite on
</Directory>

Include /etc/frontview/proftpd/ftps.conf
Include /etc/frontview/proftpd/User.conf

mdgm-ntgr · ‎2016-09-22

Was just pointing out that as tech support mode is intended for support it's designed to suit our support engineers not to be intuitive for end user use.

It could be that there is still some corrupt config as a result of your OS partition getting full.