NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
System volume 'root' usage is 100 % and system is now inactive
My RN102 started to become unresponsive. I went to the logs and found the dreaded 'System volume 'root' usage is 100%' message and noticed that it had been occurring with increasing frequency. At this point I cannot even restart my device - it shows as inactive on the Admin page and in RAIDar.
I am pretty much a novice with hands on diagnostics and tools, but searching through the forums gave me some pretty good guidelines on diagnosing the problem. Navigating through root with SSH, I think I may have identified the problem,
Below are some data using SSH commands. It looks to me like the culprit is a file(?) named 'btmp' in the var/log directory.
Welcome to ReadyNASOS 6.2.4 root@NAS:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 4.0G 4.0G 0 100% / tmpfs 10M 4.0K 10M 1% /dev /dev/md0 4.0G 4.0G 0 100% / tmpfs 248M 48K 248M 1% /dev/shm tmpfs 248M 348K 248M 1% /run tmpfs 248M 0 248M 0% /sys/fs/cgroup tmpfs 248M 0 248M 0% /media /dev/md127 2.8T 551G 2.2T 20% /data /dev/md127 2.8T 551G 2.2T 20% /home /dev/md127 2.8T 551G 2.2T 20% /apps /dev/md127 2.8T 551G 2.2T 20% /run/nfs4/home root@NAS:~# df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 65536 23113 42423 36% / tmpfs 63410 272 63138 1% /dev /dev/md0 65536 23113 42423 36% / tmpfs 63410 3 63407 1% /dev/shm tmpfs 63410 370 63040 1% /run tmpfs 63410 4 63406 1% /sys/fs/cgroup tmpfs 63410 1 63409 1% /media /dev/md127 0 0 0 - /data /dev/md127 0 0 0 - /home /dev/md127 0 0 0 - /apps /dev/md127 0 0 0 - /run/nfs4/home root@NAS:~# du -csh /var/* 4.0K /var/agentx 50M /var/backups 28M /var/cache 912K /var/cores 4.0K /var/ftp 57M /var/lib 4.0K /var/local 0 /var/lock 3.1G /var/log 4.0K /var/mail 12K /var/netatalk 4.0K /var/opt 12K /var/readydrop 2.1M /var/readynasd 0 /var/run 16K /var/spool 4.0K /var/tmp 8.0K /var/www 3.2G total root@NAS:/var/log# ls -l total 3105884 -rw-r--r-- 1 root root 472 Dec 25 2014 alternatives.log drwxr-xr-x 2 root root 4096 Dec 11 2012 apache2 drwxr-xr-x 2 root root 4096 Apr 15 2015 apt -rw------- 1 root utmp 3176878080 Nov 14 18:36 btmp -rw-r--r-- 1 root root 307200 Nov 14 18:22 ctscand.log -rw-r--r-- 1 root root 781 Nov 14 18:22 dbbroker.log -rw-r--r-- 1 root root 67802 Nov 12 15:45 dpkg.log -rw-r--r-- 1 root root 1032 Apr 4 2015 faillog drwxr-xr-x 3 root root 4096 Dec 11 2012 frontview drwxr-xr-x 2 root root 4096 Oct 14 2013 fsck drwxr-xr-x 3 root root 4096 Nov 20 2012 journal -rw-r--r-- 1 root root 12556 Nov 14 18:34 lastlog -rw-r--r-- 1 root root 0 Jul 3 2014 LeafP2P.log drwxr-s--- 2 mysql adm 4096 Jul 2 2014 mysql -rw-r----- 1 mysql adm 0 Jul 2 2014 mysql.err -rw-r----- 1 mysql adm 0 Jul 2 2014 mysql.log -rw-r--r-- 1 root root 5280 Jul 3 2014 netatalk.log -rw-r--r-- 1 root root 76 Dec 25 2014 readydropd.log drwxr-xr-x 2 admin admin 4096 Nov 22 2014 readynasd drwxr-xr-x 2 root root 4096 Aug 11 12:44 samba drwxr-xr-x 2 squeezeboxserver guest 4096 Apr 4 2015 squeezeboxserver -rw-rw-r-- 1 root utmp 31104 Nov 14 18:34 wtmp root@NAS:/var/log#
Now I need some confirmation and guidance on next steps to get my system up and running again.
Thanks in advance.
OK: looks Iike I was able to fix it.
Further research showed that the btmp records failed attempts to login to the system. I ran the command
last -f /var/log/btmp
and got a stream of messages like this:
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00So it looks like someone was trying to brute force connect to my system.
I ran the command:
sudo > /var/log/btmp
and that cleared it out. I have access to my system now.
Now the challenge will be to figure out how to better protect my system. Maybe I'll just turn off remote access.
4 Replies
Replies have been turned off for this discussion
OK: looks Iike I was able to fix it.
Further research showed that the btmp records failed attempts to login to the system. I ran the command
last -f /var/log/btmp
and got a stream of messages like this:
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00)
root ssh:notty 43.229.53.59 Tue Oct 20 00:34 - 00:34 (00:00So it looks like someone was trying to brute force connect to my system.
I ran the command:
sudo > /var/log/btmp
and that cleared it out. I have access to my system now.
Now the challenge will be to figure out how to better protect my system. Maybe I'll just turn off remote access.
Best not to port forward SSH if you don't need to access SSH remotely or can setup e.g. a VPN
Or if you need remote access read up on and setup knockd. Everyone will be quick to tell you that it's not that much higher security if someone really wants to try to get into your system, but most of the attacks are based on scans that are not directly focused on your system so if they don't get responses they go away. I just checked and I've actually not seen a single false login in the past 3 months with this setup (that's all the longer my log goes back). Basically you are only enabling traffic to sshd for nodes that send the knock sequence.
steve
Thanks for the security tips, guys. It may be moot at this point.
After I got my system back, after doing a couple of things to check that data was OK, I did the upgrade from 6.2.4 to 6.4.0. That resulted in my volumes becoming inactive. I can't get to any volumes or any shares.
I've got a ticket open with support; first tier couldn't help - it's been escalated. It will be a while before I know the outcome.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
