Re: System volume 'root' usage is 100%

kokomodrums · ‎2013-09-03

I've got a client with a Readynas 102, I just started getting this error message in my email today, every ten minutes:

System volume 'root' usage is 100 %. This condition should not occur in normal conditions. Please contact technical support.

I saw another post with the same error and it looks like they did some commands on the shell. I do not have ssh access (but can set it up if need be). Is there anyway I can troubleshoot prior to setting up ssh? They have no add-ons installed, very basic windows file server setup. They have a nightly backup that copies all data to an external drive, maybe it's a log file? On that note, is there a setting to keep logs 'trimmed'?

Thank you in advance!!!!

kokomodrums · ‎2013-09-03

In addition, I noticed that they were on 6.0 so I went ahead and updated to the latest version. Still getting the error.

mdgm-ntgr · ‎2013-09-03

It could possibly be a log file. The logs are kept trimmed automatically but if a log file grows very fast this might not happen. To look into this SSH access or tech support mode is required.

kokomodrums · ‎2013-09-04

What command should I do from ssh? I'll get that set up today.

mdgm-ntgr · ‎2013-09-04

If you can SSH in you could use the du command.

kokomodrums · ‎2013-09-04

Thank you for the advice, I will post the results once I've run the command

kokomodrums · ‎2013-09-04


root@nas-server:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          4.0G  3.9G     0 100% /
tmpfs            10M  4.0K   10M   1% /dev
/dev/md0        4.0G  3.9G     0 100% /
tmpfs           248M     0  248M   0% /dev/shm
tmpfs           248M  792K  247M   1% /run
tmpfs           248M     0  248M   0% /sys/fs/cgroup
tmpfs           248M  8.0K  248M   1% /media
/dev/md/data-0  462G  7.5G  436G   2% /data
/dev/md/data-0  462G  7.5G  436G   2% /apps
/dev/md/data-0  462G  7.5G  436G   2% /home
/dev/sdc1       597G  7.0G  590G   2% /media/USB_HDD_2
/dev/md/data-0  462G  7.5G  436G   2% /run/nfs4/home
/dev/sdc1       597G  7.0G  590G   2% /run/nfs4/media/USB_HDD_2

I do very little command line, how do I find out what files/folders are specifically too big?

mdgm-ntgr · ‎2013-09-04

There are multiple ways of doing it but I would tend to do things like:


# du -csh /var
# du -csh /var/*

kokomodrums · ‎2013-09-13

Fixed that problem, caused a bigger one. I'll let you see what I did....


root@nas-server:~# du -csh /var
3.6G    /var
3.6G    total
root@nas-server:~# du -csh /var/*
4.0K    /var/agentx
46M     /var/backups
34M     /var/cache
4.0K    /var/cores
4.0K    /var/ftp
38M     /var/lib
4.0K    /var/local
0       /var/lock
3.5G    /var/log
4.0K    /var/mail
12K     /var/netatalk
4.0K    /var/opt
368K    /var/readynasd
0       /var/run
16K     /var/spool
4.0K    /var/tmp
8.0K    /var/www
3.6G    total

So it appears that the log folder was totally full. So instead of being smart and deleting the logs from Frontview (or w/e the web interface is called in OS 6), I decided to try and be smart and delete all the logs from the shell...


root@nas-server:~# rm -rf /var/log/*
root@nas-server:~# du -csh /var/*
4.0K    /var/agentx
46M     /var/backups
11M     /var/cache
4.0K    /var/cores
4.0K    /var/ftp
38M     /var/lib
4.0K    /var/local
0       /var/lock
12K     /var/log
4.0K    /var/mail
12K     /var/netatalk
4.0K    /var/opt
368K    /var/readynasd
0       /var/run
16K     /var/spool
4.0K    /var/tmp
8.0K    /var/www
93M     total

It worked. But, it deleted all the subdirectories in the /var/log folder, which are required by ReadyNAS to write logs period. I went and tried turning on email alerts again and kept getting an error that the log/*whatever* directory didn't exist...

So, can anyone running 6.1.1 give me a list of all the subdirectories of /var/log in a clean install??? I really don't want to do a factory reset for such a minor error...

kokomodrums · ‎2013-09-13

Ok, the actual error message said that "/var/log/frontview/msmtp.log" didn't exist, so I created that directory and file.


root@nas-server:/var/log# mkdir frontview
root@nas-server:/var/log# cd frontview
root@nas-server:/var/log/frontview# touch msmtp.log

That allowed me to send alert messages now. But I'm worried that I've deleted more folders/files that are gonna cause issues down the road. If I have to do a factory reset, I can, I would just like to avoid it if I can find another solution.

mdgm-ntgr · ‎2013-09-13

Try


# touch /.os_update
# rn_shutdown -r

The above will get the NAS to do an OS Re-install which will re-install the OS but leave your data intact.

I would have recommended finding the log file(s) that filled the OS partition and overwriting them e.g.


# echo "" > /var/log/mylog.log

Clarification: Please note that an OS Re-install should only be done after space on the OS partition has been freed up. Doing an OS Re-install whilst the OS partition is very full may lead to further issues.

kokomodrums · ‎2013-09-16

Thanks for the advice, I will try to OS reinstall next. Will I have to create new users after the reinstall or does it keep all settings intact as well?

Also, what can I do to prevent the log files from getting too big in the future?

mdgm-ntgr · ‎2013-09-16

kokomodrums wrote:
Thanks for the advice, I will try to OS reinstall next. Will I have to create new users after the reinstall or does it keep all settings intact as well?

It keeps all users. Only settings for which bad values could lock you of of a system would be reset. So the admin password and some network settings would be reset.

kokomodrums wrote:

Also, what can I do to prevent the log files from getting too big in the future?

Well without knowing which log file filled up the OS partition it is difficult to say.

FLO_gruppen_AS · ‎2013-12-19

I also get the same massage on my ReadyNAS 316 (6.1.4). I'm only running owncloud application. Any solutions?

mdgm-ntgr · ‎2013-12-19

Do you have SSH access enabled?

lobbie1978 · ‎2014-04-04

Just received the same error message 15 minutes ago and 5 minutes ago. I deleted the logs from Frontview, but the error message remains... 😞 I have SSH enabled.

StephenB · ‎2014-04-04

lobbie1978 wrote:
Just received the same error message 15 minutes ago and 5 minutes ago. I deleted the logs from Frontview, but the error message remains... 😞 I have SSH enabled.

Maybe log in with ssh, and look around to see what is using the space.

lobbie1978 · ‎2014-04-04

lobbie1978 wrote:
Just received the same error message 15 minutes ago and 5 minutes ago. I deleted the logs from Frontview, but the error message remains... 😞 I have SSH enabled.

I think I found the problem. For some reason the 'Transmission' add-on installed the download folder in the root directory, filling up the root dir rapidly. Although in the config.json file, the download directory is /Data/Transmission/Incomplete

Weird...

RedBullet · ‎2015-10-03

So I'm running into this. Right now I am getting emails every 10 minutes saying root is at 100%.

However, df shows everything is fine. Rebooting didn't appear to help...

Is there some hidden space or something??

StephenB · ‎2015-10-03

Did you also check inodes? df -i

RedBullet · ‎2015-10-03

Yea, df -i looks fine:

root@Storage:/var# df -i
Filesystem     Inodes IUsed  IFree IUse% Mounted on
rootfs              0     0      0     - /
tmpfs          504422   524 503898    1% /dev
/dev/md0            0     0      0     - /
tmpfs          504422     2 504420    1% /dev/shm
tmpfs          504422   729 503693    1% /run
tmpfs          504422     4 504418    1% /sys/fs/cgroup
tmpfs          504422     2 504420    1% /media
/dev/md126          0     0      0     - /data
/dev/md126          0     0      0     - /home
/dev/md126          0     0      0     - /apps
tmpfs          504422     7 504415    1% /var/replicate/shm
tmpfs          504422     1 504421    1% /home/Andra/snapshot
tmpfs          504422     1 504421    1% /data/home/Andra/snapshot
tmpfs          504422     1 504421    1% /home/Scott/snapshot
tmpfs          504422     1 504421    1% /data/home/Scott/snapshot

So, I did have a couple of things in /var/lib which were taking up space (squeezeboxserver, and mysql). I was able to relocate those to /apps.

But now I am here (which is better):

root@Storage:/# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/md0 4.0G 2.8G 898M 77% /

and I can't figure out what is taking the space...

root@Storage:/# du -sh /var
305M    /var

Any other places I should look?

RedBullet · ‎2015-10-04

This morning I am back to nearly 100% used:

root@Storage:/var# du -sh *
0       agentx
61M     backups
100M    cache
38M     cores
0       ftp
46M     lib
0       local
4.0K    lock
66M     log
0       mail
8.0K    netatalk
0       opt
960K    readynasd
0       replicate
4.0K    run
4.0K    spool
0       tmp
4.0K    www
root@Storage:/var# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        4.0G  3.6G  123M  97% /

Where's the space going??? Help!

RedBullet · ‎2015-10-04

OK, think I found it. Appears crashplan was doing something odd (upgrade folder was pretty big). Relocating it to /apps and will investigate further.

StephenB · ‎2015-10-04

@RedBullet wrote:

OK, think I found it. Appears crashplan was doing something odd (upgrade folder was pretty big). Relocating it to /apps and will investigate further.

You need to upgrade java to 1.7 or 1.8. I provided a link on your other post.