- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Re: System volume 'root' usage is 100% and /dev/md0 is full
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I recently started getting notifications that my system volume 'root' usage is 100%
I've gone through all of the files and it seems the culprit is /dev/md0
I have no add-ons installed, and this NAS is specifically used as a backup of another NAS. It uses ReadyNAS Replicate for backups.
Here's the output from df-h and df-i
I've already tried commands similar to these:
du -csh /var
du -csh /var/*
And the problem does not seem to be in those directories. Very low usage there.
This error started appearing on the day that I got a copy failed error from ReadyNAS Replicate that gave me an Exit Code: 11 (Error in file I/O)
Can someone help me fix this?
Solved! Go to Solution.
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The reason you couldn't find it was that at some point your encrypted volume wasn't mounted (perhaps the USB key wasn't connected when you booted the NAS) and a Replicate backup job ran and filled the OS partition on the NAS.
When you looked at the system the data volume was mounted. There was data under /data on the OS partition but you couldn't tell that.
I have passed on a summary of what's happened to our engineers and am waiting for their suggestion. However as it is a long weekend in the US I don't expect a response for at least a few days.
All Replies
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
More than likely its one or two very large files in one directory.
try
cd /var
du -hsx * | sort -rh | head -10
and if you see a directory that has gigabytes of stuff, descend into it and repeat.
If you don't pick a different main directory and try again.
/var, /etc, /usr, /opt, /frontview, /bin, /tmp are all places you can look.
Though maybe double-check /var/log first.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
None of those seem to be responsible.
root@RN102Backup:~# cd /var
root@RN102Backup:/var# du -hsx * | sort -rh | head -10
106M lib
53M backups
5.5M cache
700K log
624K readynasd
16K spool
16K netatalk
8.0K www
4.0K tmp
4.0K replicate
root@RN102Backup:/var# cd /etc
root@RN102Backup:/etc# du -hsx * | sort -rh | head -10
924K ssl
508K apache2
388K default
296K frontview
276K ssh
268K init.d
172K fonts
96K pam.d
80K apt
56K network
root@RN102Backup:/etc# cd /usr
root@RN102Backup:/usr# du -hsx * | sort -rh | head -10
78M lib
20M bin
18M sbin
16M share
48K local
4.0K src
4.0K include
4.0K games
root@RN102Backup:/usr# cd /opt
root@RN102Backup:/opt# du -hsx * | sort -rh | head -10
2.2M ctch
1.2M readycloud
776K replication
740K p2p
192K readydrop
92K remote
root@RN102Backup:/opt# cd /frontview
root@RN102Backup:/frontview# du -hsx * | sort -rh | head -10
20M dashboard
2.6M conf
1.1M lib
860K bin
0 recovery
root@RN102Backup:/frontview# cd /bin
root@RN102Backup:/bin# du -hsx * | sort -rh | head -10
800K bash
660K busybox
260K tar
224K ip
184K systemctl
152K grep
152K egrep
116K fgrep
108K netstat
108K cp
root@RN102Backup:/bin# cd /tmp
root@RN102Backup:/tmp# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
There are other directories you can look in, there has to be something there.
Of course you do have the option of factory defaulting the nas.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
@vandermerwe wrote:
There are other directories you can look in, there has to be something there.
Of course you do have the option of factory defaulting the nas.
Correct.
/home, /apps, /data are actually on the data volume, so don't look in those. But it has to be on one of the others. Maybe try lost+found first???
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
I've gone through all of the directories that come up with ls.
Is there something I'm missing?
The only one I didn't check in is data because that is actually the NAS volume that stores the backup.
Here is a du -hsx of the entire root excluding data
root@RN102Backup:/# ls
apps boot dev frontview lib media opt root sbin srv tmp var
bin data etc home lost+found mnt proc run selinux sys usr
root@RN102Backup:/# du -hsx --exclude data * | sort -rh | head -10
du: cannot access `proc/22248/task/22248/fd/4': No such file or directory
du: cannot access `proc/22248/task/22248/fdinfo/4': No such file or directory
du: cannot access `proc/22248/fd/4': No such file or directory
du: cannot access `proc/22248/fdinfo/4': No such file or directory
166M var
130M usr
29M lib
24M frontview
7.9M sbin
5.5M bin
5.2M opt
4.2M etc
1.8M apps
324K run
Here are the individual folders
root@RN102Backup:/# cd /
root@RN102Backup:/# ls
apps boot dev frontview lib media opt root sbin srv tmp var
bin data etc home lost+found mnt proc run selinux sys usr
root@RN102Backup:/# cd /apps
root@RN102Backup:/apps# du -hsx * | sort -rh | head -10
0 DO_NOT_DELETE
root@RN102Backup:/apps# cd /boot
root@RN102Backup:/boot# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
root@RN102Backup:/boot# cd /dev
root@RN102Backup:/dev# du -hsx * | sort -rh | head -10
4.0K boot_reason
0 zero
0 vpd
0 urandom
0 ubi_ctrl
0 ubi0_0
0 ubi0
0 ttyS1
0 ttyS0
0 ttypf
root@RN102Backup:/dev# cd /lib
root@RN102Backup:/lib# du -hsx * | sort -rh | head -10
13M modules
11M arm-linux-gnueabi
3.0M systemd
1.2M nut
972K xtables
780K udev
208K terminfo
88K libaudit.so.0.0.0
48K libauparse.so.0.0.0
44K security
root@RN102Backup:/lib# cd /media
root@RN102Backup:/media# du -hsx * | sort -rh | head -10
16K USB_FLASH_2
root@RN102Backup:/media# cd /root
root@RN102Backup:~# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
root@RN102Backup:~# cd /sbin
root@RN102Backup:/sbin# du -hsx * | sort -rh | head -10
608K ldconfig.real
480K btrfs
464K mdadm
264K btrfs-convert
256K btrfs-image
248K mkfs.btrfs
244K mdmon
236K btrfs-show-super
236K btrfs-debug-tree
232K btrfs-zero-log
root@RN102Backup:/sbin# cd /srv
root@RN102Backup:/srv# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
root@RN102Backup:/srv# cd /home
root@RN102Backup:/home# du -hsx * | sort -rh | head -10
0 admin
root@RN102Backup:/home# cd /lost+found
root@RN102Backup:/lost+found# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
root@RN102Backup:/lost+found# cd /mnt
root@RN102Backup:/mnt# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
root@RN102Backup:/mnt# cd /proc
root@RN102Backup:/proc# du -hsx * | sort -rh | head -10
du: cannot access `21961/task/21961/fd/3': No such file or directory
du: cannot access `21961/task/21961/fdinfo/3': No such file or directory
du: cannot access `21961/fd/3': No such file or directory
du: cannot access `21961/fdinfo/3': No such file or directory
0 zoneinfo
0 vmstat
0 vmallocinfo
0 version
0 uptime
0 tty
0 timer_list
0 sysvipc
0 sysrq-trigger
0 sys
root@RN102Backup:/proc# cd /selinux
root@RN102Backup:/selinux# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
root@RN102Backup:/selinux#
root@RN102Backup:/run# du -hsx * | sort -rh | head -10
220K udev
164K lock
56K readydrop
12K systemd
8.0K mdadm
8.0K blkid
4.0K utmp
4.0K usb
4.0K sshd.pid
4.0K spindown
I was trying to avoid a factory reset because this NAS is the only backup of my other NAS, and I don't have any other drives to make an additional backup, and I'm not going to be without a backup, even if it is for a matter of hours.
This NAS is in RAID 1 , and it is encrypted using the built-in encryption Frontview provides. Not sure if this matters, but I figured I should note that.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
try replacing the -s with -a.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
As in: du -hax ?
I don't seem to be having much better luck with that in finding these files.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The reason you couldn't find it was that at some point your encrypted volume wasn't mounted (perhaps the USB key wasn't connected when you booted the NAS) and a Replicate backup job ran and filled the OS partition on the NAS.
When you looked at the system the data volume was mounted. There was data under /data on the OS partition but you couldn't tell that.
I have passed on a summary of what's happened to our engineers and am waiting for their suggestion. However as it is a long weekend in the US I don't expect a response for at least a few days.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: System volume 'root' usage is 100% and /dev/md0 is full
@mdgm wrote:
The reason you couldn't find it was that at some point your encrypted volume wasn't mounted (perhaps the USB key wasn't connected when you booted the NAS) and a Replicate backup job ran and filled the OS partition on the NAS.
Great diagnosis!
Kind of a scary problem though, hopefully development will close off that possibility in a future release.