× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: System volume 'root' usage is 100% and /dev/md0 is full

SuperNASman
Aspirant

System volume 'root' usage is 100% and /dev/md0 is full

Hello,

 

I recently started getting notifications that my system volume 'root' usage is 100%

 

I've gone through all of the files and it seems the culprit is /dev/md0

 

I have no add-ons installed, and this NAS is specifically used as a backup of another NAS. It uses ReadyNAS Replicate for backups.

 

Here's the output from df-h and df-i

 

Untitled.png

 

I've already tried commands similar to these:

 

du -csh /var
du -csh /var/*

 

And the problem does not seem to be in those directories. Very low usage there.

 

This error started appearing on the day that I got a copy failed error from ReadyNAS Replicate that gave me an Exit Code: 11 (Error in file I/O)

 

Can someone help me fix this?

Message 1 of 10

Accepted Solutions
mdgm-ntgr
NETGEAR Employee Retired

Re: System volume 'root' usage is 100% and /dev/md0 is full

The reason you couldn't find it was that at some point your encrypted volume wasn't mounted (perhaps the USB key wasn't connected when you booted the NAS) and a Replicate backup job ran and filled the OS partition on the NAS.

When you looked at the system the data volume was mounted. There was data under /data on the OS partition but you couldn't tell that.

I have passed on a summary of what's happened to our engineers and am waiting for their suggestion. However as it is a long weekend in the US I don't expect a response for at least a few days.

View solution in original post

Message 9 of 10

All Replies
StephenB
Guru

Re: System volume 'root' usage is 100% and /dev/md0 is full

More than likely its one or two very large files in one directory.

 

try

cd /var

du -hsx * | sort -rh | head -10

 

and if you see a directory that has gigabytes of stuff, descend into it and repeat.

 

If you don't pick a different main directory and try again.

 

/var, /etc, /usr, /opt, /frontview, /bin, /tmp are all places you can look.

 

Though maybe double-check /var/log first.

 

 

Message 2 of 10
SuperNASman
Aspirant

Re: System volume 'root' usage is 100% and /dev/md0 is full

None of those seem to be responsible.

 

 


root@RN102Backup:~# cd /var
root@RN102Backup:/var# du -hsx * | sort -rh | head -10
106M    lib
53M     backups
5.5M    cache
700K    log
624K    readynasd
16K     spool
16K     netatalk
8.0K    www
4.0K    tmp
4.0K    replicate
root@RN102Backup:/var# cd /etc
root@RN102Backup:/etc# du -hsx * | sort -rh | head -10
924K    ssl
508K    apache2
388K    default
296K    frontview
276K    ssh
268K    init.d
172K    fonts
96K     pam.d
80K     apt
56K     network
root@RN102Backup:/etc# cd /usr
root@RN102Backup:/usr# du -hsx * | sort -rh | head -10
78M     lib
20M     bin
18M     sbin
16M     share
48K     local
4.0K    src
4.0K    include
4.0K    games
root@RN102Backup:/usr# cd /opt
root@RN102Backup:/opt# du -hsx * | sort -rh | head -10
2.2M    ctch
1.2M    readycloud
776K    replication
740K    p2p
192K    readydrop
92K     remote
root@RN102Backup:/opt# cd /frontview
root@RN102Backup:/frontview# du -hsx * | sort -rh | head -10
20M     dashboard
2.6M    conf
1.1M    lib
860K    bin
0       recovery
root@RN102Backup:/frontview# cd /bin
root@RN102Backup:/bin# du -hsx * | sort -rh | head -10
800K    bash
660K    busybox
260K    tar
224K    ip
184K    systemctl
152K    grep
152K    egrep
116K    fgrep
108K    netstat
108K    cp
root@RN102Backup:/bin# cd /tmp
root@RN102Backup:/tmp# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory

 

Message 3 of 10
vandermerwe
Master

Re: System volume 'root' usage is 100% and /dev/md0 is full

There are other directories you can look in, there has to be something there.

Of course you do have the option of factory defaulting the nas.

Message 4 of 10
StephenB
Guru

Re: System volume 'root' usage is 100% and /dev/md0 is full


@vandermerwe wrote:

There are other directories you can look in, there has to be something there.

Of course you do have the option of factory defaulting the nas.


Correct.

 

/home, /apps, /data are actually on the data volume, so don't look in those.  But it has to be on one of the others.  Maybe try lost+found first???

Message 5 of 10
SuperNASman
Aspirant

Re: System volume 'root' usage is 100% and /dev/md0 is full

I've gone through all of the directories that come up with ls.

 

Is there something I'm missing?

 

The only one I didn't check in is data because that is actually the NAS volume that stores the backup.

 

Here is a du -hsx of the entire root excluding data

 

root@RN102Backup:/# ls
apps  boot  dev  frontview  lib         media  opt   root  sbin     srv  tmp  var
bin   data  etc  home       lost+found  mnt    proc  run   selinux  sys  usr
root@RN102Backup:/# du -hsx --exclude data * | sort -rh | head -10
du: cannot access `proc/22248/task/22248/fd/4': No such file or directory
du: cannot access `proc/22248/task/22248/fdinfo/4': No such file or directory
du: cannot access `proc/22248/fd/4': No such file or directory
du: cannot access `proc/22248/fdinfo/4': No such file or directory
166M    var
130M    usr
29M     lib
24M     frontview
7.9M    sbin
5.5M    bin
5.2M    opt
4.2M    etc
1.8M    apps
324K    run

 

 

 

Here are the individual folders

 

root@RN102Backup:/# cd /
root@RN102Backup:/# ls
apps  boot  dev  frontview  lib         media  opt   root  sbin     srv  tmp  var
bin   data  etc  home       lost+found  mnt    proc  run   selinux  sys  usr

 

root@RN102Backup:/# cd /apps
root@RN102Backup:/apps# du -hsx * | sort -rh | head -10
0       DO_NOT_DELETE

 

root@RN102Backup:/apps# cd /boot
root@RN102Backup:/boot# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory

 

root@RN102Backup:/boot# cd /dev
root@RN102Backup:/dev# du -hsx * | sort -rh | head -10
4.0K    boot_reason
0       zero
0       vpd
0       urandom
0       ubi_ctrl
0       ubi0_0
0       ubi0
0       ttyS1
0       ttyS0
0       ttypf

 

root@RN102Backup:/dev# cd /lib
root@RN102Backup:/lib# du -hsx * | sort -rh | head -10
13M     modules
11M     arm-linux-gnueabi
3.0M    systemd
1.2M    nut
972K    xtables
780K    udev
208K    terminfo
88K     libaudit.so.0.0.0
48K     libauparse.so.0.0.0
44K     security

 

root@RN102Backup:/lib# cd /media
root@RN102Backup:/media# du -hsx * | sort -rh | head -10
16K     USB_FLASH_2

 

root@RN102Backup:/media# cd /root
root@RN102Backup:~# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory

 

root@RN102Backup:~# cd /sbin
root@RN102Backup:/sbin# du -hsx * | sort -rh | head -10
608K    ldconfig.real
480K    btrfs
464K    mdadm
264K    btrfs-convert
256K    btrfs-image
248K    mkfs.btrfs
244K    mdmon
236K    btrfs-show-super
236K    btrfs-debug-tree
232K    btrfs-zero-log

 

root@RN102Backup:/sbin# cd /srv
root@RN102Backup:/srv# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory

 

root@RN102Backup:/srv# cd /home
root@RN102Backup:/home# du -hsx * | sort -rh | head -10
0       admin

 

root@RN102Backup:/home# cd /lost+found
root@RN102Backup:/lost+found# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory

 

root@RN102Backup:/lost+found# cd /mnt
root@RN102Backup:/mnt# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory

 

root@RN102Backup:/mnt# cd /proc
root@RN102Backup:/proc# du -hsx * | sort -rh | head -10
du: cannot access `21961/task/21961/fd/3': No such file or directory
du: cannot access `21961/task/21961/fdinfo/3': No such file or directory
du: cannot access `21961/fd/3': No such file or directory
du: cannot access `21961/fdinfo/3': No such file or directory
0       zoneinfo
0       vmstat
0       vmallocinfo
0       version
0       uptime
0       tty
0       timer_list
0       sysvipc
0       sysrq-trigger
0       sys

 

root@RN102Backup:/proc# cd /selinux
root@RN102Backup:/selinux# du -hsx * | sort -rh | head -10
du: cannot access `*': No such file or directory
root@RN102Backup:/selinux#

root@RN102Backup:/run# du -hsx * | sort -rh | head -10
220K    udev
164K    lock
56K     readydrop
12K     systemd
8.0K    mdadm
8.0K    blkid
4.0K    utmp
4.0K    usb
4.0K    sshd.pid
4.0K    spindown

 

 

I was trying to avoid a factory reset because this NAS is the only backup of my other NAS, and I don't have any other drives to make an additional backup, and I'm not going to be without a backup, even if it is for a matter of hours.

 

This NAS is in RAID 1 , and it is encrypted using the built-in encryption Frontview provides. Not sure if this matters, but I figured I should note that.

Message 6 of 10
StephenB
Guru

Re: System volume 'root' usage is 100% and /dev/md0 is full

try replacing the -s with -a.

Message 7 of 10
SuperNASman
Aspirant

Re: System volume 'root' usage is 100% and /dev/md0 is full

As in: du -hax ?

 

I don't seem to be having much better luck with that in finding these files.

Message 8 of 10
mdgm-ntgr
NETGEAR Employee Retired

Re: System volume 'root' usage is 100% and /dev/md0 is full

The reason you couldn't find it was that at some point your encrypted volume wasn't mounted (perhaps the USB key wasn't connected when you booted the NAS) and a Replicate backup job ran and filled the OS partition on the NAS.

When you looked at the system the data volume was mounted. There was data under /data on the OS partition but you couldn't tell that.

I have passed on a summary of what's happened to our engineers and am waiting for their suggestion. However as it is a long weekend in the US I don't expect a response for at least a few days.

Message 9 of 10
StephenB
Guru

Re: System volume 'root' usage is 100% and /dev/md0 is full


@mdgm wrote:

The reason you couldn't find it was that at some point your encrypted volume wasn't mounted (perhaps the USB key wasn't connected when you booted the NAS) and a Replicate backup job ran and filled the OS partition on the NAS.


Great diagnosis!  

 

Kind of a scary problem though, hopefully development will close off that possibility in a future release.

Message 10 of 10
Top Contributors
Discussion stats
  • 9 replies
  • 21825 views
  • 2 kudos
  • 4 in conversation
Announcements