× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

System volume root's usage is 100% and /dev/md0 is using all 4G

Craig_X
Aspirant

System volume root's usage is 100% and /dev/md0 is using all 4G

Running 6.7.4 firmware.  No applications installed.  Connected USB drive and started a backup which did start correctly, but ultimately failed (though after examining the share it does appear that the data made it to the destination).  Now the USB drive doesn't show on the overview screen, even after a reboot and the logs showing that it is connected.  Immediately after the initial backup failed the logs had the "System volume root's usage is 100%..." message, and that message appears again once every 24hrs.  After doing a df - h it appears that /dev/md0 might be the culprit.  I am not very familiar with ReadyNAS code nor with Linux, so I'm not bold/dumb enough to just rm it.

 

I took a picture of the output from df -h, but I can't figure how to post it.  The md0 line looks like this...

 

Filesystem  Size    Used    Avail    Use%   Mounted on

/dev/md0    4.0G   4.0G          0    100%    /

 

Other than backups not working, system seems to be functioning properly.  However, I do need my backups, and I understand it is dangerous to leave the root full.

 

Model: RN102|ReadyNAS 100 Series
Message 1 of 8

Accepted Solutions
StephenB
Guru

Re: System volume root's usage is 100% and /dev/md0 is using all 4G


@Craig_X wrote:

 

 

root@PRANAS-TNG:/# find / -xdev -type f -size +100M
/var/cores/core-readynasd
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/pvranchad_book.xml
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/PVRanchAD (1).xml
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/pvranchad_pax.xml
/media/USB_HDD_11/Pat/Aerial Footage - Max/Princeville Ranch Cut 3.mp4

  


You can delete /var/cores/core-readynasd for sure.

 

You can also remount the root file system, so you can search through it without mount points from muddying the water.

 

To do this you enter

mount --bind / /mnt
find /mnt -xdev -type f -size +100M

...

 

In addition to searching, etc you also would delete files directly from /mnt.  

 

Maybe also try 

mount | grep -i USB_HDD_11

 

That should let you know if USB_HDD_11 is a mount point or just a folder.  It you don't see it in the output of the above command, then it is just a folder.  You can delete it's contents if you wish, but perhaps keep the folder itself.

 

When done, you unmount with umount /mnt 

View solution in original post

Message 6 of 8

All Replies
StephenB
Guru

Re: System volume root's usage is 100% and /dev/md0 is using all 4G

Usually there's one or two huge files that need to be cleared.  Have you searched for those via ssh?

Message 2 of 8
Craig_X
Aspirant

Re: System volume root's usage is 100% and /dev/md0 is using all 4G

Not really sure how to go about searching.  Would the files be under /dev/md0?

Message 3 of 8
StephenB
Guru

Re: System volume root's usage is 100% and /dev/md0 is using all 4G

They are on /dev/md0, but /dev/md0 is the disk partition, and is not part of the path.

 

Try

cd //

du -hsx * | sort -rh | head -10

 

That will show you the biggest folders,  If (for example) the problem seems to be in /var, you can then 

cd var

du -hsx * | sort -rh | head -10

 

That usually gets you to the problem area pretty quickly.

Message 4 of 8
Craig_X
Aspirant

Re: System volume root's usage is 100% and /dev/md0 is using all 4G

Using the command you provided led me to two directories...  USB_HDD_12 and USB_HDD_11.  Those seems to be USB attached hard drives.  But here's the thing... There is only one USB attached hard drive, and exploring the shares it show as USB_HDD_12.  I wouldn't think this would be a part of the root file system, but I'm not sure how to tell.  Furthermore, it shows 59G in USB_HDD_12, which wouldn't fit in the 4G md0.  USB_HDD_11, however, shows 3.4G, which would be just enough to fill up md0.  Also, I came across another command which is supposed to show only large files in a specific directory/drive/patition/whatever (I don't really understand).  This command is:

 

find / -xdev -type f -size +100M

 

This command showed the large files in USB_HDD_11, but not those in USB_HDD-12.  So I'm guessing I need to delete/remove the files in USB-HDD_11.

 

Here is the output...

 

root@PRANAS-TNG:/# find / -xdev -type f -size +100M
/var/cores/core-readynasd
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/pvranchad_book.xml
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/PVRanchAD (1).xml
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/pvranchad_pax.xml
/media/USB_HDD_11/Pat/Aerial Footage - Max/Princeville Ranch Cut 3.mp4

 

What do you think?

 

Message 5 of 8
StephenB
Guru

Re: System volume root's usage is 100% and /dev/md0 is using all 4G


@Craig_X wrote:

 

 

root@PRANAS-TNG:/# find / -xdev -type f -size +100M
/var/cores/core-readynasd
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/pvranchad_book.xml
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/PVRanchAD (1).xml
/media/USB_HDD_11/Pat/ATCO/ATCO DATA/pvranchad_pax.xml
/media/USB_HDD_11/Pat/Aerial Footage - Max/Princeville Ranch Cut 3.mp4

  


You can delete /var/cores/core-readynasd for sure.

 

You can also remount the root file system, so you can search through it without mount points from muddying the water.

 

To do this you enter

mount --bind / /mnt
find /mnt -xdev -type f -size +100M

...

 

In addition to searching, etc you also would delete files directly from /mnt.  

 

Maybe also try 

mount | grep -i USB_HDD_11

 

That should let you know if USB_HDD_11 is a mount point or just a folder.  It you don't see it in the output of the above command, then it is just a folder.  You can delete it's contents if you wish, but perhaps keep the folder itself.

 

When done, you unmount with umount /mnt 

Message 6 of 8
Craig_X
Aspirant

Re: System volume root's usage is 100% and /dev/md0 is using all 4G

mount | grep -i USB_HDD_11 didn't produce any output, so I guess USB_HDD_11 is a folder.  There was a lot in it, and I couldn't figure out an easy way to delete all the contents without deleting the folder, so I deleted the entire folder.  Now my usage is down to 16%.  Everything seems to be working, so I'm inclined to say that fixed the problem.  I guess I'll know after another day goes by and I can examine the logs.

 

I also unplugged the external USB drive.  When I did that, the folder USB_HDD_12 disappeared.  So I'm thinking the NAS creates a folder whenever a USB drive is attached, but it doesn't (or shouldn't) write the data to the root volume.  This is consisent with the fact that I looked in the USB_HDD_12 folder in the root and it was empty even though I could browse the share and see that there was data on the external drive.

 

I'll come back and mark as solved if everything is still working tomorrow.

 

Thanks for the help.

Message 7 of 8
mdgm-ntgr
NETGEAR Employee Retired

Re: System volume root's usage is 100% and /dev/md0 is using all 4G

In future before deleting a core file I'd check the date it was created/modified doing e.g.

 

# ls -la /var/cores

If the date is recent then it may be useful for troubleshooting an issue in which case you should copy it to the data volume before deleting it if you need to delete it.

 

If the core file is very old then it's probably not very useful at all anymore.

Message 8 of 8
Top Contributors
Discussion stats
  • 7 replies
  • 6076 views
  • 1 kudo
  • 3 in conversation
Announcements