NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.

Forum Discussion

x219c's avatar
x219c
Luminary
Jan 15, 2019
Solved

Root full due to old volume

So, I need to get better with bash... But in the meantime, I've got a problem. My root is full.

 

I think I've narrowed down how it happened. I had an eda500 with a volume, eda2 on it. A few of the drives failed, and since I only had raid6, I lost the volume. No big deal, it's all backed up. I put in new drives, but it wouldn't let me use the eda2 name. Strange, but I just named it eda this time, instead.

 

Well, some of my apps didn't get the memo, and were trying to move files to this volume. I've fixed that part of the problem. Assuming that's actually the problem... Now I need to clean up the mess, but don't know the right sequence or commands.. I'm a windows guy. So here's where I'm at:

 

root@Hydra:~# df -h                                                                                                         
Filesystem      Size  Used Avail Use% Mounted on                                                                            
udev             10M  4.0K   10M   1% /dev                                                                                  
/dev/md0        4.0G  3.6G   30M 100% /                                                                                     
tmpfs           7.8G  8.0K  7.8G   1% /dev/shm                                                                              
tmpfs           7.8G  3.6M  7.8G   1% /run                                                                                  
tmpfs           3.9G  3.5M  3.9G   1% /run/lock                                                                             
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup                                                                        
/dev/md127       13T  4.9T  7.9T  39% /data                                                                                 
/dev/md127       13T  4.9T  7.9T  39% /apps                                                                                 
/dev/md127       13T  4.9T  7.9T  39% /home                                                                                 
/dev/md126       11T  4.2T  6.8T  39% /eda                                                                                  
shm              64M     0   64M   0% /apps/docker-cli-rnapp/containers/2281928361f944e1ba885f4b49fceb4061e08b46b112e0923327
4193f0a78856/mounts/shm                                                                                                     
shm              64M     0   64M   0% /apps/docker-cli-rnapp/containers/1e9868e9103924f992bb0fb3d915b2b114be0d025fa7cd9f2b6a
e33d7d5e70c0/mounts/shm
root@Hydra:/# du -hsx * | sort -rh | head -10                                                                               
du: cannot access 'proc/24047/task/24047/fd/3': No such file or directory                                                   
du: cannot access 'proc/24047/task/24047/fdinfo/3': No such file or directory                                               
du: cannot access 'proc/24047/fd/3': No such file or directory                                                              
du: cannot access 'proc/24047/fdinfo/3': No such file or directory                                                          
69G     apps                                                                                                                
2.5G    eda2                                                                                                                
539M    var                                                                                                                 
476M    usr                                                                                                                 
68M     lib                                                                                                                 
51M     data                                                                                                                
30M     frontview                                                                                                           
11M     sbin                                                                                                                
11M     etc                                                                                                                 
6.4M    bin 
root@Hydra:/# rm -rf eda2                                                                                                   
rm: cannot remove 'eda2/._share': Operation not permitted
  • There are a lot of mount points in the root partition, which can conceal the actual space usage in the partition.

     

    So start by entering 

    mount --bind / /mnt

    That mounts sysroot as /mnt.

     

    Then look at the space usage in /mnt.  For instance, 

    cd /mnt
    du -hsx * | sort -rh | head -10

    You can of course then descend into folders and repeat the du command (or use other commands to see what's taking the space).

     

    BTW, anything that is under a mount point can be deleted.

     


    x219c wrote:
    root@Hydra:/# rm -rf eda2                                                                                                   
    rm: cannot remove 'eda2/._share': Operation not permitted

    It's hard to be certain from what you posted, but I think it's likely that eda2 is actually the mount point for your new eda volume.  So check if the folders/files in eda2 match whats in eda.

     

    If I'm correct, eda2 is a btrfs volume - not the same as an ordinary folder.  That would explain why the rm command failed.  If it had worked you'd have deleted all the files on eda.

     

    When you're done with /mnt, you unmount it with

    cd /
    umount /mnt

    The cd matters, the umount will fail if you are in a folder of /mnt.

     

     

3 Replies

Replies have been turned off for this discussion
  • I could be barking up the wrong tree entirely, of course. This was my only theory, but the 69G next to apps... Well, IDK.

    • StephenB's avatar
      StephenB
      Guru - Experienced User

      There are a lot of mount points in the root partition, which can conceal the actual space usage in the partition.

       

      So start by entering 

      mount --bind / /mnt

      That mounts sysroot as /mnt.

       

      Then look at the space usage in /mnt.  For instance, 

      cd /mnt
      du -hsx * | sort -rh | head -10

      You can of course then descend into folders and repeat the du command (or use other commands to see what's taking the space).

       

      BTW, anything that is under a mount point can be deleted.

       


      x219c wrote:
      root@Hydra:/# rm -rf eda2                                                                                                   
      rm: cannot remove 'eda2/._share': Operation not permitted

      It's hard to be certain from what you posted, but I think it's likely that eda2 is actually the mount point for your new eda volume.  So check if the folders/files in eda2 match whats in eda.

       

      If I'm correct, eda2 is a btrfs volume - not the same as an ordinary folder.  That would explain why the rm command failed.  If it had worked you'd have deleted all the files on eda.

       

      When you're done with /mnt, you unmount it with

      cd /
      umount /mnt

      The cd matters, the umount will fail if you are in a folder of /mnt.

       

       

      • x219c's avatar
        x219c
        Luminary

        Thanks Stephen! Looks like I must have had a docker image in there. All good now!

NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology! 

Join Us!

ProSupport for Business

Comprehensive support plans for maximum network uptime and business peace of mind.

 

Learn More