Root full due to old volume

[x219c]

So, I need to get better with bash... But in the meantime, I've got a problem. My root is full.

I think I've narrowed down how it happened. I had an eda500 with a volume, eda2 on it. A few of the drives failed, and since I only had raid6, I lost the volume. No big deal, it's all backed up. I put in new drives, but it wouldn't let me use the eda2 name. Strange, but I just named it eda this time, instead.

Well, some of my apps didn't get the memo, and were trying to move files to this volume. I've fixed that part of the problem. Assuming that's actually the problem... Now I need to clean up the mess, but don't know the right sequence or commands.. I'm a windows guy. So here's where I'm at:

root@Hydra:~# df -h                                                                                                         
Filesystem      Size  Used Avail Use% Mounted on                                                                            
udev             10M  4.0K   10M   1% /dev                                                                                  
/dev/md0        4.0G  3.6G   30M 100% /                                                                                     
tmpfs           7.8G  8.0K  7.8G   1% /dev/shm                                                                              
tmpfs           7.8G  3.6M  7.8G   1% /run                                                                                  
tmpfs           3.9G  3.5M  3.9G   1% /run/lock                                                                             
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup                                                                        
/dev/md127       13T  4.9T  7.9T  39% /data                                                                                 
/dev/md127       13T  4.9T  7.9T  39% /apps                                                                                 
/dev/md127       13T  4.9T  7.9T  39% /home                                                                                 
/dev/md126       11T  4.2T  6.8T  39% /eda                                                                                  
shm              64M     0   64M   0% /apps/docker-cli-rnapp/containers/2281928361f944e1ba885f4b49fceb4061e08b46b112e0923327
4193f0a78856/mounts/shm                                                                                                     
shm              64M     0   64M   0% /apps/docker-cli-rnapp/containers/1e9868e9103924f992bb0fb3d915b2b114be0d025fa7cd9f2b6a
e33d7d5e70c0/mounts/shm

root@Hydra:/# du -hsx * | sort -rh | head -10                                                                               
du: cannot access 'proc/24047/task/24047/fd/3': No such file or directory                                                   
du: cannot access 'proc/24047/task/24047/fdinfo/3': No such file or directory                                               
du: cannot access 'proc/24047/fd/3': No such file or directory                                                              
du: cannot access 'proc/24047/fdinfo/3': No such file or directory                                                          
69G     apps                                                                                                                
2.5G    eda2                                                                                                                
539M    var                                                                                                                 
476M    usr                                                                                                                 
68M     lib                                                                                                                 
51M     data                                                                                                                
30M     frontview                                                                                                           
11M     sbin                                                                                                                
11M     etc                                                                                                                 
6.4M    bin 

root@Hydra:/# rm -rf eda2                                                                                                   
rm: cannot remove 'eda2/._share': Operation not permitted

[StephenB · Accepted solution]

There are a lot of mount points in the root partition, which can conceal the actual space usage in the partition.

So start by entering 

mount --bind / /mnt

That mounts sysroot as /mnt.

Then look at the space usage in /mnt.  For instance, 

cd /mnt
du -hsx * | sort -rh | head -10

You can of course then descend into folders and repeat the du command (or use other commands to see what's taking the space).

BTW, anything that is under a mount point can be deleted.

---

@x219c wrote:

root@Hydra:/# rm -rf eda2                                                                                                   
rm: cannot remove 'eda2/._share': Operation not permitted

It's hard to be certain from what you posted, but I think it's likely that eda2 is actually the mount point for your new eda volume.  So check if the folders/files in eda2 match whats in eda.

If I'm correct, eda2 is a btrfs volume - not the same as an ordinary folder.  That would explain why the rm command failed.  If it had worked you'd have deleted all the files on eda.

When you're done with /mnt, you unmount it with

cd /
umount /mnt

The cd matters, the umount will fail if you are in a folder of /mnt.

[x219c]

I could be barking up the wrong tree entirely, of course. This was my only theory, but the 69G next to apps... Well, IDK.

[StephenB · Accepted solution]

There are a lot of mount points in the root partition, which can conceal the actual space usage in the partition.

So start by entering 

mount --bind / /mnt

That mounts sysroot as /mnt.

Then look at the space usage in /mnt.  For instance, 

cd /mnt
du -hsx * | sort -rh | head -10

You can of course then descend into folders and repeat the du command (or use other commands to see what's taking the space).

BTW, anything that is under a mount point can be deleted.

---

@x219c wrote:

root@Hydra:/# rm -rf eda2                                                                                                   
rm: cannot remove 'eda2/._share': Operation not permitted

It's hard to be certain from what you posted, but I think it's likely that eda2 is actually the mount point for your new eda volume.  So check if the folders/files in eda2 match whats in eda.

If I'm correct, eda2 is a btrfs volume - not the same as an ordinary folder.  That would explain why the rm command failed.  If it had worked you'd have deleted all the files on eda.

When you're done with /mnt, you unmount it with

cd /
umount /mnt

The cd matters, the umount will fail if you are in a folder of /mnt.

[x219c]

Thanks Stephen! Looks like I must have had a docker image in there. All good now!