No space left on device (BUT THERE IS??)

[robertlight]

Firmware:  6.6.1

I get "No space left on device" errors but it appears my filesystem has room (though check out the bold text below)

I have balanced my system, scrubbed it, defrag etc...  can't seem to get any room on /dev/md0

# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 10240 4 10236 1% /dev
/dev/md0 4190208 2008612 1805084 53% /
tmpfs 1015700 0 1015700 0% /dev/shm
tmpfs 1015700 6060 1009640 1% /run
tmpfs 507852 1496 506356 1% /run/lock
tmpfs 1015700 0 1015700 0% /sys/fs/cgroup
/dev/md127 3887643136 857696088 3028480296 23% /data
/dev/md127 3887643136 857696088 3028480296 23% /apps
/dev/md127 3887643136 857696088 3028480296 23% /home

# df -i
root@nas-F6-47-9C:/# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 253307 509 252798 1% /dev
/dev/md0 0 0 0 - /
tmpfs 253925 1 253924 1% /dev/shm
tmpfs 253925 716 253209 1% /run
tmpfs 253925 145 253780 1% /run/lock
tmpfs 253925 9 253916 1% /sys/fs/cgroup
/dev/md127 0 0 0 - /data
/dev/md127 0 0 0 - /apps
/dev/md127 0 0 0 - /home

# btrfs fi show /dev/md0
Label: '43f6479c:root' uuid: d3f0f4c0-2aab-4d2b-b41c-84f257e72ed5
Total devices 1 FS bytes used 1.87GiB
devid 1 size 4.00GiB used 4.00GiB path /dev/md0

# btrfs fi show /dev/md127
Label: '43f6479c:data' uuid: 1a61de50-cc29-4839-b1d3-04a8500c5ec5
Total devices 1 FS bytes used 816.43GiB
devid 1 size 3.62TiB used 879.07GiB path /dev/md127

# cd /
# cat > foo
bash: foo: No space left on device

Any ideas on what I can do??

[mdgm-ntgr]

# df -i
is useless with BTRFS as it doesn't use inodes.

We only mention it here as we use EXT4 on ARM systems and EXT2/3/4 on legacy systems for the root volume.

The output you bolded shows that the space on the 4GB root volume is fully allocated.

If you do e.g.

# btrfs fi df /

or

# btrfs fi usage /

You should see that not only is it fully allocated an allocation is fully in use.

The root volume should be much less than 53% full. So you should find what's using up a lot of space on the root volume that shouldn't be and deal with that.

[robertlight]

You are indeed correct.

I got rid of lots of stuff in /var/cache/apt/.... freed up some space.... and then in no time... it all got used up again.

I'm now sitting full again (see below)

Is there a way of discovering who is sucking up my /dev/md0 disk space??

# btrfs fi usage /

Overall:

    Device size:   4.00GiB

    Device allocated:   4.00GiB

    Device unallocated:     0.00B

    Device missing:     0.00B

    Used:   1.89GiB

    Free (estimated):   1.73GiB (min: 1.73GiB)

    Data ratio:       1.00

    Metadata ratio:       1.96

    Global reserve:   16.00MiB (used: 0.00B)

Data,single: Size:3.57GiB, Used:1.84GiB

   /dev/md0   3.57GiB

Metadata,single: Size:8.00MiB, Used:0.00B

   /dev/md0   8.00MiB

Metadata,DUP: Size:204.56MiB, Used:26.61MiB

   /dev/md0 409.12MiB

System,single: Size:4.00MiB, Used:0.00B

   /dev/md0   4.00MiB

System,DUP: Size:8.00MiB, Used:16.00KiB

   /dev/md0   16.00MiB

Unallocated:

   /dev/md0     0.00B

[robertlight]

Perhaps a du -s * in / might help you identify where to look

# cd /

# du -s *

6800 apps

7624 bin

0 boot

66571036 data

4 dev

149612 etc

28756 frontview

34068 lib

4 lib64

0 media

0 mnt

3960 opt

du: cannot access 'proc/9427/task/9427/fd/3': No such file or directory

du: cannot access 'proc/9427/task/9427/fdinfo/3': No such file or directory

du: cannot access 'proc/9427/fd/3': No such file or directory

du: cannot access 'proc/9427/fdinfo/3': No such file or directory

0 proc

220672 root

7064 run

10364 sbin

0 selinux

0 srv

0 sys

4 syslog

0 tmp

675184 usr

630276 var

[mdgm-ntgr]

You could do e.g.

# mount --bind / /mnt
# du -csh /mnt/*

[robertlight]

I found that /root had a ton of stuff in /root/.dropbox  probably from a long time ago when I tried to install dropbox (probably as root!)

I got rid of that cruft and got things back to some semblence of normal.

I still don't quite understand the bold info below... why does it say "4.00GB used"?

# du -csh /mnt/*

0 /mnt/apps

7.5M /mnt/bin

0 /mnt/boot

0 /mnt/data

24K /mnt/dev

147M /mnt/etc

29M /mnt/frontview

0 /mnt/home

34M /mnt/lib

4.0K /mnt/lib64

0 /mnt/media

0 /mnt/mnt

3.9M /mnt/opt

0 /mnt/proc

80K /mnt/root

0 /mnt/run

11M /mnt/sbin

0 /mnt/selinux

0 /mnt/srv

0 /mnt/sys

4.0K /mnt/syslog

0 /mnt/tmp

660M /mnt/usr

616M /mnt/var

1.5G total

root@nas-F6-47-9C:/# btrfs fi show /mnt

Label: '43f6479c:root'  uuid: d3f0f4c0-2aab-4d2b-b41c-84f257e72ed5

Total devices 1 FS bytes used 1.60GiB

devid    1 size 4.00GiB used 4.00GiB path /dev/md0

[mdgm-ntgr]

That's because all 4GB of space on the root volume has been allocated to e.g. data, metadata and system data. The allocation for system data is fixed.

BTRFS dynamically allocates space to data and metadata as needed.

You should have a look under /mnt/usr and /mnt/var to see if there's anything that shouldn't be on there.

You  may need to run a balance on the root volume as well.

---