NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Tsp
Aug 14, 2016Aspirant
Balancing does not help with "no space left" despite there is capacity available.
I've run into the issue with ReadyNas 4220. Firmware 6.4.2
There were a lot of posts in community for "no space left" error but none worked for me.
# echo Test > /home/admin/test
-bash: /home/admin/test: No space left on device
# btrfs fi show
Label: '43f63420:root' uuid: 701c9fe6-995a-4572-b360-dd7986123e77
Total devices 1 FS bytes used 438.08MiB
devid 1 size 4.00GiB used 1.64GiB path /dev/md0
Label: '43f63420:prod' uuid: 96799cff-1ad8-4deb-a06b-758cacfea9bd
Total devices 1 FS bytes used 5.28TiB
devid 1 size 6.49TiB used 5.81TiB path /dev/md127
# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 985K 582 985K 1% /dev
/dev/md0 0 0 0 - /
tmpfs 985K 1 985K 1% /dev/shm
tmpfs 985K 723 985K 1% /run
tmpfs 985K 4 985K 1% /sys/fs/cgroup
tmpfs 985K 1 985K 1% /media
/dev/md127 0 0 0 - /prod
/dev/md127 0 0 0 - /home
/dev/md127 0 0 0 - /apps
/dev/md127 0 0 0 - /run/nfs4/home
/dev/md127 0 0 0 - /run/nfs4/prod/VMStorage
/dev/md127 0 0 0 - /run/nfs4/prod/ISO
# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 10M 4.0K 10M 1% /dev
/dev/md0 4.0G 467M 3.2G 13% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 13M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 3.9G 0 3.9G 0% /media
/dev/md127 6.5T 5.3T 1.2T 82% /prod
/dev/md127 6.5T 5.3T 1.2T 82% /home
/dev/md127 6.5T 5.3T 1.2T 82% /apps
/dev/md127 6.5T 5.3T 1.2T 82% /run/nfs4/home
/dev/md127 6.5T 5.3T 1.2T 82% /run/nfs4/prod/VMStorage
/dev/md127 6.5T 5.3T 1.2T 82% /run/nfs4/prod/ISO
# btrfs fi df /
Data, single: total=1.21GiB, used=425.52MiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=204.56MiB, used=12.53MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B
# btrfs fi df /prod
Data, single: total=5.80TiB, used=5.27TiB
System, DUP: total=32.00MiB, used=672.00KiB
Metadata, DUP: total=9.00GiB, used=2.65GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs filesystem usage /prod
Overall:
Device size: 6.49TiB
Device allocated: 5.83TiB
Device unallocated: 675.58GiB
Device missing: 0.00B
Used: 5.28TiB
Free (estimated): 1.18TiB (min: 871.52GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:5.79TiB, Used:5.27TiB
/dev/md127 5.79TiB
Metadata,DUP: Size:17.50GiB, Used:2.65GiB
/dev/md127 35.00GiB
System,DUP: Size:64.00MiB, Used:672.00KiB
/dev/md127 128.00MiB
Unallocated:
/dev/md127 675.58GiB
BTRFS info (device md127): 634 enospc errors during balance
md: requested-resync of RAID array md127
md: minimum _guaranteed_ speed: 30000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for requested-resync.
md: using 128k window, over a total of 995354752k.
md: md127: requested-resync done.
BTRFS info (device md127): resizing devid 1
After balancing start from UI:
# dmesg | tail
BTRFS info (device md127): relocating block group 5189960466432 flags 1
BTRFS info (device md127): found 1307 extents
BTRFS info (device md127): found 1307 extents
BTRFS info (device md127): relocating block group 5165264404480 flags 1
BTRFS info (device md127): found 91 extents
BTRFS info (device md127): found 91 extents
BTRFS info (device md127): relocating block group 4394317774848 flags 1
BTRFS info (device md127): relocating block group 3251856474112 flags 1
BTRFS info (device md127): 329 enospc errors during balance
BTRFS info (device md127): 1 enospc errors during balance
I tried to free up to 20% on /prod and run balancing again. It made progress to 75% and dropped off again with same error.
After balancing attempt /prod usage raised from 80% to 83%.
Playing with balancing filted did not help as well
# btrfs balance start -v -dusage=0 /prod
Dumping filters: flags 0x1, state 0x0, force is off
DATA (flags 0x2): balancing, usage=0
Done, had to relocate 0 out of 5844 chunks
# btrfs balance start -v -dusage=10 /prod
Dumping filters: flags 0x1, state 0x0, force is off
DATA (flags 0x2): balancing, usage=10
Done, had to relocate 0 out of 5844 chunks
# btrfs balance start -v -dusage=20 /prod
Dumping filters: flags 0x1, state 0x0, force is off
DATA (flags 0x2): balancing, usage=20
ERROR: error during balancing '/prod' - No space left on device
There may be more info in syslog - try dmesg | tail
I also tried scrubling and defragmentation.
Would you please suggest what else I should try to fix the issue?
8 Replies
- sotrackLuminary
.
- mdgm-ntgrNETGEAR Employee Retired
Do you have a backup of your data?
Can you delete some files to get volume usage below 80% and then try again?- TspAspirant
Thanks for reply.
The NAS contains mainly Virtual Machines and we do not backup them.
I’ve freed space to 72% use
# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 10M 4.0K 10M 1% /dev
/dev/md0 4.0G 472M 3.3G 13% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 13M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 3.9G 0 3.9G 0% /media
/dev/md127 6.5T 4.1T 1.7T 72% /prod
/dev/md127 6.5T 4.1T 1.7T 72% /home
/dev/md127 6.5T 4.1T 1.7T 72% /apps
/dev/md127 6.5T 4.1T 1.7T 72% /run/nfs4/home
/dev/md127 6.5T 4.1T 1.7T 72% /run/nfs4/prod/VMStorage
/dev/md127 6.5T 4.1T 1.7T 72% /run/nfs4/prod/ISO
Balancing made progress to ~70% and failed again with same error.
I’ve got 80% in use back on /prod partition as a result of balancing run and still no luck with “no space left” resolve.
How can I figure out where 600GB disappeared while balancing?
# btrfs filesystem show
Label: '43f63420:root' uuid: 701c9fe6-995a-4572-b360-dd7986123e77
Total devices 1 FS bytes used 447.16MiB
devid 1 size 4.00GiB used 1.53GiB path /dev/md0
Label: '43f63420:prod' uuid: 96799cff-1ad8-4deb-a06b-758cacfea9bd
Total devices 1 FS bytes used 4.04TiB
devid 1 size 6.49TiB used 5.59TiB path /dev/md127
As I understand free workspace is 900GB. Isn't that sufficient?
Is there a way to check what exactly locks the space and which filter to use?
Is that normal that I have to keep more than 1TB free to keep the NAS operational?
Thank you in advance.
Related Content
NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!