RN4220X shows 3TB free of 36TB volume but file writes report no free space

Question

Hi,One of our&nbsp;ReadyNAS 4220s (firmware 6.8.0)&nbsp;has started playing up and we can't write to it unless we reboot it.The problem goes away for about 48 hours then comes back again. Any writes to the SMB shares on the 36TB volume&nbsp;fail and even trying to create files via an SSH session to the console fails. Free space is&nbsp;shown at the command line and GUI as being 3TB but applications report no free space when trying to write.We can delete files but even after doing that creating new ones&nbsp;fails.Output from various commands is shown below:&nbsp;root@MPNAS01:/vol1# btrfs fi df /vol1Data, single: total=36.31TiB, used=33.37TiBSystem, DUP: total=32.00MiB, used=4.34MiBSystem, single: total=4.00MiB, used=0.00BMetadata, DUP: total=7.00GiB, used=3.74GiBGlobalReserve, single: total=347.41MiB, used=0.00B-----------------------------------------------------------------------root@MPNAS01:/vol1# btrfs filesystem usage /vol1Overall:&nbsp;&nbsp;&nbsp; Device size:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 36.34TiB&nbsp;&nbsp;&nbsp; Device allocated:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 36.33TiB&nbsp;&nbsp;&nbsp; Device unallocated:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 16.00GiB&nbsp;&nbsp;&nbsp; Device missing:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0.00B&nbsp;&nbsp;&nbsp; Used:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 33.38TiB&nbsp;&nbsp;&nbsp; Free (estimated):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.95TiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (min: 2.95TiB)&nbsp;&nbsp;&nbsp; Data ratio:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1.00&nbsp;&nbsp;&nbsp; Metadata ratio:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.00&nbsp;&nbsp;&nbsp; Global reserve:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 347.41MiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (used: 0.00B)Data,single: Size:36.31TiB, Used:33.37TiB&nbsp;&nbsp; /dev/md127&nbsp;&nbsp;&nbsp;&nbsp; 36.31TiBMetadata,DUP: Size:7.00GiB, Used:3.74GiB&nbsp;&nbsp; /dev/md127&nbsp;&nbsp;&nbsp;&nbsp; 14.00GiBSystem,single: Size:4.00MiB, Used:0.00B&nbsp;&nbsp; /dev/md127&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.00MiBSystem,DUP: Size:32.00MiB, Used:4.34MiB&nbsp;&nbsp; /dev/md127&nbsp;&nbsp;&nbsp;&nbsp; 64.00MiBUnallocated:&nbsp;&nbsp; /dev/md127&nbsp;&nbsp;&nbsp;&nbsp; 16.00GiB------------------------------------------------------------------------------------root@MPNAS01:/vol1# btrfs fi showLabel: '43f60230:root'&nbsp; uuid: 1f1056b9-4886-4c5f-bf80-94ec5ae8a07f&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Total devices 1 FS bytes used 547.11MiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; devid&nbsp;&nbsp;&nbsp; 1 size 4.00GiB used 3.90GiB path /dev/md0Label: '43f60230:vol1'&nbsp; uuid: c025ae0d-299c-49fd-ac86-76ad8225e4a1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Total devices 1 FS bytes used 33.36TiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; devid&nbsp;&nbsp;&nbsp; 1 size 36.34TiB used 36.33TiB path /dev/md127--------------------------------------------------------------------------------------I've found this in dmesg if it's any help:[171050.122965] ------------[ cut here ]------------[171050.122970] WARNING: CPU: 3 PID: 27873 at fs/btrfs/qgroup.c:2989 btrfs_qgroup_free_meta+0x92/0xa0()[171050.122970] Modules linked in: vpd(PO)[171050.122973] CPU: 3 PID: 27873 Comm: smbd Tainted: P&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; W&nbsp; O&nbsp;&nbsp;&nbsp; 4.4.79.x86_64.1 #1[171050.122974] Hardware name: NETGEAR ReadyNAS 4220/ReadyNAS 4220, BIOS 4.6.5 10/10/2013[171050.122975]&nbsp; 0000000000000000 ffff8801ea627b50 ffffffff883d9498 0000000000000000[171050.122977]&nbsp; ffffffff88cfc618 ffff8801ea627b88 ffffffff8806e41c ffff8800d480c800[171050.122979]&nbsp; 0000000000001000 ffff8800d480c800 0000000000000002 ffff8801239a5510[171050.122980] Call Trace:[171050.122983]&nbsp; [&lt;ffffffff883d9498&gt;] dump_stack+0x4d/0x65[171050.122985]&nbsp; [&lt;ffffffff8806e41c&gt;] warn_slowpath_common+0x7c/0xb0[171050.122986]&nbsp; [&lt;ffffffff8806e505&gt;] warn_slowpath_null+0x15/0x20[171050.122988]&nbsp; [&lt;ffffffff8835ef12&gt;] btrfs_qgroup_free_meta+0x92/0xa0[171050.122990]&nbsp; [&lt;ffffffff882dfe55&gt;] btrfs_delalloc_reserve_metadata+0x2b5/0x370[171050.122992]&nbsp; [&lt;ffffffff883081dd&gt;] __btrfs_buffered_write+0x19d/0x690[171050.122993]&nbsp; [&lt;ffffffff8830c2e9&gt;] btrfs_file_write_iter+0x369/0x490[171050.122995]&nbsp; [&lt;ffffffff8830bf80&gt;] ? btrfs_sync_file+0x340/0x340[171050.122997]&nbsp; [&lt;ffffffff88181389&gt;] aio_run_iocb+0x209/0x290[171050.122999]&nbsp; [&lt;ffffffff88185e12&gt;] ? fcntl_getlk+0x132/0x160[171050.123001]&nbsp; [&lt;ffffffff8815946e&gt;] ? __fdget+0xe/0x10[171050.123002]&nbsp; [&lt;ffffffff88182a84&gt;] do_io_submit+0x2a4/0x4c0[171050.123004]&nbsp; [&lt;ffffffff88182cab&gt;] SyS_io_submit+0xb/0x10[171050.123006]&nbsp; [&lt;ffffffff88a586d7&gt;] entry_SYSCALL_64_fastpath+0x12/0x6a[171050.123014] ---[ end trace 897247a73f6e674d ]-------------------------------------------------------------------------------------------------------------------&nbsp;If anyone can shed any light on the issue I'd be grateful as I've spent a couple of hours trawling the Internet and not found anything which seems to relate exactly to this problem. FYI we're staying on firmware 6.8.0 because of an issue flagged up by Veeam with later versions.&nbsp;Thanks,Mark.

markwhitmarsh · Answer

A bit of extra information:root@MPNAS01:/vol1# btrfs fi showLabel: '43f60230:root'&nbsp; uuid: 1f1056b9-4886-4c5f-bf80-94ec5ae8a07f&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Total devices 1 FS bytes used 547.11MiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; devid&nbsp;&nbsp;&nbsp; 1 size 4.00GiB used 3.90GiB path /dev/md0Label: '43f60230:vol1'&nbsp; uuid: c025ae0d-299c-49fd-ac86-76ad8225e4a1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Total devices 1 FS bytes used 33.35TiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; devid&nbsp;&nbsp;&nbsp; 1 size 36.34TiB used 36.33TiB path /dev/md127----------------------------------------------------------------------------------------------------If I'm reading this correctly it shows we need to run a balance on /dev/md127 - can anyone confirm that?Also, I forgot to add we don't have support for these devices from Netgear so this community is&nbsp;our only hope!&nbsp;

bedlam1 · Answer

Something is filling/has filled up your 4GB root directory to an extent the file system cannot operate correctly.

You need to look there for large files (probably logs from an installed app) and consider deleting them

markwhitmarsh · Answer

According to these outputs&nbsp;there is plenty of space on the root file system:&nbsp;root@MPNAS01:/vol1# btrfs fi df /Data, single: total=3.47GiB, used=535.24MiBSystem, DUP: total=8.00MiB, used=4.00KiBSystem, single: total=4.00MiB, used=0.00BMetadata, DUP: total=204.56MiB, used=11.87MiBMetadata, single: total=8.00MiB, used=0.00BGlobalReserve, single: total=16.00MiB, used=0.00B&nbsp;root@MPNAS01:/vol1# df -h /Filesystem&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Size&nbsp; Used Avail Use% Mounted on/dev/md0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.0G&nbsp; 575M&nbsp; 3.1G&nbsp; 16% /&nbsp;So why does this show not much space left? Is it to do with balancing?root@MPNAS01:/vol1# btrfs fi showLabel: '43f60230:root'&nbsp; uuid: 1f1056b9-4886-4c5f-bf80-94ec5ae8a07f&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Total devices 1 FS bytes used 547.11MiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; devid&nbsp;&nbsp;&nbsp; 1 size 4.00GiB used 3.90GiB path /dev/md0&nbsp;

bedlam1 · Answer

Sorry I don't know that, hopefully others will

StephenB · Answer

Try balancing both the OS partition and the data volume.&nbsp; You can balance the data volume from the Web UI if you like.

Forum Discussion

RN4220X shows 3TB free of 36TB volume but file writes report no free space

9 Replies

Related Content

How to get volume back to read-write mode

Issues after firmware upgrade RN4220X

Volume Expansion question

RN4220X backplane question

Lost my Volume Readynas 3138 Pulled Drives

NETGEAR Academy

ProSupport for Business