RN4220X shows 3TB free of 36TB volume but file writes report no free space

[markwhitmarsh]

Hi,

One of our ReadyNAS 4220s (firmware 6.8.0) has started playing up and we can't write to it unless we reboot it.

The problem goes away for about 48 hours then comes back again. Any writes to the SMB shares on the 36TB volume fail and even trying to create files via an SSH session to the console fails. Free space is shown at the command line and GUI as being 3TB but applications report no free space when trying to write.

We can delete files but even after doing that creating new ones fails.

Output from various commands is shown below:

root@MPNAS01:/vol1# btrfs fi df /vol1
Data, single: total=36.31TiB, used=33.37TiB
System, DUP: total=32.00MiB, used=4.34MiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=7.00GiB, used=3.74GiB
GlobalReserve, single: total=347.41MiB, used=0.00B

-----------------------------------------------------------------------

root@MPNAS01:/vol1# btrfs filesystem usage /vol1
Overall:
    Device size:                  36.34TiB
    Device allocated:             36.33TiB
    Device unallocated:           16.00GiB
    Device missing:                  0.00B
    Used:                         33.38TiB
    Free (estimated):              2.95TiB      (min: 2.95TiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              347.41MiB      (used: 0.00B)

Data,single: Size:36.31TiB, Used:33.37TiB
   /dev/md127     36.31TiB

Metadata,DUP: Size:7.00GiB, Used:3.74GiB
   /dev/md127     14.00GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/md127      4.00MiB

System,DUP: Size:32.00MiB, Used:4.34MiB
   /dev/md127     64.00MiB

Unallocated:
   /dev/md127     16.00GiB

------------------------------------------------------------------------------------

root@MPNAS01:/vol1# btrfs fi show
Label: '43f60230:root'  uuid: 1f1056b9-4886-4c5f-bf80-94ec5ae8a07f
        Total devices 1 FS bytes used 547.11MiB
        devid    1 size 4.00GiB used 3.90GiB path /dev/md0

Label: '43f60230:vol1'  uuid: c025ae0d-299c-49fd-ac86-76ad8225e4a1
        Total devices 1 FS bytes used 33.36TiB
        devid    1 size 36.34TiB used 36.33TiB path /dev/md127

--------------------------------------------------------------------------------------

I've found this in dmesg if it's any help:

[171050.122965] ------------[ cut here ]------------
[171050.122970] WARNING: CPU: 3 PID: 27873 at fs/btrfs/qgroup.c:2989 btrfs_qgroup_free_meta+0x92/0xa0()
[171050.122970] Modules linked in: vpd(PO)
[171050.122973] CPU: 3 PID: 27873 Comm: smbd Tainted: P        W  O    4.4.79.x86_64.1 #1
[171050.122974] Hardware name: NETGEAR ReadyNAS 4220/ReadyNAS 4220, BIOS 4.6.5 10/10/2013
[171050.122975]  0000000000000000 ffff8801ea627b50 ffffffff883d9498 0000000000000000
[171050.122977]  ffffffff88cfc618 ffff8801ea627b88 ffffffff8806e41c ffff8800d480c800
[171050.122979]  0000000000001000 ffff8800d480c800 0000000000000002 ffff8801239a5510
[171050.122980] Call Trace:
[171050.122983]  [<ffffffff883d9498>] dump_stack+0x4d/0x65
[171050.122985]  [<ffffffff8806e41c>] warn_slowpath_common+0x7c/0xb0
[171050.122986]  [<ffffffff8806e505>] warn_slowpath_null+0x15/0x20
[171050.122988]  [<ffffffff8835ef12>] btrfs_qgroup_free_meta+0x92/0xa0
[171050.122990]  [<ffffffff882dfe55>] btrfs_delalloc_reserve_metadata+0x2b5/0x370
[171050.122992]  [<ffffffff883081dd>] __btrfs_buffered_write+0x19d/0x690
[171050.122993]  [<ffffffff8830c2e9>] btrfs_file_write_iter+0x369/0x490
[171050.122995]  [<ffffffff8830bf80>] ? btrfs_sync_file+0x340/0x340
[171050.122997]  [<ffffffff88181389>] aio_run_iocb+0x209/0x290
[171050.122999]  [<ffffffff88185e12>] ? fcntl_getlk+0x132/0x160
[171050.123001]  [<ffffffff8815946e>] ? __fdget+0xe/0x10
[171050.123002]  [<ffffffff88182a84>] do_io_submit+0x2a4/0x4c0
[171050.123004]  [<ffffffff88182cab>] SyS_io_submit+0xb/0x10
[171050.123006]  [<ffffffff88a586d7>] entry_SYSCALL_64_fastpath+0x12/0x6a
[171050.123014] ---[ end trace 897247a73f6e674d ]---

----------------------------------------------------------------------------------------------------------------

If anyone can shed any light on the issue I'd be grateful as I've spent a couple of hours trawling the Internet and not found anything which seems to relate exactly to this problem. FYI we're staying on firmware 6.8.0 because of an issue flagged up by Veeam with later versions.

Thanks,

Mark.

[markwhitmarsh]

A bit of extra information:

root@MPNAS01:/vol1# btrfs fi show
Label: '43f60230:root'  uuid: 1f1056b9-4886-4c5f-bf80-94ec5ae8a07f
        Total devices 1 FS bytes used 547.11MiB
        devid    1 size 4.00GiB used 3.90GiB path /dev/md0

Label: '43f60230:vol1'  uuid: c025ae0d-299c-49fd-ac86-76ad8225e4a1
        Total devices 1 FS bytes used 33.35TiB
        devid    1 size 36.34TiB used 36.33TiB path /dev/md127

----------------------------------------------------------------------------------------------------

If I'm reading this correctly it shows we need to run a balance on /dev/md127 - can anyone confirm that?

Also, I forgot to add we don't have support for these devices from Netgear so this community is our only hope!

[bedlam1]

Something is filling/has filled up your 4GB root directory to an extent the file system cannot operate correctly.

You need to look there for large files (probably logs from an installed app) and consider deleting them 

[markwhitmarsh]

According to these outputs there is plenty of space on the root file system:

root@MPNAS01:/vol1# btrfs fi df /
Data, single: total=3.47GiB, used=535.24MiB
System, DUP: total=8.00MiB, used=4.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=204.56MiB, used=11.87MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B

root@MPNAS01:/vol1# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        4.0G  575M  3.1G  16% /

So why does this show not much space left? Is it to do with balancing?

root@MPNAS01:/vol1# btrfs fi show
Label: '43f60230:root'  uuid: 1f1056b9-4886-4c5f-bf80-94ec5ae8a07f
        Total devices 1 FS bytes used 547.11MiB
        devid    1 size 4.00GiB used 3.90GiB path /dev/md0 

[bedlam1]

Sorry I don't know that, hopefully others will

[StephenB]

Try balancing both the OS partition and the data volume.  You can balance the data volume from the Web UI if you like.

[markwhitmarsh]

Is there any risk to running the balance? I'm not sure we can backup the 33TB elsewhere!

[mdgm-ntgr]

Well if the data’s not important then a factory reset (wipes all data, settings, eveyrything) would be an option.

If your data’s important to you then you should backup that data regularly and factor the cost of doing so into your purchase decisions so you can budget appropriately.

Which services are you running? SMB? SSH? Anti-virus? etc.

At 90% full you’d want to free up some space before considering running a balance.

[markwhitmarsh]

Hi mdgm,

Sorry, perhaps I should have made it clear that the ReadyNAS is a backup device, ie: we store backups on it - mainly month-end ones to be kept long term just in case. We wouldn't dream of keeping live production data or even primary backups on a device like this. The more I read about btrfs the less I like it and if we were keeping the two ReadyNAS devices beyond the short term I would get rid of it and use ext4 instead. We did start running backups of the ReadyNAS in question to another NAS but that's also a ReadyNAS and isn't the most reliable storage device either.

In the meantime I'm trying to coax these things along until they get replaced. I now realise the problem is partly one of our making because we haven't been running scheduled balances and it seems like it's a very good idea to do that. I've freed up some space on the (currently) troubled NAS but despite a lot of reading around I can't find out how much free space we need before running a balance, is 11% enough?

Regards,

Mark.

[mdgm-ntgr]

We start warning about volume usage when the volume exceeds 70% full. For optimal performance it would be best to keep volume usage under 85% full.

By keeping the volume usage down to a reasonable level the system should find it easier to balance and you're less likely to find the chunks on your data volume become fully allocated. Even without doing a balance the system will return empty chunks to unallocated space.