ReadyNas 628x Mysterious Storage Consumption

Update on this one. I worked with pwptech in PM.

We can see that there are the 3 shares/LUNs of interest, that are taking most space on the NAS:

ID 0/281 = 3.47TiB = /data/Backups
ID 0/5548 = 90.09GiB = /data/Software
ID 0/31827 = 6.65TiB = /data/VeeamBackups

"VeeamBackups" is the LUN. This LUN is configured to 4.5TB (thick) as confirmed via screenshots of the config and we can see this too, from the command-line:

root@NAS-01:~# ls -lahR /data/VeeamBackups/
/data/VeeamBackups/:
total 32K
drwxr-xr-x 1 root root 12 Mar 27 2020 .
drwxr-xr-x 1 root root 142 Feb 22 10:35 ..
drwxr-xr-x 1 root root 74 Mar 27 2020 .iscsi

/data/VeeamBackups/.iscsi:
total 4.6T
drwxr-xr-x 1 root root 74 Mar 27 2020 .
drwxr-xr-x 1 root root 12 Mar 27 2020 ..
-rw-r--r-- 1 root root 4.5T Feb 28 18:29 iscsi_lun_backing_store <<<=====
-rw-r--r-- 1 root root 36 Mar 27 2020 .serial_number

Yet still, the filesystem is allocating 2TB more than expected, to house this NAS. This lines up with what pwptech reported in the first place, that he/she lost 2TB "out of the blue". I looked at the Status history on the NAS and found that the NAS was stable at around 20% space left, leading up to the episode:

[21/01/16 11:50:44 MST] warning:volume:LOGMSG_VOLUME_USAGE_WARNING Less than 20% of volume data's capacity is free. Performance on volume data will degrade if additional capacity is consumed. NETGEAR recommends that you add capacity to avoid performance degradation.
[21/01/31 11:06:42 MST] warning:volume:LOGMSG_VOLUME_USAGE_WARNING Less than 20% of volume data's capacity is free. Performance on volume data will degrade if additional capacity is consumed. NETGEAR recommends that you add capacity to avoid performance degradation.
[21/02/02 09:32:22 MST] warning:volume:LOGMSG_VOLUME_USAGE_WARNING Less than 20% of volume data's capacity is free. Performance on volume data will degrade if additional capacity is consumed. NETGEAR recommends that you add capacity to avoid performance degradation.
[21/02/12 13:05:30 MST] warning:volume:LOGMSG_VOLUME_USAGE_WARNING Less than 20% of volume data's capacity is free. Performance on volume data will degrade if additional capacity is consumed. NETGEAR recommends that you add capacity to avoid performance degradation.

Then suddenly it dropped to less 5% (essentially, the NAS filled up):

[21/02/20 16:47:25 MST] warning:volume:LOGMSG_VOLUME_USAGE_CRITICAL Less than 5% of volume data's capacity is free. data's performance is degraded and you risk running out of usable space. To improve performance and stability, you must add capacity or make free space.

What preceeded this was a defragmentation on the NAS - which would also defrag the LUN file (iscsi_lun_backing_store):

[21/02/13 05:11:04 MST] notice:volume:LOGMSG_DEFRAGSTART_VOLUME Defragmentation started for volume data.
[21/02/13 13:16:14 MST] notice:volume:LOGMSG_DEFRAGEND_VOLUME Defragmentation complete for volume data.

This is the only time a defrag was ever run on the NAS and a few days later, the NAS reported out of space condition. Keep in mind that these space warnings don't happen all the time so it is quite possible that the space issues happened just after the LUN was defragged or very shortly thereafter. So, the filesystem is using 6.65TiB to house a thick LUN of 4.5TB. This reminds of an issue other ReadyNAS users reported where defragging a LUN would balloon the space utilization on the NAS. mdgm might remember more about this but we saw several reports of it. It has something to do with extents on the LUN increasing/breaking after the defrag and likely more of a BTRFS issue than a NAS issue, per se. I very much suspect exactly the same thing happened here. The only possible remedy I think there is, is to defrag the LUN again, using a smaller block size:

btrfs fi defragment -t 8192 -v /data/VeeamBackups/.iscsi/iscsi_lun_backing_store

Or even using a 4K (4096) size could maybe help even more. It is not a guarantee to work but it is probably worth a shot, I think. mdgm could have some insight here too, if he remembers these issues - but it has been a couple of years now :)

Cheers