× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ISSUE - Unallocated/free space Volume DECREASING with no reason

Jota79
Aspirant

ISSUE - Unallocated/free space Volume DECREASING with no reason

I have a rn4420x array. My config is very simple:

 

   12 x SATA HDD 4TB total
        6 x HDD in RAID 10
        6 x HDD in RAID 10

   8 LUNS of 2TB in Thick provision - iSCSI

 

I have upgraded firmware from 6.4.2 to 6.9.6 four months ago.
Some days after upgrade date, I have realized to some strange behavior...

My unallocated space from my second volume is decreasing!!!

It seems there is some process that is eating my unallocated space.
I do NOT have snapshot.

 

To solve this strange issue, I have tried:

 

* Defrag
* Scrub
* Balance
* Reboot

This actions does not fix the problem.

I have tried to upgrade firmware one more time, from 6.9.6 to 6.10.2 ( last one )

No luck, second upgrade neither solve this strange issue.

 

What it is eating my unallocated/free space from my second volume? What can I do?

 

Thanks in advance

Model: RN422X124|ReadyNAS 4220 10Gbase-T (12x4TB Enterprise)
Message 1 of 15
Marc_V
NETGEAR Employee Retired

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

@Jota79

 

Welcome to the Community!

 

It seems the LUNs you have created almost consume 80%+ of the total volume capacity. Have you used Snapshot or installed any Apps? It seems you have tried accessing the NAS through SSH, have you checked the volume for shares or possible log files that consumed the space? Also, having fragmentation on the volume, defragging it will move data around and might show different space allocations /infoirmation.

 

As always, we advise as much as possible to allocate more than 20% of your total storage for other processes/services (Snapshots,Apps, logs, copies) small to no space allocated for these might degrade the performance and also the space accounting of your NAS. 80% of usage may seem to be too low but it usually is just the right amount to be considered as max to avoid any performance and accounting issues

 

Are you experiencing any performance degradation or any issues on your connections? It is best to contact Support and have them check it on the backend since you are also using LUNs.

 

Other users might be able to suggest for steps on what to check and what to do.

 

HTH

 

 

Regards

Message 2 of 15
StephenB
Guru

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason


@Marc_V wrote:

 

It seems the LUNs you have created almost consume 80%+ of the total volume capacity.

 


Based on the first post, @Jota79 has 24 TB of space in two volumes (12 TB each), and 16 TB of LUNs - which would be 67%.  That assumes the LUN size is in TB - if it is TiB, then the space used should be about 73%.

 

@Jota79: do you have 4 LUNs on each volume?  How full is each volume (per the Web UI?)

Message 3 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

We have Thick Provision, so we use 2TB per LUN in ReadyNas.

 

If you check this config at low level in BTRFS, we can see:

 

Total   Exclusive  Set shared  Filename

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/.purge

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/.vault

   2.00TiB   899.55GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01/.iscsi/iscsi_lun_backing_store

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01/.iscsi/.serial_number

   2.00TiB   899.55GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01/.iscsi

   2.00TiB   899.55GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/snapshot.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/iscsi.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/custom_snapshot_management.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/custom_snapshot_schedule.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/snapshot.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/iscsi.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/custom_snapshot_management.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/custom_snapshot_schedule.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/snapshot.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/iscsi.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/custom_snapshot_management.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/custom_snapshot_schedule.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/snapshot.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/iscsi.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/custom_snapshot_management.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/custom_snapshot_schedule.conf

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/.snapshot_prune

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/.tier_data_migration_threshold

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share

   2.00TiB     1.99TiB           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01/.iscsi/iscsi_lun_backing_store

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01/.iscsi/.serial_number

   2.00TiB     1.99TiB           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01/.iscsi

   2.00TiB     1.99TiB           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01

   2.00TiB    33.14GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02/.iscsi/iscsi_lun_backing_store

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02/.iscsi/.serial_number

   2.00TiB    33.14GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02/.iscsi

   2.00TiB    33.14GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02

   2.00TiB    52.53GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03/.iscsi/iscsi_lun_backing_store

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03/.iscsi/.serial_number

   2.00TiB    52.53GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03/.iscsi

   2.00TiB    52.53GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03

   8.00TiB     2.96TiB     5.04TiB  /VOLUME_PERFORMANCE2

  

We use this LUNS (iSCSI Share) in two VMware vCenter environments. We use one LUN per DataStore, so at vmware level we can see:

 

ARA_VOL01_LUN01 1648GB used of 2TB

NOSUP_VOL01_LUN01 1840GB used of 2TB

NOSUP_VOL01_LUN02 1875GB used of 2TB

NOSUP_VOL01_LUN03 1529GB used of 2TB

 

ARA_VOL02_LUN01 1657GB used of 2TB

NOSUP_VOL02_LUN01  996GB used of 2TB

NOSUP_VOL02_LUN02 1564GB used of 2TB

NOSUP_VOL02_LUN03 1398GB used of 2TB 

 

We have opened a case with netgear support, but not luck by the moment.

Gui is slower than normal, productions vms seems not to be affected in performance.

 

We have only 22GB free per 10.90TB ( 8TB used, 4x2TB Thick LUN ). Not many time to get 100% full. Perhaps

we have one day or two more until 100%.

 

Thanks in advance

 

 

 

 

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 4 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

Volumes screenshot

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 5 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

shares screenshot

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 6 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

system screenshot

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 7 of 15
StephenB
Guru

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

Can you try btrfs subvolume list -s ?

 

I'm wondering if there is a snapshot that isn't showing up in the web ui.

Message 8 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

No snapshots:

 

root@Netgear-rn4220:~# btrfs subvolume list -s /VOLUME_PERFORMANCE2

root@Netgear-rn4220:~# btrfs subvolume list /VOLUME_PERFORMANCE2
ID 257 gen 2028563 top level 5 path .purge
ID 258 gen 2028563 top level 5 path .vault
ID 259 gen 2104228 top level 5 path NOSUP_VOL02_LUN01
ID 260 gen 2102113 top level 5 path ._share
ID 2310 gen 2104228 top level 5 path ARA_VOL02_LUN01
ID 4359 gen 2104228 top level 5 path NOSUP_VOL02_LUN02
ID 6408 gen 2104228 top level 5 path NOSUP_VOL02_LUN03

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 9 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

No snapshots:

 

root@Netgear-rn4220:~# btrfs subvolume list -s /VOLUME_PERFORMANCE2

root@Netgear-rn4220:~# btrfs subvolume list /VOLUME_PERFORMANCE2
ID 257 gen 2028563 top level 5 path .purge
ID 258 gen 2028563 top level 5 path .vault
ID 259 gen 2104228 top level 5 path NOSUP_VOL02_LUN01
ID 260 gen 2102113 top level 5 path ._share
ID 2310 gen 2104228 top level 5 path ARA_VOL02_LUN01
ID 4359 gen 2104228 top level 5 path NOSUP_VOL02_LUN02
ID 6408 gen 2104228 top level 5 path NOSUP_VOL02_LUN03

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 10 of 15
StephenB
Guru

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason


@Jota79 wrote:

No snapshots:

 


Thanks for confirming.

 

I am wondering about the exclusive categorization in your earlier btrfs fi -du post.

Total       Exclusive   Set shared  Filename   
8.00TiB     2.96TiB     5.04TiB     /VOLUME_PERFORMANCE2

It seems to me that should be zero.  Though that doesn't really explain it.

 

I think the real issue is likely here:

/VOLUME_PERFORMANCE2/._share/.tier_data_migration_threshold
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share
   2.00TiB     1.99TiB           - 

I'm assuming you aren't using data tiering now.  Is this something you were using before?

 

Message 11 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

We have never used data tiering. What can we do ?  Any suggestions?

 

Thanks a lot

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 12 of 15
StephenB
Guru

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason


@Jota79 wrote:

We have never used data tiering. What can we do ?  Any suggestions?

 

Thanks a lot


I'd look in the folder, and see what is in there.

 

I'm not sure if it can be safely deleted or not - @JohnCM_S , @Marc_V : is this something you've seen before?

Message 13 of 15
Jota79
Aspirant

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason

We can check  .snapshot_prune and .tier_data_migration_threshold files in both volumes:

 

 root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE2/._share/.snapshot_prune
10
root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE2/._share/.tier_data_migration_threshold
90
root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE/._share/.snapshot_prune
10
root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE/._share/.tier_data_migration_threshold
90
root@Netgear-rn4220:~#

 

4 KB size both files.

 

Both files seems to be config files, it is only a threshold value.

 

 

 

 

 

 

 

Model: RN4220X|ReadyNAS 4220 10Gbase-T (chassis only)
Message 14 of 15
StephenB
Guru

Re: ISSUE - Unallocated/free space Volume DECREASING with no reason


@Jota79 wrote:

We can check  .snapshot_prune and .tier_data_migration_threshold files in both volumes:

 

 root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE2/._share/.snapshot_prune
10
root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE2/._share/.tier_data_migration_threshold
90
root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE/._share/.snapshot_prune
10
root@Netgear-rn4220:~# cat /VOLUME_PERFORMANCE/._share/.tier_data_migration_threshold
90
root@Netgear-rn4220:~#

 

4 KB size both files.

 


Sorry - a line break in your du post led me down the wrong path there.  You might want to use the </> tool as it often formats lists like this better. Here's a reformatted one with a couple of blank lines added:

    Total   Exclusive  Set shared  Filename

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/.purge
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/.vault

   2.00TiB   899.55GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01/.iscsi/iscsi_lun_backing_store
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01/.iscsi/.serial_number
   2.00TiB   899.55GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01/.iscsi
   2.00TiB   899.55GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN01

     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/snapshot.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/iscsi.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/custom_snapshot_management.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01/custom_snapshot_schedule.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN01
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/snapshot.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/iscsi.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/custom_snapshot_management.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01/custom_snapshot_schedule.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/ARA_VOL02_LUN01
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/snapshot.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/iscsi.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/custom_snapshot_management.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02/custom_snapshot_schedule.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN02
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/snapshot.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/iscsi.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/custom_snapshot_management.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03/custom_snapshot_schedule.conf
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/NOSUP_VOL02_LUN03
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/.snapshot_prune
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share/.tier_data_migration_threshold
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/._share

   2.00TiB     1.99TiB           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01/.iscsi/iscsi_lun_backing_store
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01/.iscsi/.serial_number
   2.00TiB     1.99TiB           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01/.iscsi
   2.00TiB     1.99TiB           -  /VOLUME_PERFORMANCE2/ARA_VOL02_LUN01

   2.00TiB    33.14GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02/.iscsi/iscsi_lun_backing_store
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02/.iscsi/.serial_number
   2.00TiB    33.14GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02/.iscsi
   2.00TiB    33.14GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN02

   2.00TiB    52.53GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03/.iscsi/iscsi_lun_backing_store
     0.00B       0.00B           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03/.iscsi/.serial_number
   2.00TiB    52.53GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03/.iscsi
   2.00TiB    52.53GiB           -  /VOLUME_PERFORMANCE2/NOSUP_VOL02_LUN03

   8.00TiB     2.96TiB     5.04TiB  /VOLUME_PERFORMANCE2

Looking again - all the storage is shown as being in iscsi_lun_backing_store as it should be.

Message 15 of 15
Top Contributors
Discussion stats
  • 14 replies
  • 1278 views
  • 0 kudos
  • 3 in conversation
Announcements