× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

dgaliffa
Aspirant

ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

Hello,

 

I'm having an issue with our ReadyNAS 4312, we're using it as VEEAM target, set up an iSCSI LUN 49TB in size. We had about 5-6TB of space left on the NAS outside of that LUN. We recently tried to turn on snapshots for the iSCSI LUN and I must have chose too high of a retention rate or something. I now have no space left out of that 5-6TB and since the LUN is thick provisioned even though I have space left there I have no way to clear off space or delete these snapshots or half snaps or whatever is happening.

 

I don't have software support on this NAS anymore, I couldn't even contact support. Anyone have any ideas? I'll attach a screenshot of the volume overview, because it doesn't even show me a yellow section for where the snaps are taking up space. I would prefer not to have to delete everything on this target these are our VEEAM backups for a large portion of our environment.

Model: RN12S1220|READYNAS 4200 24TB 12-BAY 2U RACKMOUNT HIGH-PERFORMANCE STORAGE WITH 10 GE (SFP+)|EOL
Message 1 of 13
StephenB
Guru

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN


I don't have software support on this NAS anymore, I couldn't even contact support.

If you (or your company) were the original purchasers,  you should be able to purchase a support contract through my.netgear.com.  You would need to use the account of the person who registered the NAS (or arrange to get that transfered somehow).  @JohnCM_S or @Marc_V should be able to assist.

 


@dgaliffa wrote:

I'm having an issue with our ReadyNAS 4312, we're using it as VEEAM target, set up an iSCSI LUN 49TB in size. We had about 5-6TB of space left on the NAS outside of that LUN. We recently tried to turn on snapshots for the iSCSI LUN and I must have chose too high of a retention rate or something. I now have no space left out of that 5-6TB and since the LUN is thick provisioned even though I have space left there I have no way to clear off space or delete these snapshots or half snaps or whatever is happening.

 


What firmware are you running?

 

You could enable ssh, and see any snapshots by entering

# btrfs subvolume list /data
Message 2 of 13
dgaliffa
Aspirant

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

If possible I'd like to make adding software support contract to be the last resort that's sort of why I'm on here. The firmware version is 6.10.1 which I recently updated. I will turn on SSH and try to check that.

Message 3 of 13
dgaliffa
Aspirant

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

I attached a screenshot, i was able to get ssh'd in but the command doesn't work

Message 4 of 13
dgaliffa
Aspirant

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

Scratch that, I needed to be logged in as root, so I see the veeam repo/.snapshots is that what I'm looking for? 

 

 

root@MVH-RNAS72:~# btrfs subvolume list /data
ID 257 gen 8278595 top level 5 path home
ID 258 gen 8260736 top level 5 path .apps
ID 259 gen 10 top level 5 path .vault
ID 260 gen 8261962 top level 5 path ._share
ID 41223 gen 4952707 top level 5 path .purge
ID 82186 gen 8281815 top level 5 path Veeam-Repo-01
ID 132364 gen 8279931 top level 5 path .timemachine
ID 132365 gen 8245691 top level 82186 path Veeam-Repo-01/.snapshots
ID 141963 gen 8280829 top level 257 path home/admin
root@MVH-RNAS72:~#

Message 5 of 13
bedlam1
Prodigy

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

you need to login as User root using the nas admin password

Message 6 of 13
dgaliffa
Aspirant

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

Yes I tried responding to this post but i think it kept deleting because of spam. I got the feedback here below, i see i have this Veeam-Repo-01/.snapshots  is this what I'm looking for? Can I delete this? Not sure what the command would be? Any ideas?

 

root@-RNAS72:~# btrfs subvolume list /data
ID 257 gen 8278595 top level 5 path home
ID 258 gen 8260736 top level 5 path .apps
ID 259 gen 10 top level 5 path .vault
ID 260 gen 8261962 top level 5 path ._share
ID 41223 gen 4952707 top level 5 path .purge
ID 82186 gen 8282712 top level 5 path Veeam-Repo-01
ID 132364 gen 8279931 top level 5 path .timemachine
ID 132365 gen 8245691 top level 82186 path Veeam-Repo-01/.snapshots
ID 141963 gen 8280829 top level 257 path home/admin
root@-RNAS72:~#

Message 7 of 13
dgaliffa
Aspirant

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

I logged in as root and ran the command, it came back with a listing of the volumes, there is one in there for my veeam-repo-01/.snapshots. I'm attaching the feedback from the command. Is it possible to delete this from the command line? I keep getting commit errors when I try to disable the snapshots. I assume because the NAS is completely full, but if I can't delete these snapshots then i dont know how I'll reclaim any space.

Message 8 of 13
StephenB
Guru

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN


@dgaliffa wrote:

I logged in as root and ran the command, it came back with a listing of the volumes, there is one in there for my veeam-repo-01/.snapshots.

Yes, but there are no snapshots in there, so there are no subvolumes to delete.

 

I'm not convinced that your space issue is actually due to snapshots (though I don't know what is causing it),

 

You might want to run

# btrfs filesystem df /data

and post the results on a reply.

Message 9 of 13
dgaliffa
Aspirant

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

Yeah, see that is the peculiar thing, I ran that command and this was the output. The thing is, the only data on this NAS is the 49TB thick provisioned LUN. No other applications or devices are writing to this device as a source. When the iSCSI drive shows up on our VEEAM server it only shows up as a 49TB drive, not the full size 54TB. 

 

Thats the part that has me confused on how its taking up more space all of a sudden, the only thing I changed was the snapshot settings, have you ever seen snapshots running on systems without enough space? Maybe it's in the middle of running a snapshot and filling up but running out of space? So the drive overview isn't showing the yellow section or separating it into the snapshot section.

 

 

root@-RNAS72:~# btrfs filesystem df /data
Data, single: total=54.53TiB, used=54.53TiB
System, DUP: total=8.00MiB, used=6.03MiB
Metadata, DUP: total=1.50GiB, used=787.94MiB
GlobalReserve, single: total=329.94MiB, used=0.00B

Message 10 of 13
StephenB
Guru

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN


@dgaliffa wrote:

Data, single: total=54.53TiB, used=54.53TiB

This is saying that the blocks are totally allocated (consistent with what the web UI is saying).  "used" doesn't necessarily match the sum of the file sizes.

 

This is normally something a balance would fix, but you don't have any free blocks.

 

One thing that might work is expanding the array (followed by a balance).  What RAID format are you using?

Message 11 of 13
dgaliffa
Aspirant

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN

12x 6TB drives in RAID 6, I'm lost on how anything could be taking up that remainder 5-6 TB. If the only thing writing to this thing is a single server connected through iSCSI and it's to a thick provisioned LUN of 49TB. Do you think support would be able to figure this out? Expansion doesn't seem like it would be an option without a decent cost.

Message 12 of 13
StephenB
Guru

Re: ReadyNAS 4312 - 54TB usable full even though the only thing stored 49TB LUN


@dgaliffa wrote:

12x 6TB drives in RAID 6, I'm lost on how anything could be taking up that remainder 5-6 TB. 


BTRFS allocates space in chunks (usually 1 GB).  I think in your case there is free space within some chunks, but that all the chunks are used (that is, none of them are completely empty).  This would happen if the LUN became fragmented over time.  Though it's not clear how a thick LUN would have been fragmented.

 

If there are no free chunks in your filesystem, there there isn't much you (or support) can do to fix it.  If you can't expand the volume, I guess you could back up the contents of the iSCSI drive from the VEEAM server.  Then delete the LUN, recreate it, and then restore the contents from the backup.  That would be very time consuming (and of course assumes you have enough storage).

 

If there are some files outside the LUN (at least a few gigabytes), you could try deleting those and see if you end up with any unused or unallocated chunks.  Then balances could help (initially run from ssh, with some thresholds set manually).

 

FWIW, there is another command that gives you a somewhat different breakdown

# btrfs filesystem usage /data

That will give you the amount of unallocated space as well as an estimate of free space (which is a slippery concept with btrfs).

 

Hopefully @JohnCM_S or @Marc_V will chime in - they might have some other ideas.

 

 

Message 13 of 13
Top Contributors
Discussion stats
  • 12 replies
  • 1805 views
  • 3 kudos
  • 3 in conversation
Announcements