This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
Had a 11TB Lun with 1.7TB free space / 2 recent snapshots using about 5.7TB
Seemed like the snapshots were taking up a lot of space so I ran a Balance operation to see if it would free up any space. When the balance operation hit 71% I got an out of memory error and had to hard reboot. Very sluggish getting to the web admin page on reboot and I soon kept getting 403 browser errors. Blue drive LEDS were all flashing. From RAIDar I was able to run diagnostics:
System
Volume data is inactive
Volume data is inactive
Logs
RIP []_del_reloc_root+0x31/0xf0
IP: []_del_reloc_root+0x31/0xf0
BTRFS:error (device md127) in cleanup_transaction:1856: errno=12 Out of memory
data.mount: Directory /data to mount over is not empty, mounting anyway
I rebooted again and now have web admin control but my volume is marked inactive. Data is using 18.17TB and Free Space is 0. I guess I need to free up some space (maybe delete a snapshot) and get the volume back online. Not sure how to gain access to do that; I would appreciate any advice. I have fresh logs if needed.
I just want to follow up with my solution in case this helps anyone in the future. My data was intact on the iSCSI target and I disabled snapshots. I was unable to delete the existing snapshots through the web interface so my available space was still less than 20%. I used btrfs commands to find and delete those snapshots, rebooted and I am happy to report I have recovered all the space formerly used by the snapshots. The NAS has been stable and running for several weeks now. I appreciate the advice I was provided here and I found my solution from reading some of the other community posts.
You do need to be careful when using snapshots with iSCSI LUNs. There's very limited situations in which using them would be advisable.
A balance is not going to make snapshots use less space. All a balance does is move around data and metadata to empty chunks.
Also balancing works best when the volume usage is below 85%.
With 5.7TB consumed by recent snapshots the LUN must be changing very significantly. This could be e.g. due to running a defrag of the filesystem on the LUN (e.g. if you're using the Microsoft Initiator in Windows, I'd suspect you may have run a defrag of the NTFS filesystem on the LUN using Windows). Another possibility would be modifying/adding/deleting a huge number of files on the LUN.
Is your backup up to date?
If not you may wish to contact support. Support does sell data recovery services however do note that the attempt may be completely unsuccessful.
I was able to start the 316 in read only mode and it seemed the storage totals were pretty much as they were before I started the balance operation. My LUN was still there but offline since I was in the read only mode. I restarted again in the normal mode with no errors and the LUN was active and my iSCSI initiator was able to connect. There are far too many folders and files to know if they are all there and readable but on the surface it looks like everything is there. I shut off all snapshots and am considering deleting the 2 existing snapshots taking up a huge amount of space. Do you think this would comprimise my data? I don't have a backup because this NAS serves as a backup but I still would prefer not to loose anything. Some of my users may have inadvertantly stored primary data there. My users backup video editing projects on this 316 until the project is complete and then it is deleted. You mentioned "modifying/adding/deleting a huge number of files on the LUN" could cause large snapshot data size. Will this be solved by disabling snapshots in general or is there something inherent to modifying a lot of data while connected via LUN/iSCSI?
Will this be solved by disabling snapshots in general or is there something inherent to modifying a lot of data while connected via LUN/iSCSI?
It's general, but is usually more problematic with LUNs than with shares.
When you first take a snapshot, it takes no space - all the data blocks are held in common with the main share. As the main share changes, any updated blocks are written as new blocks (e.g., "copy on write"). The original block then is in the snapshot only - so it now starts using space. And the updated file (or LUN) becomes fragmented.
So if you have a lot of updates happening in the share, the snapshots take a lot of space, and the main share (or LUN) quickly becomes fragmented. Defragmenting the file (LUN) in the main share actually makes the snapshot space go up. The main file is re-written, so it ends up sharing no blocks in common with the snapshot.
A LUN is simply raw block storage for the NAS - so operations that rewrite metadata or otherwise reorganize the storage are detrimental when snapshots are on. For instance, when snapshots are enabled, defragmenting the file system in the LUN will create fragmentation in the LUN container itself, and every disk block that is modified increases the snapshot space usage. So you end up with poor performance combined with a big loss of free space in the RAID array.
There are other cases where snapshots shouldn't be enabled. A torrent download creates similar "churn", as do updates to SQL databases.
I just want to follow up with my solution in case this helps anyone in the future. My data was intact on the iSCSI target and I disabled snapshots. I was unable to delete the existing snapshots through the web interface so my available space was still less than 20%. I used btrfs commands to find and delete those snapshots, rebooted and I am happy to report I have recovered all the space formerly used by the snapshots. The NAS has been stable and running for several weeks now. I appreciate the advice I was provided here and I found my solution from reading some of the other community posts.
