NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Whole-volume defragmentation on volume with nothing but auto-defrag shares?
Hello ReadyNAS community, I have owned a RN526X and EDA500 expansion for several years now, and both have been working great so far. I have a general maintenance question about whole-volume disk ...
ReadyNASser wrote:
I use ReadyDR for scheduled backups to the EDA500. I don't have any of its read-only ReadyDR shares set to auto-defrag, as it seems to not be needed (a manual whole-volume defragmentation typically completes in a few seconds, indicating that nothing much was fragmented).
There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share.
When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share.
ReadyNASser wrote:
But is this still advisable to do from time to time, even with auto-defrag enabled on every share? Is it doing something that per-share auto-defrag cannot?
That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this.
If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).
StephenBwrote: "When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share."
If that is true, I'm drawing the following two conclusions:
(1) With snapshots enabled, the more you defrag the share the more your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind.
(2) With snapshots enabled, you urgently want to find a way of minimizing defragmentation activities on the share at all.
If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients:
Application "smb plus" has a parameter to preallocate disk space before a file is written. This minimizes the level of fragmentation as it helps to (a) keep defragmentation processes short and (b) reduces the amount of space allocated for those snapshots.
Kind regards
Retired_Member wrote:
If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients:
Application "smb plus" has a parameter to preallocate disk space before a file is written.
The pre-allocate helps avoid fragmentation in the main share when files are created. But it won't stop fragmentation when the file is modified after a snapshot is taken (the case I talked about above).
For example, fragmentation can become extreme when you
- frequently update a SQL database
- download torrents
- frequently update an iSCSI LUN
We already have guidance here that snapshots should be disabled for shares/LUNs with that kind of content.
Of course there are other examples that are much less extreme - they generally won't add an unmanagable amount of storage to the share.
- documents that are sometimes updated
- music files that are sometimes retagged
Retired_Member wrote:
your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind.
I think it's exactly what the inventors did have in mind. You can roll back to earlier versions of files or folders if needed. It's also what ReadyDR does.
I can live with some fragmentation, and I've found it isn't that difficult to manage retention to keep the the snapshot space usage reasonable.
StephenB: "I think it's exactly what the inventors did have in mind. You can roll back to earlier versions of files or folders if needed. It's also what ReadyDR does."
I disagree with part of that.
1) yes, they wanted to create an easy way of roll-back of files and folders.
2) no, they did not want that to happen at the file-level, but merely at the block-level, because in this way the efficiency of the backup is maximized due to minimized data redundancy during incremental backup.
In a world, where spinning disks are still the mainstream, if it comes to big(er) data (e.g. nas), you now have two strong antagonists working against each other making their lives difficult in their competition for resources:
A) defragmentation agents, which try to help maximizing the speed data can be accessed on spinning drives and
B) redundancy agents (snapshot mechanisms), which try to minimize the amount of data, which need to go into incremental backups
While a defragmentation agent is improving access times it is also causing a redundancy agent to waste space on shares and volumes when storing the related snapshots, as the defragmentation agent is reading and writing a file completely while doing its work. (I learned that from you, thanks, StephenB :-)
So, the best of all possible worlds to my understanding would be to store all data on media where fragmention is not an issue anymore. After the defragmentation agents will have been disappeared into history, the snapshots will be able to show their full power.
Kind regards
Retired_Member wrote:Application "smb plus" has a parameter to preallocate disk space before a file is written. This minimizes the level of fragmentation as it helps to (a) keep defragmentation processes short and (b) reduces the amount of space allocated for those snapshots.
Thanks RolandWausE. I looked into Netgear's SMB Plus app, but unfortunately it seems to work only with Windows-connectivity at the moment, and I'm usually on a Mac. I saw your suggestion on this forum to integrate its functionality with ReadyNAS's OS (that seems like a good idea), so hopefully Netgear's working on doing that (or has plans to bring it to the Mac).
ReadyNASserwrote: "I looked into Netgear's SMB Plus app, but unfortunately it seems to work only with Windows-connectivity at the moment, and I'm usually on a Mac."
Just to clarify, the smb plus does not care about the operating system the workstation is on. As long the network protocol between client and nas is smb you will benefit of the app. Should you still use afp for mac-to-nas communication you could also think about switching this to smb, as to my knowledge afp might be discontinued by apple in the near future.
Kind regards
StephenB wrote:There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share.
When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share.
Thanks StephenB. That would explain why my free space decreased by about 600 GB just after I last ran volume defrag. Snapshots occupy just over 770 GB on the volume.
StephenB wrote:
That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this.
If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).
I noticed after I ran the most recent volume defrag, the nightly ReadyDR backup to the EDA500 of my music share took nearly four hours to complete (and I hadn't modified any files), probably because the volume defrag had moved files and snapshots around, and those changes then needed to be copied to the EDA500.
I like ReadyDR because it works in the background, can run multiple jobs at once, and is a copy of a snapshot of a share; I've never used the file-based backup functionality of the NAS, but presume it would need to copy things over in the same way as ReadyDR after volume defrag is run.
ReadyNASser wrote:
I've never used the file-based backup functionality of the NAS, but presume it would need to copy things over in the same way as ReadyDR after volume defrag is run.The file-based backups (for instance rsync) just copy over files that have changed since the previous backup. You can use snapshots on the destination share to get some versioning. Defragging a file shouldn't result in copying it, since the file attributes (size, permissions, timestamps, etc) wouldn't have changed.
FWIW, if the backup source is local to the NAS, then the NAS will create a temporary snapshot of the source and back that up. That does ensure coherency.
ReadyDR will copy all the data blocks that have changed in the snapshot, so it will re-copy defragged files. On the other hand, it only copies the changed data blocks (while Rsync needs to copy the entire file).
ReadyNASser wrote:
Thanks StephenB. That would explain why my free space decreased by about 600 GB just after I last ran volume defrag. Snapshots occupy just over 770 GB on the volume.
If you are using the "Smart" snapshots, then the monthly snapshots are retained indefinitely - they eventually fill the NAS, and require manual deletion. Personally I use the "Custom" setting - taking snapshots only when there are changes, and explicitly limiting retention to 3 months. That takes up about 5% of the space used (though of course that will vary depending on the amount of change).
StephenB wrote:The file-based backups (for instance rsync) just copy over files that have changed since the previous backup. You can use snapshots on the destination share to get some versioning. Defragging a file shouldn't result in copying it, since the file attributes (size, permissions, timestamps, etc) wouldn't have changed.
FWIW, if the backup source is local to the NAS, then the NAS will create a temporary snapshot of the source and back that up. That does ensure coherency.
Interesting, thanks for the info. I may try some file-based backup operations eventually (probably using some external drives plugged into the NAS via USB), so that's good to know.
StephenB wrote:If you are using the "Smart" snapshots, then the monthly snapshots are retained indefinitely - they eventually fill the NAS, and require manual deletion. Personally I use the "Custom" setting - taking snapshots only when there are changes, and explicitly limiting retention to 3 months. That takes up about 5% of the space used (though of course that will vary depending on the amount of change).
I use "Custom" snapshots only when there are changes as well. After trying out "Smart" snapshots for a little while, I received a few warnings about volume capacity being less than 30% free, and switched. After experimenting with snapshot retention settings, I now no longer need to manually remove any, which is a real timesaver.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
