Re: Whole-volume defragmentation on volume with nothing but auto-defrag shares?

ReadyNASser · ‎2019-06-01

Hello ReadyNAS community,

I have owned a RN526X and EDA500 expansion for several years now, and both have been working great so far. I have a general maintenance question about whole-volume disk defragmentation:

Is it necessary or recommended to run or schedule a whole-volume disk defragmentation job on my primary volume when every share it contains was created from the start with auto-defrag enabled? I have been manually defragmenting the whole volume about once every 3-6 months or so, and the last time I did this it took about eight hours. But is this still advisable to do from time to time, even with auto-defrag enabled on every share? Is it doing something that per-share auto-defrag cannot?

I use ReadyDR for scheduled backups to the EDA500. I don't have any of its read-only ReadyDR shares set to auto-defrag, as it seems to not be needed (a manual whole-volume defragmentation typically completes in a few seconds, indicating that nothing much was fragmented).

Thanks for your advice!

StephenB · ‎2019-06-02

@ReadyNASser wrote:

I use ReadyDR for scheduled backups to the EDA500. I don't have any of its read-only ReadyDR shares set to auto-defrag, as it seems to not be needed (a manual whole-volume defragmentation typically completes in a few seconds, indicating that nothing much was fragmented).

There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share.

When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share.

@ReadyNASser wrote:

But is this still advisable to do from time to time, even with auto-defrag enabled on every share? Is it doing something that per-share auto-defrag cannot?

That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this.

If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).

Retired_Member · ‎2019-06-02

@StephenBwrote: "When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share."

If that is true, I'm drawing the following two conclusions:

(1) With snapshots enabled, the more you defrag the share the more your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind.

(2) With snapshots enabled, you urgently want to find a way of minimizing defragmentation activities on the share at all.

If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients:

Application "smb plus" has a parameter to preallocate disk space before a file is written. This minimizes the level of fragmentation as it helps to (a) keep defragmentation processes short and (b) reduces the amount of space allocated for those snapshots.

Kind regards

StephenB · ‎2019-06-02

@Retired_Member wrote:

If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients:

Application "smb plus" has a parameter to preallocate disk space before a file is written.

The pre-allocate helps avoid fragmentation in the main share when files are created. But it won't stop fragmentation when the file is modified after a snapshot is taken (the case I talked about above).

For example, fragmentation can become extreme when you

frequently update a SQL database
download torrents
frequently update an iSCSI LUN

We already have guidance here that snapshots should be disabled for shares/LUNs with that kind of content.

Of course there are other examples that are much less extreme - they generally won't add an unmanagable amount of storage to the share.

documents that are sometimes updated
music files that are sometimes retagged

@Retired_Member wrote:

your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind.

I think it's exactly what the inventors did have in mind. You can roll back to earlier versions of files or folders if needed. It's also what ReadyDR does.

I can live with some fragmentation, and I've found it isn't that difficult to manage retention to keep the the snapshot space usage reasonable.

ReadyNASser · ‎2019-06-02

@StephenB wrote:
There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share.

When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share.

Thanks StephenB. That would explain why my free space decreased by about 600 GB just after I last ran volume defrag. Snapshots occupy just over 770 GB on the volume.

@StephenB wrote:

That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this.

If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).

I noticed after I ran the most recent volume defrag, the nightly ReadyDR backup to the EDA500 of my music share took nearly four hours to complete (and I hadn't modified any files), probably because the volume defrag had moved files and snapshots around, and those changes then needed to be copied to the EDA500.

I like ReadyDR because it works in the background, can run multiple jobs at once, and is a copy of a snapshot of a share; I've never used the file-based backup functionality of the NAS, but presume it would need to copy things over in the same way as ReadyDR after volume defrag is run.

ReadyNASser · ‎2019-06-02

@Retired_Member wrote:

Application "smb plus" has a parameter to preallocate disk space before a file is written. This minimizes the level of fragmentation as it helps to (a) keep defragmentation processes short and (b) reduces the amount of space allocated for those snapshots.

Thanks RolandWausE. I looked into Netgear's SMB Plus app, but unfortunately it seems to work only with Windows-connectivity at the moment, and I'm usually on a Mac. I saw your suggestion on this forum to integrate its functionality with ReadyNAS's OS (that seems like a good idea), so hopefully Netgear's working on doing that (or has plans to bring it to the Mac).

Retired_Member · ‎2019-06-03

@ReadyNASserwrote: "I looked into Netgear's SMB Plus app, but unfortunately it seems to work only with Windows-connectivity at the moment, and I'm usually on a Mac."

Just to clarify, the smb plus does not care about the operating system the workstation is on. As long the network protocol between client and nas is smb you will benefit of the app. Should you still use afp for mac-to-nas communication you could also think about switching this to smb, as to my knowledge afp might be discontinued by apple in the near future.

Kind regards

Retired_Member · ‎2019-06-03

@StephenB: "I think it's exactly what the inventors did have in mind. You can roll back to earlier versions of files or folders if needed. It's also what ReadyDR does."

I disagree with part of that.

1) yes, they wanted to create an easy way of roll-back of files and folders.

2) no, they did not want that to happen at the file-level, but merely at the block-level, because in this way the efficiency of the backup is maximized due to minimized data redundancy during incremental backup.

In a world, where spinning disks are still the mainstream, if it comes to big(er) data (e.g. nas), you now have two strong antagonists working against each other making their lives difficult in their competition for resources:

A) defragmentation agents, which try to help maximizing the speed data can be accessed on spinning drives and

B) redundancy agents (snapshot mechanisms), which try to minimize the amount of data, which need to go into incremental backups

While a defragmentation agent is improving access times it is also causing a redundancy agent to waste space on shares and volumes when storing the related snapshots, as the defragmentation agent is reading and writing a file completely while doing its work. (I learned that from you, thanks, @StephenB 🙂

So, the best of all possible worlds to my understanding would be to store all data on media where fragmention is not an issue anymore. After the defragmentation agents will have been disappeared into history, the snapshots will be able to show their full power.

Kind regards

StephenB · ‎2019-06-03

@ReadyNASser wrote:
I've never used the file-based backup functionality of the NAS, but presume it would need to copy things over in the same way as ReadyDR after volume defrag is run.

The file-based backups (for instance rsync) just copy over files that have changed since the previous backup. You can use snapshots on the destination share to get some versioning. Defragging a file shouldn't result in copying it, since the file attributes (size, permissions, timestamps, etc) wouldn't have changed.

FWIW, if the backup source is local to the NAS, then the NAS will create a temporary snapshot of the source and back that up. That does ensure coherency.

ReadyDR will copy all the data blocks that have changed in the snapshot, so it will re-copy defragged files. On the other hand, it only copies the changed data blocks (while Rsync needs to copy the entire file).

@ReadyNASser wrote:

Thanks StephenB. That would explain why my free space decreased by about 600 GB just after I last ran volume defrag. Snapshots occupy just over 770 GB on the volume.

If you are using the "Smart" snapshots, then the monthly snapshots are retained indefinitely - they eventually fill the NAS, and require manual deletion. Personally I use the "Custom" setting - taking snapshots only when there are changes, and explicitly limiting retention to 3 months. That takes up about 5% of the space used (though of course that will vary depending on the amount of change).

StephenB · ‎2019-06-03

@Retired_Member wrote:

2) no, they did not want that to happen at the file-level, but merely at the block-level, because in this way the efficiency of the backup is maximized due to minimized data redundancy during incremental backup.

But in fact, they implemented the file system so defragmenting the main copy would result in a separate copy of the file in the snapshot. So saying they "didn't want that to happen" seems wrong to me. It does what they designed it to do - it is not accidental behavior.

Hypothetically they could have designed the system differently - for example, they could have fragmented the snapshot files when you defragged the main copy. That wouldn't increase the snapshot space needed, but it would have taken longer to defragment if you had a lot of snapshots. I think they made the right tradeoff in deciding not to do that.

@Retired_Member wrote:

So, the best of all possible worlds to my understanding would be to store all data on media where fragmention is not an issue anymore.

Which of course is happening as SSDs take hold.

ReadyNASser · ‎2019-06-03

@Retired_Member wrote:

Just to clarify, the smb plus does not care about the operating system the workstation is on. As long the network protocol between client and nas is smb you will benefit of the app. Should you still use afp for mac-to-nas communication you could also think about switching this to smb, as to my knowledge afp might be discontinued by apple in the near future.
Kind regards

Thanks, I'll check it out then. I'm on the latest macOS (10.14 Mojave) with SMB3 and required encryption set up on the NAS.

ReadyNASser · ‎2019-06-04

@StephenB wrote:
The file-based backups (for instance rsync) just copy over files that have changed since the previous backup. You can use snapshots on the destination share to get some versioning. Defragging a file shouldn't result in copying it, since the file attributes (size, permissions, timestamps, etc) wouldn't have changed.

FWIW, if the backup source is local to the NAS, then the NAS will create a temporary snapshot of the source and back that up. That does ensure coherency.

Interesting, thanks for the info. I may try some file-based backup operations eventually (probably using some external drives plugged into the NAS via USB), so that's good to know.

@StephenB wrote:
If you are using the "Smart" snapshots, then the monthly snapshots are retained indefinitely - they eventually fill the NAS, and require manual deletion. Personally I use the "Custom" setting - taking snapshots only when there are changes, and explicitly limiting retention to 3 months. That takes up about 5% of the space used (though of course that will vary depending on the amount of change).

I use "Custom" snapshots only when there are changes as well. After trying out "Smart" snapshots for a little while, I received a few warnings about volume capacity being less than 30% free, and switched. After experimenting with snapshot retention settings, I now no longer need to manually remove any, which is a real timesaver.

StephenB · ‎2019-06-04

@ReadyNASser wrote:
Interesting, thanks for the info. I may try some file-based backup operations eventually (probably using some external drives plugged into the NAS via USB), so that's good to know.

One useful trick here - you can use rsync for USB backups if you use the loopback IP address (127.0.0.1). If you use the loopback address on the backup source, the NAS won't make a temporary snapshot. Using loopback for the destination USB will result in the temporary snapshot - but it is a bit trickier to set up. You'd have to create a share on the USB drive, and enable rsync for it.

My own backups (NAS->NAS) generally run on the destination NAS. I don't worry about coherency, since the backups are run off-hours (and I'm not backing up anything where loss of coherency is a big deal).

@ReadyNASser wrote:
After experimenting with snapshot retention settings, I now no longer need to manually remove any, which is a real timesaver.

Totally agree. I really don't understand why the "Smart" snapshots have no retention setting. It makes the feature completely useless to me (actually it's worse than useless, it actually does damage by filling up the file system).