Hello ReadyNAS community, I have owned a RN526X and EDA500 expansion for several years now, and both have been working great so far. I have a general maintenance question about whole-volume disk defragmentation: Is it necessary or recommended to run or schedule a whole-volume disk defragmentation job on my primary volume when every share it contains was created from the start with auto-defrag enabled? I have been manually defragmenting the whole volume about once every 3-6 months or so, and the last time I did this it took about eight hours. But is this still advisable to do from time to time, even with auto-defrag enabled on every share? Is it doing something that per-share auto-defrag cannot? I use ReadyDR for scheduled backups to the EDA500. I don't have any of its read-only ReadyDR shares set to auto-defrag, as it seems to not be needed (a manual whole-volume defragmentation typically completes in a few seconds, indicating that nothing much was fragmented). Thanks for your advice!

ReadyNASser wrote: I use ReadyDR for scheduled backups to the EDA500. I don't have any of its read-only ReadyDR shares set to auto-defrag, as it seems to not be needed (a manual whole-volume defragmentation typically completes in a few seconds, indicating that nothing much was fragmented). There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share. When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share. ReadyNASser wrote: But is this still advisable to do from time to time, even with auto-defrag enabled on every share? Is it doing something that per-share auto-defrag cannot? That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this. If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).

StephenBwrote: "When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share." If that is true, I'm drawing the following two conclusions:(1) With snapshots enabled, the more you defrag the share the more your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind.(2) With snapshots enabled, you urgently want to find a way of minimizing defragmentation activities on the share at all. If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients:Application "smb plus" has a parameter to preallocate disk space before a file is written. This minimizes the level of fragmentation as it helps to (a) keep defragmentation processes short and (b) reduces the amount of space allocated for those snapshots. Kind regards

Retired_Member wrote: If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients: Application "smb plus" has a parameter to preallocate disk space before a file is written. The pre-allocate helps avoid fragmentation in the main share when files are created. But it won't stop fragmentation when the file is modified after a snapshot is taken (the case I talked about above). For example, fragmentation can become extreme when you frequently update a SQL database download torrents frequently update an iSCSI LUN We already have guidance here that snapshots should be disabled for shares/LUNs with that kind of content. Of course there are other examples that are much less extreme - they generally won't add an unmanagable amount of storage to the share. documents that are sometimes updated music files that are sometimes retagged Retired_Member wrote: your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind. I think it's exactly what the inventors did have in mind. You can roll back to earlier versions of files or folders if needed. It's also what ReadyDR does. I can live with some fragmentation, and I've found it isn't that difficult to manage retention to keep the the snapshot space usage reasonable.

Retired_Member wrote: Application "smb plus" has a parameter to preallocate disk space before a file is written. This minimizes the level of fragmentation as it helps to (a) keep defragmentation processes short and (b) reduces the amount of space allocated for those snapshots.Thanks RolandWausE. I looked into Netgear's SMB Plus app, but unfortunately it seems to work only with Windows-connectivity at the moment, and I'm usually on a Mac. I saw your suggestion on this forum to integrate its functionality with ReadyNAS's OS (that seems like a good idea), so hopefully Netgear's working on doing that (or has plans to bring it to the Mac).

StephenB wrote:There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share. When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share.Thanks StephenB. That would explain why my free space decreased by about 600 GB just after I last ran volume defrag. Snapshots occupy just over 770 GB on the volume.StephenB wrote: That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this. If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).I noticed after I ran the most recent volume defrag, the nightly ReadyDR backup to the EDA500 of my music share took nearly four hours to complete (and I hadn't modified any files), probably because the volume defrag had moved files and snapshots around, and those changes then needed to be copied to the EDA500. I like ReadyDR because it works in the background, can run multiple jobs at once, and is a copy of a snapshot of a share; I've never used the file-based backup functionality of the NAS, but presume it would need to copy things over in the same way as ReadyDR after volume defrag is run.

Whole-volume defragmentation on volume with nothing but auto-defrag shares?

12 Replies

StephenB
Guru
Jun 02, 2019
ReadyNASser wrote:

I use ReadyDR for scheduled backups to the EDA500. I don't have any of its read-only ReadyDR shares set to auto-defrag, as it seems to not be needed (a manual whole-volume defragmentation typically completes in a few seconds, indicating that nothing much was fragmented).

There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share.

When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share.

ReadyNASser wrote:

But is this still advisable to do from time to time, even with auto-defrag enabled on every share? Is it doing something that per-share auto-defrag cannot?

That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this.

If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).
- Retired_Member
  Jun 02, 2019
  StephenBwrote: "When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share."
  
  If that is true, I'm drawing the following two conclusions:
  (1) With snapshots enabled, the more you defrag the share the more your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind.
  (2) With snapshots enabled, you urgently want to find a way of minimizing defragmentation activities on the share at all.
  
  If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients:
  Application "smb plus" has a parameter to preallocate disk space before a file is written. This minimizes the level of fragmentation as it helps to (a) keep defragmentation processes short and (b) reduces the amount of space allocated for those snapshots.
  
  Kind regards
  - StephenB
    Guru
    Jun 02, 2019
    Retired_Member wrote:
    
    If we agree on those two, there is at least one solution for those writes to a share conducted by smb-clients:
    
    Application "smb plus" has a parameter to preallocate disk space before a file is written.
    
    The pre-allocate helps avoid fragmentation in the main share when files are created. But it won't stop fragmentation when the file is modified after a snapshot is taken (the case I talked about above).
    
    For example, fragmentation can become extreme when you
    
    frequently update a SQL database
    
    download torrents
    
    frequently update an iSCSI LUN
    
    We already have guidance here that snapshots should be disabled for shares/LUNs with that kind of content.
    
    Of course there are other examples that are much less extreme - they generally won't add an unmanagable amount of storage to the share.
    
    documents that are sometimes updated
    
    music files that are sometimes retagged
    
    Retired_Member wrote:
    
    your snapshot turns into some kind of incremental backup at the file level. Something the inventors of the snapshot concept probably did not have in mind.
    
    I think it's exactly what the inventors did have in mind. You can roll back to earlier versions of files or folders if needed. It's also what ReadyDR does.
    
    I can live with some fragmentation, and I've found it isn't that difficult to manage retention to keep the the snapshot space usage reasonable.
- ReadyNASser
  Tutor
  Jun 03, 2019
  
  StephenB wrote:
  There is a trade-off here that you might not be aware off. Snapshots only hold changes from the main share. When you modify a file in the main share (for example a update a database), then the system fragments the file. The original blocks end up in the snapshots, and the changed blocks are in the main share.
  
  When you defrag that share, then the full original file ends up in the snapshot - which increases the total disk space needed for the share.
  Thanks StephenB. That would explain why my free space decreased by about 600 GB just after I last ran volume defrag. Snapshots occupy just over 770 GB on the volume.
  StephenB wrote:
  
  That's a good question. The only difference I can see is that auto-defrag is likely triggered by some threshold, and the manual defrag isn't. But there's nothing documented on this.
  
  If the manual defrag is taking 8 hours, it clearly is doing something. The last time I ran it on my NAS it took about 3 hours (and autodefrag is set on most of my shares).
  I noticed after I ran the most recent volume defrag, the nightly ReadyDR backup to the EDA500 of my music share took nearly four hours to complete (and I hadn't modified any files), probably because the volume defrag had moved files and snapshots around, and those changes then needed to be copied to the EDA500.
  
  I like ReadyDR because it works in the background, can run multiple jobs at once, and is a copy of a snapshot of a share; I've never used the file-based backup functionality of the NAS, but presume it would need to copy things over in the same way as ReadyDR after volume defrag is run.
  - StephenB
    Guru
    Jun 03, 2019
    ReadyNASser wrote:
    I've never used the file-based backup functionality of the NAS, but presume it would need to copy things over in the same way as ReadyDR after volume defrag is run.
    
    The file-based backups (for instance rsync) just copy over files that have changed since the previous backup. You can use snapshots on the destination share to get some versioning. Defragging a file shouldn't result in copying it, since the file attributes (size, permissions, timestamps, etc) wouldn't have changed.
    
    FWIW, if the backup source is local to the NAS, then the NAS will create a temporary snapshot of the source and back that up. That does ensure coherency.
    
    ReadyDR will copy all the data blocks that have changed in the snapshot, so it will re-copy defragged files. On the other hand, it only copies the changed data blocks (while Rsync needs to copy the entire file).
    
    ReadyNASser wrote:
    
    Thanks StephenB. That would explain why my free space decreased by about 600 GB just after I last ran volume defrag. Snapshots occupy just over 770 GB on the volume.
    
    If you are using the "Smart" snapshots, then the monthly snapshots are retained indefinitely - they eventually fill the NAS, and require manual deletion. Personally I use the "Custom" setting - taking snapshots only when there are changes, and explicitly limiting retention to 3 months. That takes up about 5% of the space used (though of course that will vary depending on the amount of change).