Rsync is a robust Linux file copy program with lots of options http://linux.die.net/man/1/rsync is a great discussion of the features and options. .http://serverfault.com/questions/138287/how-does-rsync-does-incremental-backups is a thread that discusses rsync. The effect is that Rsync does not do true incremental backup but instead captures the state of the file differences from a base copy at the time of the backup operation. The Netgear documentation regarding backup is sparse. The ReadyNAS backup capability is a GUI for the most common file copy needs. The Netgear documentation discusses incremental backup but that is not what happens since there are no lots of incremental files created. What actual rsync options are enabled during a Netgear copy? BsckgroundThe terms ‘incremental’ and differential’ backup deal with having a base level copy of a set of files and then maintaining the difference between the data today and the base.Incremental backups track the difference between the base and essentially build on each other. For example if there are five incremental backups to restore the file the backups must be restored sequentially as they were taken – first #1, then #2, etc. The amount of data in each increment is typically less than other methods. In robust File backup software, this is accomplished by log files, and associated data that is captured.Differential backups track the difference between the base and what the current state is today. Each time a differential backup is taken, all of the changes that have occurred are contained in the backup data. To restore the data only the most recent version of the ‘difference’ needs to be restored. The amount of data in each differential backup typically gets larger as the changes accumulate.In both scenarios there typically is a complete base level copy done periodically so there are no differences and the process of incremental/differential backup start again. Purges of old data need to occur. Some backup programs do thls automatically.

Sounds like you may be on your way to a solution. FWIW, I got my ReadyNAS up into operating condition today and have the following to report regarding the rsync command as issued from the GUIrsync -v -8 --timeout=30 --recursive --links --times --devices --specials --one-file-system --modify-window=1 --owner --group --no-acls --no-perms --exclude CrashPlan --exclude Downloads --exclude Library/logs --exclude Library/Caches --rsh=ssh -o SrticHostKeyChecking=no -PasswordAuthentication=no -p 22 name@host:/SourceShare /data/DestShareGood luck! EDITED TO ADD:I forgot to say that this is Rsync over ssh with the following optionsNO - enable compressionNO - remove deleted files on targetI DO have a list of excluded files/folders, but it seems the gui is programmed also to include .snapshots in that exclusion listSchedule Full backup: "First Time Only"NO - Remove Contents before NO - Change ownership

Forum Discussion

Bains

Guide

Nov 13, 2015

Solved

Backup using rsync – what options are used

Rsync is a robust Linux file copy program with lots of options

http://linux.die.net/man/1/rsync is a great discussion of the features and options.

http://serverfault.com/questions/138287/how-does-rsync-does-incremental-backups is a thread that discusses rsync. The effect is that Rsync does not do true incremental backup but instead captures the state of the file differences from a base copy at the time of the backup operation.

The Netgear documentation regarding backup is sparse. The ReadyNAS backup capability is a GUI for the most common file copy needs. The Netgear documentation discusses incremental backup but that is not what happens since there are no lots of incremental files created.

What actual rsync options are enabled during a Netgear copy?

Bsckground

The terms ‘incremental’ and differential’ backup deal with having a base level copy of a set of files and then maintaining the difference between the data today and the base.

Incremental backups track the difference between the base and essentially build on each other. For example if there are five incremental backups to restore the file the backups must be restored sequentially as they were taken – first #1, then #2, etc. The amount of data in each increment is typically less than other methods. In robust File backup software, this is accomplished by log files, and associated data that is captured.
Differential backups track the difference between the base and what the current state is today. Each time a differential backup is taken, all of the changes that have occurred are contained in the backup data. To restore the data only the most recent version of the ‘difference’ needs to be restored. The amount of data in each differential backup typically gets larger as the changes accumulate.

In both scenarios there typically is a complete base level copy done periodically so there are no differences and the process of incremental/differential backup start again.

Purges of old data need to occur. Some backup programs do thls automatically.

14 Replies

Replies have been turned off for this discussion

metapaso
Apprentice
Nov 14, 2015
Hi Bains,

My RN316 is currently not working properly due to some issues in 6.4.0, so I can't go paste images of what I have set up in the UI. But I do have some information for you.

I enabled SSH and installed HTOP so I could see how the processes are run when rsync is triggered from the UI system. Mostly, I was interested in how the --exclude parameters worked, because I too found the Netgear documentation very sparse on this issue. I posted some of my results in another thread. Unfortunately, I forget what the other options were on the command line, but I'm sure you could fire up rsync and peek around yourself to see what's there. I'd be happy to do it for you once my machine is back up and running. I recall most of the options being pretty sane. I really do recommend using excludes, though, as some files such as VMs, and the dot-files on OSX systems for spotlight can be a real headache.

I had a nice discussion with Stephen B about the --in-place option available in rsync, and although I think there will be some future developments within RSYNC for recognizing COW filesystems, for now I'm not going to worry about it.

I think your implied question in regards to rsync on Readynas is how to control incremental / differential / base backups. I'm not sure if you mean copying FROM another computer TO the ReadyNAS or FROM the Readynas TO another computer. I'm only familiar with copying FROM another computer TO the ReadyNAS, so I can describe what I do now and also how I used to do it on an older ReadyNAS. Sorry if I'm being too didactic here, but just for completeness:

First, everyone on this forum will tell you over and over again, ReadyNAS is not a backup. I'm learning about that today as my system is caught in a boot rebalance that I can't get it out of. Hopefully the techs will be able to get it to work. For us, the ReadyNAS is simply a local backup copy for easy recovery if our main systems go down. We have a second backup copy onsite and 2 offsite backups.

The various configuration options for rsync mean for me that you can convince rsync to do some pretty sophisticated backing up, including base copies, incremental copies, or hardlinked incremental copies. The basic algorithm remains the same, however, when copying from one machine to another.

We have a 6.5TB set of image files from our clients. The main working copy is on the Mac that my wife uses for her image processing. The "backup" copy is on the ReadyNAS. First, rsync runs on EACH machine. That is: the MAC runs a copy of rsync AND the ReadyNAS runs a copy of rsync (for OS X, you'll want to grab the latest binary of rsync as the one bundled with OSX is very out of date and incompatible). The two copies of RSYNC talk to each other and figure out the differences between the two sets of files.. For mathematical reasons, the computation of the differences is quite a bit faster than you might imagine and requires only minimal information to be sent to each other. By default, RSYNC uses only timestamps and file sizes to compare differences initially, then if it finds any discrepancy it will look at actual binary differences, but you can force it to check binary differences with the --checksum option if you like.

After the differences have been figured out, the rsync version on the MAC will start sending the file and folder differences to the rsync version on the ReadyNAS. You can tell rsync how you want those differences to be stored. You can put the changed files in a different folder using --backup-dir option, so you DO keep a backup history of old files. OR, you can use the magic of Hard Links and --link-dest so that you keep a fully intact version of the directory tree for each time you run rsync. Unfortunately, you don't get to use any of these options with the ReadyNAS UI, but that's OK because of btrfs.

This is where ReadyNAS OS 6 and BTRFS steps in. Currently, I don't rely on rsync to do any of the heavy lifting in regards to keeping track of past versions. I simply turn snapshots on for the share that I'm using. BTRFS will manage the version history of the files in a sane way. It will keep hourly snapshots for a day, then daily snapshots for a week, weekly snapshots for a month, etc etc.

Because btrfs handles all the version history, rsync has only one job: copy the files over. If I want to go back in time on a particular file, I will use the snapshot browse feature. Rsync just keeps the most recent files up to date.

Rsync is fast. It scans our 6.5TB image library in a few minutes. We have approximately 20GB of new and/or changed files a day. Rsync finds them and copies them over in a hurry. I run it every 6 hours.

We used to have an old ReadyNAS NV+ Sparc version. I created a backup script that ran every 4 hours and did essentially the same thing, but required the use of --link-dest and hard links. Actually, it was great. The only problem is that it didn't manually clear out old copies, and sometimes it broke and started re-sending the entire file-set again. But with a little weekly maintenance it was functional. If you're interested, I would be happy to share it.

Hope that helps!
- Bains
  Guide
  Nov 14, 2015
  Thanks for taking the time to reply and outline what you know. Very thoughtful.
  
  We are using our ReadyNAS systems as the data repository in all of our branch operations – there are over 50 systems installed scattered across the country. What that means is that we do not have ‘IT folks” with any detailed knowledge to maintain those systems. Stuff has to be intuitive and simple.
  
  Workstations use SMB protocols and folks do their work – sales, billing, inventory, etc. All data is captured on the ReadyNAS units. That data in turn is captured via backup to one or more Windows workstation during the night. CryptoLocker malware infecting the workstations has scared us to death – I figure it is only a matter of time before we get hit. We want backup that we can fall back on after the malware has wreaked havoc. The backup data shares will be RSYNC protocol only so in theory CryptoLocker will not find the data.
  
  I do not understand the Linux environment well and all of our experience is in the Windows area where the terms Incremental and Differential have very precise meanings. And that is why I wanted to know what the RSYNC options selected were – to check and see if it meets our needs.
  
  We will actually be backing up internally to the ReadyNAS – volume to volume. It will be our data repository and our backup in case of malware. For other circumstances we will have the workstation backup and use it as we currently do.
  
  I am thinking about trying the snapshot approach but we will need to see if non-technical personnel can cope with that capability and the different ReadyNAS UI. Our trial implementation so far has been disappointing – folks do not understand the paradigm.
  
  Thanks again
  - metapaso
    Apprentice
    Nov 15, 2015
    Although Incremental and Differential backups have specific meaning, I don't think in your case that you need to focus on whether rsync does one or the other. It can be configured to do either, though as I understand, not "easily" through the ReadyNAS UI in version 6.
    
    If your ReadyNAS units are running OS 6, then the simplest solution for your various users may be to enable snapshotting on the volume or volumes in question and make sure that snapshots are NOT browseable by the users. That means the snapshots won't be exposed via SMB and CryptoLocker or other malware won't find them. If you are running large databases or have large files with frequent changes, you might look into how well this performs. I also cannot recommend this with the current 6.4.0 release as there are a lot of performance issues related to purging snapshots.
    
    It's a good idea to practice your restore procedures too so that If you do get struck with malware of ANY kind you can be sure that your backups remain "read-only" and cannot be edited or infiltrated by the malware as you try to do your restore.
    
    Older versions of the ReadyNAS software (before version 6) do not use the btrfs filesystem and so do not have the unlimited snapshot capability. If you are using an older version of the ReadyNAS software then you can definitely do a volume-to-volume rsync with options that suit your needs. The key to undestanding the most powerful feature here is to read up on how hard links work in Linux and how to take advantage of hard links with rsync using the --link-dest option. In a nutshell, --link-dest can give you either incremental or differential backups (depending on how you set it up) that are a breeze to restore because the UNCHANGED files for each incremental backup are hardlinked.
    
    To explain in detail. A properly designed rsync command or script can be used to create a time-stamped backup folders in a format that looks like:
    
    ..
    Volume1_Backup_20151115-0900/ Volume1_Backup_20151115-1300/ Volume1_Backup_20151115-1700/
    Inside each folder would be a COMPLETE copy of all the files on the original Volume1 at the date and time indicated. Any files that are the SAME would be hardlinked to each other to save space, any files that are different in each folder would not be linked. Of course, if THESE folders were exposed to malware, they could get corrupted, but the idea is that these folders are locked down perhaps with only read permission enabled and definitely not accessible via SMB. So, your concern with a malware infection on the original Volume1 would mean that rsync would see the corrupted file as changed and when copying it to the backup, it would NOT overwrite the originals.
    
    I have two final thoughts:
    
    1) We've had very good luck with CrashPlan. It's not officially supported on ReadyNAS, but perhaps your Windows Servers or Workstations could run the application and move your backups safely to offsite. It's something like $10/month per computer, but even for 50 servers I think it's totally a bargain to get rock-solid offsite backups for an entire nationwide organization for only $500/month. I can't say I advocate for CrashPLan over BackBlaze or Carbonite, they all seem pretty decent. CrashPlan has a fairly sophisticated central control application for their Pro line of software, which could mean a kind of easy "central monitoring" of how the backups are working.
    
    2) Whatever backup strategy you do use, please make sure whoever is performing the backups uses a retention policy that includes keeping older versions of the backup for a long time. On our old backup server, I kept weekly backups for half a year, monthly versions going back as far as 15 months, and then half-year versions going back as far as 3 or 4 years. If you do get a malware infection, you may have to go a long way back in time to find a version that wasn't corrupted.
    
    best of luck