NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
Backup using rsync – what options are used
Rsync is a robust Linux file copy program with lots of options http://linux.die.net/man/1/rsync is a great discussion of the features and options. . http://serverfault.com/questions...
Sounds like you may be on your way to a solution.
FWIW, I got my ReadyNAS up into operating condition today and have the following to report regarding the rsync command as issued from the GUI
rsync -v -8 --timeout=30 --recursive --links --times --devices --specials --one-file-system --modify-window=1 --owner --group --no-acls --no-perms --exclude CrashPlan --exclude Downloads --exclude Library/logs --exclude Library/Caches --rsh=ssh -o SrticHostKeyChecking=no -PasswordAuthentication=no -p 22 name@host:/SourceShare /data/DestShare
Good luck!
EDITED TO ADD:
I forgot to say that this is Rsync over ssh with the following options
NO - enable compression
NO - remove deleted files on target
I DO have a list of excluded files/folders, but it seems the gui is programmed also to include .snapshots in that exclusion list
Schedule Full backup: "First Time Only"
NO - Remove Contents before
NO - Change ownership
Hi Bains,
My RN316 is currently not working properly due to some issues in 6.4.0, so I can't go paste images of what I have set up in the UI. But I do have some information for you.
I enabled SSH and installed HTOP so I could see how the processes are run when rsync is triggered from the UI system. Mostly, I was interested in how the --exclude parameters worked, because I too found the Netgear documentation very sparse on this issue. I posted some of my results in another thread. Unfortunately, I forget what the other options were on the command line, but I'm sure you could fire up rsync and peek around yourself to see what's there. I'd be happy to do it for you once my machine is back up and running. I recall most of the options being pretty sane. I really do recommend using excludes, though, as some files such as VMs, and the dot-files on OSX systems for spotlight can be a real headache.
I had a nice discussion with Stephen B about the --in-place option available in rsync, and although I think there will be some future developments within RSYNC for recognizing COW filesystems, for now I'm not going to worry about it.
I think your implied question in regards to rsync on Readynas is how to control incremental / differential / base backups. I'm not sure if you mean copying FROM another computer TO the ReadyNAS or FROM the Readynas TO another computer. I'm only familiar with copying FROM another computer TO the ReadyNAS, so I can describe what I do now and also how I used to do it on an older ReadyNAS. Sorry if I'm being too didactic here, but just for completeness:
First, everyone on this forum will tell you over and over again, ReadyNAS is not a backup. I'm learning about that today as my system is caught in a boot rebalance that I can't get it out of. Hopefully the techs will be able to get it to work. For us, the ReadyNAS is simply a local backup copy for easy recovery if our main systems go down. We have a second backup copy onsite and 2 offsite backups.
The various configuration options for rsync mean for me that you can convince rsync to do some pretty sophisticated backing up, including base copies, incremental copies, or hardlinked incremental copies. The basic algorithm remains the same, however, when copying from one machine to another.
We have a 6.5TB set of image files from our clients. The main working copy is on the Mac that my wife uses for her image processing. The "backup" copy is on the ReadyNAS. First, rsync runs on EACH machine. That is: the MAC runs a copy of rsync AND the ReadyNAS runs a copy of rsync (for OS X, you'll want to grab the latest binary of rsync as the one bundled with OSX is very out of date and incompatible). The two copies of RSYNC talk to each other and figure out the differences between the two sets of files.. For mathematical reasons, the computation of the differences is quite a bit faster than you might imagine and requires only minimal information to be sent to each other. By default, RSYNC uses only timestamps and file sizes to compare differences initially, then if it finds any discrepancy it will look at actual binary differences, but you can force it to check binary differences with the --checksum option if you like.
After the differences have been figured out, the rsync version on the MAC will start sending the file and folder differences to the rsync version on the ReadyNAS. You can tell rsync how you want those differences to be stored. You can put the changed files in a different folder using --backup-dir option, so you DO keep a backup history of old files. OR, you can use the magic of Hard Links and --link-dest so that you keep a fully intact version of the directory tree for each time you run rsync. Unfortunately, you don't get to use any of these options with the ReadyNAS UI, but that's OK because of btrfs.
This is where ReadyNAS OS 6 and BTRFS steps in. Currently, I don't rely on rsync to do any of the heavy lifting in regards to keeping track of past versions. I simply turn snapshots on for the share that I'm using. BTRFS will manage the version history of the files in a sane way. It will keep hourly snapshots for a day, then daily snapshots for a week, weekly snapshots for a month, etc etc.
Because btrfs handles all the version history, rsync has only one job: copy the files over. If I want to go back in time on a particular file, I will use the snapshot browse feature. Rsync just keeps the most recent files up to date.
Rsync is fast. It scans our 6.5TB image library in a few minutes. We have approximately 20GB of new and/or changed files a day. Rsync finds them and copies them over in a hurry. I run it every 6 hours.
We used to have an old ReadyNAS NV+ Sparc version. I created a backup script that ran every 4 hours and did essentially the same thing, but required the use of --link-dest and hard links. Actually, it was great. The only problem is that it didn't manually clear out old copies, and sometimes it broke and started re-sending the entire file-set again. But with a little weekly maintenance it was functional. If you're interested, I would be happy to share it.
Hope that helps!
Thanks for taking the time to reply and outline what you know. Very thoughtful.
We are using our ReadyNAS systems as the data repository in all of our branch operations – there are over 50 systems installed scattered across the country. What that means is that we do not have ‘IT folks” with any detailed knowledge to maintain those systems. Stuff has to be intuitive and simple.
Workstations use SMB protocols and folks do their work – sales, billing, inventory, etc. All data is captured on the ReadyNAS units. That data in turn is captured via backup to one or more Windows workstation during the night. CryptoLocker malware infecting the workstations has scared us to death – I figure it is only a matter of time before we get hit. We want backup that we can fall back on after the malware has wreaked havoc. The backup data shares will be RSYNC protocol only so in theory CryptoLocker will not find the data.
I do not understand the Linux environment well and all of our experience is in the Windows area where the terms Incremental and Differential have very precise meanings. And that is why I wanted to know what the RSYNC options selected were – to check and see if it meets our needs.
We will actually be backing up internally to the ReadyNAS – volume to volume. It will be our data repository and our backup in case of malware. For other circumstances we will have the workstation backup and use it as we currently do.
I am thinking about trying the snapshot approach but we will need to see if non-technical personnel can cope with that capability and the different ReadyNAS UI. Our trial implementation so far has been disappointing – folks do not understand the paradigm.
Thanks again
Although Incremental and Differential backups have specific meaning, I don't think in your case that you need to focus on whether rsync does one or the other. It can be configured to do either, though as I understand, not "easily" through the ReadyNAS UI in version 6.
If your ReadyNAS units are running OS 6, then the simplest solution for your various users may be to enable snapshotting on the volume or volumes in question and make sure that snapshots are NOT browseable by the users. That means the snapshots won't be exposed via SMB and CryptoLocker or other malware won't find them. If you are running large databases or have large files with frequent changes, you might look into how well this performs. I also cannot recommend this with the current 6.4.0 release as there are a lot of performance issues related to purging snapshots.
It's a good idea to practice your restore procedures too so that If you do get struck with malware of ANY kind you can be sure that your backups remain "read-only" and cannot be edited or infiltrated by the malware as you try to do your restore.
Older versions of the ReadyNAS software (before version 6) do not use the btrfs filesystem and so do not have the unlimited snapshot capability. If you are using an older version of the ReadyNAS software then you can definitely do a volume-to-volume rsync with options that suit your needs. The key to undestanding the most powerful feature here is to read up on how hard links work in Linux and how to take advantage of hard links with rsync using the --link-dest option. In a nutshell, --link-dest can give you either incremental or differential backups (depending on how you set it up) that are a breeze to restore because the UNCHANGED files for each incremental backup are hardlinked.
To explain in detail. A properly designed rsync command or script can be used to create a time-stamped backup folders in a format that looks like:
..
Volume1_Backup_20151115-0900/ Volume1_Backup_20151115-1300/ Volume1_Backup_20151115-1700/Inside each folder would be a COMPLETE copy of all the files on the original Volume1 at the date and time indicated. Any files that are the SAME would be hardlinked to each other to save space, any files that are different in each folder would not be linked. Of course, if THESE folders were exposed to malware, they could get corrupted, but the idea is that these folders are locked down perhaps with only read permission enabled and definitely not accessible via SMB. So, your concern with a malware infection on the original Volume1 would mean that rsync would see the corrupted file as changed and when copying it to the backup, it would NOT overwrite the originals.
I have two final thoughts:
1) We've had very good luck with CrashPlan. It's not officially supported on ReadyNAS, but perhaps your Windows Servers or Workstations could run the application and move your backups safely to offsite. It's something like $10/month per computer, but even for 50 servers I think it's totally a bargain to get rock-solid offsite backups for an entire nationwide organization for only $500/month. I can't say I advocate for CrashPLan over BackBlaze or Carbonite, they all seem pretty decent. CrashPlan has a fairly sophisticated central control application for their Pro line of software, which could mean a kind of easy "central monitoring" of how the backups are working.
2) Whatever backup strategy you do use, please make sure whoever is performing the backups uses a retention policy that includes keeping older versions of the backup for a long time. On our old backup server, I kept weekly backups for half a year, monthly versions going back as far as 15 months, and then half-year versions going back as far as 3 or 4 years. If you do get a malware infection, you may have to go a long way back in time to find a version that wasn't corrupted.
best of luck
What a great answer -- you took a lot of time and I really appreciate your effort.
My issue is not technical as much as it is personnel. The individual(s) responsible for the ReacyNAS unit at each location is most likely an accounting clerk or the most junior sales person. Their knowledge of Lunix and any capability to deal with issues in non-existent. Think about a well-intentioned person who would start to 'push' buttons to get the office back operational. Then think about me on an airplane.
At this time we do not allow field personnel any access to the ReadyNAS UI – too much opportunity to’ push’ the wrong buttons. Instead we access the remote site with TeamViewer connected to a local workstations and then use Frontview.
How could I make the snapshots not browseable – the idea is intriguing. I want to explore.
Your cautionary warning about the newness of the BRFS technology to the platform is very useful – it says wait a while.
I like your idea of off-site backup but many of our operations have only 1.5 GBPS Internet access. For that reason I have avoided Internet based transportation solutions. We have an off-site backup plan and approach for each site currently. Not the best but certainly adequate.
Again, thanks for your comments and the time you spent.
Bains wrote:
How could I make the snapshots not browseable – the idea is intriguing. I want to explore.
.
There is a checkbox for each share for "Allow Snapshot Access". All you need to do is make sure it is unchecked.Though the Windows systems still have some access to the snapshots. If users right-click on a changed file or folder, they will see "restore previous versions", which accesses the snapshots. Some malware might be able to access them also. However, I'd still recommend disallowing snapshot access (assuming of course that you enable snapshots in the first place).
Bains wrote:
Your cautionary warning about the newness of the BRFS technology to the platform is very useful – it says wait a while.
The OS 6 platforms with BTRFS were introduced about 2 years ago. So its not that new.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
