This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
The Netgear documentation regarding backup is sparse. The ReadyNAS backup capability is a GUI for the most common file copy needs. The Netgear documentation discusses incremental backup but that is not what happens since there are no lots of incremental files created.
What actual rsync options are enabled during a Netgear copy?
Bsckground
The terms ‘incremental’ and differential’ backup deal with having a base level copy of a set of files and then maintaining the difference between the data today and the base.
Incremental backups track the difference between the base and essentially build on each other. For example if there are five incremental backups to restore the file the backups must be restored sequentially as they were taken – first #1, then #2, etc. The amount of data in each increment is typically less than other methods. In robust File backup software, this is accomplished by log files, and associated data that is captured.
Differential backups track the difference between the base and what the current state is today. Each time a differential backup is taken, all of the changes that have occurred are contained in the backup data. To restore the data only the most recent version of the ‘difference’ needs to be restored. The amount of data in each differential backup typically gets larger as the changes accumulate.
In both scenarios there typically is a complete base level copy done periodically so there are no differences and the process of incremental/differential backup start again.
Purges of old data need to occur. Some backup programs do thls automatically.
My RN316 is currently not working properly due to some issues in 6.4.0, so I can't go paste images of what I have set up in the UI. But I do have some information for you.
I enabled SSH and installed HTOP so I could see how the processes are run when rsync is triggered from the UI system. Mostly, I was interested in how the --exclude parameters worked, because I too found the Netgear documentation very sparse on this issue. I posted some of my results in another thread. Unfortunately, I forget what the other options were on the command line, but I'm sure you could fire up rsync and peek around yourself to see what's there. I'd be happy to do it for you once my machine is back up and running. I recall most of the options being pretty sane. I really do recommend using excludes, though, as some files such as VMs, and the dot-files on OSX systems for spotlight can be a real headache.
I think your implied question in regards to rsync on Readynas is how to control incremental / differential / base backups. I'm not sure if you mean copying FROM another computer TO the ReadyNAS or FROM the Readynas TO another computer. I'm only familiar with copying FROM another computer TO the ReadyNAS, so I can describe what I do now and also how I used to do it on an older ReadyNAS. Sorry if I'm being too didactic here, but just for completeness:
First, everyone on this forum will tell you over and over again, ReadyNAS is not a backup. I'm learning about that today as my system is caught in a boot rebalance that I can't get it out of. Hopefully the techs will be able to get it to work. For us, the ReadyNAS is simply a local backup copy for easy recovery if our main systems go down. We have a second backup copy onsite and 2 offsite backups.
The various configuration options for rsync mean for me that you can convince rsync to do some pretty sophisticated backing up, including base copies, incremental copies, or hardlinked incremental copies. The basic algorithm remains the same, however, when copying from one machine to another.
We have a 6.5TB set of image files from our clients. The main working copy is on the Mac that my wife uses for her image processing. The "backup" copy is on the ReadyNAS. First, rsync runs on EACH machine. That is: the MAC runs a copy of rsync AND the ReadyNAS runs a copy of rsync (for OS X, you'll want to grab the latest binary of rsync as the one bundled with OSX is very out of date and incompatible). The two copies of RSYNC talk to each other and figure out the differences between the two sets of files.. For mathematical reasons, the computation of the differences is quite a bit faster than you might imagine and requires only minimal information to be sent to each other. By default, RSYNC uses only timestamps and file sizes to compare differences initially, then if it finds any discrepancy it will look at actual binary differences, but you can force it to check binary differences with the --checksum option if you like.
After the differences have been figured out, the rsync version on the MAC will start sending the file and folder differences to the rsync version on the ReadyNAS. You can tell rsync how you want those differences to be stored. You can put the changed files in a different folder using --backup-dir option, so you DO keep a backup history of old files. OR, you can use the magic of Hard Links and --link-dest so that you keep a fully intact version of the directory tree for each time you run rsync. Unfortunately, you don't get to use any of these options with the ReadyNAS UI, but that's OK because of btrfs.
This is where ReadyNAS OS 6 and BTRFS steps in. Currently, I don't rely on rsync to do any of the heavy lifting in regards to keeping track of past versions. I simply turn snapshots on for the share that I'm using. BTRFS will manage the version history of the files in a sane way. It will keep hourly snapshots for a day, then daily snapshots for a week, weekly snapshots for a month, etc etc.
Because btrfs handles all the version history, rsync has only one job: copy the files over. If I want to go back in time on a particular file, I will use the snapshot browse feature. Rsync just keeps the most recent files up to date.
Rsync is fast. It scans our 6.5TB image library in a few minutes. We have approximately 20GB of new and/or changed files a day. Rsync finds them and copies them over in a hurry. I run it every 6 hours.
We used to have an old ReadyNAS NV+ Sparc version. I created a backup script that ran every 4 hours and did essentially the same thing, but required the use of --link-dest and hard links. Actually, it was great. The only problem is that it didn't manually clear out old copies, and sometimes it broke and started re-sending the entire file-set again. But with a little weekly maintenance it was functional. If you're interested, I would be happy to share it.
Thanks for taking the time to reply and outline what you know. Very thoughtful.
We are using our ReadyNAS systems as the data repository in all of our branch operations – there are over 50 systems installed scattered across the country. What that means is that we do not have ‘IT folks” with any detailed knowledge to maintain those systems. Stuff has to be intuitive and simple.
Workstations use SMB protocols and folks do their work – sales, billing, inventory, etc. All data is captured on the ReadyNAS units. That data in turn is captured via backup to one or more Windows workstation during the night. CryptoLocker malware infecting the workstations has scared us to death – I figure it is only a matter of time before we get hit. We want backup that we can fall back on after the malware has wreaked havoc. The backup data shares will be RSYNC protocol only so in theory CryptoLocker will not find the data.
I do not understand the Linux environment well and all of our experience is in the Windows area where the terms Incremental and Differential have very precise meanings. And that is why I wanted to know what the RSYNC options selected were – to check and see if it meets our needs.
We will actually be backing up internally to the ReadyNAS – volume to volume. It will be our data repository and our backup in case of malware. For other circumstances we will have the workstation backup and use it as we currently do.
I am thinking about trying the snapshot approach but we will need to see if non-technical personnel can cope with that capability and the different ReadyNAS UI. Our trial implementation so far has been disappointing – folks do not understand the paradigm.
Although Incremental and Differential backups have specific meaning, I don't think in your case that you need to focus on whether rsync does one or the other. It can be configured to do either, though as I understand, not "easily" through the ReadyNAS UI in version 6.
If your ReadyNAS units are running OS 6, then the simplest solution for your various users may be to enable snapshotting on the volume or volumes in question and make sure that snapshots are NOT browseable by the users. That means the snapshots won't be exposed via SMB and CryptoLocker or other malware won't find them. If you are running large databases or have large files with frequent changes, you might look into how well this performs. I also cannot recommend this with the current 6.4.0 release as there are a lot of performance issues related to purging snapshots.
It's a good idea to practice your restore procedures too so that If you do get struck with malware of ANY kind you can be sure that your backups remain "read-only" and cannot be edited or infiltrated by the malware as you try to do your restore.
Older versions of the ReadyNAS software (before version 6) do not use the btrfs filesystem and so do not have the unlimited snapshot capability. If you are using an older version of the ReadyNAS software then you can definitely do a volume-to-volume rsync with options that suit your needs. The key to undestanding the most powerful feature here is to read up on how hard links work in Linux and how to take advantage of hard links with rsync using the --link-dest option. In a nutshell, --link-dest can give you either incremental or differential backups (depending on how you set it up) that are a breeze to restore because the UNCHANGED files for each incremental backup are hardlinked.
To explain in detail. A properly designed rsync command or script can be used to create a time-stamped backup folders in a format that looks like:
Inside each folder would be a COMPLETE copy of all the files on the original Volume1 at the date and time indicated. Any files that are the SAME would be hardlinked to each other to save space, any files that are different in each folder would not be linked. Of course, if THESE folders were exposed to malware, they could get corrupted, but the idea is that these folders are locked down perhaps with only read permission enabled and definitely not accessible via SMB. So, your concern with a malware infection on the original Volume1 would mean that rsync would see the corrupted file as changed and when copying it to the backup, it would NOT overwrite the originals.
I have two final thoughts:
1) We've had very good luck with CrashPlan. It's not officially supported on ReadyNAS, but perhaps your Windows Servers or Workstations could run the application and move your backups safely to offsite. It's something like $10/month per computer, but even for 50 servers I think it's totally a bargain to get rock-solid offsite backups for an entire nationwide organization for only $500/month. I can't say I advocate for CrashPLan over BackBlaze or Carbonite, they all seem pretty decent. CrashPlan has a fairly sophisticated central control application for their Pro line of software, which could mean a kind of easy "central monitoring" of how the backups are working.
2) Whatever backup strategy you do use, please make sure whoever is performing the backups uses a retention policy that includes keeping older versions of the backup for a long time. On our old backup server, I kept weekly backups for half a year, monthly versions going back as far as 15 months, and then half-year versions going back as far as 3 or 4 years. If you do get a malware infection, you may have to go a long way back in time to find a version that wasn't corrupted.
What a great answer -- you took a lot of time and I really appreciate your effort.
My issue is not technical as much as it is personnel. The individual(s) responsible for the ReacyNAS unit at each location is most likely an accounting clerk or the most junior sales person. Their knowledge of Lunix and any capability to deal with issues in non-existent. Think about a well-intentioned person who would start to 'push' buttons to get the office back operational. Then think about me on an airplane.
At this time we do not allow field personnel any access to the ReadyNAS UI – too much opportunity to’ push’ the wrong buttons. Instead we access the remote site with TeamViewer connected to a local workstations and then use Frontview.
How could I make the snapshots not browseable – the idea is intriguing. I want to explore.
Your cautionary warning about the newness of the BRFS technology to the platform is very useful – it says wait a while.
I like your idea of off-site backup but many of our operations have only 1.5 GBPS Internet access. For that reason I have avoided Internet based transportation solutions. We have an off-site backup plan and approach for each site currently. Not the best but certainly adequate.
Again, thanks for your comments and the time you spent.
How could I make the snapshots not browseable – the idea is intriguing. I want to explore.
.
There is a checkbox for each share for "Allow Snapshot Access". All you need to do is make sure it is unchecked.
Though the Windows systems still have some access to the snapshots. If users right-click on a changed file or folder, they will see "restore previous versions", which accesses the snapshots. Some malware might be able to access them also. However, I'd still recommend disallowing snapshot access (assuming of course that you enable snapshots in the first place).
My original question was what options the ReadyNAS rsync implementation uses for its copy when activated via the UI.
I don't recall off hand - perhaps netgear can comment. If you have ssh access, you can see the options if you run top while a backup is in progress. There's a "c" toggle that shows the command line (or not).
The other question is if it is possible to change those options somehow without going deep into the system with kernel access.
You would need ssh access to change them. I've never tried to do that, but I suspect it might be difficult.
An easier path might be to directly create your own back-up cron jobs, and not use frontview backup. You could write the logs to a share, so status could be easily seen. Though you wouldn't get alerts on failure.
Your suggestions are stretching my technical knowledge and most definitely my available time.
50+ remote sites and no technical personnel means no SSH access will be allowed.
No alerts, not really feasible from a management perspective.
We like supported solutions – money is less an issue than reasonable support for the integrated solutions. Unfortunately, aside from a great NAS product, the current ReadyNAS environment is more like a toolbox – good solutions for some stuff but not integrated and RTG.
I was/am hoping someone will see this post and offer/suggest their solution that is already integrated with the appropriate GUI – similar to what we find in the Windows environment.
OK, so as I understand it, you want to copy a volume or share from within the ReadyNAS, just to make a kind of backup copy that's "isolated" from SMB. Perhaps this method would work, since you are backing up from within the readynas. It would rely only on the Web interface for Local to Local backup. Rsync is not used:
Step 1: On the "Shares" page, create a new share. Let's call this share "MasterBackup".
In the options for "MasterBackup", choose "bitrot protection" AND set "snapshot schedule" for Daily or Hourly depending on how often you think you need a backup. Disable ALL access protocols (rsync itself my not even be needed, but you may want it for later)
Step 2: For each folder or share that you want to back up, create a subfolder in MasterBackup. For example, if you have a share named DesignTeam, then create a folder in MasterBackup named "DesignTeamBak" or something like that.
Step 3: For each folder or share that you want to back up, in the "Backup" page, create a Local to Local backup. Select the SOURCE folder as "Design Team" and the Destination as "/MasterBackup/DesignTeamBak". Set the backup schedule according to your needs, but remember that backing up MORE often than your snapshot schedule is counterproductive because you'll be relying on the snapshots for the real backing up part.
Note: Perhaps someone from Netgear can tell you what options are used for this backup. It might actually be rsync under the hood, but it might just be a plain old Linux copy command "cp". That might influence whether you want to set the "full backup" option to "every time" or only "first time"
You'll need a good recovery procedure in place in case a viral or worm disaster does strike...my recommendation is to roll the snapshots back to a known good snapshot, then plug in a clean external USB drive and copy the known good snapshot onto the clean USB drive. Maybe a couple of USB drives so you are sure you always maintain a safe one.
Thanks again for spending the time to offer your suggestions.
Let me be clear, there is no ‘if CryptoLocker hits’ in our contingency planning. Instead we have updated our contingency plan to discuss “when CryptoLocker hits”.
What I did not emphasize or explain properly is that we have non-technical folks responsible for the ReadyNAS system at our branch locations. They try hard to meet what they think are the objectives we want (keeping the branch operational) but the result is often not what is optimal. Think about a typical well intentioned person who tries to help by pushing the ‘buttons’ that they know little about.
We expect to have our data scrambled and unusable at one or more sites. Depending on events and email communications we expect that the event will spread to at least five internal sites and possibly HQ. So, using a technical term, the “fit has hit the shan”. I am not concerned with cost but instead a plan that can be followed by the essential equivalent of entry level technicians with low technical skill level.
Our current plan is to isolate the workstations, remove the malware and then fall back to the last good backup and manually recover the data that is not in the backup, Not elegant, probably will cost a couple thousand at each branch for additional labor costs but lots better than paying ransom.
Snapshots are interesting but probably not the right answer for our distributed environment. At this time I have elected to not use them. That may change in the future when I gain more confidence in the implementation.
StephenB has provided a way for rsync to be used internally and not be equivalent to the Linux cp command. We have tested it and it seems to work. It does not do a ‘copy’ but instead does the typical rsync ‘difference’ for files that have changed. We plan to have several cyclical jobs that will enable multiple copies of the data at different times/stages. Restoration seems to work in our test cases.
The process is a little clutsy/nerdy but it does work. It can all be done from HQ with TeamViewer and RAIDar. I would like to change some of the rsync parameters but the Netgear I/F does not allow that.
As I mentioned previously, we like supported solutions – money is less an issue than reasonable support for the integrated solutions. Unfortunately, aside from a great NAS product, the current ReadyNAS environment is more like a toolbox – good solutions for some stuff but not integrated and RTG.
To this point, the Netgear folks have not responded with a technical answer to my initial question on this thread.
My guess is that the information is either ‘secret’ or they are too busy to pay attention to this type of technical questions for their customers or they do not care. The result is the same.
I was/am hoping a software vendor would see this post and offer/suggest their solution that is already integrated with the appropriate GUI – similar to what we find in the Windows environment.
