NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Step-by-Step: Rsync to Local USB
UPDATE: 11/16/2010
LAST UPDATE: 11/17/2010
--------------------------------------
It would be great to create a step-by-step guide to backing up a ReadyNAS to a local USB drive for those of us newbies who are, well, new to the whole thing :) I'm a bit at a loss, because my searches of the forums are coming up with contradictory information. For example:
viewtopic.php?p=240107#p240107
viewtopic.php?p=266997#p266997
One says that the source should be a 'remote: rsync server' the other says that the destination should be an 'rsync server'. It's not clear if both methods work, or if one is preferred over the other.
(Per below & experience: both methods DO work and neither is preferred, however, if you want to backup private home shares with rsync, then you must set the destination to be the 'remote: rsync server')
(Another situation where 'remote: rsync server' must be the destination is if you are trying to backup your an entire ReadyNAS volume at once. In that case, your 'souce' becomes 'volume: <volumeLetter' and the destination is configured as described below.
I also can find no documentation about the syntax allowed in the 'exclusion list'. Is it only for files and directories? Are wildcards allowed? How are the directories notated?
(Wildcards--at least '*' and '?' are indeed valid.)
And so on....
Here's what I've come up with so far. I'd love it if folks can add in the missing steps or clarify any of the missing pieces (currently based on my setup--ReadyNAS Pro Business and RAIDiator 4.2.15). Since I am focused on LOCAL backups, I will ignore any of the possibilities of using Rsync over SSH or the need to set specific rsync users & passwords:
1. Enable Rsync service in Frontview: Services->Standard File Protocols->Rsync
2. Set Rsync permissions on the shares you want to backup (your source shares): Shares->Share Listing
NOTE: if rsync only has 'read-only' permissions to your source share, you will need to give rsync 'read/write' access if you ever want to use rsync to restore to that share.
3. Set Rsync permissions on the USB Storage Device(s) you are backing up to (your destination shares). The default of 'read/write' is fine.
4. Define your backup job(s) in Frontview: Backup->Add New Backup Job.
4a) Set up the Source:
- I find it easiest to think of the source being the rsync server, so unless I'm backing up a home share, I always use 'remote: rsync server' up here.
- Since we are backing up your local ReadyNAS, your hostname can be any of the following: <ReadyNAS name>, localhost, 127.0.0.1
- The path to your backup source: <shareName>/<folderStructure> (note that the trailing '/' is not strictly needed, but the beginning of the path should NOT include a '/') Also note that this path is CASE-SENSITIVE.
- Your login & password should be blank if you haven't set any of the shares in step 2 to require an rsync user/password.
- The option to tunnel rsync over ssh is not needed since were only concerned with local backups
- The option to 'enable compression' should remain unchecked. It's not useful when backing up to a local destination since bandwidth is not a concern.
- The option to 'remove deleted files' is worded in a slightly misleading fashion. This option will remove files from your backup destination that no longer exist on the source when doing incremental backups. (Also, the same thing holds true when using the rsync server as a destination, although the wording is still confusing.) Using this option means that your backup will basically be an exact duplicate of your source as of the backup time.
- Enter your exclusion list. For example, if I wanted to exclude my lossy photos and only keep my RAW versions, I could use "*.jpg,*.JPG" The exclusion list is case-sensitive, so if your file extensions are a hodge-podge of capitalization, it's possible that your results will not be as you expect. File paths work here too--also in conjunction with files themselves.
4b) Set up the Destination:
- Choose the share on your USB drive that you defined rsync permissions for in Step 3.
- You can leave the 'host' field blank.
- Enter the path to your backup destination: <folderStructure> (again, note that there is not a leading '/'). Case-sensitivity is still important.
- Unless you defined an rsync login/password for your USB share, you can leave these blank.
NOTE: If you are using the rsync server as the destination, you will have to enter your USB share as the first part of the path. This is ALSO case-sensitive (ie. USB_HDD_1_1 is not the same as usb_hdd_1_1).
4C) Set up the schedule:
- The frequency is how often the backup job will be triggered in a single day.
- The hours are when it will begin trying to trigger, and when it will stop trying. This is important if you have multiple jobs scheduled. If Job1 continues past both the START and END times of Job2, then Job2 will never trigger.
- Check off which days you want the preceding rules in effect.
4D) Set up Full/Incremental & Special options:
- Choose how often you would like a full backup to be performed as opposed to incremental backups.
- Choose your Log file verbosity
* Note: the option to remove the contents of the backup destination ONLY takes effect when a full backup is being done (per the schedule in the first option).
- Because we are only backing up locally, the ownership change option doesn't really apply, but it is an important tool to have if you back up remotely or plan on moving your USB drive to a different unit to restore files.
Thoughts & Clarifications Welcome!
brad
LAST UPDATE: 11/17/2010
--------------------------------------
It would be great to create a step-by-step guide to backing up a ReadyNAS to a local USB drive for those of us newbies who are, well, new to the whole thing :) I'm a bit at a loss, because my searches of the forums are coming up with contradictory information. For example:
viewtopic.php?p=240107#p240107
viewtopic.php?p=266997#p266997
One says that the source should be a 'remote: rsync server' the other says that the destination should be an 'rsync server'. It's not clear if both methods work, or if one is preferred over the other.
(Per below & experience: both methods DO work and neither is preferred, however, if you want to backup private home shares with rsync, then you must set the destination to be the 'remote: rsync server')
(Another situation where 'remote: rsync server' must be the destination is if you are trying to backup your an entire ReadyNAS volume at once. In that case, your 'souce' becomes 'volume: <volumeLetter' and the destination is configured as described below.
I also can find no documentation about the syntax allowed in the 'exclusion list'. Is it only for files and directories? Are wildcards allowed? How are the directories notated?
(Wildcards--at least '*' and '?' are indeed valid.)
And so on....
Here's what I've come up with so far. I'd love it if folks can add in the missing steps or clarify any of the missing pieces (currently based on my setup--ReadyNAS Pro Business and RAIDiator 4.2.15). Since I am focused on LOCAL backups, I will ignore any of the possibilities of using Rsync over SSH or the need to set specific rsync users & passwords:
1. Enable Rsync service in Frontview: Services->Standard File Protocols->Rsync
2. Set Rsync permissions on the shares you want to backup (your source shares): Shares->Share Listing
NOTE: if rsync only has 'read-only' permissions to your source share, you will need to give rsync 'read/write' access if you ever want to use rsync to restore to that share.
3. Set Rsync permissions on the USB Storage Device(s) you are backing up to (your destination shares). The default of 'read/write' is fine.
4. Define your backup job(s) in Frontview: Backup->Add New Backup Job.
4a) Set up the Source:
- I find it easiest to think of the source being the rsync server, so unless I'm backing up a home share, I always use 'remote: rsync server' up here.
- Since we are backing up your local ReadyNAS, your hostname can be any of the following: <ReadyNAS name>, localhost, 127.0.0.1
- The path to your backup source: <shareName>/<folderStructure> (note that the trailing '/' is not strictly needed, but the beginning of the path should NOT include a '/') Also note that this path is CASE-SENSITIVE.
- Your login & password should be blank if you haven't set any of the shares in step 2 to require an rsync user/password.
- The option to tunnel rsync over ssh is not needed since were only concerned with local backups
- The option to 'enable compression' should remain unchecked. It's not useful when backing up to a local destination since bandwidth is not a concern.
- The option to 'remove deleted files' is worded in a slightly misleading fashion. This option will remove files from your backup destination that no longer exist on the source when doing incremental backups. (Also, the same thing holds true when using the rsync server as a destination, although the wording is still confusing.) Using this option means that your backup will basically be an exact duplicate of your source as of the backup time.
- Enter your exclusion list. For example, if I wanted to exclude my lossy photos and only keep my RAW versions, I could use "*.jpg,*.JPG" The exclusion list is case-sensitive, so if your file extensions are a hodge-podge of capitalization, it's possible that your results will not be as you expect. File paths work here too--also in conjunction with files themselves.
4b) Set up the Destination:
- Choose the share on your USB drive that you defined rsync permissions for in Step 3.
- You can leave the 'host' field blank.
- Enter the path to your backup destination: <folderStructure> (again, note that there is not a leading '/'). Case-sensitivity is still important.
- Unless you defined an rsync login/password for your USB share, you can leave these blank.
NOTE: If you are using the rsync server as the destination, you will have to enter your USB share as the first part of the path. This is ALSO case-sensitive (ie. USB_HDD_1_1 is not the same as usb_hdd_1_1).
4C) Set up the schedule:
- The frequency is how often the backup job will be triggered in a single day.
- The hours are when it will begin trying to trigger, and when it will stop trying. This is important if you have multiple jobs scheduled. If Job1 continues past both the START and END times of Job2, then Job2 will never trigger.
- Check off which days you want the preceding rules in effect.
4D) Set up Full/Incremental & Special options:
- Choose how often you would like a full backup to be performed as opposed to incremental backups.
- Choose your Log file verbosity
* Note: the option to remove the contents of the backup destination ONLY takes effect when a full backup is being done (per the schedule in the first option).
- Because we are only backing up locally, the ownership change option doesn't really apply, but it is an important tool to have if you back up remotely or plan on moving your USB drive to a different unit to restore files.
Thoughts & Clarifications Welcome!
brad
10 Replies
bspachman wrote:
One says that the source should be a 'remote: rsync server' the other says that the destination should be an 'rsync server'. It's not clear if both methods work, or if one is preferred over the other.
Both will work.bspachman wrote:
I also can find no documentation about the syntax allowed in the 'exclusion list'. Is it only for files and directories? Are wildcards allowed? How are the directories notated?
The exclusion list is passed directly to the rsync command line. Consult the documentation for rsync (not ReadyNAS specific) for details.bspachman wrote:
* It's unclear to me whether it is a good practice to allow rsync to have 'read-only' access to the source shares, or should it have 'read/write' acess to the source shares. I have currently set my source shares to allow rsync 'read-only' access.
Read-only is fine. If it's the backup source, then rsync will only read from it.bspachman wrote:
3. Set Rsync permissions on the USB Storage Device(s) you are backing up to (your destination shares).
* Again, it's unclear what level of access Rsync should have to the destination shares. Certainly, it will need write access, but does it need read access? I have currently left my destination shares at 'read/write' for rsync.
You only need to do this if you're selecting rsync as the destination protocol. If you're selecting the USB drive share, then there's no need to configure rsync access to it.bspachman wrote:
4a) Set up the Source:
* Source as Rsync Server?
* Host: localhost? 127.0.0.1? the name of the ReadyNAS? the IP address of the ReadyNAS? -- all options seem to give me "Error connecting to xxxx"
Any of those should work as long as the path is correct.bspachman wrote:
* what does 'enable compression' do? is it valuable in these circumstances?
It enabled data compression. It won't help here since you're not trying to push the data over a slow link.bspachman wrote:
* how does the 'remove deleted files on source' option function?
If your destination is an rsync share, files that no longer exist on the source will be deleted on the destination.bspachman wrote:
* Note: the hours are when it will begin trying to trigger, and when it will stop trying. This is important if you have multiple jobs scheduled. If Job1 continues past the END TIME of Job2, then Job2 will never trigger.
Not necessarily. If a job is triggered while another is running, it will get queued.bspachman wrote:
* I don't understand the utility of changing file ownership on the backup, but if you need it, the option is there...
Let's say you backup a remote share, the permissions on the remote share are kept, and these permissions prevent a NAS user from accessing the contents of the share. You would then need this option.- Thanks for the response! As I learn more, I will update the original post at the top of the thread….
ewok wrote: bspachman wrote:
One says that the source should be a 'remote: rsync server' the other says that the destination should be an 'rsync server'. It's not clear if both methods work, or if one is preferred over the other.
Both will work.
Understood, however, is one more advantageous than the other? Consider one likely case: where a user wants to do a partial backup of a share. Does the SOURCE have to be the rsync entry so that the exclusion feature can be used? What happens if you 'exclude' on an rsync destination?ewok wrote: The exclusion list is passed directly to the rsync command line. Consult the documentation for rsync (not ReadyNAS specific) for details.
Is it sufficient to look at the man page for rsync on my OSX machine? How is it passed? via the "--exclude" option? My documentation says that it only takes one rule/pattern…. If I want to exclude all transport stream and MPEG files from the backup, can I use "*.ts,*.mpg"?ewok wrote: bspachman wrote:
4a) Set up the Source:
* Source as Rsync Server?
* Host: localhost? 127.0.0.1? the name of the ReadyNAS? the IP address of the ReadyNAS? -- all options seem to give me "Error connecting to xxxx"
Any of those should work as long as the path is correct.
It seems that you are right when you actually run the backup job, however, when I click "Test Connection" with any of the above entries in the host field, a pop-up appears telling me that the connection failed.ewok wrote: bspachman wrote:
* how does the 'remove deleted files on source' option function?
If your destination is an rsync share, files that no longer exist on the source will be deleted on the destination.
So does the option apply at all if you are using rsync as a source? If it applies, what happens? For the destination, it would seem to be clearer language to say "remove files already deleted from source" or if it applies in either case, just "Remove deleted files from backup"ewok wrote: bspachman wrote:
* Note: the hours are when it will begin trying to trigger, and when it will stop trying. This is important if you have multiple jobs scheduled. If Job1 continues past the END TIME of Job2, then Job2 will never trigger.
Not necessarily. If a job is triggered while another is running, it will get queued.
So what's the purpose of having an end time for any of the jobs? Something else I had read here on the boards tried to explain the utility of the end times, but I guess I didn't grasp it.
Thanks,
Brad - if your test connection is not working,
double check that rsync is enabled for the shares,
paths and shares are case sensitive,
you don't need rsync authentication but if you enable it make sure its the same on the rsync share page and your backup job (this is separate/independent of your nas users).
if you want the whole share to be rsynced, you do not need to specify a path
you should list the exact details or post a picture of your backup job if your having problems:
source: readynas share: stuff
path: /backmeup
destination: rsync server: 127.0.0.1/usb_hdd_01
path: /stuffbackup
compression: disabled
remove files deleted from source: enabled
this will essentially sync all files and folders in the stuff/backmeup folder to /usb_hdd_01/stuffbackup
any files that were deleted from /stuff/backmeup will also be deleted from /usb_hdd_01/stuffbackup, making the backup and exact duplicate
note: rsync must be enabled for the usb_hdd_01 share in frontview - Is there any way to setup a single rsync backup job that will backup an entire volume? I have my "C" volume with 7 shares on it. I can setup a separate rsync job for each share, but I'd prefer to just setup a single job that will backup the entire "C" volume. However, it won't seem to accept "c" in the path. Any ideas?
- You need to specify volume C as the source and the USB disk as the remote rsync target.
See Rsync backup to USB disk mdgm wrote: You need to specify volume C as the source and the USB disk as the remote rsync target.
See Rsync backup to USB disk
I guess that's what I'm saying is that I don't know how to specify volume C as the source. According to the guides I've read, you specify the share without a leading / as the target. I have found nowhere where it tells how to specify the volume.
If I leave the path blank, I can't save the backup job (even though I can successfully connect via the rsync test).
If I try to specify "c", then I get an error with the rsync test.
Any hints? Do I need to create the job via FrontView, then SSH to the Duo and edit the file?- I'm not sure which version of RAIDiator the Duo runs, but on my Pro, in the Source step (when creating a FrontView backup), the drop down list has "Volume: c" as an option. I believe everything else in the 'Source' step can then remain blank, and the destination can be configured as the rsync job.
If 'volume: c' is not an option in your drop-down menu, one of the gurus will have to weigh in... :)
Hope that helps....
brad bspachman wrote: I'm not sure which version of RAIDiator the Duo runs, but on my Pro, in the Source step (when creating a FrontView backup), the drop down list has "Volume: c" as an option. I believe everything else in the 'Source' step can then remain blank, and the destination can be configured as the rsync job.
If 'volume: c' is not an option in your drop-down menu, one of the gurus will have to weigh in... :)
Hope that helps....
brad
This definitely helps. I was trying to create the backup job with rsync in the source, and the USB drive as the destination. It didn't occur to me to try and switch them. Duh. Thanks, Brad.bspachman wrote:
Understood, however, is one more advantageous than the other? Consider one likely case: where a user wants to do a partial backup of a share. Does the SOURCE have to be the rsync entry so that the exclusion feature can be used? What happens if you 'exclude' on an rsync destination?
It normally doesn't matter either way. I prefer writing to local share (so rsync source) to minimize permissions issues. Exclude works the same way regardless of which side is rsync (excluded files are not transferred to the destination).bspachman wrote:
Is it sufficient to look at the man page for rsync on my OSX machine? How is it passed? via the "--exclude" option? My documentation says that it only takes one rule/pattern…. If I want to exclude all transport stream and MPEG files from the backup, can I use "*.ts,*.mpg"?
The OSX rsync man page should have the info you need. The exclude field is split by comma and each element is passed to an --exclude option.bspachman wrote:
It seems that you are right when you actually run the backup job, however, when I click "Test Connection" with any of the above entries in the host field, a pop-up appears telling me that the connection failed.
It works on my Pro regardless of which of the three formats used.bspachman wrote:
So does the option apply at all if you are using rsync as a source? If it applies, what happens? For the destination, it would seem to be clearer language to say "remove files already deleted from source" or if it applies in either case, just "Remove deleted files from backup"
If this option is enabled on the source, files that exist on the destination but not on the source are deleted. Basically, it keeps the source and destination in sync.bspachman wrote:
So what's the purpose of having an end time for any of the jobs? Something else I had read here on the boards tried to explain the utility of the end times, but I guess I didn't grasp it.
It's not an end time for a job, it's the end time for the schedule. For example, if you set a start time of 1:00, and end time of 3:00, and a backup every hour, backups will run at 1, 2, 3, and then not run again until 1 of the next scheduled day.- Thank you, bspachman!
Great article, exactly what I was looking for :)
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
