Backup ReadyNas to ReadyNas with rsync and exclude lists

I would also like to know if this is possible too. Thank you, Tom

You would have to ssh in and make the change manually.

ewok wrote:
You would have to ssh in and make the change manually.

I, too, need the ability to pass an arbitrary number of extra flags and parameters to rsync when it performs back-ups.

For example, I need to back up a chroot'ed BIND installation that features a /proc directory within the chroot'ed environment. If I don't use '--exclude chroot/proc' when backing this up, madness ensues and the job fails.

FrontView is inadequate for this. There really ought to be a text field in which one can add the necessary extra options for insertion in the rsync command line that is ultimately invoked.

Lacking this, I finally succumbed to the urge to install the EnableRootSSH patch and go poking around in search of something I could easily hack to get the job done.

Well, I found it. /frontview/bin/backup is the Perl script that handles the back-ups. It's invoked by each of the scheduled back-up jobs, which are listed in /etc/cron.d/frontview-backup.

Happily, the extra functionality can be added to /frontview/bin/backup in about 15 lines of Perl. Here's how I chose to implement it: The patch below works on RAIDiator 4.1.3 (and possibly others).

--- backup.orig	2008-09-25 15:56:13.000000000 +0200
+++ backup	2008-09-25 22:19:35.000000000 +0200
@@ -244,6 +244,22 @@
   $rsync_option .= " --delete";
 }
 
+my $rsync_option_file = '/frontview/conf/rsync.conf';
+if( $backup_source_share_proto eq 'rsync' && -f $rsync_option_file )
+{
+  open(RSYNC, $rsync_option_file);
+  my @rsync = <RSYNC>;
+  close(RSYNC);
+
+  my $row = (grep(/$backup_source_path/, @rsync))[0];
+  if( $row )
+  {
+    my ($extra_options) = (split $FS, $row)[1];
+    chomp($extra_options);
+    $rsync_option .= " $extra_options";
+  }
+}
+
 my $mount_point = "/job_${job}";
 my $login_param;
 my $source_path = "/$backup_source_share_proto/$backup_source_path";

I wanted to avoid having to repeatedly hack system files when a back-up job with extra rsync options needs to be modified, so I chose to have the new functionality use its own configuration file, /frontview/conf/rsync.conf.

That file should contain one line of additional rsync configuration per back-up job. Each line should contain two fields, separated by the standard FrontView field separator, two exclamation marks (!!).

The first field is the back-up source, which uniquely identifies the back-up job by host::module/path.

The second field consists of the options that should be inserted in the rsync command line before it is invoked.

In my case, /frontview/conf/rsync.conf looks like this:

foo.caliban.org::slash/var/named!!--exclude chroot/proc --delete-excluded

Now, when an rsync back-up job is run that has foo.caliban.org::slash/var/named as its source, '--exclude chroot/proc --delete-excluded' is added to the rsync command line.

The nice thing about this approach is that /frontview/conf/rsync.conf can be maintained and edited on some other machine and ssh'ed into place on the ReadyNAS. There's no need to work directly on the ReadyNAS, which is important to me. It's an appliance and I want to treat it as such. Logging into it and performing this hack was the last resort for me.

Some would say that, if you need this level of control, you should be configuring the back-up job on the client and pushing your data to the ReadyNAS. However, I need to back-up multiple machines and I want all of my back-up jobs centralised on the ReadyNAS. This is an entirely reasonable approach to managing back-ups.

Obviously, I would like to see this functionality added to FrontView. I don't like the fact that I've potentially endangered my right to tech support by installing EnableRootSSH. I also don't like poking around on an appliance that wasn't really intended to be manipulated at this level. I consider it a dangerous activity.

Anyway, this works for me. If you know what you're doing, it will work for you, too.

Here's an improved version of my patch. Previously, the extra rsync options would only be used if the back-up source was an rsync server. The new version will also use the options when the back-up destination is an rsync server.

Let's suppose your /frontview/conf/rsync.conf file looks like this:

foo.caliban.org::slash/var/named!!--exclude chroot/proc --delete-excluded
127.0.0.1::USB_HDD_1/backup!!--size-only

The first line is the configuration for the job described in my last posting.

The second line shows a new job, that backs up a local ReadyNAS share to a USB-attached hard drive. The job could also be configured as a simple share back-up, but that wouldn't allow files that have since been removed from the source to also be removed from the destination. Only rsync jobs allow this (the --delete flag), so we have to use that, even though both source and destination are actually local.

The --size-only flag is often needed when the source and destination file-systems aren't the same time. If copying from Linux ext3 to Windows FAT, for example, there will be discrepancies in the timestamps that cause all files to be copied from the source to the destination, defeating the main purpose of rsync. --size-only tells rsync to use only the size of the file to determine whether or not the file has changed on the source.

--- backup.orig 2008-09-25 15:56:13.000000000 +0200
+++ backup      2008-10-04 00:43:28.000000000 +0200
@@ -244,6 +244,29 @@
   $rsync_option .= " --delete";
 }
 
+my $rsync_option_file = '/frontview/conf/rsync.conf';
+if( -f $rsync_option_file && ( $backup_source_share_proto eq 'rsync' ||
+                              $backup_dest_share_proto eq 'rsync' ) )
+{
+  open(RSYNC, $rsync_option_file);
+  my @rsync = <RSYNC>;
+  close(RSYNC);
+
+  my $row = (grep(/$backup_source_path/, @rsync))[0];
+
+  unless( $row )
+  {
+    $row = (grep(/$backup_dest_path/, @rsync))[0];
+  }
+
+  if( $row )
+  {
+    my ($extra_options) = (split $FS, $row)[1];
+    chomp($extra_options);
+    $rsync_option .= " $extra_options";
+  }
+}
+
 my $mount_point = "/job_${job}";
 my $login_param;
 my $source_path = "/$backup_source_share_proto/$backup_source_path";

Ian-

What's the prospect for adding a service to the current array of default services on these readynas devices using the basic hacking method that you've outlined above...?

Specifically, I'm thinking of how we could add the tivodesktop server so that our LANs could utilize the readynas to serve up photos and music to tivo boxes in our home, which is currently being done by a pc. The readynas is already storing / backing up all of our music and photo library, so why not take the next step?

Thanks for your thoughts on this....

StGeorge wrote:
What's the prospect for adding a service to the current array of default services on these readynas devices using the basic hacking method that you've outlined above...?

You'd probably want to use the add-on mechanism for this rather than hacking files on the NAS.

Ian,

Thank you so much for posting this.

I am in a similar boat: I totally agree with you about the purpose of appliances (and the need to not void rights to tech support as well), but i need to get my ReadyNAS to run rsync with parameters that I can't control through FrontView.

Your solution is about as uninvasive as it gets so I am preparing to ride on your coat tails (thanks again for bothering to post the solution). I just wanted to see if you had run into any problems since implementing this before I jump in.

Also, I actually need to get FrontView to STOP adding the "-l" paramter because the rsync server i'm backing up to (from the ReadyNAS) is ibackup and their server rejects connections with this parameter (a documented feature). I'm not familiar with perl at the moment so it's not clear to me if this will be doable/easy.

Lastly, to the tech support folks: is there an effort underway to build an add-on for this? If so I might wait for that as the cleanest way to get the changes.

Thanks in advance for any feedback you can offer.

Jared

jleinbach wrote:

Ian, Thank you so much for posting this. I am in a similar boat: I totally agree with you about the purpose of appliances (and the need to not void rights to tech support as well), but i need to get my ReadyNAS to run rsync with parameters that I can't control through FrontView. Your solution is about as uninvasive as it gets so I am preparing to ride on your coat tails (thanks again for bothering to post the solution). I just wanted to see if you had run into any problems since implementing this before I jump in. Also, I actually need to get FrontView to STOP adding the "-l" paramter because the rsync server i'm backing up to (from the ReadyNAS) is ibackup and their server rejects connections with this parameter (a documented feature). I'm not familiar with perl at the moment so it's not clear to me if this will be doable/easy. Jared

Hello Jared,

Thanks for your kind words. Yes, the lack of rsync control is frustrating. Flags like --exclude are absolutely essential to me.

To remove the -l option from the back-up jobs, just do the following. This solution assumes RAIDiator 4.1.4, but should be similar or identical for other versions.

Go to line 68 of /frontview/bin/backup, where you should find the following:

my $rsync_option = "v --links ";

Change this to:

my $rsync_option = "v ";

Of course, this will affect all of your back-up jobs. If this is unacceptable, you'll need to add code to change the options on a per job basis. That's trickier.

And yes, my hack is still working well for me months later. The only issue with it is that installing a new version of the firmware blows it away.

Ian, Thanks so much. That's perfect. I never need the --links switch so that fix will be do the trick. I feel safe making an alteration as innocuous as that. I will actually need the exclude too ultimately and will probably use your approach to have a seperate file for the additional paramter(s), but for now just getting things going with removal of the --links switch will do the trick. It does seem (given how it is all implemented) that adding a text box to FrontView to allow for the manual setting of switches in rsync would be a trivial enhancement for them to implement. Maybe we'll see it in the next release. Thanks again.

jleinbach wrote:
It does seem (given how it is all implemented) that adding a text box to FrontView to allow for the manual setting of switches in rsync would be a trivial enhancement for them to implement.

It's on our todo list, along with hundreds of other things. :D