Backup ReadyNas to ReadyNas with rsync and exclude lists

I'd like to implement this on my ReadyNAS, but /frontview/bin/backup on mine is a compiled file:

readynas:/etc# file /frontview/bin/backup
/frontview/bin/backup: perl script text executable

My NAS is a ReadyNAS Pro Business Edition with RAIDiator 4.2.4

Any suggestions where I should look for the raw perl code?

Thanks!

dmcguire62 wrote:
I'd like to implement this on my ReadyNAS, but /frontview/bin/backup on mine is a compiled file: readynas:/etc# file /frontview/bin/backup /frontview/bin/backup: perl script text executable My NAS is a ReadyNAS Pro Business Edition with RAIDiator 4.2.4 Any suggestions where I should look for the raw perl code?

It's executable, yes, but that doesn't mean to say that it's compiled code. Just edit the file and you'll see the Perl code. It's interpreted at run-time.

Thank you so much for this code update. I was looking into it and considering using it as a roadmap - but I kept thinking the lack of the settings being visible to the UI would be problematic to someone other than me looking at the interface. I then looked at the config files and considered that maybe there is even a slightly better way to do this vs having a separate config file.

The /etc/cron.d/frontview-backup file for rsync contains lines that look like...

5 19-19/24 * * sun,mon,tue,wed,thu,fri,sat root /frontview/bin/backup 001 enable 'remote!!rsync!!192.168.1.10::folder!!!!!!' 'local!!backup!!serverx-backup/folder!!!!!!' 'FIRST_TIME!!0!!1!!1!!0' &> /dev/null

Ultimately the "192.168.1.10::folder" variable gets put into $backup_source_path which is then used on 1 of 2 lines similar to...

${password_param} $nice rsync -aA${rsync_option} '${login_param}$backup_source_path/.' '$dest_path' &> \${TEMP_COPYLOG}

It seems it would be possible to use the web-ui "Path" field to quietly embed the command line options. e.g. "folder --exclude blah --delete" - these would make it into the backup file as "!!192.168.1.10::folder%20--exclude%20blah%20--delete!!"

Someone more familiar with perl string parsing could then split the path at the first "--" and auto append the options to the rsync options and remove it from the path...(excuse the pseudo code - I know zero perl)...

if $backup_source_path contains "--" then
$rsync_option += [ everything in $backup_source_path starting after the first -- ]
$backup_source_path = [ everything to the left of the first -- in $backup_source_path ]
end if

The same could be done with the dest path as well. So without requiring separate config files you could override options and see the overrides in the UI. Obviously there are some issues with this - possibly some special characters that can't be saved using the UI (e.g. I would have suggested a "folder[--exclude blah --delete]" syntax except that the [ ( { characters are not allowed by the UI and I am assumming that the path doesn't have a "--" in it.

It could also be extended to have something like "folder --exclude blah --delete_" where the the "_" at the end of the parameters signifies that rather than $rsync_option += it should be a complete override of the options.

I would happily do this myself and post it to the site if I was even remotely competent in Perl - I am hoping that this model makes sense to someone who knows perl and they may want to take a swing at it.

gdietz wrote:
Thank you so much for this code update. I was looking into it and considering using it as a roadmap - but I kept thinking the lack of the settings being visible to the UI would be problematic to someone other than me looking at the interface. I then looked at the config files and considered that maybe there is even a slightly better way to do this vs having a separate config file.

I decided to use a separate config file precisely because it was an uninstrusive approach. I needed to patch only one file and no changes were required to the FrontView UI.

As you point out, the integration could be better, using the UI instead of ssh, but the changes to the code would be greater and more difficult to maintain across firmware releases. As it stands, the same patch has worked for me across several firmware upgrades.

Maintaining the extra parameters also has the advantage that the file can be managed on a separate system and rsync'ed into place after changes are made.

I hope to eventually see the feature in a future firmware release.

I appreciate your desire to keep patching to a minimum, although the solution I suggested doesn't change in any substantial way from yours. Both involve patching a single file /frontview/bin/backup in the exact same place in the code. (see my solution below).

The only difference in your solution and the one that I just tried is who you determine what rsync options to append. Your solution uses the $backup_source/dest_path variable to lookup up in a separate config file. Mine uses the variables themselves to see if they have options embedded in them and if so parses them out.

This solution allows you to enter the options directly into the UI (with no changes to the UI code) - just put all your options at the end of the path field using <sp>--optionx -optiony valuey.... The backup code will detect the " --" in the path and remove them - placing them into the options variable.

The advantage of this is that you can modify and see the options in the UI viewer, you can disable SSH access after patching the backup script, and you don't need to maintain a separate config file.

I suspect it is just a matter of personal preference and as you say I hope that netgear adds advanced options into the UI. Thank you again for doing the heavy lifting on figuring out where to do the changes.

-- backup.orig
++ backup
if( $backup_option_rsync_remove_missing_source_files )
{
$rsync_option .= " --delete";
}

if( $backup_option_rsync_use_compression )
{
$rsync_option .= " --compress";
}

+if ($backup_source_share_proto eq 'rsync' && $backup_source_path =~ /^.*?\s--/)
+{
+ @parts = split(/\s--/, $backup_source_path);
+ $backup_source_path = shift(@parts);
+ foreach $option (@parts) { $rsync_option .= " --$option" }
+}
+if ($backup_dest_share_proto eq 'rsync' && $backup_dest_path =~ /^.*?\s--/)
+{
+ @parts = split(/\s--/, $backup_dest_path);
+ $backup_dest_path = shift(@parts);
+ foreach $option (@parts) { $rsync_option .= " --$option" }
+}

gdietz wrote:

The only difference in your solution and the one that I just tried is who you determine what rsync options to append. Your solution uses the $backup_source/dest_path variable to lookup up in a separate config file. Mine uses the variables themselves to see if they have options embedded in them and if so parses them out. This solution allows you to enter the options directly into the UI (with no changes to the UI code) - just put all your options at the end of the path field using <sp>--optionx -optiony valuey.... The backup code will detect the " --" in the path and remove them - placing them into the options variable.

As you point out, though, this means you can't have '--' in your paths or option lists. There may be other side effects, too.

That wouldn't be a big deal for most people, but may cause unexpected results in an isolated number of cases. The more solutions, the better, though.

I have a new NVX unit here and /frontview/bin/backup is a Perl executable alright, but it appears to have been compiled with Perl::ByteLoader. So editing it is no longer possible, at least not with RAIDiator 4.2.5. Interestingly /frontview/bin/backup is the only file in that folder compiled that way. What gives?

Cheers... Mike

faceless wrote:
I have a new NVX unit here and /frontview/bin/backup is a Perl executable alright, but it appears to have been compiled with Perl::ByteLoader. So editing it is no longer possible, at least not with RAIDiator 4.2.5. Interestingly /frontview/bin/backup is the only file in that folder compiled that way. What gives?

That's interesting. I hope they haven't done that in the new 4.1.6, too, because that would prevent me from upgrading when I get back from holiday.

Perhaps they did this to protect proprietary information, but I'm just guessing. It would be a step backwards if we can no longer hack that file.

ianmacd wrote:

faceless wrote: I have a new NVX unit here and /frontview/bin/backup is a Perl executable alright, but it appears to have been compiled with Perl::ByteLoader. So editing it is no longer possible, at least not with RAIDiator 4.2.5. Interestingly /frontview/bin/backup is the only file in that folder compiled that way. What gives? That's interesting. I hope they haven't done that in the new 4.1.6, too, because that would prevent me from upgrading when I get back from holiday. Perhaps they did this to protect proprietary information, but I'm just guessing. It would be a step backwards if we can no longer hack that file.

I've just added a ReadyNAS Pro alongside my NV+ and can confirm that backup is now a byte-compiled file. Bummer.

The good news is that RAIDiator 4.2.8 (and maybe a couple of earlier revisions) adds built-in support for excluding directories from rsync back-ups. No more need to hack that in.

The bad news is that the new support isn't a generic solution, so there's still no way to add arbitrary options to the rsync command line. In particular, I still need --size-only.

ianmacd wrote:

gdietz wrote: Thank you so much for this code update. I was looking into it and considering using it as a roadmap - but I kept thinking the lack of the settings being visible to the UI would be problematic to someone other than me looking at the interface. I then looked at the config files and considered that maybe there is even a slightly better way to do this vs having a separate config file. I decided to use a separate config file precisely because it was an uninstrusive approach. I needed to patch only one file and no changes were required to the FrontView UI. As you point out, the integration could be better, using the UI instead of ssh, but the changes to the code would be greater and more difficult to maintain across firmware releases. As it stands, the same patch has worked for me across several firmware upgrades. Maintaining the extra parameters also has the advantage that the file can be managed on a separate system and rsync'ed into place after changes are made. I hope to eventually see the feature in a future firmware release.

Thanks for your posts. Really helpful hacks.