Reply

Re: Documentation for Replicate, specifically backup and restoring data

nickjames
Luminary

Documentation for Replicate, specifically backup and restoring data

Hello,

 

I've been using Replicate for about 3 years now and luckily (knock on wood), I'v never had to restore my backup NAS to my primary NAS. That being said, I can't find any documentation on how to restore my data if I had too. I've been following through this PDF, which seems outdated-

 

http://www.downloads.netgear.com/files/GDC/READYNAS-COMMON/ReadyNAS%20Replicate%20UM_10Jul2013.pdf

 

→ Is there any more documentation regarding this specifically?

 

Also, if its full/incremental backups, how does the file structure work on the backup device? There is a new folder each time it runs.

 

Is there something that works better than Replicate? Or easier to understand?

Model: RN51600|ReadyNAS 516 6-Bay
Message 1 of 25

Accepted Solutions
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data

Thanks againg for your input, StephenB and TeknoJnky.

 

I got everything backed up this last week and so far as good.

 

StephenB, I know when I called into Netgear originally to setup Replicate, they suggested NOT to use snapshots on the destination (backupnas) device. Should I also not be creating snapshots on the destination device, when using Rsync?

 

By the way, I'm going with the "pull" method as I like the idea of putting the NAS on a power schedule. That being said, do I just figure how long the backup jobs should take, schedule them within the hours that the NAS is powered on, and that is it? Will it power off automatically if schedule backups are pending or in the middle of finishing one?

 

Here is a picture of my current schedule, which by the way, can you just have them all run at the same time and it will run each one, one-by-one until completion?

 

backup_jobs.JPG

 

Thanks again for your suggestions!

View solution in original post

Message 21 of 25

All Replies
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:

 

 

Is there something that works better than Replicate? Or easier to understand?


What ReadyNAS models, and what firmware are they running?  Is the backup NAS on the same network as the primary NAS?

Message 2 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data


@StephenB wrote:

@nickjames wrote:

 

 

Is there something that works better than Replicate? Or easier to understand?


What ReadyNAS models, and what firmware are they running?  Is the backup NAS on the same network as the primary NAS?


RN516s

6.6.0 and 6.6.1

They are BOTH on the same LAN (broadcast domain)

Message 3 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:


RN516s

6.6.0 and 6.6.1

They are BOTH on the same LAN (broadcast domain)


ReadyDR is one possibility.  It backs up snapshots only, so it is quite efficient - especially for iSCSI luns.

 

The main disadvantage is that you can't immediately cut over to the backup NAS - you need to restore the backup instead.  If that's an issue, then rsync backups are better (and are what I use myself).  You can combine that with custom snapshots on the destination share(s) to give you some retention.

 

I like to use a separate rsync backup for each share (and since I don't use the home folders, I don't bother to back those up).  Set up email notification on failure.  

Message 4 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data


@StephenB wrote:

@nickjames wrote:


RN516s

6.6.0 and 6.6.1

They are BOTH on the same LAN (broadcast domain)


ReadyDR is one possibility.  It backs up snapshots only, so it is quite efficient - especially for iSCSI luns.

 

The main disadvantage is that you can't immediately cut over to the backup NAS - you need to restore the backup instead.  If that's an issue, then rsync backups are better (and are what I use myself).  You can combine that with custom snapshots on the destination share(s) to give you some retention.

 

I like to use a separate rsync backup for each share (and since I don't use the home folders, I don't bother to back those up).  Set up email notification on failure.  


Thanks for the reply, StephenB.

 

I currently do not use ReadyDR. I do not use LUNS either.

 

My main concern is how do I get the backup NAS online, if the main NAS fails? What if I don't have another NAS to restore the backup from Replicate to? Can you elaborate on the RSYNC method outlined? That sounds like the best bet for me as I don't use home folders either.

Message 5 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:

I currently do not use ReadyDR. I do not use LUNS either.

 


ReadyDR is a new backup method.  It copies btrfs snapshots to the destination system, which allows you to restore either the most recent snapshot or an older version.  This is similar to replicate. It's available on business-class OS-6 only (which includes the RN516).  It's not my preferred method, because I'd have to restore the ReadyDR backup to a share in order to use it.  


nickjames wrote: Can you elaborate on the RSYNC method outlined? That sounds like the best bet for me as I don't use home folders either.

I've created one backup job for each share on each backup NAS, and those are scheduled to run daily. They are all rsync, and all have "Remove deleted files on source" checked.  Each destination share has custom snapshots enabled, generally creating daily snapshots with 3 month retention.   Some shares with more churn have shorter retention.  So I can roll back to older backups of any share (up to the retention limit).

 

I run these jobs on the destination ReadyNAS, though it works just as well if you run them on the source ReadyNAS.

 

Since the destinations are all shares (with the same name as the original share), I can directly access the backup NAS if the main NAS fails.

 

 

 

Message 6 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data


@StephenB wrote:

@nickjames wrote:

I currently do not use ReadyDR. I do not use LUNS either.

 


ReadyDR is a new backup method.  It copies btrfs snapshots to the destination system, which allows you to restore either the most recent snapshot or an older version.  This is similar to replicate. It's available on business-class OS-6 only (which includes the RN516).  It's not my preferred method, because I'd have to restore the ReadyDR backup to a share in order to use it.  


nickjames wrote: Can you elaborate on the RSYNC method outlined? That sounds like the best bet for me as I don't use home folders either.

I've created one backup job for each share on each backup NAS, and those are scheduled to run daily. They are all rsync, and all have "Remove deleted files on source" checked.  Each destination share has custom snapshots enabled, generally creating daily snapshots with 3 month retention.   Some shares with more churn have shorter retention.  So I can roll back to older backups of any share (up to the retention limit).

 

I run these jobs on the destination ReadyNAS, though it works just as well if you run them on the source ReadyNAS.

 

Since the destinations are all shares (with the same name as the original share), I can directly access the backup NAS if the main NAS fails.

 

 

 


Thank you, StephenB.

 

The only problem I see with this is the fact that in order for the data to backup each night, it would take upwards of 36 hours, which is going to be before the start of the next business day. With Replicate, it was only backing up the "changed" data. I'll disclose that I dont know how RSYNC works so perhaps this is normal function of RSYNC (only backup changed data). Would that be true? Otherwise, if I run a backup job on an attached USB device, it takes around 36 hours to complete. Running this every single night is not ideal; it would never finish before the start of day.

 

In a perfect world, if I could run a "backup" for all the shares on the 6TB "main NAS" and have backup only include the changed data this would be the same as the Replicate process and normally takes about 2-3 hours each night (which is perfectly OK).

 

How could I get similar function to Replicate, using RSYNC but only the changed data and still have a mirror copy of the main NAS ready to accessed at the start of each day, if need be?

 

Were getting closer! Thanks for your help!

Message 7 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data

Looks like I answered my own questions!!

 

https://en.wikipedia.org/wiki/Rsync

 

I will attempt to set this up and report back-- this is perfect!

Message 8 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:

In a perfect world, if I could run a "backup" for all the shares on the 6TB "main NAS" and have backup only include the changed data this would be the same as the Replicate process and normally takes about 2-3 hours each night (which is perfectly OK).

 


The default backup method is incremental, so only changed files are written after that.

 

Start with one share, then kick off the second backup job when the first one completes, etc.  Then enable the schedules when you have the initial backups all done.

 

It will take a couple days to get all 6 TB copied initially. 

Message 9 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data

Message 10 of 25

Re: Documentation for Replicate, specifically backup and restoring data

interresting topîc!

Message 11 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data

Hi Stephen,

 

I was able to get the "push" and "pull" method setup. I did  a few tests to understand how that checkbox option works when deleting the files on the destination as well. I did have quick question - suggested puling the data whereas the KB article I was using suggested pushing the data as a snapshot is created at the time of push?

 

Here are the KB articles that I ended using:

 

http://kb.netgear.com/29741/How-do-I-back-up-data-between-two-ReadyNAS-OS-6-systems-using-the-backup...

http://kb.netgear.com/29929/ReadyNAS-OS-6-Setting-up-a-backup-job-with-rsync-over-SSH

http://kb.netgear.com/29929/ReadyNAS-OS-6-Setting-up-a-backup-job-with-rsync-over-SSH?cid=wmt_netgea...

 

Is there really a "better" method between the two? Could you give a scenario where you might want one but not the other?

 

Cheers,

Nick

Message 12 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:
http://kb.netgear.com/29929/ReadyNAS-OS-6-Setting-up-a-backup-job-with-rsync-over-SSH?cid=wmt_netgea...

Is this the best KB for this setup, Stephen?

Your article is rsync-over-ssh, which isn't needed since your backup NAS is local.

 

Try this one: http://kb.netgear.com/29741/How-do-I-back-up-data-between-two-ReadyNAS-OS-6-systems-using-the-backup...

 

I use "pull" jobs (running on the destination ReadyNAS).  Here are screen shots of the settings I use.

 

Backup-OS6-1.png

 

Backup-OS6-2.png

Backup-OS6-3.png

Backup-OS6-4.png

Backup-OS6-5.png

Message 13 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:

I was able to get the "push" and "pull" method setup. I did  a few tests to understand how that checkbox option works when deleting the files on the destination as well. I did have quick question - suggested puling the data whereas the KB article I was using suggested pushing the data as a snapshot is created at the time of push?

The "push" backup does create a snapshot on the source share first.  That is an advantage - it ensures that you get a coherent backup even if users are changing files in the share during the backup, or if there is an on-line database that is being updated during the backup.  Another small advantage - if the backup NAS isn't working, the "errors" notification email will kick in, and you'll get notified that the backups failed.

 

 

My backup NAS are on a power schedule though - and with a "push" backup the NAS will do it's normal power scheduled power down.  A "pull" backup will run to completion - the NAS waits for the completion before it shuts down.  So that overrides the advantages of "push" for me.  Since the NAS is for personal use, I can schedule the "pull" backups for off-hours times, when the data isn't likely to be changed.  One backup job on each backup NAS is configured to always send me status, so I do get positive daily email notifications that the backup NAS are functioning.

 

 

If your backup NAS is turned on 24/7, then it's probably better to use "push".   But in practice, "push" works out for me, and I like keeping the backups off when they are not in use.

Message 14 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data


@StephenB wrote:

@nickjames wrote:

I was able to get the "push" and "pull" method setup. I did  a few tests to understand how that checkbox option works when deleting the files on the destination as well. I did have quick question - suggested puling the data whereas the KB article I was using suggested pushing the data as a snapshot is created at the time of push?

The "push" backup does create a snapshot on the source share first.  That is an advantage - it ensures that you get a coherent backup even if users are changing files in the share during the backup, or if there is an on-line database that is being updated during the backup.  Another small advantage - if the backup NAS isn't working, the "errors" notification email will kick in, and you'll get notified that the backups failed.

 

 

My backup NAS are on a power schedule though - and with a "push" backup the NAS will do it's normal power scheduled power down.  A "pull" backup will run to completion - the NAS waits for the completion before it shuts down.  So that overrides the advantages of "push" for me.  Since the NAS is for personal use, I can schedule the "pull" backups for off-hours times, when the data isn't likely to be changed.  One backup job on each backup NAS is configured to always send me status, so I do get positive daily email notifications that the backup NAS are functioning.

 

 

If your backup NAS is turned on 24/7, then it's probably better to use "push".   But in practice, "push" works out for me, and I like keeping the backups off when they are not in use.


Thanks for taking the time to reply with that detailed update!

 

I like what you are saying as I'm in the same boat-- I only ran backups when the business is closed so no-one should be accessing the data. It sounds like a good idea to use the power schedule so the device is not running all the time as well. This will save on the wear and tear too. Genius!

 

Those screen shots are sweet! Hopefully, this thread helps other in the future too!

 

I'll keep you posted on the progress with this project. From the sounds of it, Replicate is not as useful as RSYNC? I'm having a hard time understanding why you would want to use Replicate over RSYNC? RSYNC seems a lot more robust and readily available, ie- not having to restore your Replicate instance in a true "down" or "hardware failure" state

Message 15 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:
I'll keep you posted on the progress with this project. From the sounds of it, Replicate is not as useful as RSYNC? I'm having a hard time understanding why you would want to use Replicate over RSYNC? RSYNC seems a lot more robust and readily available, ie- not having to restore your Replicate instance in a true "down" or "hardware failure" state

 Well, rsync is a better fit for me better than Replicate or ReadyDR.  The other two work more like traditional backup software, and many people do want that model.  

 

ReadyDR has one additional advantage - it's the only practical way to back up iSCSI Luns from the server.  But I don't use those, and I like having direct access to the backed up files.

Message 16 of 25
TeknoJnky
Hero

Re: Documentation for Replicate, specifically backup and restoring data

replicate lets you backup securely *over the internet* without having to worry about vpn or ssh certificates etc.

 

the backup jobs are created and managed via the 'cloud' portal instead of on either nas.

 

replicate essentially uses resync over the netgear private vpn between the nas's

 

as far as restore, it is essentially the same as rsync restore.

 

in a hardware failure,  you would need to setup a working device if you don't already have one (ie create shares/users/etc), then restore the data to the applicable share/folder.

 

ReadyNAS 528X, Pro 6, ProBusiness, Ultra 4, NV+
Message 17 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@TeknoJnky wrote:

 

in a hardware failure,  you would need to setup a working device if you don't already have one (ie create shares/users/etc), then restore the data to the applicable share/folder.

 


Right.

 

But with my use case, I can simply switch to the backup NAS and all the data is immediately available with no need to restore.  Then repair/replace the failed hardware, and reverse the backup jobs to get it running again.  So there's near-zero down time.

 

Of course my use case works out that way because the backup and primary NAS are co-located.  My disaster recovery depends on Crashplan, and there I would need to do a full restore over the internet.

 

 

Message 18 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data

I like where this discussion is going.

 

I would have to disagree though, that Replicate is basically the same thing, because its not, unless I have it setup wrong (which is probably the case).

 

When my data Replicates, I'm left with a folder dated the date of the Replication. I don't have an equal backup of the NAS that it came from either. Honestly, I find this worthless in the event that my main NAS has failed- I can't restore the replicated data as I dont have a 3rd NAS and the replicated data is stored across "changed" replications (folders/shares)? Here is a screen shot of what I'm talking about, notice all the folders.. I'm assuming these are only the "changed" files from each previous backup?

 

replicate.JPG

 

So that said, how do you get Replicate to "mirror" my main NAS onto the backup NAS? I find that rysnc will hopefully do a better job of this. Also, in this case, my devices are located locally, on the same physical network. The original plan was to deploy the backup device off site, but the data was never "usable" as its spanned across multiple Replications/folders.

 

Now, my thought is if the main NAS were to fail now while using the rsync backup method, all I have to do is power it off and change the IP address of the backup device to match the main device, and users would not be affected or even know anything happened. With Replicate, this is not possible.

 

→ Why would I have to do a rsync restore, if the backup device is essentially set to mirror the main NAS via rsync?

Message 19 of 25
TeknoJnky
Hero

Re: Documentation for Replicate, specifically backup and restoring data

Well you are correct, in that replicate is not a 'mirroring' solution. And its not designed so that the backup device can simply stand in for the original source device.

 

If I remember correctly, the backup destination is divided into snapshots, depending on how you have the backup job configured.

 

I think you can set the backup job to not use any snapshots, and it should perform more closely to how a traditional rsync.

 

Otherwise, again if I remember correctly, during a replicate job, the source is snapshotted, the destination is snapshotted, and the difference is rsynced into a new destination subfolder.

 

If you did not already know, that replicate was designed back on raidator 4.x, so we did not the btrfs snapshot capabilities we have now.

 

I am not sure if replicate is orphaned internally or not, but I know it hasn't been migrated to the mynetgear account system.

 

If it had some more development time, it could probably be updated and streamlined and made a little more user friendly.

 

ReadyNAS 528X, Pro 6, ProBusiness, Ultra 4, NV+
Message 20 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data

Thanks againg for your input, StephenB and TeknoJnky.

 

I got everything backed up this last week and so far as good.

 

StephenB, I know when I called into Netgear originally to setup Replicate, they suggested NOT to use snapshots on the destination (backupnas) device. Should I also not be creating snapshots on the destination device, when using Rsync?

 

By the way, I'm going with the "pull" method as I like the idea of putting the NAS on a power schedule. That being said, do I just figure how long the backup jobs should take, schedule them within the hours that the NAS is powered on, and that is it? Will it power off automatically if schedule backups are pending or in the middle of finishing one?

 

Here is a picture of my current schedule, which by the way, can you just have them all run at the same time and it will run each one, one-by-one until completion?

 

backup_jobs.JPG

 

Thanks again for your suggestions!

Message 21 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:

StephenB, I know when I called into Netgear originally to setup Replicate, they suggested NOT to use snapshots on the destination (backupnas) device. Should I also not be creating snapshots on the destination device, when using Rsync?

 

 


Snapshots work well on the destination shares for rsync backups, and I've used them for some years with mine.  I do use custom snapshots (not the default "smart snapshots").  I check the "only take snapshots with changes" box.  Keep on eye on the total space used for snapshots, and adjust the retention to keep it reasonable.  3 months works for me generally.  I have one share that holds PC image backups (very large fies), and I've set retention there to 2 weeks.

Message 22 of 25
nickjames
Luminary

Re: Documentation for Replicate, specifically backup and restoring data

Awesome, thanks for your input. I'll setup snapshots as well then as we have the free space.

 

Any thoughts on the other part of my question?

 

By the way, I'm going with the "pull" method as I like the idea of putting the NAS on a power schedule. That being said, do I just figure how long the backup jobs should take, schedule them within the hours that the NAS is powered on, and that is it? Will it power off automatically if schedule backups are pending or in the middle of finishing one?

 

Here is a picture of my current schedule, which by the way, can you just have them all run at the same time and it will run each one, one-by-one until completion?

Message 23 of 25
crazy_toy
NETGEAR Expert

Re: Documentation for Replicate, specifically backup and restoring data

Message 24 of 25
StephenB
Guru

Re: Documentation for Replicate, specifically backup and restoring data


@nickjames wrote:

That being said, do I just figure how long the backup jobs should take, schedule them within the hours that the NAS is powered on, and that is it? Will it power off automatically if schedule backups are pending or in the middle of finishing one?

 

Here is a picture of my current schedule, which by the way, can you just have them all run at the same time and it will run each one, one-by-one until completion?


The NAS will delay its power down until the backups are complete.  So if you start the backups at 12:05 am you can turn the power down at 01:00. .It will also do this for some maintenance functions (scrub for sure).   I power up the NAS at 11:00, just to make sure I don't miss snapshot creation/deletion and other such tasks

 

You can schedule all the backups to start at the same time, the backup job manager will queue them.

Message 25 of 25
Top Contributors
Discussion stats
  • 24 replies
  • 5234 views
  • 9 kudos
  • 5 in conversation
Announcements