I have performed a factory reset my ReadyNAS Pro Pioneer (RNPP) unit. and now want to ensure the restore of the original data from an external USB drive (connected to the NAS' USB port) is copied without any data corruption. I am trying to confirm color=#0000FF]whether RSYNC is a good tool to use to verify the copied data's integrity[/color].I have ready many posts that suggest using RSYNC to verify a backup. One such example:viewtopic.php?f=66&t=51339&p=292769mdgm wrote:...After doing the backup to the USB disk you can verify it using a Rsync backup to USB diskMy questions are:'1) After the initial restore to the RNPP unit (using FrontView's backup job, Source=Share-USBDrive; Target=Share-RNPP Share Drive), I set the backup type to RSYNC.Do I then perform a FULL backup again?2) The expectation is nothing changes during this first RSYNC backup. Question - so, how does RSYNC actually verify the backup? Does it to a byte-for-byte comparison or just a date/time comparison?

You do an incremental backup. It compares the files on the source and destination. If the files are different the changes will be transferred across.Edit: not comparison of checksums, see below.

mdgm wrote:You do an incremental backup. It compares the files on the source and destination. If the files are different (I.e. checksum doesn't match) the changes will be transferred across.First, THANKS SO MUCH for the fast response!!!So, you are saying the (first) RSYNC backup (after the initial full backup/transfer of data) does a checksum verification on each file (source versus target) to determine if two same-named files are different? I am not clear on this because several posts on this forum state that RSYNC uses timestamp/filesize to determine if a file (between source and target) has changed, then if a difference is detected, RSYNC does the delta calculation to determine what delta gets backed up. Here is one such post:viewtopic.php?f=31&t=27318&p=152960ewok wrote:bru wrote:What I want to happen is for rsync to compare dates and checksums of the file andto not backup files that have not changed so that the network does not get hit veryhard and the disks to not get too much usage.That's what rsync does, using timestamp and filesize to check if a file needs to be transferred. Using checksums would be very slow....

Oops. I guess I was wrong then. Using file size and timestamp should be sufficient because one of those is slightly off if the file wasn't copied successfully.

mdgm wrote:Oops. I guess I was wrong then. Using file size and timestamp should be sufficient because one of those is slightly off if the file wasn't copied successfully.LOL, are you sure you are wrong? Again, I have seen BOTH sides of the story. OK, let me ask this. IS there a definitive way to do the checksums verification natively on the NAS? I mean, I can use/run FreeFileSync (http://freefilesync.com/) from my network-connected PC to compare the USB version of the file versus that on the RNPP, but that takes ALONG time because of the time it takes to pull data across my network (100 not 1000Mbps) and my PC is OLD (~4+ yrs laptop), so I prefer not to go that route.

Just had a look and there is an option for rsync to compare (to decide which files have changed) by checksum. From reading your quote of ewok's it sounds like this option is not used by Frontview backup jobs.If you have SSH access you could run rsync manually from the command line with the relevant option specified.

How does RSYNC "verify" integrity of a backup???

mdgm wrote:
... After doing the backup to the USB disk you can verify it using a Rsync backup to USB disk

23 Replies

Replies have been turned off for this discussion

winger

Tutor

Sep 23, 2011

ewok wrote:
Rsync, by default, will use the timestamp and filesize to determine if a file has changed. The Backup Manager has no facility to use rsync checksums, but feel free to create a post in the Feature Requests forum.

From the rsync man page:
Rsync  finds  files  that  need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed  in  size
or   in  last-modified  time.

ewok wrote:
Rsync, by default, will use the timestamp and filesize to determine if a file has changed. The Backup Manager has no facility to use rsync checksums, but feel free to create a post in the Feature Requests forum. From the rsync man page: Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.

Thanks for responding, Ewok.

Do you know the answer to my question above (when using the FrontView RSync), which is:

.... Will even the slightest corruption result in a file size difference that can/will be detected by the Rsync process?

For your suggestion on placing a post in the requested features section, will do.

ewok

NETGEAR Expert

Sep 23, 2011

winger wrote:

Do you know the answer to my question above (when using the FrontView RSync), which is:
.... Will even the slightest corruption result in a file size difference that can/will be detected by the Rsync process?

The answer is no. Consider, for example, the transfer of a text file that contains "hello world". If the file gets corrupted such that the file now contains "hallo wurld", the filesize and timestamp would still be the same. The best you could do without opening up the ReadyNAS would probably be to run checksumming software from your PC, but that would involve a lot of network traffic. An interesting idea for an addon would be one that does an md5sum on every file in the data volume and makes the output available.

TeknoJnky
Hero
Sep 23, 2011
or run rsync with -c and then it will compare checksums of both source and destination to determine whether to copy the file/delta data.

:)
ewok
NETGEAR Expert
Sep 23, 2011
Of course, that would be a good option, but you'd have to open up the NAS. :)

winger

Tutor

Sep 23, 2011

ewok wrote:
winger wrote:

Do you know the answer to my question above (when using the FrontView RSync), which is:
.... Will even the slightest corruption result in a file size difference that can/will be detected by the Rsync process?

The answer is no. Consider, for example, the transfer of a text file that contains "hello world". If the file gets corrupted such that the file now contains "hallo wurld", the filesize and timestamp would still be the same. The best you could do without opening up the ReadyNAS would probably be to run checksumming software from your PC, but that would involve a lot of network traffic. An interesting idea for an addon would be one that does an md5sum on every file in the data volume and makes the output available.

I wonder what checksum freefilesync uses. I just completed freefilesync checks on over 800+ GB of data (this took a few days!), and only two files were found to be different. I simply re-copied (teracopy) those two back files to the NAS and used freefilesync to re-check.

winger

Tutor

Sep 23, 2011

ewok wrote:

TeknoJnky wrote:
or run rsync with -c and then it will compare checksums of both source and destination to determine whether to copy the file/delta data.

:)

Of course, that would be a good option, but you'd have to open up the NAS. :)

Guys, so I have SSH enabled now. Is this Rsync with -c option something I can run directly from the NAS using Putty (comparing NAS data versus data on a USB-connected HardDrive) ? If yes:

1) what are the exact commands to run for my setup? Here is one set of folders I would want to compare:
NAS:
Share name = Data01

USB Harddrive:
NAS volume name: USB_HDD_3
Folder: /backups/Data01

2) would running this -c Rsync process produce an output of differences ? or will the Rsync process just copy (from the USB to the NAS) when differences are detected?

ewok
NETGEAR Expert
Sep 23, 2011
TeknoJnky wrote:
or run rsync with -c and then it will compare checksums of both source and destination to determine whether to copy the file/delta data.

:)

One problem I see with this is that if rsync detects differences due to corruption, it really doesn't know on which side the corruption exists. For example, if the corruption exists on the source side, then it will just copy the corrupted file over to the destination.
TeknoJnky
Hero
Sep 23, 2011
yes, the source always takes precedence, but that would be the same regardless if backed up via -c or by timestamp/size.
TeknoJnky
Hero
Sep 24, 2011
I have no way to verify if it did or didn't work, but putting --checksum in the rsync exclusion fields didn't cause any errors and in theory would be passed along to the backup job command line

TeknoJnky wrote:
or run rsync with -c and then it will compare checksums of both source and destination to determine whether to copy the file/delta data. :)

winger

Tutor

Sep 24, 2011

TeknoJnky wrote:
I have no way to verify if it did or didn't work, but putting --checksum in the rsync exclusion fields didn't cause any errors and in theory would be passed along to the backup job command line

TeknoJnky wrote:
I have no way to verify if it did or didn't work, but putting --checksum in the rsync exclusion fields didn't cause any errors and in theory would be passed along to the backup job command line

Maybe I can test this. The question is, how can I get two files, maybe different only by a bit or two AND also have same date/timestamp last updated?

So, that "--checksum", where is the "-c" ? (btw, I am not a programmer, by any means.)

Forum Discussion

How does RSYNC "verify" integrity of a backup???

23 Replies

Related Content

RBR50: Orbi app returns "Unknown error." verifying security account.

Dropbox integration

Backup / Restore

R7800 error during firmware update "MD5 verify error."

ReadyNAS backup to USB EXT4 - How to check USB disk integrity

NETGEAR Academy

ProSupport for Business