Re: Duo v1 / Radiator 4.1.9 always takes "FULL" backup!

jvanhambelgium · ‎2012-08-29

Hi,

I took my old Duo v1 from the shelves and wanted to use as a backup system for my custom made NAS.
It's running 2 x 1TB disks in X-RAID setup on Radiator 4.1.9
I'm using a local "backup" shared on the Duo and the remote target is a NFS-mount*

* I added some parameters to the /frontview/backup file so my NFS-mount is now much faster than the "stock" (10mbytes/sec <-> 4.8mbytes/sec) also I tuned the TCP/IP considerably (eg. sysctl values etc)

Now everything works quite well, I've created 5 jobs that are configured to do a "FULL" only the first time.
They sure did, but now I notice they are doing everything all over again ... and offcourse not a file changed on my NAS in that area !

When logging into the Duo, looking at the mount it made ;
172.29.45.253://DATAVAULT/FOTO on /job_001 type nfs (ro,noatime,soft,intr,rsize=32768,wsize=32768,udp,addr=172.29.45.253)

Then you see the process running that actually is making the backup ;
root 3976 3911 74 20:05 ? 00:44:19 cp -vua /job_001/. /backup/FOTO

So it uses a basic copy, with the parameter to indeed only copy files if the source is NEWER then the target.Clocks on both system are NTP-synced and identical.

Also, looking at the proces-list I see my backup-jobs ;
root 3890 3888 0 20:05 ? 00:00:00 /bin/sh -c /frontview/bin/backup 001 enable 'remote!!nfs!!172%2E29%2E45%2E253%3A%2F%2FDATAVAULT%2FFOTO!!!!!!' 'local!!backup!!FOTO!!!!!!' 'FIRST_TIME!!0!!0!!0!!1
root 3894 3890 0 20:05 ? 00:00:04 /usr/bin/perl /frontview/bin/backup 001 enable remote!!nfs!!172%2E29%2E45%2E253%3A%2F%2FDATAVAULT%2FFOTO!!!!!! local!!backup!!FOTO!!!!!! FIRST_TIME!!0!!0!!0!!1!!

...why is there the string "FIRST TIME" in there ? It sure was not the first time it ran...

So, anyone of development that can shine some light on this ? It would be appreciated, since I have some 500GB of usefull data to copy and each time a FULL is really not workable :twisted:

Also, I tried the RSYNC approach but I did not like the performance (50% slower) and I just expect this simple method to work!

From what I read there seems a lot problems with this ?

Thanks!

JohnnyB11 · ‎2012-08-29

Please check whether your custom made NAS and your Duo v1 use the same file system type. Different file systems have different time stamp resolutions. If the target file system (i.e., the Duo v1) offers lesser granularity, the cp command will almost always thinks that files on the source file system are newer. See, e.g., viewtopic.php?f=31&t=65736

jvanhambelgium · ‎2012-08-29

JohnnyB wrote:
Please check whether your custom made NAS and your Duo v1 use the same file system type. Different file systems have different time stamp resolutions. If the target file system (i.e., the Duo v1) offers lesser granularity, the cp command will almost always thinks that files on the source file system are newer. See, e.g., viewtopic.php?f=31&t=65736

Yes, I've ready about that.
My Duo offcourse has the EXT3 filesystem

...my other NAS is running...crap...ext4 !

So even the latest & greatest Radiator 4.19 suffers from this known bug ? Is there any forecast on a fix ?
In the meantime, I'll try to make my Duo a EXT4 system ... see what I can do there because I won't downgrade my other NAS from EXT4 -> EXT3

Thanks! for the info

jvanhambelgium · ‎2012-08-29

Oh man, this ain't gonna work...

Since ext4 is only in the Linux kernel as of 2.6.28 and the Duo v1 is running 2.6.17.14 (Sparc)

So I'll probably have to live with it and just perform the "FULL" backup sequence by pushing the backup-button on the front :evil: :evil:

JohnnyB11 · ‎2012-08-29

You could use rsync: even if it is 50% slower, the incremental backups should be faster than always performing a full backup via NFS.

jvanhambelgium · ‎2012-08-29

JohnnyB wrote:
You could use rsync: even if it is 50% slower, the incremental backups should be faster than always performing a full backup via NFS.

Yes, I'm thinking about that now since it seems quite not possible to get EXT4 onto the Duo v1/SPARC platform without some real voodoo...
I've already setup the RSYNC-server on my custom NAS and everything works since I ran some tests to compare performance.
I wonder if it will accept all data that lives already at /c/backup/folderX etc or will the RSYNC start from scratch, even if I change my existing 5 backup jobs from "NFS" to "RSYNC" as source ...

Thanks!

mdgm-ntgr · ‎2012-08-29

If you change the properties of your existing backup jobs it should just copy the changes across.

jvanhambelgium · ‎2012-08-30

Knowing that the Duo v1 has not so much CPU power (compared to my other custom NAS which runs Core2Duo) I'm going make some modifications to the backup-script in order to see if reverting to MD4 hashing (versus MD5) will speed things up.
A large part of RSYNC is really hash-calculations so forcing the "older" MD4 calculation might really speed things up noticeable on the little Sparc Duo v1 😉

I'll post some info if there really is a usefull increase in performance. Might be interesting for others.

JohnnyB11 · ‎2012-08-30

Without the "-c" option, rsync skips files based on mod-time and size. While this might yield to the same problems as with "cp -u", rsync offers the "--modify-window" option that allows you to add some fuzziness regarding time stamps. E.g., the man page recommends "--modify-window=1" for FAT destinations that has only a 2 second resolution. This might not be a perfect solution but might help you to prevent the CPU intensive check sum calculations on the little Sparc Duo v1.

jvanhambelgium · ‎2012-08-30

JohnnyB wrote:
Without the "-c" option, rsync skips files based on mod-time and size. While this might yield to the same problems as with "cp -u", rsync offers the "--modify-window" option that allows you to add some fuzziness regarding time stamps. E.g., the man page recommends "--modify-window=1" for FAT destinations that has only a 2 second resolution. This might not be a perfect solution but might help you to prevent the CPU intensive check sum calculations on the little Sparc Duo v1.

Hi, I'll give that a try what you suggested.
Yesterday I quickly converted my 5 backup jobs to RYNC -> LOCAL and added the "--protocol=29" option in the /frontview/bin/backup file.
In terms of speed, I had not the impression it would go faster. It got stuck at about 5.5Mbytes/sec when really transferring chuncks of data (I deleted some 4Gbytes ISO's from the target hence it would "re-transfer" from the source)
Since that majority of this content is pretty "static" it is not the end of the world...

jvanhambelgium · ‎2012-08-31

JohnnyB wrote:
Without the "-c" option, rsync skips files based on mod-time and size. While this might yield to the same problems as with "cp -u", rsync offers the "--modify-window" option that allows you to add some fuzziness regarding time stamps. E.g., the man page recommends "--modify-window=1" for FAT destinations that has only a 2 second resolution. This might not be a perfect solution but might help you to prevent the CPU intensive check sum calculations on the little Sparc Duo v1.

Ive been looking into the /frontview/bin/backup script but it seems the "-c" (or --checksum) it not used anyway ? So this is already "faster" than if using the "-c" switch...nothing to optimize there I guess....

StephenB · ‎2012-08-31

You can't add the -c via frontview (the script filters it out).

JohnnyB11 · ‎2012-08-31

I have to admit that I do not really understand why rsync works when backing up from ext4 to ext3 without adding "--modify-window=1". See also viewtopic.php?f=31&t=65768 that states incremental backups are working fine when using rsync.