Backup job copies ALL files instead only changes

What protocol are you using for backup? What format are you using on the USB drive?

It's a standard Backup Job via ReadyNAS interface from a ReadyNAS "Share: Name" to a ReadyNAS "USB Device: Port". There is no option to select a specific protocol in the job, and I have no idea what protocol ReadyNAS uses behind the scenes to perform the backup.

The USB drive is formatted NTFS.

Perhaps the ESXI server is altering some attribute of the existing files that is making them appear to the Readynas as needing backing up. I don't know how the ESXI backup works.
There has also been an issue described where timestamps on different filesystems had different resolutions, leading to similar behaviour to that you have described. I thought it had been fixed though and it occurred in os6

viewtopic.php?f=31&t=74087&p=411580&hilit=Rsync+to+USB#p411580



Have you considered setting up an rsync backup to the USB disk?

The backup is actually done by Acronis vmProtect. Each backup is done to a separate file, and to my knowledge the old files do not change. Thus, all I need is to copy the non-existent files. This should be the easiest incremental backup possible...and yet it copies EVERYTHING. Ugh!

I have looked at rsync using 127.0.0.1 as the "remote" server. However, I'm skeptical that rsync can do better. I know rsync is supposed be faster because it copies only the changed portions of files, but how does it determine what's changed? Seems to me there's only two possible ways.

1. Does it do a timestamp and size comparison and skip ones that are the same? That would be nice if it worked, but since the current backup can't accomplish that--possibly because of the problems with different filesystems that you mention--why would rsync perform any better?


2. Or does it read both the source and destination files, perform some sort of checksum, and then copy only the pieces that changed? I can see that being quite efficient with a client/server over a low-bandwidth connection with mostly static data. However, within the same box that takes almost as long as flatout copying everything because everything in both the source and destination will need to be read once, checksum calculated, plus a possible write to the destination. In fact, worst case it could actually take longer if all files changed.

Also, with rsync I have another failure point: username and password. With Share > USB, I don't have to worry about disabled accounts or changed passwords.

However, I can't continue as is, so unless someone can offer a fix, I guess I'll have to try rsync after the current job gets done copying everything again.

rsync has the capability of updating portions of files based on checksums, but the ReadyNAS command line used in frontview backup doesn't set those options. So it should only copy changed files (based on the usual attributes of timestamp and size).

You shouldn't need a username/password in your case, and the data remains in the unit, since the address is 127.0.0.1.

Your problem might be the use of NTFS. If for some reason the timestamp granularity differs, then the timestamps might never match. Did you check the full timestamps using ssh?

Hmmm...I didn't think to check full timestamps beyond seconds. Turns out the NVX (ext3?) stores seconds to a precision of 9 decimal places, while NTFS stores to only 7. If both the standard backup and rsync are checking seconds to greater than 7 decimal points accuracy (e.g., HH:MM:SS.123456789...that's a billionth of a second...i.e., a nanosecond. Seriously??? Isn't ten millionths of second enough???), then I'll have to sync ReadyNAS Share to ReadyNAS USB via a remote computer and take the network hit. Unfortunately, the main backups run at night and these secondary backups are scheduled during the day. That's not a problem when they're isolated to the ReadyNAS, but if I have to copying gigabytes and sometimes terabytes of data across the network TWICE while users are there, the huge network hit will NOT be appreciated.

But before I do that, I'll give rsync a try. Just have to wait until the current backup is done re-copying the 465GB backup file it already started. :-P

One option would be to format (note this wipes all data) the USB disk to use EXT4.

There is also a way to set rsync to use 1 second granularity (--modify-window=1). I think there might be a way to edit the default settings the NAS uses to include this - but am not sure. mdgm likely knows. It's also possible that frontview backup already uses this with rsync - there are similar problems with different versions of linux, including the linux used in OS6 and OS4.

Alternatively you could probably write a simple script to truncate the timestamps on the ReadyNAS volume, and put that in a cron job. If it runs before the backup, then the timestamps would match (after an initial sync).

ext4 format will also work (using your existing job), and will be faster than ntfs. ext3 will not. Though as mgdm says, using ext4 requires formatting/data loss (and also means you need ext support on whatever machine might be used to restore the backups or verify them).

I'd be interested in learning how to set time granularity.

Formatting EXT is NOT an option. My clients run Windows only. If something happens to the ReadyNAS, I need a way to restore backups (yes, I do have restore backups on a regular basis...my clients are extremely computer-illiterate and find all kinds of unique ways to screw things up). And no, I'm not about to install EXT drivers on my clients' Windows PCs and have my clients complain that "My computer doesn't work ever since you <updated IE/installed that printer/cleaned my monitor/...>" They complain enough as it is and that's just adding fuel to the fire. :-)

How is vmProtect writing to the ReadyNAS share? Is it set up to use smb/cifs, FTP, or something else?