All FTP backup jobs become Full = "Every Time" with upgrade

all my FTP backup jobs are just re-running the same data and never get to the new data before failing out!

Are you sure? (Though I don't know what you mean by "failing out")

Yes there appears to have been a change to the FTP backup options *BUT* you should find that FTP backup is now always incremental. (Rather than saying "Everytime" that option should probably be disabled)

To force a full backup you must select the option "Remove the contents of the backup destination before a full backup is performed."

At least - that's how it works for me and the FTP backup jobs I run every day.

Yes, it is indeed performing a full backup each time. It begins at the oldest files and overwrites the existing files in the backup location. I verified via the logs that it is uploaded the same files each time!

I already have opened cases regarding the fact that my FTP backups constantly fail "for unknown reasons," and the reasons are truly unknown because support has never figured out the problem.

1) So the issue is, my FTP backups fail after 6-12 hours continuously. Previously, I gave up fighting about this since they kick off every 12 hours and would eventually catch up with the new data on the device.
2) Now, the FTP backups upload the same (old) data EVERY time, and never get to the new data.

-Robert

Regarding the fail - nothing I can comment as I have no information

Regarding incremental backups: I backup 2 computers every day via FTP. As a test I uploaded a file to one of the computers and kicked off the backup again. Here's the resultant log:

Job: 017
Protocol: ftp
Source: ftp://xxxxxxxx.dyndns.org/Backup/Laptop2
Destination: [mad]/Laptop2

/sph_test.rtf  -> "sph_test.rtf" 

FINISHED --2011-05-20 07:24:08--
Downloaded: 1 files, 356 in 0s (663 GB/s)

Backup finished. Fri May 20 07:24:08 CEST 2011

As you see - only one file is reported as backed up which is the file I uploaded, indicating an incremental backup. (As does the unrealistic effective transfer rate reported)

So clearing the NAS does implement incremental FTP backups and so his would suggest that there is something particular to your scenario that is causing the full backup run. There is of course the possibility of a firmware issue, but perhaps something related to time differences between source & destination? Or could it be that the timestamps of many, but perhaps not all, files in the source are somehow being changed?

You could try creating a new job that backs up a small amount of the same source data and run multiple times to figure if incremental backups are just not working for you (ie indicating a firmware related problem) But if the job is implemented as incremental, it would then point to the problem likely being related to data changes or your network setup and at least narrow the issue down

The incremental backups to a local, usb-attached drive are working properly and incrementally or fully, as I set them.

The backups to a remote FTP site that worked fine on 4.2.15 and now can only be set to "Every Time" on 4.2.17 are, indeed, full backups every time. I ran them multiple consecutive times and each time they backed up every file. Nothing has changed on my FTP server in the last four days, so I am pretty convinced it is a problem with the new firmware. After all, last week I could set the job to "First time only" and now I can't!

-Robert

The difference I'm seeing here is that alldidasmc is doing an FTP push backup while sphardy is doing a pull. In the backend, each type of FTP backup is handled by a different utility. The reason we changed the full backup frequency to "every time" is because incrementals didn't seem to work with the FTP server(s) we tried, so perhaps this is FTP server dependent. Which FTP server are you using as your backup destination?

A quick look at the changes in the backend between 4.2.15 and 4.2.17 suggest that it was only a UI change, and that the backend itself hasn't changed, so if 4.2.15 could push incrementals to an FTP server, 4.2.17 should also be doing it despite what the full backup frequency option says.

I upload to a remote backup account that I don't control, so I don't know what server software they are using. I've been using the FTP incrementally with success for about 4-6 months. I have uploads scheduled every 12 hours. With this latest build, it is trying to upload everything every time, so my twice-daily incrementals are destroyed! This was a very important part of my backup strategy as this is critical data!

ewok is right about the difference being push vs. pull, but the push functionality is definitely hosed for users like me that perform frequent incrementals. As a supplement to my main backups, I also pull DB backups down from a server twice a day and then push them up to another server (using two separate jobs, scheduled every 12 hours, staggered by one hour). There are two DB's, so each job should move two files. See logs, below.

FULL Backup started. Thu May 26 13:05:02 EDT 2011
Job: 007
Protocol: ftp
Source: ftp://[removed]
Destination: [removed]

[2 lines removed]

FINISHED --2011-05-26 13:05:21--
Downloaded: 2 files, 282K in 0s (537777 GB/s)
Backup finished. Thu May 26 13:05:21 EDT 2011

vs.

FULL Backup started. Thu May 26 02:05:02 EDT 2011

Job: 011
Protocol: ftp
Source: [removed]
Destination: ftp://[removed]

[282 lines removed]

Backup finished. Thu May 26 02:19:37 EDT 2011

This still works for my tiny DB backups directory, but my main backup operates on a directory that grows about 50GB/mo. So it is of no use to me now.

Hmm, according to our code logs, the change that forces full FTP backups to every time was made in 4.2.14. This area hasn't really changed since, but incremental pushes worked for you with 4.2.15? The only inconsistency I can think of is that if you created the backup job pre-4.2.14 and hadn't editted it, the backup job would still have a full backup frequency that was not "every time". This shouldn't have any non-cosmetic effect with FTP however.

Does your server support anything other than FTP? We'll still look into the issue, but I'm not sure what to make of it yet.

I have been editing and using these jobs consistently with 4.2.15; I file data by month and each month I change the job to incrementally work only on that month's data (folder).

The backup server I am using does support rsync, which I've never tried. I guess I'll give that a shot; do you have any suggestions or primers for someone who has never used rsync before?

Thanks!
-Robert

AHA! I had some trouble figuring out how to load my public key onto my server (the account does not allow SSH access), but the job is setup and running. I'll see how this works; thanks!

-Robert