I have multiple shares that are backed-up weekly from one NAS to another. Recently, this has become an issue because all the backups are configured to do a full followed by 3 incremental, which means that they are all doing a full at the same time. At the moment this means they are running for 4 or 5 days continuously!!
What I would like to do is stagger the full backups so that they don't all happen at the same time.
Maybe I'm missing it, but is there a way to do that?
I'm running an RN104 which is backing-up an RN314, both running f/w 6.10.7 btw.
Unfortunately, there is no way to do it via the GUI. You can try creating the full backups as separate jobs and then editing /etc/cron.d/frontview-backup to change the frequency, but I don't know how that will affect the GUI. And you'd need to be comfortable with the Linux command line to implement it. Any other change via the GUI would likely undo your manual edits, so you'd have to do them again.
I think that not scheduling the full backups via the GUI and then manually creating a separate cron.d entry (/etc/cron.d/monthly-backup, as an example, though putting it in cron.weekly may be better) would likely be a better method. But I don't know if fvbackup (the executable that does the backups) will prevent that from causing multiple simultaneous backups, which could bog down a 104. But since nothing in the cron job seems to prevent that, I suspect that fvbackup is responsible for "stacking up" the backups instead of doing them simultaneously.
I have multiple shares that are backed-up weekly from one NAS to another. Recently, this has become an issue because all the backups are configured to do a full followed by 3 incremental, which means that they are all doing a full at the same time. At the moment this means they are running for 4 or 5 days continuously!!
What I would like to do is stagger the full backups so that they don't all happen at the same time.
Current settings let you specify a full backup every 2, 3, or 4 weeks. Is that what you mean?
You could change the settings to "never" for a bit, and then set some of the shares to 3 weeks. Then try setting some more back right after the first full backups resume.
You can try creating the full backups as separate jobs and then editing /etc/cron.d/frontview-backup to change the frequency, but I don't know how that will affect the GUI. And you'd need to be comfortable with the Linux command line to implement it. Any other change via the GUI would likely undo your manual edits, so you'd have to do them again.
I've done this something like this with frontview-volumeschedule, though for a different purpose (starting up maintenance tasks on days when my backup NAS were turned on, since they are on a power schedule).
You would need to examine backup_jobs.conf in order to determine the job number associated with each backup.
I've also got my target NAS (the RN104) on a power schedule, so currently it switches on once a week, pulls the backups and then shuts down again (eventually!).
I'll have a look at configuring it using cron.d and no doubt come back with more questions! 😉
where '007' correlates to the 7th backup job in the list shown in the GUI.
So I looked in /frontview/bin expecting to see a file called '007' that would contain the definition of the backup job, including when to do a full and when to do an incremental, but there isn't one.
Does anyone know where '007' is? (and don't say "ask Ms Moneypenny" 😁).
/etc/frontview/backup_jobs.conf. This appears to be something read by the GUI and fvbackup to set up and execute the backups. Items like the frequency appear to be only for the GUI, as /etc/cron.d/frontview-backup does the actual scheduling.
Ah, of course. StephenB did mention that earlier 🙈
So it looks like the mechanism for keeping track of when the full backups are done must be internal to fvbackup, as backup_jobs.conf only has a key for <opt_full_backup_freq>. This key just has one of the GUI options in it (never, every 4 weeks or whatever) so there's nothing saying "next full backup in 2 weeks" or "last full backup on this date" some such.
So it looks like there isn't a way to stagger the full backups other than to set one to 'every 4 weeks' with all the others set to 'never', wait a week then set another to 'every 4 weeks' and so on until schedule is right. Or am I missing something obvious?
So it looks like there isn't a way to stagger the full backups other than to set one to 'every 4 weeks' with all the others set to 'never', wait a week then set another to 'every 4 weeks' and so on until schedule is right. Or am I missing something obvious?
@Sandshark was suggesting creating two backups for each share - one that always does a full backup, and another that always does incremental. Then set the schedules you want for each in cron.d.
My alternative idea was just to stagger the existing backup jobs so they weren't all running the full backup job at the same time.
So it looks like there isn't a way to stagger the full backups other than to set one to 'every 4 weeks' with all the others set to 'never', wait a week then set another to 'every 4 weeks' and so on until schedule is right. Or am I missing something obvious?
I'm not even sure if that will work or if the "every X weeks" always uses the first week of the year (or some other) as the reference. The former would be easier to implement. If it really is based on when the last full backup actually occurred, there is either a file somewhere or a database that keeps track of that. If there is a file, I've not found it.
@Sandshark wrote: 'm not even sure if that will work or if the "every X weeks" always uses the first week of the year (or some other) as the reference. The former would be easier to implement. If it really is based on when the last full backup actually occurred, there is either a file somewhere or a database that keeps track of that. If there is a file, I've not found it.
While I am not 100% sure it will work, I think it is worth trying as it is simpler than fiddling with cron.d
FWIW, db.sq3 definitely holds info on backup jobs, though I haven't attempted to figure out exactly what is in there.
db.sq3 has a 'Backup' table in it, which contains the same information as the backup_jobs.conf file. Couldn't find anything that records dates or anything in there.
I've started writing a script to do what I want as I can't see a way of doing it either through the GUI or in pure cron. It's obviously not finished, so don't be too critical 😀
#!/bin/bash
# Required backup pattern is defined below:
# F=Full, I=Incremental, S=Skip
# In order of Wk1, Wk2, Wk3, Wk4
job_1=("F" "I" "I" "I")
job_2=("F" "I" "I" "I")
job_3=("F" "I" "I" "I")
job_4=("F" "I" "I" "I")
job_5=("I" "F" "I" "I")
job_6=("I" "I" "F" "I")
job_7=("I" "I" "I" "F")
# Check if there is an existing timekeeper file
tspath="/data/Backup/timekeeper"
# If the timekeeper file exists...
if [ -f "$tspath" ]; then
# For debug
echo "Timekeeper file exists"
else
# But if it doesn't...
# For debug
echo "Timekeeper file does not exist"
echo "Initialising with this week as Wk1"
# ...assume that this is week 1, so get the week number...
wk1=`date +%U`
# ...and write that to the timekeeper file.
echo $wk1 > "$tspath"
# For debug
echo "First week = $wk1"
# Work out week numbers
# This is done by dividing the week number by 4 and recording the remainder. This allows us to work out which week of the 4 week cycle we're on, for instance:
# Say we're in week 38 when we first run, so that becomes cycle week 1. 38/4 = 9.50, so any week divided by 4 that has a 50 remainder is a cycle week 1.
# Which means that a week number with a remainder of 75 is cycle week 2, 00 is cycle week 3 and 25 is cycle week 4.
# Loop for a 4 week cycle
for ((i = 1 ; i < 5 ; i++)); do
# Bash can only do integer maths, so multiply week number by 100 before dividing by 4...
wkrem=$((${wk1}00/4))
# For debug
echo "Week $i remainder = ${wkrem: -2}"
# ...and write the remainder (either 00, 25, 50 or 75) for this week number to the timekeeper file.
echo ${wkrem: -2} >> "$tspath"
# Increment firstweek to work out the remainder for the next week.
wk1=$((++wk1))
done
fi
# Find out what week number number we're in
thisweek=`date +%U`
# Bash can only do integer maths, so multiply week number by 100 before dividing by 4.
thisweek=$((${thisweek}00/4))
# Just need the remainder to work out which week we're in.
tweek="${thisweek: -2}"
# Find out which line number in the timekeeper file the same remainder appears on.
thisweeknumber=$(grep -n $tweek timekeeper | cut -d: -f1)
# The first line of the timekeeper file is the first week number, so the cycle week number is actually one less
thisweeknumber=$((--thisweeknumber))
# For debug
echo "This is week $thisweeknumber"
# Decrement thisweeknumber again, as arrays indexes start at 0, not 1
thisweeknumber=$((--thisweeknumber))
# Generate this weeks jobs
echo "Job 1 = ${job_1[${thisweeknumber}]}"
echo "Job 2 = ${job_2[${thisweeknumber}]}"
echo "Job 3 = ${job_3[${thisweeknumber}]}"
echo "Job 4 = ${job_4[${thisweeknumber}]}"
echo "Job 5 = ${job_5[${thisweeknumber}]}"
echo "Job 6 = ${job_6[${thisweeknumber}]}"
echo "Job 7 = ${job_7[${thisweeknumber}]}"
But any efficiencies that could be added without it making it too difficult to understand would be appreciated.
Again, the simplest approach is to create a full backup job and a purely incremental backup job in the web ui. Then create a custom frontview-backup file in /etc/cron.d that schedules both jobs as you wish. No need for a timekeeper script.
frontview-backup entries on one of my systems look like this:
This starts the backup at 5 minutes after the hour on monday, wednesday and friday (that particular NAS's power schedule tells it to turn on at 11 pm Sunday, Tues, and Thurs, and turn off at 1 am Monday, Weds, and Friday).
It's pretty easy to modify this to run the backup jobs on any schedule you want.
Cron will run this job on the first 7 days of the month, the date check added to the command itself results in the backup portion of the command only running on Monday.
While you might think that you can avoid the date check with 1 0 1-7 * 1, you can't. It's been a while since I played with this, but my recollection is that results in the cron job running on days 1-7 of the month and also on every Monday. You can add in months if you need more spacing between the full backups though.
I don't think you started with a schedule created by the NAS web ui. I suggest you do that, and use the first two parameters you see in the NAS-created schedule. My sample was an excerpt of what I see ( 5 00-23/24 * * ) - not sure if the details depend on whether snapshots are running or not, but I am thinking the NAS more likely always uses these parameters for daily backups .
If I'm reading it right then that almost works, but falls over when a month has five Mondays in it. You end up doing four incremental backups instead of three (plus an additional full backup of the last job).
An example of this is January 2023:
02/01/23 Monday - Full
09/01/23 Monday - Inc
16/01/23 Monday - Inc
23/01/23 Monday - Inc
30/01/23 Monday - Inc
Next year this would also happen in May, July & November so 30% of the time it doesn't work as intended.
Does that really matter? No, probably not, but I set myself the task of figuring out how to make it as difficult as possible do it.
EDIT: It's probably obvious, but I forgot to mention that my script would be fired by a cron entry every Monday at 00:01.
If I'm reading it right then that almost works, but falls over when a month has five Mondays in it. You end up doing four incremental backups instead of three (plus an additional full backup of the last job).
That is almost correct (there are four months each year with 5 weeks). No additional full backups though, just four months with 4 incremental ones.
But as you say doesn't really matter. And it's far simpler than what you are attempting.
Rewriting the backup schedule in cron.d is a little tricky when the system is running - you'll need to restart the crond daemon.
EDIT: It's probably obvious, but I forgot to mention that my script would be fired by a cron entry every Monday at 00:01.
Your sample looked like it was triggering a backup job every Monday at 00:01. As I mentioned, I'd go with the parameters the NAS web ui uses instead - especially if you are putting the NAS on a power schedule.
When I said an additional Full, I meant that the last job - the one which was Full on week 4 (job_7) - would be repeated on week 5 thereby doing an additional Full.
Again, it was confusing when I mentioned that the timekeeper would be scheduled using cron. I didn't mean by the example entry that I'd previously given, the one with fvbackup in it, but a simple one something like this added in it's own file in cron.d (or directly in crontab even):
1 0 * * 1 /path/to/timekeeper.sh
That's all that would be needed, as the script will (when it's finished) do the rest including kicking off fvbackup with the appropriate backup definition number.
I'm going to carry on torturing myself working on the script anyway, for education if nothing else.
When I said an additional Full, I meant that the last job - the one which was Full on week 4 (job_7) - would be repeated on week 5 thereby doing an additional Full.
To clarify, my approach is using days of the month combined with a day of the week to drive the full backup jobs - the four date ranges being
1-7 (week 1)
8-14 (week 2)
21-27 (week 3)
22-28 (week 4)
No full backups scheduled for the 29-31, so a full backup would only be run once a month. If you did extend the fourth date range to 22-31, then you would sometimes get repeated full backups for the extended week 4.
You could run incremental backups similarly (running 3 incremental backups a month, and skipping the 4th in 5-week months), or you could extend the date range for incremental to 31.
Note I don't do this for backups, but I have made this mod for the 4 maintenance functions (disk test, balance, scrub, and defrag). It ensures that they start during the window when the NAS is powered up (per its power schedule). I made another mod to the power off script to make sure the NAS remains powered until the maintenance functions complete (which unfortunately wasn't done by Netgear).
That's all that would be needed, as the script will (when it's finished) do the rest including kicking off fvbackup with the appropriate backup definition number.
I use daily custom snapshots on my backup NAS to give retention (currently set to 3 weeks on most of my shares). If you do that, you want to make sure the snapshot schedule is set to either make the snapshot before the backup job starts, or after it finishes (not in the middle of the backup). Mine are set to make the snapshots before the backups start.
