Backups not working yet status says they are working
I have been using the built-in Rsync feature to back up my ReadyNAS (firmware version 6.10.4) to a directly attached external USB drive for a number of years now. I get nightly notices via email from my ReadyNAS indicating "Successfully completed backup job nightly rsync of data folder." However, nothing has been written to my external USB drive in over two months. I did a little investigating and found that the external USB drive is only 68% full, buy my ReadyNAS is 100% full (but see next paragraph). I presume this is preventing rsyncing to the external drive. I'm curious as to why a full ReadyNAS would break the backup process (although I can image several reasons why). I am even more curious as to why the backup process would tell me that a backup was succesfully when it was not. Is this a bug that should be fixed for the entire ReadyNAS community?
When I say that the ReadyNAS is full, I'm looking at the output of the "df" command (from an ssh session) which shows 20G available with 100% use. I have not gotten any notices that the ReadyNAS is full and I'm still able to write to it just fine (I mostly write small text files), so I'm guessing that samba runs as root so it can still write my files as it likely has access to the 20G remaining..
This is a very dangerous senario considering that I rely on these emails to let me know that my backups are functioning. I have had no backups now for over two months and I've been told (via email) they have been successful every night. Thank goodness I didn't have any data loss and that I caught this before such an event occurred as this could have been catastrophic.
Here is an example of a nightly email that I receive from the ReadyNAS (notice that thte backup time is less than one second):
I have been using the built-in Rsync feature to back up my ReadyNAS (firmware version 6.10.4) to a directly attached external USB drive for a number of years now. I get nightly notices via email from my ReadyNAS indicating "Successfully completed backup job nightly rsync of data folder." However, nothing has been written to my external USB drive in over two months.
This is a known bug with rsync "push" backups in 6.10.4 and 6.10.5 No idea why Netgear hasn't addressed it (perhaps @JohnCM_S should make sure it is on their fix list???)
Since you are using "push" backups (backing up from a local source to a "remote" destination), you can either downgrade back to 6.10.3 or you can switch to using "pull" backups. Though you need multiple backup jobs to use "pull" (one for each share).
...but my ReadyNAS is 100% full (but see next paragraph).
When I say that the ReadyNAS is full, I'm looking at the output of the "df" command (from an ssh session) which shows 20G available with 100% use.
You should expand your storage! BTRFS behaves very badly when it runs out of free space - particularly when it can't allocate space for metadata. Your volume is at risk. After you expand it, I suggest running a balance from the volume settings wheel. (I don't recommend running a balance now - it likely will fail).
FWIW, df isn't a good tool for exploring disk usage with btrfs. "Free space" is a rather complex subject with btrfs and zfs. There are some tools built into btrfs that give better information. For instance, btrfs fi df.
I have been using the built-in Rsync feature to back up my ReadyNAS (firmware version 6.10.4) to a directly attached external USB drive for a number of years now. I get nightly notices via email from my ReadyNAS indicating "Successfully completed backup job nightly rsync of data folder." However, nothing has been written to my external USB drive in over two months.
This is a known bug with rsync "push" backups in 6.10.4 and 6.10.5 No idea why Netgear hasn't addressed it (perhaps @JohnCM_S should make sure it is on their fix list???)
Since you are using "push" backups (backing up from a local source to a "remote" destination), you can either downgrade back to 6.10.3 or you can switch to using "pull" backups. Though you need multiple backup jobs to use "pull" (one for each share).
Wouldn't this require the backup device to be "smart" so one can command it to do the "pull"? I am just using an external USB drive with no brains.
...but my ReadyNAS is 100% full (but see next paragraph).
When I say that the ReadyNAS is full, I'm looking at the output of the "df" command (from an ssh session) which shows 20G available with 100% use.
You should expand your storage! BTRFS behaves very badly when it runs out of free space - particularly when it can't allocate space for metadata. Your volume is at risk. After you expand it, I suggest running a balance from the volume settings wheel. (I don't recommend running a balance now - it likely will fail).
I will likely delete some large old Windoze backups to free up space rather than expanding. I will run the balance after I remove some files and free up some space. Thank you for the suggestion on "balancing". I will have to study what that does.
FWIW, df isn't a good tool for exploring disk usage with btrfs. "Free space" is a rather complex subject with btrfs and zfs. There are some tools built into btrfs that give better information. For instance, btrfs fi df.
Thank you. I will study those.
BTW, the "quoting" mechanism in this forum software isn't very intuitive - it was VERY time consuming and difficult to figure out a way to enter a response part way through a quote without it looking like the response was part of the original quote. I won't belabor this since it is not the topic at hand, but it needs fixing (or I need some lessons).
Re: Backups not working yet status says they are working
"Push" and "Pull" with both work with a USB drive and using the LocalHost address for the "remote" system. For a pull, you want the source to be "remote" at the LocalHost address and the destination to be the local share. Note that if you don't want the backup to be in a folder within the target share you need to set the desitnation path to / after you have created the backup job.
"Push" and "Pull" with both work with a USB drive and using the LocalHost address for the "remote" system. For a pull, you want the source to be "remote" at the LocalHost address and the destination to be the local share. Note that if you don't want the backup to be in a folder within the target share you need to set the desitnation path to / after you have created the backup job.
Right, that is the way it is supposed to work. Push will fail with 6.10.4 and 6.10.5, but pull will work.
"Push" just means the destination is remote, it doesn't require any intelligence in the drive itself. It does require an rsync server on the remote device. With this setup, that just requires rsync to be enabled for the destination share on the USB drive.
I have already forwarded this to the appropriate NETGEAR team. I will send a follow-up to them since I have not received any feedback yet.
Regards,
StephenB (or anyone) - can I get a status on this? John and I were going back and forth in PM but it appears he retired and I can't PM him any longer. I have (intentionally) not cleared off space so that the problem remains in case programmers at Netgear needed to see the problem real-time, but I'm feeling nervous about leaving it in this state so was hoping for update to see what is next.
Re: Backups not working yet status says they are working
Currently rsync backp on latest ReadyNAS firmware doesn't support to backup the full data volume (Backup Source: [volume:data]/) by default, please enable multiple file systems on rsync backup option, which will back up the whole data volume to destination.
Re: Backups not working yet status says they are working
I don't think you understood my question. I'm using the same method I've always used. Your answer doesn't address the issue. Can you please re-read the entire thread in detail?
Re: Backups not working yet status says they are working
StephenB, when you say "Push will fail with 6.10.4 and 6.10.5, but pull will work." do you mean that push will fail in my circumstance (ie. NAS reports full even though backup drive has plenty of space), or that pushes won't work under any condition?
I've read the above over and over and I'm confused over whether I'm pushing or pulling. I've always done it this way:
Under "source" tab, "Name:" is set to "volume: data" (there are no other options in this drop-down) and under "destination" tab, "Name:" is set to "remote: Rsync Server" and "Host:" is "localhost". Am I pushing or pulling?
I'm confused over the comment, "Currently rsync backup on latest ReadyNAS firmware doesn't support to backup the full data volume (Backup Source: [volume:data]/)". I have been backing up using the above configuration for years now and until the ReadyNAS reported 100% full (per above discussion), I've been under the impression that my entire NAS has been backed up. When I ssh onto my NAS, and change directories to my attached backup drive (cd /media/USB_HDD_7) I see the "data" directory in the root folder which appears to have all my files in it (including the subdirectory "home" which has all my individual users in it with their data in their directories). What am I missing here?
I am happy to enable "multiple file systems", but can someone explain exactly what this does (that WAS NOT already happening)? (It isn't going to solve the problem of the reporting that backups are successful when, indeed, they are not, is it)?
Finally, I'd like to wrap around to the original issue - the fact that the system emails me every night telling me that backups are successful, when, in fact, the backups are not completing. Above, I was told that this is a known bug in 6.10.4 and 6.10.5 and I see that 6.10.6 has since been released, but the release notes do not address this issue. With @JohnCM_S being gone, is someone aware of this and working on it, and is there an ETA? I've left my NAS in its current state to help with debugging but I'm starting to feel like I may be doing this in vain. Please let me know status on the fix.
please enable multiple file systems on rsync backup option, which will back up the whole data volume to destination.
....
@kenalker1 - it might be worth going into the advanced options page in your backup job, and setting that option.
Note that the original solution came from @dingjs1421 - who works for Netgear.
I don't know if you ever tried my suggestion to try his solution or not. After I suggested it, other users have verified that it works. So the problem is already fixed.
I've read the above over and over and I'm confused over whether I'm pushing or pulling. I've always done it this way:
Under "source" tab, "Name:" is set to "volume: data" (there are no other options in this drop-down) and under "destination" tab, "Name:" is set to "remote: Rsync Server" and "Host:" is "localhost". Am I pushing or pulling?
You are pushing. Pushing simply is a shorthand way of saying that the backup job has a local source and a remote destination. If the backup job has a remote source and a local destination, then you would be pulling.
I see how this could be confusing, since in your case the destination is localhost. But still, as far as the NAS is concerned it is remote.
I am happy to enable "multiple file systems", but can someone explain exactly what this does (that WAS NOT already happening)? (It isn't going to solve the problem of the reporting that backups are successful when, indeed, they are not, is it)?
I have some guesses on what it might be doing, but I haven't tried to verify them. So I won't speculate, as I might well be wrong.
But I do think you should simply set the option, and let us know if it works in your case.
please enable multiple file systems on rsync backup option, which will back up the whole data volume to destination.
....
@kenalker1 - it might be worth going into the advanced options page in your backup job, and setting that option.
Note that the original solution came from @dingjs1421 - who works for Netgear.
I don't know if you ever tried my suggestion to try his solution or not. After I suggested it, other users have verified that it works. So the problem is already fixed.
I am happy to enable "multiple file systems", but can someone explain exactly what this does (that WAS NOT already happening)? (It isn't going to solve the problem of the reporting that backups are successful when, indeed, they are not, is it)?
I have some guesses on what it might be doing, but I haven't tried to verify them. So I won't speculate, as I might well be wrong.
But I do think you should simply set the option, and let us know if it works in your case.
OK - sorry. I did not understand from the way that it was written that enabling multiple file systems would solve the actual backup issue. I tried it just now, and sure enough, backups are now working again. That is fantastic!
However, if that new option is left OFF, one still gets the email stating that the backups were successful when they really were not. I just upgraded to 6.10.6 and the issue remains. I feel pretty strongly that this should be fixed - it is misleading and dangerous.
BTW, it sure seems like anyone in my situation is now potentially in danger of having no backups after upgrading to whatever version introduced that "feature." I wonder if there is some way to reach out to those people and let them know, OR introduce a new critical firmware version that has that feature enabled and hope and pray that they all upgrade before they have a melt-down and discover they haven't had backups in months.
I'm also very curious to know more about this new feature. I am hoping it is documented in detail somewhere and someone can point me to that documentation. I'd like to know exactly what it does and how it is to be used.
However, if that new option is left OFF, one still gets the email stating that the backups were successful when they really were not. I just upgraded to 6.10.6 and the issue remains. I feel pretty strongly that this should be fixed - it is misleading and dangerous.
I'm also very curious to know more about this new feature. I am hoping it is documented in detail somewhere and someone can point me to that documentation. I'd like to know exactly what it does and how it is to be used.
@JeraldM , @Marc_V: Seems to me that this setting should be in at least one of the KB articles.
I'm also very curious to know more about this new feature. I am hoping it is documented in detail somewhere and someone can point me to that documentation. I'd like to know exactly what it does and how it is to be used.
Thank you for the help.
I seriously doubt this is a new "feature". It looks pretty clearly like a bug. The rsysnc options set for a normal backup were likely changed inadvertently, and turning on "multiple file systems" adds it back in (probably with others), so it's useful as a work-around for the bug.
Re: Backups not working yet status says they are working
@Sandshark wrote: I seriously doubt this is a new "feature". It looks pretty clearly like a bug. The rsysnc options set for a normal backup were likely changed inadvertently, and turning on "multiple file systems" adds it back in (probably with others), so it's useful as a work-around for the bug.
It seems unlikely that they'd add the option to work around the bug, it would have been simpler just to always use the multiple file systems setting internally.
I haven't checked 6.10.4, but it would not surprise me if the option was added in that release (when the bug was introduced). Either way, they messed up.
