I am trying to backup from a remote shared network drive as Source to a folder in a NAS Share. The settings and permissions are all good and the test connection works fine. Problem comes when I execute the backup. It runs forever and is apparently copying huge amounts of data - after >24h it had copied more than 900GB - but the size of the source drive was <500GB - I cancelled the backup at this point. There are no files listed in the Share folder and I cannot see anything in that folder. The overall available size available was also reduced by 900GB - after I deleted the apparently empty Share folder, the 900GB came back to the system. Snaphsots are disabled on this share
The only issue I can see is that the Remote drive has a .snapshots folder. The directories in this folder seem to be writing when I am trying to backup with the NAS - they are not part of the remote drive. The log for the backup reports errors trying and failing to access files in these .snapshot directories. I can't find a way to delete the .snapshots folder on the remote drive.
Is there anything I can change in the backup settings to get it to ignore this folder? I can't use RSync as I can't load a public key on the remote host.
The only issue I can see is that the Remote drive has a .snapshots folder. The directories in this folder seem to be writing when I am trying to backup with the NAS - they are not part of the remote drive. The log for the backup reports errors trying and failing to access files in these .snapshot directories. I can't find a way to delete the .snapshots folder on the remote drive.
Is this a remote ReadyNAS?
I'm not sure what you mean when you say .snapshots is not part of the remote drive. Are you saying that the remote share doesn't have a hidden .snapshots subfolder?
Or are you saying that the local destination has a .snapshots subfolder? If this is what you mean, then what snapshot settings are you using for the local destination share?
I am trying to backup from a remote shared network drive as Source to a folder in a NAS Share. ... I can't use RSync
What backup protocol and settings are you using?
I was using the Windows/NAS(Timestamp)
I'm assuming you mean that you can't use rsync over ssh, as vanilla rsync doesn't need a public key.
OK - not an expert - tried looking at RSync as it apparently has options to exclude folders, so thought this might be a workaround. The remote is a corporate SAN facility so I do not have any administrative permissions on this system. I am creating a backup for longer term storage of data as the SAN isn't big enough for this and is a more active working space. I didn't try the 'vanilla' RSync
The only issue I can see is that the Remote drive has a .snapshots folder. The directories in this folder seem to be writing when I am trying to backup with the NAS - they are not part of the remote drive. The log for the backup reports errors trying and failing to access files in these .snapshot directories. I can't find a way to delete the .snapshots folder on the remote drive.
What I mean is that I don't think (could be wrong) that the remote drive natively keeps snapshots. If it does, then it seems odd that the only directories in this folder align with the times where I have been trying to run backups from the NAS. It looks like the NAS is making it write to the .snapshots or it is the NAS that is writing it. Either way it doesn't make much sense to me.
UPDATE
I switched to Windows Archive Bit - I also noticed a 1h time discrepancy with the NAS system as it was linked to GMT rather than London time so wasn't on British Summer Time - fixed this issue. The Backup now appears to be working. Not sure if it is the different protocol or if the time fix prevented some infinite loop of snapshots.
Not all of my post seemed to get posted - got confused with attempting to including quotes in replies...
I was previously using Windows(NAS) Timestamp
I didn't try the vanilla form of RSync. I'm not an expert - as you probably gathered - I read that RSync can exclude specific directories so thought this might be a workaround. The remote is a corporate SAN drive so I don't have any administrtive priveleges.
I read that RSync can exclude specific directories so thought this might be a workaround.
Rsync would let you exclude directories. Ordinary rsync backup jobs are not encrypted, so they don't need a key installed. But of course that means you shouldn't use ordinary rsync over the internet. It'd be fine the NAS and SAN drive are connecting over your corporate network.
What I mean is that I don't think (could be wrong) that the remote drive natively keeps snapshots. If it does, then it seems odd that the only directories in this folder align with the times where I have been trying to run backups from the NAS. It looks like the NAS is making it write to the .snapshots or it is the NAS that is writing it. Either way it doesn't make much sense to me.
I don't use Windows backup protocols myself, so I don't recall off-hand how it handles hidden folders like .snapshots.
On the ReadyNAS side of things, shares (and snapshots of shares) aren't ordinary folders - they are BTRFS subvolumes. Normal linux commands won't delete them.
But on my ReadyNAS (currently running 6.10.4), I don't see a .snapshots folder when I look at the share with ssh. I do see a snapshot folder if I have "allow snapshot access" checked in the snapshot settings, but not .snapshot.
As an aside, I don't check the "allow snapshot access". It not only makes the snapshots accessible, it also makes them writable. That defeats their purpose as far as I am concerned. I do check "allow access to Windows Previous Versions".
Unfortunately I cannot connect with the vanilla RSync - it won't accept my login details. So I guess the SAN isn't setup to accept this protocol.
When I did the backup with the Windows Archive bit, it seemed to work, but the GB of data was somewhat different and less on the backup than on the source (as measured in Windows Explorer).
Is there a way of getting Directory size in the NAS OS? That would be really useful!
I am rerunning now having gone back to to Windows(NAS)Timestamp protocol. Will see if it runs forever....
Is there a way of getting Directory size in the NAS OS? That would be really useful!
If volume quota is enabled on the volume settings wheel, you can see the on-disk space for each share. This is in a "consumed" column in the shares page. This size includes snapshot space (if snapshots are enabled). The file system is likely different from the SANs, so the results might not tie out exactly.
FWIW, when you access the share via Windows, there is some translation going on in Samba (running on the NAS), and the space won't be that accurate. That might also be true with the SAN. You could download a sample from both, and then diff them I guess.
It ran forever, once again accumulating invisible dark data chewing space on the NAS. Only way to recover this is to delete the destination folder on the NAS and start again.
So the problem wasn't to do with the time difference between remote host and NAS. I'm convinced it is something to do with the .snapshots - is it trying to unpack them?
Work around this time is to revert to the Windows archive bit, but to reset it on all files before I start using attrib "Path:*.*" /s +a in a command line. It looks like this does not touch hidden and system files like thumbs.db, which is good as far as I'm concerned. The previous effort at using the archive bit missed a lot of files as they didn't have the archive bit set.
So the problem wasn't to do with the time difference between remote host and NAS. I'm convinced it is something to do with the .snapshots - is it trying to unpack them?
I still am confused about whether the .snapshots folder is on the SAN or not. It is possible that the Windows Timestamp method is trying to back them up.
But if you want to pursue this further, I think looking at what's in the folder on the SAN (and whether it is useful to back it up) would be the right next step.
There are some less efficient ways you could back up the SAN. For instance, you could mount the SAN volume on a Windows PC as a drive letter, and use Windows tools to back up the files to the NAS. FreeFileSync is one of many tools you could use. That would give you explicit control over files/folders to exclude.
I ran some tests using Windows Timestamp backup and found the following:
Whether on a Windows 10 desktop source (I don't have a Windows server to test on), the ReadyNAS will back up folders that start with a dot (which isn't a hidden folder in Windows, but is in Linux), those with the Windows Hidden attribute set, and those with both.
With a Linux source (another ReadyNAS), the same is true. It will also back up any files in snapshots made visible, and not as snapshots -- as another copy of the file.
The fact that the folder is named .snapshot may indicate that the source is a NetApp server. That's not the name Windows Server uses. To the best of my knowledge, rsync is not a part of and cannot be added to the NetApp OS. On a NetApp server, .snapshot can be made invisible, but it's "all or nothing". All snapshots are visible to all users or no snapshots are visible to any users. And, of course, only an admin can change it.
I think you are correct that the contents of the .snapshot folder are being copied as separate folders and files, which is going to go on for a long time and fill the destination. I don't think there is a native ReadyNAS backup job that will work for you unless you want to create a separate backup job for each folder except .snapshot.. But you could look at Duplicati as a potential solution. I've not used it, but I've seen it recommended for similar situations, though not on a ReadyNAS. It has a web-based interface to run on headless systems like a ReadyNAS, though you have to install it via SSH. As @StephenB suggested, there are a lot of solutions that would run on a separate PC and "bucket brigade" the files from the server to the NAS. Even a simple script using robocopy would work.
Thanks for all the input and help on this, which has been really helpful.
Just to follow up on where I got to on this:
It was confirmed to me that the .snapshots folder is part of the backup file system so cannot be removed.
It seems clear that the Windows(NAS) Timestamp mode cannot deal with this folder and is likely attempting to unpack it.
I interact with the SAN folder by having it mounted as a drive on my Windows machine. I implemented the attrib "path:*.*" /s +a via cmd line to add the archive bit to all of the files and then used the Windows Archive Bit backup from the NAS, this worked well and it seemed to capture all of the files and just some error messages I can ignore on the .snapshots folder. I will use this for now and keep an eye on how well this captures file changes.
The alternatives are to organise the drives into subfolders and backup those individually, or run a backup from the Windows environment using e.g. robocopy. My problem with the latter is that I am not good at remembering to do this and in the past Windows Task Scheduler has not worked as intended for this type of task. Problem with the former is that other users of the folder are not going to appreciate it being restructured and there are too many folders to backup individually.
The alternatives are to organise the drives into subfolders and backup those individually, or run a backup from the Windows environment using e.g. robocopy.
With some customization of the NAS, you could also create a backup script for the NAS.
The idea here is that you could mount the SAN using SMB in the NAS. Then either use rsync or a cp script that excludes .snapshots (or all hidden files). rsync is probably the better mechanism for incremental backups.
If the system is a Netapp, then .snapshot can be hidden from non-admin users, so ask about that. That means an admin may need to get involved to recover any inadvertently deleted files or folders, but that's not always a bad idea. Like the ReadyNAS, Netapp supports both a snapshot folder, which can be visible or not, and Windows previous versions. Unlike a ReadyNAS, the snapshot folder can be made visible to admins only.
What does a NAS Windows Timestamp backup do with Windows hard and symbolic links? Maybe an option is to create a folder of links and back that up if the NAS follows rather than just copies the links. Since Acronis True Image for PC to NAS and Rsync for NAS to NAS work for me, I've just never tried it.
