This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
Scripting the snapshots (RN626X, RN516, RN316, RN204, etc.)
I'm interested in recovering all changed files on our RN316. This is the current model under discussion, but the concept is relevant to any ReadyNAS device running OS 6.* (or maybe more, for that matter).
Background: we have six weeks worth of snapshots on our RN316, and we need to replace the drives with new and larger models. This will of course result in the loss of our volume. The latest versions of the files are on another RN device, but only for the past 12 days. We'd like to capture that six-week history before we destroy the data.
My idea is to write a script to loop through each snapshot and build a list along the way of every file that's changed since the previous snapshot, and then copy those new/changed files to a new folder with a date-specific name that's also created by the script.
The logic behind such a script is simple enough, but what I don't know yet is how to access those snapshots in my code (in a read-only manner, gathering first the files' metadata, comparing it, and then copying the file when necessary).
I'll probably do this in PowerShell, since I'm more familiar with its syntax. Visual Studio Code will be the ideal IDE for the task. I've already installed and tested PS on the device, using this guidance:
It works a peach. Right now I'm running a quick one-liner to recurse a certain share, capture the full path of each file in it and append that location to a text file. It's up to 600MB so far, without a hitch.
I've turned on snapshot access with the GUI and I see a separate pair for each file, e.g.:
Is it simply a matter of looping through the files in these snapshot folders and comparing each file's metadata with the same from the previous snapshot? It sure looks like it. If so, my task ends up being a lot easier than I'd originally considered when I started discussing the notion with everyone.
A little clarification on this will go a long way.
This will of course result in the loss of our volume.
As @rn_enthusiast says, it normally won't result in loss of the volume (unless you are planning to destroy it for some reason, or are wanting to do a factory reset).
So you should first sort out if you really need to do this.
Is it simply a matter of looping through the files in these snapshot folders and comparing each file's metadata with the same from the previous snapshot? It sure looks like it. If so, my task ends up being a lot easier than I'd originally considered when I started discussing the notion with everyone.
Each snapshot shows the full contents of the share at the time the snapshot was taken - including the metadata. So you could just compare file dates (and maybe size) with the previous (or next) snapshot. Of course you also need to be handling deletions and adds.
Any tool that allows you to diff two different folders should be usable for this (though tools that actually compare the files would work very slowly).
Is your goal to recreate the snapshots on the new volume?
From what I can read, you are looking to expand the storage capacity of the NAS. You can add larger drives without destroying the volume. Non-destructive volume expansion is one of the best features of these NAS units.
As you found yourself, you can make snapshots available over SMB. Go to settings on a share and find the "Snapshots" tab. From here you can enable "Allow snapshot access". This will mount the snapshots but not in a complete read-only mode. I just tested it and I can delete files from within my snapshot, over SMB - using this method. However, I don't think you can make changes to files within the snapshot though.
When you set "Allow snapshot access" it will make a folder visible within the share. That folder is simply called: "snapshot" and from there you can browse all your snapshots and the data within them.
But again, if you want to vertically expand the capacity, just add larger drives and keep your volume intact 🙂 Replace one drive with a larger one and allow the raid to sync, replace the next drive with a larger one and wait for that to sync, and so forth.
This will of course result in the loss of our volume.
As @rn_enthusiast says, it normally won't result in loss of the volume (unless you are planning to destroy it for some reason, or are wanting to do a factory reset).
So you should first sort out if you really need to do this.
Is it simply a matter of looping through the files in these snapshot folders and comparing each file's metadata with the same from the previous snapshot? It sure looks like it. If so, my task ends up being a lot easier than I'd originally considered when I started discussing the notion with everyone.
Each snapshot shows the full contents of the share at the time the snapshot was taken - including the metadata. So you could just compare file dates (and maybe size) with the previous (or next) snapshot. Of course you also need to be handling deletions and adds.
Any tool that allows you to diff two different folders should be usable for this (though tools that actually compare the files would work very slowly).
Is your goal to recreate the snapshots on the new volume?
The RN316 is destined to become an offsite mirror of our onsite RN6256X. That's why we purchased 13 WD 4TB Reds—identical drives throughout, with one cold spare. (I didn't discuss this earlier because I didn't want to detract from the real question, which of course is about the proposed scripting solution.)
To clarify, the RN626X holds the most recent snapshots going back two weeks; the RN316 holds the snapshots covering from three weeks back to nine weeks. It's those six weeks of changes that we want to move to the onsite RN626X (as actual full files inside a date-driven folder hierarchy, since it's not possible to migrate the snapshots themselves).
I suppose the best way to find out whether it works like I think it must is to try it. Isn't that always the case? So far I can't think of any reason why it wouldn't work, and since it's all read-only against the snapshots I can always blow away the resulting filesets as I make my scripting mistakes along the way. (That ALWAYS happens—I don't care how good one is. Except for Bill Evans. He never made mistakes. What an incredible genius he was.)
Frankly, I'm not even sure whether turning on snapshot access in the GUI made any difference as to whether my script—again, running directly on the device and not across SMB—could access those folders. I'd turned it on well before I started all of this, and I haven't tested it otherwise. I'm guessing not, but you may know differently.
FYI I'm not at all pleased with your discovery—that SMB-based deletions within a snapshot are possible. But that's for a different discussion at a different time. I just thought I'd mention it briefly while we were here.
unless you are planning to destroy it for some reason
Yes, such is the case. The RN316 that's presently holding these six weeks worth of snapshots ultimately will be traveling offsite to become a mirror of our onsite RN626X. But we want to recover that history before we recreate the volume with the new larger drives. (This is new info, I realize, but I didn't feel it relevant to the scripting question. I didn't want to stray too far off-topic before I even got started.)
So you should first sort out if you really need to do this.
I have and I do. I'm pretty sure.
Each snapshot shows the full contents of the share at the time the snapshot was taken - including the metadata
Bingo. That's exactly what I wanted to hear. It looked like that very thing to me at first glance, but your second opinion makes for a wonderful confirmation. Thank you.
Of course you also need to be handling deletions and adds.
Yes, that complicates matters significantly. But I'm confident I'll get it worked out. Twenty-five years of software development can't have gone to complete waste 😉
Is your goal to recreate the snapshots on the new volume?
I didn't think that was possible. Could you elaborate?
When you enable the "Allow Snapshot Access" option from the GUI, the NAS will mount all the snapshots as RW.
Before enabling the option on my "Misc" share:
root@Datastore:~# mount | grep Misc
root@Datastore:~#
After enabling the option (notice how they are mounted as RW now):
root@Datastore:~# mount | grep Misc
tmpfs on /data/Misc/snapshot type tmpfs (rw,noatime,nodiratime,size=4k)
/dev/md127 on /data/Misc/snapshot/c_2019_03_30__00_00_09 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1001,subvol=/Misc/.snapshots/3/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_04_27__00_00_07 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1059,subvol=/Misc/.snapshots/7/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_05_25__00_00_16 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1090,subvol=/Misc/.snapshots/11/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_06_29__00_00_14 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1121,subvol=/Misc/.snapshots/16/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_07_27__00_00_23 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1165,subvol=/Misc/.snapshots/20/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_08_31__00_00_17 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1206,subvol=/Misc/.snapshots/25/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_09_28__00_00_00 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1232,subvol=/Misc/.snapshots/29/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_10_19__00_00_16 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1252,subvol=/Misc/.snapshots/32/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_11_30__00_00_26 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1284,subvol=/Misc/.snapshots/37/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2019_12_28__00_00_19 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1372,subvol=/Misc/.snapshots/41/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_01_25__00_00_11 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1400,subvol=/Misc/.snapshots/45/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_02_29__00_00_27 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1460,subvol=/Misc/.snapshots/50/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_03_28__00_00_05 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1538,subvol=/Misc/.snapshots/54/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_04_25__00_00_16 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1730,subvol=/Misc/.snapshots/58/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_05_30__00_00_26 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1891,subvol=/Misc/.snapshots/63/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_06_27__00_00_09 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=1955,subvol=/Misc/.snapshots/67/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_07_25__00_00_04 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2017,subvol=/Misc/.snapshots/71/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_08_29__00_00_19 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2122,subvol=/Misc/.snapshots/76/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_09_26__00_00_15 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2232,subvol=/Misc/.snapshots/80/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_10_31__00_00_16 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2509,subvol=/Misc/.snapshots/85/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_11_07__00_00_28 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2518,subvol=/Misc/.snapshots/86/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_11_14__00_00_25 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2542,subvol=/Misc/.snapshots/87/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_11_21__00_00_19 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2551,subvol=/Misc/.snapshots/88/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_11_28__00_00_23 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2594,subvol=/Misc/.snapshots/89/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_12_05__00_00_06 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2606,subvol=/Misc/.snapshots/90/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_12_12__00_00_20 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2645,subvol=/Misc/.snapshots/91/snapshot)
/dev/md127 on /data/Misc/snapshot/c_2020_12_19__00_00_05 type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=2678,subvol=/Misc/.snapshots/92/snapshot)
root@Datastore:~#
Is your goal to recreate the snapshots on the new volume?
I didn't think that was possible. Could you elaborate?
I was just trying to understand your intent (that is, whether capturing the changed files in the snapshot was the only thing you wanted to do).
Back on the main topic - you might be able to use btrfs send/receive to extract the incremental changes in each snapshot. That's not something I've ever played with though. You might check out this page if you want to explore that path: https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
Re: Scripting the snapshots (RN626X, RN516, RN316, RN204, etc.)
I agree. In my opinion a snapshot should always be R/O so this isn't an option I ever use either. If I need to restore files from a snap I will typically just use the CLI or GUI to do so.
I agree. In my opinion a snapshot should always be R/O so this isn't an option I ever use either. If I need to restore files from a snap I will typically just use the CLI or GUI to do so.
So it appears that the setting governs SMB access only, as other types of access are possible even when the setting is turned off.
