This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
I have just moved most of my collection of photos from one folder to another on my PC, as part of my recent 'tidy' project. The PC is backed up regularly to the ReadyNAS system, which does snapshots every night and once a week. I also manually do snapshots after significant changes to the data. So there are many snapshots of the original source for those photos on the NAS, which with the COW switched on will change these to 1 real copy plus any changes there might be, on this evenings snapshot. Since I will have the new source for the photos backed up, ALL the photos in the old snapshots, which take up at least as much space as the original and maybe more are superfluous, and will under normal circumstances never be removed until all snapshots related to them are deleted. This MAY be years away, possibly never. Hence my question: Is there an easy way of deleting the contents of a folder throughout ALL snapshots that contain it? Equally could this be done for 1 file as well? I am thinking here of some enormous files that reside on my system, like disc and system dumps etc. I have a feeling the answer is NO, but I will pose the question anyway. Thanks in advance.
Certainly I confirmed this on the NAS today (see below). What I find interesting is that you say that is what we should NOT be doing, and then show us how to do it :o. I'm not a Linux expert, so would not be going down that route anyway.
btaroli wrote:
But on another Linux box, I find all my snapshots are by default read-write. Need to see if I can default that the other way. Given how these are created, they really shouldn't be modifiable. Though it might be handy if some were.
It's nice to know that someone else might find the facility I am suggesting useful.
btaroli wrote:
So, BaJohn, I suspect that whatever you are seeing is not coming from the NAS... perhaps something running on the PC is creating those folders you see?
I double checked today and No the snapshots are NOT deletable from the PC, they only appear to be deleted. When viewing on the PC, the properties are 'full access' to everyone and if you highlight and delete, it asks "do you want to permanently delete this file - Yes or No"?. When I click yes, the screen refreshes with the file shown as deleted from the snapshot folder. No other messages about it did not happen, or about needing authority or anything. It seems to me that the PC (Windows 7 Ultimate 64bit) and the NAS (RN516) are NOT talking to each other in the same language, as the NAS knows I cannot delete the snapshot file, BUT the PC thinks it can. Wierd ay, perhaps it's a case of "Can't see the wood for the B... Trees."
Hence my question: Is there an easy way of deleting the contents of a folder throughout ALL snapshots that contain it? Equally could this be done for 1 file as well? I am thinking here of some enormous files that reside on my system, like disc and system dumps etc. I have a feeling the answer is NO, but I will pose the question anyway.
Hence my question: Is there an easy way of deleting the contents of a folder throughout ALL snapshots that contain it? Equally could this be done for 1 file as well? I am thinking here of some enormous files that reside on my system, like disc and system dumps etc. I have a feeling the answer is NO, but I will pose the question anyway.
On the NAS box, individual files or folders can be deleted. The system knows what snapshots are linked to that file or folder. Why can't the system give an option on deletion to delete all associated snapshots? Doesn't seem that difficult to do. Perhaps this could be a new functionality request. I can't be the only one that thinks this might be a good idea, especially for those who's system is getting tight on space. Any comments?
Currently the method to do that is quite cumbersome. Firstly our snapshots are read-only (this makes a lot of sense for security purposes and maintaining the integrity of the snapshots). So you would need to make a read/write clone of each snapshot and delete the files from there. Then clone the snapshots back to read-only ones. Even then it may not free up the space unless you delete the entire oldest snapshot that contained the file.
Cloning snapshots to read/write ones could potentially be dangerous if things are messed up doing this procedure. You would want to take a new snapshot before doing this and then stop automatically taking/deleting new snapshots till finished.
If you want to free up space from deleted files I would suggest deleting the snapshots that contain the file or let the oldest automatic ones be automatically purged when the free space falls below 5%
It sounds like you are wanting to delete all previous versions of the file/folder?
Correct.
StephenB wrote:
It would be reasonable, though it seems hard to do when the file is deleted with windows explorer or finder (or nfs).
I WAS suggesting it be done from the NAS box. At the <Shares><Browse> pages, for each folder right clicking shows options 'Browse', 'Rename', 'Cut', 'Copy' or 'Delete', and for each file right clicking shows options 'Download',' 'Rename', 'Versions', 'Cut', 'Copy' or 'Delete'. If the 'Delete' key is used the NAS system knows exactly which file is required to be deleted and is in control. I was suggesting that an option could be 'Delete all versions' as well as the 'yes', 'no' options currently available. The NAS system has all the information (somewhere) to action what I am suggesting. Obviously the technicalities of doing it may be problematical.
Firstly our snapshots are read-only (this makes a lot of sense for security purposes and maintaining the integrity of the snapshots).
Up to now, I have carried out what I wanted to do by deleting files from snapshots remotely. i.e. Deleting from snapshots as a remote windows user on a shared drive. Since I am able to delete from snapshots (not read only to the windows user), I assumed (incorrectly) that I would be able to delete from the snapshots when logged into the NAS as admin. Slightly quirky, as the admin user should have more authority than the remote user. This seems to be the crux of the problem and the major sticking point.
mdgm wrote:
So you would need to make a read/write clone of each snapshot and delete the files from there. Then clone the snapshots back to read-only ones. Even then it may not free up the space unless you delete the entire oldest snapshot that contained the file.
Cloning snapshots to read/write ones could potentially be dangerous if things are messed up doing this procedure. You would want to take a new snapshot before doing this and then stop automatically taking/deleting new snapshots till finished.
Obviously this looks like an extremely difficult (probably impossible) to achieve. Thanks for the explanation.
mdgm wrote:
If you want to free up space from deleted files I would suggest deleting the snapshots that contain the file or let the oldest automatic ones be automatically purged when the free space falls below 5%
This would defeat my objective (delete file/folder from ALL snapshots), as the snapshots contain other data that I want to retain.
My last question then, is "Do you see any problems with my current way of deleting files/folders from snapshots one by one, by using a shared drive on my PC?".
fwiw, what I am doing is using the automatic snapshot pruning, and then manually deleting snapshots older then 3 months about once a month. That gives me 3 months of previous versions, and (so far) a good amount of free space. With the automatic algorithm, this only requires deleting one or two monthly snapshots from each share.
fwiw, what I am doing is using the automatic snapshot pruning, and then manually deleting snapshots older then 3 months about once a month. That gives me 3 months of previous versions, and (so far) a good amount of free space. With the automatic algorithm, this only requires deleting one or two monthly snapshots from each share.
FWIW I have nightly snapshots scheduled, with backups from my PC to NAS every day or so. I also occasionally do manual snapshots, usually after a significant change to data. The nightly snapshots take care of themselves, and I have manually pruned these now and again. The intention is to retain some manual snapshots for a long while, probably more than 3 months (I will probably get less paranoid with time). I currently do not have a problem with space, being at about 10% (hmmm was nearer 15% a little while ago, but I do housekeeping, and defrag, balance etc) Going back on subject, the photos that were moved from effectively from one virtual disk to another, consume space on the NAS unnecessarily in potentially ALL snapshots. Hence why I wanted to slice through the snapshots (without deleting them in their entirety), so that ALL trace of the photos in their old position was removed. As I say, I have been deleting these from one snapshot at a time, by hand accessing the NAS from my PC. My belief was that this was okay and probably more efficient if I did the oldest first and worked towards the present day. BUT it would have been nice to have an option on the NAS to delete all associated files in all snapshots. I needed to ask to discover if it was possible or not. 'mdgm' might have replied "Great idea BaJohn, we will implement in the next version" 🙂 .
Course it's unlikely that the semantics of btrfs snapshots are going to change to accommodate that. It sounds like what you really want isn't a snapshot at all. Perhaps a subvolume clone. Or even cp -R with --reflink option, which effectively does the same thing (multiple references to the same blocks until one of them is modified (CoW behavior); yes, multiple copies of a file without consuming extra storage (at first). And then you needn't be bothered to take the whole subvolume. You could just "snapshot" the directories or even single files you truly do want.
But a subvolume snapshot's whole purpose in life is to be a read-only copy for safe keeping. If that's not what you want then don't use snapshots. 🙂 there are other CoW supported approaches.
Course it's unlikely that the semantics of btrfs snapshots are going to change to accommodate that. It sounds like what you really want isn't a snapshot at all. Perhaps a subvolume clone. Or even cp -R with --reflink option, which effectively does the same thing (multiple references to the same blocks until one of them is modified (CoW behavior); yes, multiple copies of a file without consuming extra storage (at first). And then you needn't be bothered to take the whole subvolume. You could just "snapshot" the directories or even single files you truly do want.
But a subvolume snapshot's whole purpose in life is to be a read-only copy for safe keeping. If that's not what you want then don't use snapshots. 🙂 there are other CoW supported approaches.
I think you are missing the point. I can do want I want 'by hand', but it is time consuming and repetitive and prone to error. When I have to do something more than twice, I look for an automated answer. I am not suggesting a change to the workings of BTRFS and snapshots, but an App that would do the job that is runnable and linked into the OS. If I have enough time, I can probably write a program to do the job (that I want) from my PC, but how much better to have something similar that is built in to the OS, AND available to all. STOP trying to tell me how difficult it is and why I should not want it. If no one else can see the benefits, then unfortunately I am a lone voice, BUT it does not stop me from seeing it as being advantagous to me.
I think you are missing the point. I can do want I want 'by hand', but it is time consuming and repetitive and prone to error. When I have to do something more than twice, I look for an automated answer.
Admirable. Ever heard of SSH access and shell scripts? I actually *do* exactly what I described above in different scenarios with media, not for precisely the same purposes as you but with similar effect. I've never perceived this being something more widely useful and have no plans to package it into anything. But shell scripts and cron jobs (if you like automation) are pretty low barriers to entry.
BaJohn wrote:
STOP trying to tell me how difficult it is and why I should not want it. If no one else can see the benefits, then unfortunately I am a lone voice, BUT it does not stop me from seeing it as being advantagous to me.
Perhaps you're confusing me with someone else. I believe the point I made might be paraphrased as "don't try to nicely slice tomato with a paper shredder, but here are some other things you might try." It's perfectly fine if you don't like the possibilities offered, as there is always room for you to create your own.
... I have been deleting these from one snapshot at a time, by hand accessing the NAS from my PC. ...
I noticed you mentioned this a few times in this thread. At first, I was thinking "he enabled snapshot access in the share properties and is trying to delete the files that way?" but then it occurred to me that this wouldn't magically make the snapshots writable. Even down in the OS layer, the snapshots are fundamentally read-only.
root@nas:~# cd /data/Transmission/.snapshots
root@nas:/data/Transmission/.snapshots# ls -l
total 0 drwxr-xr-x 1 root root 32 May 5 04:59 1 drwxr-xr-x 1 root root 32 May 5 14:00 10 drwxr-xr-x 1 root root 32 May 5 15:00 11 drwxr-xr-x 1 root root 32 May 5 16:00 12 drwxr-xr-x 1 root root 32 May 5 17:00 13 drwxr-xr-x 1 root root 32 May 5 18:00 14 drwxr-xr-x 1 root root 32 May 5 19:00 15 drwxr-xr-x 1 root root 32 May 5 20:00 16 drwxr-xr-x 1 root root 32 May 5 21:00 17 drwxr-xr-x 1 root root 32 May 5 22:01 18 drwxr-xr-x 1 root root 32 May 5 23:00 19 drwxr-xr-x 1 root root 32 May 5 06:01 2 drwxr-xr-x 1 root root 32 May 6 00:00 20 drwxr-xr-x 1 root root 32 May 6 01:00 21 drwxr-xr-x 1 root root 32 May 6 02:00 22 drwxr-xr-x 1 root root 32 May 6 03:00 23 drwxr-xr-x 1 root root 32 May 6 04:00 24 drwxr-xr-x 1 root root 32 May 6 05:00 25 drwxr-xr-x 1 root root 32 May 6 06:00 26 drwxr-xr-x 1 root root 32 May 6 07:00 27 drwxr-xr-x 1 root root 32 May 6 08:00 28 drwxr-xr-x 1 root root 32 May 6 09:01 29 drwxr-xr-x 1 root root 32 May 5 07:00 3 drwxr-xr-x 1 root root 32 May 6 10:00 30 drwxr-xr-x 1 root root 32 May 6 11:00 31 drwxr-xr-x 1 root root 32 May 6 12:00 32 drwxr-xr-x 1 root root 32 May 5 08:01 4 drwxr-xr-x 1 root root 32 May 5 09:00 5 drwxr-xr-x 1 root root 32 May 5 10:01 6 drwxr-xr-x 1 root root 32 May 5 11:01 7 drwxr-xr-x 1 root root 32 May 5 12:01 8 drwxr-xr-x 1 root root 32 May 5 13:01 9
root@nas:/data/Transmission/.snapshots# cd 5/snapshot
root@nas:/data/Transmission/.snapshots/5/snapshot# touch test touch: cannot touch `test': Read-only file system
root@nas:/data/Transmission/.snapshots/5/snapshot# cd save/config
root@nas:/data/Transmission/.snapshots/5/snapshot/save/config# ls blocklists dht.dat email_alert resume save settings.json stats.json torrents
root@nas:/data/Transmission/.snapshots/5/snapshot/save/config# rm *json rm: cannot remove `settings.json': Read-only file system rm: cannot remove `stats.json': Read-only file system
So if you have been somehow creating snapshots of some kind and modifying them, how are you accomplishing that? I haven't tried using SMB for snapshot access, but I'm wondering if the apparent file permissions suggest it can do something it can't, appear to let you do it, but never really commit that change... because that subvolume (for the snapshot) is indeed read-only.
... I have been deleting these from one snapshot at a time, by hand accessing the NAS from my PC. ...
I noticed you mentioned this a few times in this thread. At first, I was thinking "he enabled snapshot access in the share properties and is trying to delete the files that way?" but then it occurred to me that this wouldn't magically make the snapshots writable. Even down in the OS layer, the snapshots are fundamentally read-only.
So if you have been somehow creating snapshots of some kind and modifying them, how are you accomplishing that? I haven't tried using SMB for snapshot access, but I'm wondering if the apparent file permissions suggest it can do something it can't, appear to let you do it, but never really commit that change... because that subvolume (for the snapshot) is indeed read-only.
Until a few months ago, I knew nothing about NAS boxes and I am not a Linux expert, so I cannot answer your question on how it was accomplished. I don't even know if it is meant to work that way. All I know (in this regard) is that my A: drive is setup as a share on the NAS box and I can see my data and a snapshot for each share. Within the snapshots are dated folders and files can be deleted from there. I didn't think this was NOT supposed to happen. Perhaps I was naïve in expecting data that was put there from the PC to be able to be deleted from the PC. Seemed a reasonable expectation at the time. If you can explain how I get an image into this post, I will send an image which shows file properties with 'full control' for everyone against a typical file on the NAS, viewed from my PC. I obviously slice tomatoes different to everyone else.
... my A: drive is setup as a share on the NAS box and I can see my data and a snapshot for each share. Within the snapshots are dated folders and files can be deleted from there. ... Perhaps I was naïve in expecting data that was put there from the PC to be able to be deleted from the PC. Seemed a reasonable expectation at the time.
Hmm... this may be something other than the NAS snapshots then. I'll try turning on snapshot access for a share or two and see if it presents itself that way on mine. My understanding of how that works is that the snapshots would show up in a subfolder of the share, default "snapshot". But I can't really check how it appears within that until I'm home. 😞
BaJohn wrote:
If you can explain how I get an image into this post, I will send an image which shows file properties with 'full control' for everyone against a typical file on the NAS, viewed from my PC. I obviously slice tomatoes different to everyone else.
I'm not aware of an easy way to embed images or attach files in this forum. When I've done it, I post the image in my public dropbox folder, and then link to it using the "img" tag here. Wish there was an easier way. Maybe there is?
Interesting indeed. Certainly I confirmed this on the NAS
# btrfs property get /data/media/.snapshots/30/snapshot ro ro=true
But on another Linux box, I find all my snapshots are by default read-write. Need to see if I can default that the other way. Given how these are created, they really shouldn't be modifiable. Though it might be handy if some were.
# btrfs property get /fedora20_20141215 ro ro=false
After reading a bit, it seems that support for marking btrfs snapshots read-only was a much more recent addition. Used to be they were always writable. 🙂 Anyway, I did confirm that the "snapshot" folder that enabling access on the share does indeed NOT allow for deleting files. So, BaJohn, I suspect that whatever you are seeing is not coming from the NAS... perhaps something running on the PC is creating those folders you see? And in that case, perhaps you might well delete a common file from all those subdirectories.
Certainly I confirmed this on the NAS today (see below). What I find interesting is that you say that is what we should NOT be doing, and then show us how to do it :o. I'm not a Linux expert, so would not be going down that route anyway.
btaroli wrote:
But on another Linux box, I find all my snapshots are by default read-write. Need to see if I can default that the other way. Given how these are created, they really shouldn't be modifiable. Though it might be handy if some were.
It's nice to know that someone else might find the facility I am suggesting useful.
btaroli wrote:
So, BaJohn, I suspect that whatever you are seeing is not coming from the NAS... perhaps something running on the PC is creating those folders you see?
I double checked today and No the snapshots are NOT deletable from the PC, they only appear to be deleted. When viewing on the PC, the properties are 'full access' to everyone and if you highlight and delete, it asks "do you want to permanently delete this file - Yes or No"?. When I click yes, the screen refreshes with the file shown as deleted from the snapshot folder. No other messages about it did not happen, or about needing authority or anything. It seems to me that the PC (Windows 7 Ultimate 64bit) and the NAS (RN516) are NOT talking to each other in the same language, as the NAS knows I cannot delete the snapshot file, BUT the PC thinks it can. Wierd ay, perhaps it's a case of "Can't see the wood for the B... Trees."
