Copy/move between shares is sloooooooooooow...

This is really awesome - you saved me so much time it's not even funny!

Thanks!

For posterity, the post from that thread reads:

Skywalker wrote:

StephenB wrote: Xophile wrote: Isn't it possible to move data between shares directly with BTRFS? Each share is set up as a btrfs subvolume. That allows you to control snapshots on a share-by-share basis. A direct consequence is that moving files from one share to another is done as a copy/delete - the same as moving files between two disks on a PC. Well, not exactly. You can "instantly" copy files around between subvolumes using cp --reflink, and then remove the original. So, to move a file from the Backup share to the Documents share, you could do: # cp -a --reflink /data/Backup/file /data/Documents/ # rm /data/Backup/file

Seriously? Really?!

So which folder should I put my downloads into? Transmission downloads to the /data/Downloads/ share on my system.

Now I have to move all files using an SSH terminal? All because I am using the /data/Videos /data/Music /data/Pictures shares that were there out of the box...?

On the NV+, I could of course map to /c/ and use this to move between shares. I had assumed this was the case with the 104 - simply map to /data/

How can it be acceptable that ordinary move operations are not possible on this device? What is being done to address this? Aside from forcing users to use a single share? The fact that BTRFS has this limitation is irrelevant - I need to be able to move large files without coping them...!

rotordave wrote:
How can it be acceptable that ordinary move operations are not possible on this device? What is being done to address this? Aside from forcing users to use a single share? The fact that BTRFS has this limitation is irrelevant - I need to be able to move large files without coping them...!

You apparently missed a post further down:

Skywalker wrote:
...future firmware releases will support this sort of operation via SMB2's FSCTL_SRV_COPYCHUNK command, which is for server-side copies. But I don't know which SMB clients use that command, other than Windows Server 2012.

It seems to me that the web ui could also use the --reflink option.

The bottom line is that it works this way for now. Netgear might be able to put in limited support for --reflink in various spots. But this is a tradeoff - the snapshots are also a feature that many users want. The ability to turn snapshots on/off on a share-by-share basis is also needed, and supporting that means that moves between shares (subvolumes) are treated as copy/delete by the file system. Some users will prefer the snapshot control, others don't care about snapshots, and would rather have the direct move.

I imagine there could also be a setting when you create the share that doesn't set it up as a subvolume - permanently disabling snapshots. There'd be no way to undo that without deleting the share.

You could attempt to do this manually in ssh. Rename "Downloads" temporarily in SSH, and create a Downloads folder manually with the normal permissions. There might be some side effects, but if it works you'd end up with a downloads folder that is not a subvolume.

StephenB wrote:

You apparently missed a post further down: Skywalker wrote: ...future firmware releases will support this sort of operation via SMB2's FSCTL_SRV_COPYCHUNK command, which is for server-side copies. But I don't know which SMB clients use that command, other than Windows Server 2012. It seems to me that the web ui could also use the --reflink option. The bottom line is that it works this way for now. Netgear might be able to put in limited support for --reflink in various spots. But this is a tradeoff - the snapshots are also a feature that many users want. The ability to turn snapshots on/off on a share-by-share basis is also needed, and supporting that means that moves between shares (subvolumes) are treated as copy/delete by the file system. Some users will prefer the snapshot control, others don't care about snapshots, and would rather have the direct move.

I agree that it would be a lot simpler for Netgear to include support for the --reflink option in the GUI, particularly as they can refuse support if SSH access is enabled. As the prime purpose of a NAS is file storage and management I think this is a fairly basic request. I move files between different folders and shares on a pretty regular basis on my main machine and also a WHS server, I'd like to be able to do the same efficiently and immediately on the ReadyNAS.

I don't yet fully understand all the implications of the snapshot capability but I believe I read another thread in which a user found that his disk usage seemed too high after moving some files. It turned out that the reason was snapshot storage left over in the folder from which the files had been moved. Deleting those snapshots solved the excess usage issue.

If Netgear were to implement the "--reflink" method in the GUI for a move command and add an option to choose to either move or delete the related snapshots that would seem to solve both the above requirements you mention? As I understand it after a "--reflink" move the files would still be stored in the same physical location/blocks so the snapshots would still be relevant, just referenced from a different share?

Or am I missing a complication which makes that not possible?

Snapshots are built into btfrs at a low level, using its "copy on write" facility. Users sometimes need to delete some snapshots to manage space.

But deleting the snapshots doesn't solve the problem you are trying to solve. The shares are configured as btrfs subvolumes - that is why btrfs itself is copy/deleting. The configuration as btrfs subvolumes is what allows share by share enabling of snapshots. But the copy/delete happens even if snapshots are disabled in both shares. It's more like moving files between two drive letters on a PC. Even if the two drive letters are partitions on the same physical disk, Windows still copy/deletes if you move files from one to the other. Anyway, my setting idea upon share creation was to tell the NAS not to create the share as a subvolume at all, that instead it should create an ordinary folder.

I'm not sure I understand your question in the --reflink paragraph. The cp --reflink command tells btrfs to "clone" the file. New metadata is created which points to the data in the old file. Then the "rm" command deletes the original metadata, but since btrfs knows about the second metadata the data blocks remain allocated.

If I understand what's happening correctly, if snapshots were enabled in the two shares, then the cp --reflink has no impact at all on snapshots in either share (unless there happens to be a file with the same name already in the destination share). But the "rm" command would result in the deleted file being saved in the source snapshot.

The question is would the new files appear in the new destination snapshot ?
If I understood they wouldn't since it's only a shortcut. But still the source will keep a copy of them as deleted (which isn't itself a problem as long as you have some disk space left and you can still manually delete snapshots). Now What I don't understand is if you update those files after that they've been deleted from source, would they appear in the destination snapshot, the source snapshot or neither ? I would bet none of them, and that may be the problem.
That said Maybe moving the Data afterwards on the same share would regenerate everything needed for the snapshots (every metadata should be updated including those for snapshots), but this would need to create a temporary folder to do that.

I prefer per share snapshots myself than fast copy, you do not move that much stuff on a NAS anyway, at least not from a share to another. So worst case scenario, you spend 2 days moving the data one or twice in your NAS live time. Accidentally deleting a file, or wanting to get back a copy that was corrupted or wanting to get an older version because modifications didn't go well, that's really useful stuff for day to day business and really having that and time machine on top of that really make me feel a lot safer. I many shares, most of them with daily snapshot, some with weekly and one with monthly snapshots, so I guess using a single setting wouldn't fit my needs.

To keep the thread constructive, here is a script I wrote to at least reduce the effort to one command line:

#!/bin/bash

tail=${@:$#}
echo -- Moving $(($#-1)) files to "$tail"

for i in "$@"
do
    if [ "$i" != "$tail" ]; then
        echo -- Copying "$i" to "$tail"
        cp -av --reflink "$i" "$tail"
        echo -- Deleting "$i"
        rm -irv "$i"
    fi
done

I've left the -i switch in the "rm" command to keep it safe when testing the first time. The last argument is the destination...

Example:

script-name file1 second\ file\ with\ spaces /data/another\ share 

Some folks in other linux forums also suggest "alias cp='cp --reflink=auto --sparse=always'"
http://unix.stackexchange.com/questions ... -behaviour

I assume that this won't work across volumes such as main unit to chassis, right?