Re: Copy files/folders via dashboard OS6.10.3 / RN214

aks-2 · ‎2020-09-28

After transferring files from a NV+v2 to a RN214 in a bulk 'backup' transfer, I then wanted to move the files and folders to specic areas on the RN214. I did this via the dashboard>shares.

Firstly, my old NV+v2 used OS5 and I mostly managed files/folders via SSH mv when I needed to move large data sets. Obviously, the moves were pretty instant, even on this older platform. Because OS6 works a bit differently, translate that to "I don't know the quirks with OS6 ", I decided to use the web I/F instead - drag+drop.

Moving files takes an absolute age, 39GB took ~20 minutes, that's effectively 2GB/min or 33MB/sec. Wow, that seems extremely slow for an on drive move. However, I was moving from a share to a user home area, and unlike OS5 that uses ext4 (I think), OS6 is using BTRFS, and user home areas an created differently (again, I think).

Nothing else was active on the RN214, and it has OS6.10.3 installed.

Via ssh I could not see a copy/move process, but I did see this "fsbroker.cgi" process during moves:

20928 ?        Dsl    0:08 /frontview/lib/fsbroker.cgi

This is clearly not an efficient way to move files. Thoughts?

StephenB · ‎2020-09-28

I haven't timed copying files in the web ui, and usually just use drag/drop.

Do you have snapshots enabled on the source share? If you did, then a move would result in the original files ending up in a snapshot on the source, and with a copy placed in the destination. Free space would be reduced as a result.

In general, moves using drag-and-drop are instant when you are reorganizing a share, but moves turn into copy-delete when you are moving files between shares.

The absolute fastest way to move files quickly is to use cp --reflink from ssh, and then delete the originals. The --reflink bit tells btrfs not to copy the datablocks, but instead to clone the metadata.

aks-2 · ‎2020-09-28

No snapshots, just plain shares, of course from a share to a home share breaks the boundary you mention.

But, 2GB/min seems extremely slow, it's even slower than a LAN copy! I'll do some more testing, so far this is not impressing me over the freedom and flexibility of OS5!

StephenB · ‎2020-09-28

@aks-2 wrote:

But, 2GB/min seems extremely slow, it's even slower than a LAN copy! I'll do some more testing, so far this is not impressing me over the freedom and flexibility of OS5!

It does seem very slow. I just did a copy/paste of an 8.7 GB file on my own RN202 using the web browser interface, and the paste took just a few seconds.

aks-2 · ‎2020-09-29

After more testing via the web I/F, I am seeing extremely slow moves across share boundaries, in an ssh window running 'top' I see fsbroker.cgi consuming 50-60% cpu. I've no idea what this is doing, well I think we do know, read-write-delete, but why is not clear.

Moves within the same share are practically instant (1-2 seconds), as you mention.

From an ssh window, I moved ~30GB across a share boundary using the mv command, it was virtually instant. I checked that the data was really moved, and as far as I could tell it was. The directories/files looked correct, du reported the right result.

Have I created any problems with file system integrity by moving data this way in a terminal session, or will the file system just figure it out?

Sandshark · ‎2020-09-29

In OS4.x and 5.x, shares are simply directories at the Linux level. In OS6, they are BTRFS sub-volumes. While this does give you a lot more control over each share (snapshots, COW, strict sync, etc.), one piece of baggage that comes with it is that standard file moves are, as @StephenB said, a copy-and-delete function, not just moving a pointer to the data from one directory to another. Netgear could implement the use of the --reflink option for share to share copies, but the logic to make sure it's only used for that would likely not be trivial.

So, one thing to think about is Do you really need different shares?, or are different directories sufficient? If you don't need different permissions, protocols, or other things I mentioned earlier, then you probably don't need them to be separate shares.

You can, of course, continue to use the mv --reflink command from SSH as long as you are copying to an existing share. All will be fine using it, but don't create a new folder under /data (or whatever you named your volume) and expect it to show up as a share, because it won't. And don't delete one, either. You can create or remove directories within a share without any issues.

aks-2 · ‎2020-09-29

So, one thing to think about is Do you really need different shares?

Yes, this is how I manage access and media content sharing, piling everything in to one directory would be limiting.

What I don't need is all the BTRFS baggage, I don't need snapshots and teh like, but I suppose OS6 will not use ext4 or similar?

The mv I performed in the ssh terminal, was a plain mv (without --reflink), and that was super fast even across shares. Perhaps there is already an extension to the underlying mv command to add --reflink by default, I don't know.

In reality, moving stuff around is just whilst I am setting things up, getting familiar with the new platform - yes I have come to OS6 several years after many of you, but I in fact prefer the simpler (more Linux like) OS5 - I can understand that much more easily !

Thanks for the tips to you @Sandshark and @StephenB for the pointers whilst I 'discover' OS6/BTRFS for the first time.

Sandshark · ‎2020-09-30

If you don't use snapshots, then there is no downside to using a straight mv command.

Actually, what you are experiencing isn't from BTRFS per se, it's the fact that Netgear chose to make shares sub-volumes instead of just directories. But, no, OS6 has no provisions for an internal volume to be EXT or for a share to be a simple folder. More complex file structures like BTRFS and ZFS are very common in Linux today when used for storage appliances and are becoming moreso for just standard use. With drives getting bigger and bigger, the extra protection they provide is very important. So OS5 isn't so much "more Linux like" compared to OS6, it's "old Linux". Once you figure it out, it really doesn't get in the way, even at the SSH command prompt level. And the day may come when you really do end up using the extra it provides.

aks-2 · ‎2020-09-30

Thanks @Sandshark, much appreciated.