Copy Speed Test Results to USB 3.0 Drive w/ SSD - why so slow?

Question

NAS: RN102&nbsp;I need to copy off about 3TB of data...I know not a ton, but was curious to test various copy methods to see which was fastest and just to learn a little more in the process.&nbsp;SOURCE:&nbsp;The NAS disks are WD Red 5,400rpm, capable of around 110 MB/s reads each. This is a RAID 1 of 2 disks, so assuming it could read in parallel from each disk to have a potential of double that right?&nbsp;DESTINATION:A usb NVME drive to the rear USB 3.0 port. It is a Samsung 970 Evo SSD with write speeds of around 2,500 MB/s .&nbsp;TEST RESULTS:ssh &gt; rsync: 7.5 MB/sssh &gt; cp: 18 MB/sWindows Drag and Drop: 18 MB/sWindows command prompt &gt; Robocopy: 64 MB/s&nbsp;Here are the specific commands in case the switches have an impact.In all cases, the destination file did not exist.rsync -hvrPt --info=progress2 --ignore-existing "/data/Videos/2022/12/2022-12-20-063917.mkv" "/media/USB_HDD_7/Videos/2022/12/"&nbsp;cp -r -u -v "/data/Videos/2022/12/2022-12-20-063917.mkv" "/media/USB_HDD_7/Videos/2022/12/"&nbsp;Robocopy \READYNAS\Videos\2022\12 \Readynas\usb_hdd_7\Videos\2022\12 2022-12-20-063917.mkv /s /zb /r:1 /w:1 /XO /MT:32&nbsp;The BUT WHYs 🙂 ?But Why is rsync so slow and since it is via ssh, I thought this was a direct copy from the internal disks &gt; usb port &gt; NVME ssd without a round trip over the network to the client and back?&nbsp;Interestingly cp, also via ssh and hence I believe a direct hardware copy like rsync, was about 2.5 times faster than rsync.&nbsp;Windows drag and drop...isn't this a copy over the network, to the client and then back? NAS &gt; Windows Client &gt; NAS &gt; USB Port &gt; NVME SSD? This shocked me!&nbsp;Robocopy, wouldn't this be over the network too? Yet in the end it was 8+x faster than ssh rsync?????&nbsp;But Why, since it is a direct link over USB and sourced from 2 disks in RAID 1, was the speed not near the theorhetical 220 MB/s (read from 2 disks in parallel at 110 MB/s each)? I tried copying multiple files to try to take advantage of this, but&nbsp;I guess these copy commands are sequential in nature...I'll have to look to see how you could actually copy 2 files in parallel to test this aspect further.&nbsp;So I guess outside of parallel reads from the RAID 1, the speed should still be closer to 110 MB/s (the read limit of the source disks being the bottleneck), no?

StephenB · Answer

The slow CPU and limited memory in the RN102 likely is also a factor.
&nbsp;
chadwixk&nbsp;wrote:
&nbsp;
SOURCE:&nbsp;
The NAS disks are WD Red 5,400rpm, capable of around 110 MB/s reads each. This is a RAID 1 of 2 disks, so assuming it could read in parallel from each disk to have a potential of double that right?
&nbsp;

Theoretically, yes.&nbsp; But from what I've read (posts on other forums), a single file is only read from one disk.&nbsp; A second file copy should be read from the second disk if it is done at the same time.&nbsp; But this is not something I've tested.
&nbsp;
chadwixk&nbsp;wrote:
&nbsp;
But Why, since it is a direct link over USB and sourced from 2 disks in RAID 1, was the speed not near the theorhetical 220 MB/s (read from 2 disks in parallel at 110 MB/s each)? I tried copying multiple files to try to take advantage of this, but&nbsp;I guess these copy commands are sequential in nature...I'll have to look to see how you could actually copy 2 files in parallel to test this aspect further.
&nbsp;

Use two ssh sessions to test this.
&nbsp;
chadwixk&nbsp;wrote:
&nbsp;
DESTINATION:
A usb NVME drive to the rear USB 3.0 port. It is a Samsung 970 Evo SSD with write speeds of around 2,500 MB/s .

But USB 3.0 is ~600 MB/s, so it would be the write bottleneck.&nbsp; Or possibly the NVME adapter you used???
&nbsp;
Did you measure the read and write speeds on the RN102 using dd?&nbsp; Might be worth doing (for both source and destination).
&nbsp;
FWIW, the NVME formatting might also be relevant here.&nbsp; NTFS is likely slower than ext.
&nbsp;
chadwixk&nbsp;wrote:
&nbsp;
Windows drag and drop...isn't this a copy over the network, to the client and then back? NAS &gt; Windows Client &gt; NAS &gt; USB Port &gt; NVME SSD?&nbsp;
&nbsp;

Yes.&nbsp; As is robocopy.
&nbsp;
chadwixk&nbsp;wrote:
rsync -hvrPt --info=progress2 --ignore-existing "/data/Videos/2022/12/2022-12-20-063917.mkv" "/media/USB_HDD_7/Videos/2022/12/"
&nbsp;

No one ever accused rsync of being fast.&nbsp; It also verifies, which will of course make quite a difference in transfer times.&nbsp; But might be worth the performance penalty.
&nbsp;
If you add --checksum-choice="None", it should be somewhat faster.
&nbsp;
chadwixk&nbsp;wrote:
&nbsp;
TEST RESULTS:
ssh &gt; cp: 18 MB/s
cp -r -u -v "/data/Videos/2022/12/2022-12-20-063917.mkv" "/media/USB_HDD_7/Videos/2022/12/"
&nbsp;

This one is surprising.&nbsp; It might be worth copying to /dev/null to test the read speed (in addition using dd to test read and write speeds of both devices).

chadwixk · Answer

Thank you&nbsp;StephenB&nbsp;for your time in responding and helping me learn and investigate this.&nbsp;&lt;&nbsp;I see how to quote you're entire reply, but I don't know how to break up the quote to then insert my replies...curious how if you don't mind...so bear with my formatting below to mimic this...well I could use the raw html editing, but that would take a while...and I'm assuming there is an easier way to do this how you did in your replies &gt;&nbsp;The slow CPU and limited memory in the RN102 likely is also a factor.&nbsp;I'm not a hardware or linux guy (more a microsoft web dev), so not sure if I'm interpreting these properly, but CPU and Mem or not at 100%, but maybe they are effectively at 100%, limiting throughput?&nbsp;&nbsp;Theoretically, yes.&nbsp; But from what I've read (posts on other forums), a single file is only read from one disk.&nbsp; A second file copy&nbsp;should&nbsp;be read from the second disk if it is done at the same time.&nbsp; But this is not something I've tested.&nbsp;Tried this, one session running RoboCopy and one running cp, IOSTAT still shows a 20MB/s read and write (in this view CPU also does seem to be pegged).&nbsp;But USB 3.0 is ~600 MB/s, so it would be the write bottleneck.&nbsp; Or possibly the NVME adapter you used???&nbsp;True, but we're waaaay less than the 600MB/s bottleneck. The NVME adapter is well doing well over 1,000MB/s from Crystal Disk tests.&nbsp;Did you measure the read and write speeds on the RN102 using dd?&nbsp; Might be worth doing (for both source and destination).&nbsp;Looked that up, I'll try after my 2 current copy sessions end.&nbsp;FWIW, the NVME formatting might also be relevant here.&nbsp; NTFS is likely slower than ext.&nbsp;True, but I can't imaging that would reduce the 110 MB/s read of the disks down to 1/5.&nbsp; Also doesn't account for the wide range of speeds seen in the test results.&nbsp;This one is surprising.&nbsp; It might be worth copying to /dev/null to test the read speed (in addition using dd to test read and write speeds of both devices).This was in response to:&nbsp;ssh &gt; cp: 18 MB/scp -r -u -v "/data/Videos/2022/12/2022-12-20-063917.mkv" "/media/USB_HDD_7/Videos/2022/12/"&nbsp;Just curious why this one was surprising to you...because it was also ssh and 2.5x the speed of rsync?&nbsp;To me, the Biggest Surprise&nbsp;is how an over-the-network copy was several factors faster than a direct device copy?!?!? That makes zero sense...in fact opposite&nbsp;a reasonable hypothesis.

chadwixk · Answer

Actually, I missed the 12% si utilization on the CPU summary in the TOP command pic. So CPU is pegged.

Another interesting thing I see on IOSTAT output is the %utilization of the read HDDs vs the write SSD and also the WAIT on the SSD?

The HDDs show pretty low utilization, and it's % util sort of corresponds with the % of actual to rated read speed...what would be holding this back? CPU/Memory is not designed in this such as to take full advantage of these slow HDDs?

The SSD shows very high utilization when the throughput is way less than capacity. I have no theories as to why this would be. Also note the WAITs for this. No clues on my end.

StephenB · Answer

chadwixk&nbsp;wrote:
I see how to quote you're entire reply, but I don't know how to break up the quote to then insert my replies...curious how if you don't mind...so bear with my formatting below to mimic this...well I could use the raw html editing, but that would take a while...and I'm assuming there is an easier way to do this how you did in your replies 
&nbsp;

You can delete stuff out of the quote, and you can quote more than once.&nbsp; There are some situations where the forum software misbehaves - when that happens I work around the issue using the html model in the expanded toolbar.
&nbsp;
One thing that can't be quoted is inserted code - &lt;/&gt; in the toolbar.&nbsp; Netgear changed the way that works a couple months ago, and introduced a bug.&nbsp; (&nbsp;Marc_V&nbsp;,&nbsp;JeraldM&nbsp;: it would be very nice if that was fixed!).
&nbsp;
chadwixk&nbsp;wrote:
I'm not a hardware or linux guy (more a microsoft web dev), so not sure if I'm interpreting these properly, but CPU and Mem or not at 100%, but maybe they are effectively at 100%, limiting throughput?

These numbers can be hard to interpret.&nbsp; In your case, the load average is quite high (over 4).&nbsp; That suggests that you are CPU bound.

https://www.site24x7.com/blog/load-average-what-is-it-and-whats-the-best-load-average-for-your-linux-servers

chadwixk&nbsp;wrote:
&nbsp;
Thank you&nbsp;StephenB&nbsp;for your time in responding and helping me learn and investigate this.
This was in response to:&nbsp;
ssh &gt; cp: 18 MB/s
cp -r -u -v "/data/Videos/2022/12/2022-12-20-063917.mkv" "/media/USB_HDD_7/Videos/2022/12/"
&nbsp;
Just curious why this one was surprising to you...because it was also ssh and 2.5x the speed of rsync?

Surprisingly slow.
&nbsp;
FWIW, I don't have a comparable NMVE drive to what you have.&nbsp; My RN102 is for testing only, and has two 1 TB Seagate Ironwolf drives in it.&nbsp; When I copy a ~1 GB file between two shares, I see
&nbsp;
root@RN102:/data/Pictures# rsync -P /data/Videos/test.avi /data/Pictures
test.avi
    944,136,192 100%   20.34MB/s    0:00:44 (xfr#1, to-chk=0/1)
root@RN102:/data/Pictures#

&nbsp;
&nbsp;
and
&nbsp;
&nbsp;
root@RN102:/data/Pictures# rsync -P --checksum-choice="None" /data/Videos/test.avi /data/Pictures
test.avi
    944,136,192 100%   31.15MB/s    0:00:28 (xfr#1, to-chk=0/1)
root@RN102:/data/Pictures#
&nbsp;
So noticeably faster with --checksum-choice.
&nbsp;
cp is faster:
&nbsp;
root@RN102:/data/Pictures# time cp /data/Videos/test.avi /data/Pictures

real    0m12.461s
user    0m0.030s
sys     0m7.330s
root@RN102:/data/Pictures#

&nbsp;
Doing the math on the total times, I get 70 MB/sec for cp throughput.
&nbsp;
Testing the read speed with cp, I see this
&nbsp;
root@RN102:/data/Pictures# time cp /data/Videos/test.avi /dev/null

real    0m5.359s
user    0m0.020s
sys     0m3.400s
root@RN102:/data/Pictures#
&nbsp;
Computed throughput here works out to about 160 MB/sec.&nbsp;&nbsp;
&nbsp;
&nbsp;
FWIW, dd read speed depends on the block size:
&nbsp;
root@RN102:/data/Pictures# time dd if=/data/Pictures/test.avi of=/dev/null
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.5109 s, 69.2 MB/s

real    0m15.536s
user    0m2.120s
sys     0m11.180s

root@RN102:/data/Pictures# time dd if=/data/Pictures/test.avi of=/dev/null bs=1M
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.7195 s, 188 MB/s

real    0m5.727s
user    0m0.040s
sys     0m3.920s
&nbsp;
The first test uses a default block size (512 bytes), the second uses 1M.&nbsp;
&nbsp;
&nbsp;
dd write speed similarly depends on block size (and is much slower for the 512 byte size).
&nbsp;
root@RN102:/data/Pictures# time dd if=/dev/zero of=/data/Pictures/test.avi bs=512 count=2097152
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.2124 s, 13.1 MB/s

real    1m22.276s
user    0m2.610s
sys     1m9.120s
root@RN102:/data/Pictures#

root@RN102:/data/Pictures# time dd if=/dev/zero of=/data/Pictures/test.avi bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.21715 s, 116 MB/s

real    0m9.263s
user    0m0.030s
sys     0m5.700s
root@RN102:/data/Pictures#
&nbsp;
&nbsp;
&nbsp;
I don't know what block size cp is using on the NAS, but I can approximately match the cp speed with a block size of 1M in dd. (Actually any block size over 128k gives me about 70 MB/s).
&nbsp;
root@RN102:/data/Pictures# time dd if=/data/Videos/test.avi of=/data/Pictures/test.avi bs=1M
900+1 records in
900+1 records out
944136192 bytes (944 MB, 900 MiB) copied, 13.7353 s, 68.7 MB/s

real    0m13.744s
user    0m0.030s
sys     0m6.930s
root@RN102:/data/Pictures#
&nbsp;
&nbsp;
&nbsp;
rsync appears to be using 128K (which is the max block size you can specify with -B).&nbsp; But still much slower than cp (or dd).

Sandshark · Answer

Your NAS is also quite limited in RAM, so Linux cannot assign as many read/write buffers as it might want to.&nbsp; I'm not sure if it's a big contributor, but it's surely a factor.

Forum Discussion

Copy Speed Test Results to USB 3.0 Drive w/ SSD - why so slow?

5 Replies

Related Content

AX1800 (RAX20) network speed testing results

CM1000v2 poor speed test results

CM1000v2 having poor speed test results

Wifi 7 speeds vs Wifi 6E speeds (Samsung Phones)

SFP speed

NETGEAR Academy

ProSupport for Business