NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Slow rsync between esata and /data
Because I do not have a Gigabit Switch so I thought I'd transfer data from my external HDD (NTFS) drives to /data with rsync (all done on the NAS by ssh into the NAS as root and run the command), but the speed seems to be quite disappointing
What do you think is holding it back ?, I have done some research and I believe it has something to do with the btrfs file format ? and because of this format it needs the CPU to do something to the files (xor-ing or something) and thus the CPU is holding back the speed ? or the speed is slow because this is NTFS , top did show high CPU usage for mount.ntfs ?
Top is showing this
Am I doing something wrong, or this is expected ? I did however tried copy files from my OS to the NAS via a Gigabit Switch earlier and the speed was steady around 40MB/s - I thought rsycn between esata and /data internally should atleast match that ?
7177991340 100% 14.76MB/s 0:07:43 (xfer#1, to-check=862/864)
What do you think is holding it back ?, I have done some research and I believe it has something to do with the btrfs file format ? and because of this format it needs the CPU to do something to the files (xor-ing or something) and thus the CPU is holding back the speed ? or the speed is slow because this is NTFS , top did show high CPU usage for mount.ntfs ?
Top is showing this
top - 18:33:46 up 24 min, 2 users, load average: 2.64, 2.75, 2.05
Tasks: 123 total, 2 running, 121 sleeping, 0 stopped, 0 zombie
%Cpu(s): 32.3 us, 55.7 sy, 0.0 ni, 0.0 id, 5.0 wa, 0.0 hi, 7.0 si, 0.0 st
KiB Mem: 507240 total, 481764 used, 25476 free, 81304 buffers
KiB Swap: 1047420 total, 0 used, 1047420 free, 203536 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1912 root 20 0 30352 1740 388 R 26.0 0.3 5:03.21 rsync
1910 root 20 0 30456 2228 880 S 25.4 0.4 5:23.83 rsync
1712 root 20 0 5628 1300 668 S 22.5 0.3 2:55.49 mount.ntfs
850 root 20 0 0 0 0 S 6.4 0.0 1:03.65 md127_raid5
3 root 20 0 0 0 0 S 2.6 0.0 0:05.46 ksoftirqd/0
1480 root 20 0 0 0 0 S 1.6 0.0 0:25.09 flush-btrfs-1
6 root -2 0 0 0 0 S 1.0 0.0 0:07.71 rcu_kthread
236 root 20 0 0 0 0 S 1.0 0.0 0:09.94 kswapd0
1630 guest 20 0 85916 12m 4356 S 1.0 2.4 0:08.74 transmission-da
1479 root 19 -1 173m 25m 5804 S 0.6 5.1 0:21.43 readynasd
1706 root 20 0 5368 1580 1056 R 0.6 0.3 0:07.07 top
Am I doing something wrong, or this is expected ? I did however tried copy files from my OS to the NAS via a Gigabit Switch earlier and the speed was steady around 40MB/s - I thought rsycn between esata and /data internally should atleast match that ?
4 Replies
Replies have been turned off for this discussion
- Voidness,
You have not given a NAS model# or F/ware rev. I guess you have OS6.
If HDD was plugged into a PC, could you do a similar data copy for comparison.?
Thanks,Marto - I have a ReadyNAS 104 / FW 6.1.8 - copying files from my own Linux Box > Gigabit Router > NAS I did achieved expected speed ~ 40MB/s, but not rsync locally between esata and /data
No.voidness wrote: What do you think is holding it back ?, I have done some research and I believe it has something to do with the btrfs file format ? and because of this format it needs the CPU to do something to the files (xor-ing or something) and thus the CPU is holding back the speed ?
[quote="voidness"or the speed is slow because this is NTFS , top did show high CPU usage for mount.ntfs ?[/quote]. Again no. SmallNetBuilder.com got ~35 MB/s in their backup test to ntfs (though they were probably not using rsync).
I am thinking that you should probably try NFS to make the backup.- The CPU is simply saturated by using the combination of (full-)rsync and ntfs
%Cpu(s): 32.3 us, 55.7 sy, 0.0 ni, 0.0 id, 5.0 wa, 0.0 hi, 7.0 si, 0.0 st
5.0% waiting => theres not much time spent waiting on the disk, your not I/O Bound
the nas is spending roughly 1/3 of its time in each rsync process, and in the ntfs process
ntfs performance is cpu bound in the linux implementation, AFAIK. I simply think that code isn't very optimized (and hard to do, as they had to reverse engineer it. If i recall correctly, ntfs still involves userspace parts, which means there is a lot of kernel-userspace-kernel transition needed.)
rsync does heavy-checksumming on default, to see which parts of a file might have changed.
You're using rsync from the webgui as abackup job? then i think you can't modify the options...
if you can, try "--sizes-only". This will degrade rsync to a copy, that will assume files haven't changed when they're the same size in destination as in source, but this should speed up the process quite a bit...
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
