NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Push Gigabit NIC teaming to the limit for NAS backup in SOHO
I have both Pro4 and Pro6 NAS unit which both support NIC teaming. However I'm not happy about the NAS to NAS backup speed, For full backup, around 30-40MB/s is what I can get, teaming or not. How to leverage NIC team to speed up file copying? I suppose this is one of the FAQ in NAS community. I searched various NAS community from different vendors, and didn't get answers. Some veterans even claim that you will never get beyond 1Gbps for point to point communication(aka one PC to one NAS, or one NAS to one NAS, of course, we are not talking about 10Gbps NIC). I managed to find out a way to break through the limitation. Let's jump to the numbers at first.
single huge file copy from NAS to NAS: around 130-140MB/s, Note: both NAS are 7200rpm HDD in RAID0
two concurrent files copy from NAS to NAS: around 85MB+85MB/s, Note: both files are on different RAID0 volume
memory copy across the switch : stay at 220MB/s, Note: benchmark mode, copy from local /dev/zero to remote /dev/null, in case storage system performance is the bottleneck
How ?
ReadyNAS(or linux box with decent CPU/NIC) : build NIC teaming with 2 NICs, Balanced XOR mode(for static LAG) or 802.3ad LACP mode(for dynamic LAG). xmit_hash_policy must be Layer3+4
Switch: Cisco SG200-08 (or SG200-08P, no SG200-18, 26 etc, they run different firmware). configure LAG as normal, LAG settings-Load Balance Algorithm must be "Src/Dest IP and TCP/UDP Port Fields" Some managed switches which support layer3+4 hashing should work as well, which are listed in the last page of this doc: http://www.readynas.com/wp-content/uploads/2008/09/ReadyNAS-Teaming.pdf . But whatever these switches are not in my list.
Tool: bbcp (binary available at http://www.slac.stanford.edu/~abh/bbcp/ , usage tutorial at http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm) which could launch multiple connections for single file and whole directory. You need to use ssh to install bbcp on BOTH NAS(/usr/bin folder) and then execute it.
the command I use to backup files is
bbcp -P 2 –s 8 /c/hugefile root@192.168.1.2:/c/hugefile
the command I use to run benchmark memory copy is
bbcp -P 2 –s 8 /dev/zero root@192.168.1.2:/dev/null
Any findings and comments are welcome.
single huge file copy from NAS to NAS: around 130-140MB/s, Note: both NAS are 7200rpm HDD in RAID0
two concurrent files copy from NAS to NAS: around 85MB+85MB/s, Note: both files are on different RAID0 volume
memory copy across the switch : stay at 220MB/s, Note: benchmark mode, copy from local /dev/zero to remote /dev/null, in case storage system performance is the bottleneck
How ?
ReadyNAS(or linux box with decent CPU/NIC) : build NIC teaming with 2 NICs, Balanced XOR mode(for static LAG) or 802.3ad LACP mode(for dynamic LAG). xmit_hash_policy must be Layer3+4
Switch: Cisco SG200-08 (or SG200-08P, no SG200-18, 26 etc, they run different firmware). configure LAG as normal, LAG settings-Load Balance Algorithm must be "Src/Dest IP and TCP/UDP Port Fields" Some managed switches which support layer3+4 hashing should work as well, which are listed in the last page of this doc: http://www.readynas.com/wp-content/uploads/2008/09/ReadyNAS-Teaming.pdf . But whatever these switches are not in my list.
Tool: bbcp (binary available at http://www.slac.stanford.edu/~abh/bbcp/ , usage tutorial at http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm) which could launch multiple connections for single file and whole directory. You need to use ssh to install bbcp on BOTH NAS(/usr/bin folder) and then execute it.
the command I use to backup files is
bbcp -P 2 –s 8 /c/hugefile root@192.168.1.2:/c/hugefile
the command I use to run benchmark memory copy is
bbcp -P 2 –s 8 /dev/zero root@192.168.1.2:/dev/null
Any findings and comments are welcome.
2 Replies
Replies have been turned off for this discussion
- Well, one key thing users here need to understand is that bbcp uses multiple streams. With teamed gigabit connections, any single stream is limited to 1 gbps.
I think the main issue is that you need a switch that supports layer 3+layer 4 hashing. Mine (GS724T) doesn't let me set the hashing mode.
One concern on layer 3+layer 4 hashing is whether un-teamed clients will experience packet loss. If you disconnect an ethernet cable on one of the NAS and re-run your benchmarks, what performance do you see? Is there any evidence of packet loss in the switch stats? - Not sure whether anybody get similar result with my configurations. For myself, I copied over 10TB data between two NAS(a few hard drive RAID set) in last week. Generally I'm able to achieve at least 130MB/s, the highest result is 180MB/s from 4-disk RAID0 set to 6 disk RAID5 set for 2.5TB data. Considering the cost of SG200-08(less than $100 from local reseller), I'm very happy about the result.
Something I need to mention
1. bbcp is not for novice NAS user, need basic linux knowledge. After bbcp copying, some file ownership/permission fix is required.
2. bbcp could bs installed on both OS4 and OS6, however, if the destination NAS box runs OS6, bbcp performance is very poor(30-40MB/s at most). If same box run OS4, everything OK. I had tried PC run Debian 7 x86_64 as destination, everything OK as well. OS6 box as source is OK as well. Not sure what changes ReadyNAS developers commited to kernel or network settings.
3. bbcp could be slow for huge amounts of small files, the workaround is to use tar as pipe on both side. An interesting article to read: http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html also covered other file copy tools available.
StephenB, regarding your question, if I disconnect one cable on destination during the benchmark testing, the throughput numbers will drop slowly, but after I connected it back , the throughput numbers will get back slowly. Tested under dynamic LAG mode. If the benchmark test is start with one cable on destination only, the throughput is around 90MB/s, less than expectation of single ethernet connection.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
