choice of file protocol (AFP vs CIFS vs NFS)

As a point of comparison to the prior post, here are the same tests performed using the same client and same ReadyNAS server, this time with jumbo frames enabled on both ends. Note that the ReadyNAS thinks jumbo frames means "MTU is 7936 bytes". Mac OS thinks jumbo frames mean "MTU is 9000 bytes". That won't work. If you use TCP (which AFP and CIFS do), path MTU discovery will mean the machines will automatically negotiate the smaller MTU that works. If you use UDP (which NFS does), all the traffic will get dropped on the floor and you're in for some confusing troubleshooting.

Note again that the same relative performance between the remote filesystem protocols holds -- AFP has the fastest reads, CIFS gets penalized heavily for small reads and writes, and NFS comes out somewhere in between, beating AFP in a couple tests and losing in others.


Testbed:

Client testbed: Mac Pro, OS X 10.4.9, 5GB RAM.
Server testbed: MacBook Pro, OS X 10.4.9, 2GB RAM.
Server testbed: ReadyNAS NV, 1GB RAM, 3.01c1-p6 firmware, AFP update, oplocks update.
ReadyNAS disk hardware: 4x Seagate Barracuda 7200.10 drives, 400GB each, configured as one X-RAID volume.
Network: gigabit ethernet, linksys switch, jumbo frames (MTU=7936).


Test methodology:

Same as above.


Test results:

Large writes: 1GB, written 1MB at a time.
(dd bs=1048576 count=1024 if=/dev/zero of=/t/afp/testfile)

AFP: 132.2 sec (8.1 MB/sec) - small MTU was 148.7 sec (7.2 MB/sec)
CIFS: 101.1 sec (10.6 MB/sec) - small MTU was 157.7 sec (6.8 MB/sec)
NFS: 140.5 sec (7.6 MB/sec) - small MTU was 137.5 sec (7.8 MB/sec)


Small writes: 10MB, written 1KB at a time.
(dd bs=1024 count=10000 if=/dev/zero of=/t/afp/testfile)

AFP: 0.95 sec (10.8 MB/sec) - small MTU was 1.1 sec (9.6 MB/sec)
CIFS: 7.8 sec (1.3 MB/sec) - small MTU was 7.5 sec (1.4 MB/sec)
NFS: 1.2 sec (8.4 MB/sec) - small MTU was 1.2 sec (8.4 MB/sec)

The conclusion holds -- small transactions are heavily penalized under CIFS (at least with Apple's implementation).

Small reads: 100MB, read 1KB at a time.
(dd bs=1024 count=100000 of=/dev/null if=/t/afp/testfile)

AFP: 17.6 sec (5.8 MB/sec) - small MTU was 7.5 sec (13.7 MB/sec)
CIFS: 65.6 sec (1.6 MB/sec) - small MTU was 64.0 sec (1.6 MB/sec)
NFS: 8.6 sec (11.9 MB/sec) - small MTU was 7.3 sec (14.0 MB/sec)


Large reads: 1GB, read 1MB at a time.
(dd bs=1048576 count=1024 of=/dev/null if=/t/afp/testfile)

AFP: 200.0 sec (5.4 MB/sec) - small MTU was 76.7 sec (14.0 MB/sec).
CIFS: 81.9 sec (13.1 MB/sec) - small MTU was 78.6 sec (13.6 MB/sec).
NFS: 90.2 sec (11.9 MB/sec) - small MTU was 89.3 sec (12.0 MB/sec).


Conclusion:

Effects of jumbo frames on each protocol: CIFS gains significantly on large writes, and was slightly slower on other tests. NFS loses a little across the board. AFP got a little faster on writes, but much slower on reads. Given the configuration hassle and mixed results, I'm not sure jumbo frames are a win.

Relative protocol results with jumbo frames: Even with jumbo frames enabled, AFP still looks like it performs the best on Mac OS.