choice of file protocol (AFP vs CIFS vs NFS)

Introduction:

What's the right file sharing protocol to use between your computer and your ReadyNAS? You have the choice of at least 3 network filesystem protocols (Apple's AFP, native protocol of Macs; Microsoft's CIFS, also known as SMB, native protocol of Windows but supported by most OSes; Sun's NFS, commonly used with Unix), depending on what OS you run on your computers.

If you run Windows, the choice is probably pretty simple -- use CIFS, because Windows doesn't support AFP or NFS out of the box. But if you run Linux, it's roughly equally easy to use CIFS and NFS, and if you run Mac OS X, your computer supports all 3 -- which to choose?

The choice comes down to a combination of:
- performance -- how fast is it for the work you do? I'll attempt to answer that question here.
- features -- does your client OS support it? Does the ReadyNAS fully support it? Does it preserve things you care about, like access permissions, Spotlight metadata, other metadata? I won't touch this topic here either, except to note that the ReadyNAS won't export user shares over NFS, so if you use the ReadyNAS in User security mode, you probably can't use NFS.

If you have questions about the methodology or results, or think I did something wrong, please comment!


Goals:

- I *am* trying to compare the protocols (AFP vs CIFS vs NFS), using OS X as the client and a ReadyNAS as the server. The remote filesystem protocol is the variable in these experiments, for a bunch of different access patterns.
- I am *not* trying to measure the raw performance of the ReadyNAS.
- I am *not* trying to distinguish whose fault it is (protocol itself, Apple's client implementation, Infrant's server implementation) when something is slow or works poorly. I'm sure you could increase the raw numbers below by using jumbo frames, disabling all journaling, etc on the ReadyNAS, but I'd also expect the relative performance of each protocol would remain the same.


Testbed:

Client testbed: Mac Pro, OS X 10.4.9, 5GB RAM.
Server testbed: ReadyNAS NV, 1GB RAM, 3.01c1-p6 firmware, AFP update, oplocks update.
ReadyNAS disk hardware: 4x Seagate Barracuda 7200.10 drives, 400GB each, configured as one X-RAID volume.
Network: gigabit ethernet, linksys switch, normal Ethernet MTU (1500).


Test methodology:

I mounted the same ReadyNAS share via AFP, CIFS and NFS, using mount commands (/sbin/mount_afp, /sbin/mount_nfs, /sbin/mount_smbfs) as root, to mount the "Backup" share on the ReadyNAS as /t/afp, /t/nfs, and t/cifs, respectively.

I ran the tests with 'dd', which itself reports the transfer time. I also kept an eye on the network traffic on the OS X client, as measured by Activity Monitor or MenuMeters, during the transfers, to see if it roughly agreed with the throughput calculated by dd; if dd reported a lot of traffic and MenuMeters reported much less, I suspected caching effects on the client (see notes in the individual tests).

These are all sequential reads or writes; I'm trying to measure the network filesystem, not the ReadyNAS disk performance.

All the results here are extremely reproducible for me, plus or minus a couple percent. In cases where I suspected a caching effect, I ran the same test repeatedly to see what happened, and reported both the cached and uncached results.


Test results:

Large writes: 1GB, written 1MB at a time.
(dd bs=1048576 count=1024 if=/dev/zero of=/t/afp/testfile)

AFP: 148.7 sec (7.2 MB/sec)
CIFS: 157.7 sec (6.8 MB/sec)
NFS: 137.5 sec (7.8 MB/sec)


Small writes: 10MB, written 1KB at a time.
(dd bs=1024 count=10000 if=/dev/zero of=/t/afp/testfile)

AFP: 1.1 sec (9.6 MB/sec)
CIFS: 7.5 sec (1.4 MB/sec)
NFS: 1.2 sec (8.4 MB/sec)


Small reads: 100MB, read 1KB at a time.
(dd bs=1024 count=100000 of=/dev/null if=/t/afp/testfile)

With this test, I had to unmount and remount the share before each run, because NFS and AFP cache it too well. It's easy to tell the difference, watching traffic on the physical interface though, between a run that's filled from the cache and one that actually goes to the ReadyNAS.

AFP: 7.5 sec (13.7 MB/sec). If repeated: 0.5 sec (evidence of local caching).
CIFS: 64.0 sec (1.6 MB/sec). If repeated: same.
NFS: 7.3 sec (14.0 MB/sec). If repeated: 2.9 sec (evidence of local caching, but not cached as well as AFP).


Large reads: 1GB, read 1MB at a time.
(dd bs=1048576 count=1024 of=/dev/null if=/t/afp/testfile)

AFP: 76.7 sec (14.0 MB/sec). If repeated: 0.48 sec (obviously cached locally)
CIFS: 78.6 sec (13.6 MB/sec). If repeated: 78 seconds again (obviously NOT cached locally)
NFS: 89.3 sec (12.0 MB/sec). If repeated: 0.53 sec (obviously cached locally)


Opening 155 .jpg files (415 MB total) in Preview, via "Select All, Open" in Finder. This is basically a real-world case of mid-side reads, and something I do a lot.

local disk: Preview window opens almost immediately, displays sidebar with thumbnails about 3 seconds later, switching between images is almost instant.
AFP: Preview window opens after 2 seconds, displays sidebar with thumbnails after 7 seconds; switching between images has barely perceptible lag (about .5 sec). Repeat experiment: Preview window opens and displays thumbnail sidebar almost instantly -- cached on client side.
CIFS: Preview window opens after 5 seconds, displays sidebar with thumbnails after 14 seconds; switching between images has notable lag (approx 2 seconds). Repeat experiment: same results (not cached).
NFS: Preview window opens after 26 seconds, displays sidebar with thumbnails after 28 seconds; switching between images has barely perceptible lag (approx .5 sec). Repeat experiment: same results (not cached).

Editorial comment: The CIFS test (like all tests here) was with the Infrant "ToggleOplocks" addon installed. Without it, performance on this test was downright unusable; the oplock support helps hugely, but still doesn't come close to keeping up with AFP.


Conclusions:
- AFP: fastest reads, not far behind on writes, caches better. No pathologically bad cases.
- CIFS: good for raw I/O throughput (large reads or writes that actually have to go to the server), but bad for small reads/writes, and doesn't cache reads.
- NFS: speed is in the middle, no access to user shares. No pathologically bad cases for read performance. Note that the ReadyNAS doesn't support NFS over TCP, so it always uses UDP, so you don't get TCP's path MTU discovery, so if you mismatch MTUs on server and client, you'll get really flaky results. (In general, NFS is the least user-friendly of these protocols on OS X.)

In my opinion, AFP is the clear winner; it's cached much better by an OS X client, with significantly better read performance even if it does have to cross the network.

Again: If you have questions about the methodology or results, or think I did something wrong, please comment!