× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

choice of file protocol (AFP vs CIFS vs NFS)

magi1
Aspirant

choice of file protocol (AFP vs CIFS vs NFS)

Introduction:

What's the right file sharing protocol to use between your computer and your ReadyNAS? You have the choice of at least 3 network filesystem protocols (Apple's AFP, native protocol of Macs; Microsoft's CIFS, also known as SMB, native protocol of Windows but supported by most OSes; Sun's NFS, commonly used with Unix), depending on what OS you run on your computers.

If you run Windows, the choice is probably pretty simple -- use CIFS, because Windows doesn't support AFP or NFS out of the box. But if you run Linux, it's roughly equally easy to use CIFS and NFS, and if you run Mac OS X, your computer supports all 3 -- which to choose?

The choice comes down to a combination of:
- performance -- how fast is it for the work you do? I'll attempt to answer that question here.
- features -- does your client OS support it? Does the ReadyNAS fully support it? Does it preserve things you care about, like access permissions, Spotlight metadata, other metadata? I won't touch this topic here either, except to note that the ReadyNAS won't export user shares over NFS, so if you use the ReadyNAS in User security mode, you probably can't use NFS.

If you have questions about the methodology or results, or think I did something wrong, please comment!


Goals:

- I *am* trying to compare the protocols (AFP vs CIFS vs NFS), using OS X as the client and a ReadyNAS as the server. The remote filesystem protocol is the variable in these experiments, for a bunch of different access patterns.
- I am *not* trying to measure the raw performance of the ReadyNAS.
- I am *not* trying to distinguish whose fault it is (protocol itself, Apple's client implementation, Infrant's server implementation) when something is slow or works poorly. I'm sure you could increase the raw numbers below by using jumbo frames, disabling all journaling, etc on the ReadyNAS, but I'd also expect the relative performance of each protocol would remain the same.


Testbed:

Client testbed: Mac Pro, OS X 10.4.9, 5GB RAM.
Server testbed: ReadyNAS NV, 1GB RAM, 3.01c1-p6 firmware, AFP update, oplocks update.
ReadyNAS disk hardware: 4x Seagate Barracuda 7200.10 drives, 400GB each, configured as one X-RAID volume.
Network: gigabit ethernet, linksys switch, normal Ethernet MTU (1500).


Test methodology:

I mounted the same ReadyNAS share via AFP, CIFS and NFS, using mount commands (/sbin/mount_afp, /sbin/mount_nfs, /sbin/mount_smbfs) as root, to mount the "Backup" share on the ReadyNAS as /t/afp, /t/nfs, and t/cifs, respectively.

I ran the tests with 'dd', which itself reports the transfer time. I also kept an eye on the network traffic on the OS X client, as measured by Activity Monitor or MenuMeters, during the transfers, to see if it roughly agreed with the throughput calculated by dd; if dd reported a lot of traffic and MenuMeters reported much less, I suspected caching effects on the client (see notes in the individual tests).

These are all sequential reads or writes; I'm trying to measure the network filesystem, not the ReadyNAS disk performance.

All the results here are extremely reproducible for me, plus or minus a couple percent. In cases where I suspected a caching effect, I ran the same test repeatedly to see what happened, and reported both the cached and uncached results.


Test results:

Large writes: 1GB, written 1MB at a time.
(dd bs=1048576 count=1024 if=/dev/zero of=/t/afp/testfile)

AFP: 148.7 sec (7.2 MB/sec)
CIFS: 157.7 sec (6.8 MB/sec)
NFS: 137.5 sec (7.8 MB/sec)


Small writes: 10MB, written 1KB at a time.
(dd bs=1024 count=10000 if=/dev/zero of=/t/afp/testfile)

AFP: 1.1 sec (9.6 MB/sec)
CIFS: 7.5 sec (1.4 MB/sec)
NFS: 1.2 sec (8.4 MB/sec)


Small reads: 100MB, read 1KB at a time.
(dd bs=1024 count=100000 of=/dev/null if=/t/afp/testfile)

With this test, I had to unmount and remount the share before each run, because NFS and AFP cache it too well. It's easy to tell the difference, watching traffic on the physical interface though, between a run that's filled from the cache and one that actually goes to the ReadyNAS.

AFP: 7.5 sec (13.7 MB/sec). If repeated: 0.5 sec (evidence of local caching).
CIFS: 64.0 sec (1.6 MB/sec). If repeated: same.
NFS: 7.3 sec (14.0 MB/sec). If repeated: 2.9 sec (evidence of local caching, but not cached as well as AFP).


Large reads: 1GB, read 1MB at a time.
(dd bs=1048576 count=1024 of=/dev/null if=/t/afp/testfile)

AFP: 76.7 sec (14.0 MB/sec). If repeated: 0.48 sec (obviously cached locally)
CIFS: 78.6 sec (13.6 MB/sec). If repeated: 78 seconds again (obviously NOT cached locally)
NFS: 89.3 sec (12.0 MB/sec). If repeated: 0.53 sec (obviously cached locally)


Opening 155 .jpg files (415 MB total) in Preview, via "Select All, Open" in Finder. This is basically a real-world case of mid-side reads, and something I do a lot.

local disk: Preview window opens almost immediately, displays sidebar with thumbnails about 3 seconds later, switching between images is almost instant.
AFP: Preview window opens after 2 seconds, displays sidebar with thumbnails after 7 seconds; switching between images has barely perceptible lag (about .5 sec). Repeat experiment: Preview window opens and displays thumbnail sidebar almost instantly -- cached on client side.
CIFS: Preview window opens after 5 seconds, displays sidebar with thumbnails after 14 seconds; switching between images has notable lag (approx 2 seconds). Repeat experiment: same results (not cached).
NFS: Preview window opens after 26 seconds, displays sidebar with thumbnails after 28 seconds; switching between images has barely perceptible lag (approx .5 sec). Repeat experiment: same results (not cached).

Editorial comment: The CIFS test (like all tests here) was with the Infrant "ToggleOplocks" addon installed. Without it, performance on this test was downright unusable; the oplock support helps hugely, but still doesn't come close to keeping up with AFP.


Conclusions:
- AFP: fastest reads, not far behind on writes, caches better. No pathologically bad cases.
- CIFS: good for raw I/O throughput (large reads or writes that actually have to go to the server), but bad for small reads/writes, and doesn't cache reads.
- NFS: speed is in the middle, no access to user shares. No pathologically bad cases for read performance. Note that the ReadyNAS doesn't support NFS over TCP, so it always uses UDP, so you don't get TCP's path MTU discovery, so if you mismatch MTUs on server and client, you'll get really flaky results. (In general, NFS is the least user-friendly of these protocols on OS X.)

In my opinion, AFP is the clear winner; it's cached much better by an OS X client, with significantly better read performance even if it does have to cross the network.

Again: If you have questions about the methodology or results, or think I did something wrong, please comment!
Message 1 of 19
magi1
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

As a point of comparison to the prior post, here are the same tests performed using the same client, this time against my laptop as the server. This is mostly just for fun, and also to show that the network itself and the test client weren't at their limits when testing against the ReadyNAS. It's not relevant to directly compare the actual speeds observed between the ReadyNAS and the MacBook as the server -- the MacBook has a much faster CPU and much slower hard drive.

Note that the same relative performance between the remote filesystem protocols holds -- AFP has the fastest reads, CIFS gets penalized heavily for small reads and writes, and NFS comes out somewhere in between, beating AFP in a couple tests.


Testbed:

Client testbed: Mac Pro, OS X 10.4.9, 5GB RAM.
Server testbed: MacBook Pro, OS X 10.4.9, 2GB RAM.
Network: gigabit ethernet, linksys switch, normal Ethernet MTU (1500).


Test methodology:

Same as above.


Test results:

Large writes: 1GB, written 1MB at a time.
(dd bs=1048576 count=1024 if=/dev/zero of=/t/afp/testfile)

AFP: 38.6 sec (27.8 MB/sec) - ReadyNAS was 148.7 sec (7.2 MB/sec)
CIFS: 44.0 sec (24.4 MB/sec) - ReadyNAS was 157.7 sec (6.8 MB/sec)
NFS: 44.5 sec (24.1 MB/sec) - ReadyNAS was 137.5 sec (7.8 MB/sec)


Small writes: 10MB, written 1KB at a time.
(dd bs=1024 count=10000 if=/dev/zero of=/t/afp/testfile)

AFP: 0.15 sec (65.8 MB/sec) - ReadyNAS was 1.1 sec (9.6 MB/sec)
CIFS: 4.21 sec (2.4 MB/sec) - ReadyNAS was 7.5 sec (1.4 MB/sec)
NFS: 0.44 sec (23.1 MB/sec) - ReadyNAS was 1.2 sec (8.4 MB/sec)

That's pretty much too fast to measure, so let's try it with count=100000 (10x the size):

AFP: 3.3 sec (31.5 MB/sec)
CIFS: 42.3 sec (2.4 MB/sec)
NFS: 5.0 sec (20.6 MB/sec)

The conclusion holds -- small transactions are heavily penalized under CIFS (at least with Apple's implementation).

Small reads: 100MB, read 1KB at a time.
(dd bs=1024 count=100000 of=/dev/null if=/t/afp/testfile)

AFP: 3.4 sec (30.1 MB/sec) - ReadyNAS was 7.5 sec (13.7 MB/sec)
CIFS: 39.6 sec (2.6 MB/sec) - ReadyNAS was 64.0 sec (1.6 MB/sec)
NFS: 3.8 sec (26.9 MB/sec) - ReadyNAS was 7.3 sec (14.0 MB/sec)


Large reads: 1GB, read 1MB at a time.
(dd bs=1048576 count=1024 of=/dev/null if=/t/afp/testfile)

AFP: 34 sec (30.1 MB/sec) - ReadyNAS was 76.7 sec (14.0 MB/sec).
CIFS: 38.8 sec (27.7 MB/sec) - ReadyNAS was 78.6 sec (13.6 MB/sec).
NFS: 26.1 sec (41.1 MB/sec) - ReadyNAS was 89.3 sec (12.0 MB/sec).
Message 2 of 19
bhoar
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

magi wrote:
The conclusion holds -- small transactions are heavily penalized under CIFS (at least with Apple's implementation).


I'd be surprised if they were using something other than samba, which leads me to think there must be something wrong performance-wise in recent samba releases.

-brendan
Message 3 of 19
magi1
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

bhoar wrote:
I'd be surprised if they were using something other than samba, which leads me to think there must be something wrong performance-wise in recent samba releases.


smbclient reports 3.0.10, smbd reports version 3.0.10, so that seems to be what Apple is shipping.

I don't know if it's a bug, or it's mistuned on one end or the other... I'll try from a native Windows machine too and see if I can get comparable numbers. The problem with that is (1) it's harder without simple tools like dd and /dev/zero, and (2) I don't have nfs or afp clients for Windows to set a comparison point and control for any other differences on the Windows system. It'll still be useful to see if there's as much falloff between large files and small ones though.
Message 4 of 19
magi1
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

As a point of comparison to the prior post, here are the same tests performed using the same client and same ReadyNAS server, this time with jumbo frames enabled on both ends. Note that the ReadyNAS thinks jumbo frames means "MTU is 7936 bytes". Mac OS thinks jumbo frames mean "MTU is 9000 bytes". That won't work. If you use TCP (which AFP and CIFS do), path MTU discovery will mean the machines will automatically negotiate the smaller MTU that works. If you use UDP (which NFS does), all the traffic will get dropped on the floor and you're in for some confusing troubleshooting.

Note again that the same relative performance between the remote filesystem protocols holds -- AFP has the fastest reads, CIFS gets penalized heavily for small reads and writes, and NFS comes out somewhere in between, beating AFP in a couple tests and losing in others.


Testbed:

Client testbed: Mac Pro, OS X 10.4.9, 5GB RAM.
Server testbed: MacBook Pro, OS X 10.4.9, 2GB RAM.
Server testbed: ReadyNAS NV, 1GB RAM, 3.01c1-p6 firmware, AFP update, oplocks update.
ReadyNAS disk hardware: 4x Seagate Barracuda 7200.10 drives, 400GB each, configured as one X-RAID volume.
Network: gigabit ethernet, linksys switch, jumbo frames (MTU=7936).


Test methodology:

Same as above.


Test results:

Large writes: 1GB, written 1MB at a time.
(dd bs=1048576 count=1024 if=/dev/zero of=/t/afp/testfile)

AFP: 132.2 sec (8.1 MB/sec) - small MTU was 148.7 sec (7.2 MB/sec)
CIFS: 101.1 sec (10.6 MB/sec) - small MTU was 157.7 sec (6.8 MB/sec)
NFS: 140.5 sec (7.6 MB/sec) - small MTU was 137.5 sec (7.8 MB/sec)


Small writes: 10MB, written 1KB at a time.
(dd bs=1024 count=10000 if=/dev/zero of=/t/afp/testfile)

AFP: 0.95 sec (10.8 MB/sec) - small MTU was 1.1 sec (9.6 MB/sec)
CIFS: 7.8 sec (1.3 MB/sec) - small MTU was 7.5 sec (1.4 MB/sec)
NFS: 1.2 sec (8.4 MB/sec) - small MTU was 1.2 sec (8.4 MB/sec)

The conclusion holds -- small transactions are heavily penalized under CIFS (at least with Apple's implementation).

Small reads: 100MB, read 1KB at a time.
(dd bs=1024 count=100000 of=/dev/null if=/t/afp/testfile)

AFP: 17.6 sec (5.8 MB/sec) - small MTU was 7.5 sec (13.7 MB/sec)
CIFS: 65.6 sec (1.6 MB/sec) - small MTU was 64.0 sec (1.6 MB/sec)
NFS: 8.6 sec (11.9 MB/sec) - small MTU was 7.3 sec (14.0 MB/sec)


Large reads: 1GB, read 1MB at a time.
(dd bs=1048576 count=1024 of=/dev/null if=/t/afp/testfile)

AFP: 200.0 sec (5.4 MB/sec) - small MTU was 76.7 sec (14.0 MB/sec).
CIFS: 81.9 sec (13.1 MB/sec) - small MTU was 78.6 sec (13.6 MB/sec).
NFS: 90.2 sec (11.9 MB/sec) - small MTU was 89.3 sec (12.0 MB/sec).


Conclusion:

Effects of jumbo frames on each protocol: CIFS gains significantly on large writes, and was slightly slower on other tests. NFS loses a little across the board. AFP got a little faster on writes, but much slower on reads. Given the configuration hassle and mixed results, I'm not sure jumbo frames are a win.

Relative protocol results with jumbo frames: Even with jumbo frames enabled, AFP still looks like it performs the best on Mac OS.
Message 5 of 19
magi1
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

As a point of comparison to the prior post, here are the same tests performed using the same client, this time against an older Windows machine as the server. I wanted to see how Microsoft's SMB server compares, still using OS X as the client.


Testbed:

Client testbed: Mac Pro, OS X 10.4.9, 5GB RAM.
Server testbed: homebuilt PC, Athlon XP processor, Windows XP, 1GB RAM.
Network: gigabit ethernet, linksys switch, normal Ethernet MTU (1500).


Test methodology:

Same as above. Testing CIFS only since that's all Windows supports out of the box.


Test results:

Large writes: 1GB, written 1MB at a time.
(dd bs=1048576 count=1024 if=/dev/zero of=/t/afp/testfile)

CIFS: 49.7 sec (21.6 MB/sec) - ReadyNAS was 157.7 sec (6.8 MB/sec)


Small writes: 10MB, written 1KB at a time.
(dd bs=1024 count=10000 if=/dev/zero of=/t/afp/testfile)

CIFS: 3.4 sec (3.0 MB/sec) - ReadyNAS was 7.5 sec (1.4 MB/sec)

Small reads: 100MB, read 1KB at a time.
(dd bs=1024 count=100000 of=/dev/null if=/t/afp/testfile)

CIFS: 36.9 sec (2.8 MB/sec) - ReadyNAS was 64.0 sec (1.6 MB/sec).


Large reads: 1GB, read 1MB at a time.
(dd bs=1048576 count=1024 of=/dev/null if=/t/afp/testfile)

CIFS: 73.6 sec (14.6 MB/sec) - ReadyNAS was 78.6 sec (13.6 MB/sec).


Conclusion:

Again, OS X's CIFS performance is much worse for small reads/writes than large reads/writes. (I did notice one difference: the read tests were faster if performed back to back, for data sizes that are smaller than the server's memory, and not for data sizes that are larger than the client's memory. This might indicate that the server is disk-bound and caching on the server helps. It doesn't seem to indicate client-side caching, because that wouldn't depend on the amount of memory available on the server.)
Message 6 of 19
magi1
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

As a point of comparison to the prior post, here are the same tests performed using a real Windows machine as the client, against the same ReadyNAS.


Testbed:

Client testbed: homebuilt PC, Athlon XP processor, Windows XP, 1GB RAM.
Server testbed: ReadyNAS NV, 1GB RAM, 3.01c1-p6 firmware, AFP update, oplocks update.
ReadyNAS disk hardware: 4x Seagate Barracuda 7200.10 drives, 400GB each, configured as one X-RAID volume.
Network: gigabit ethernet, linksys switch, normal Ethernet MTU (1500).


Test methodology:

Same as above. Testing CIFS only since that's all Windows supports out of the box.


Test results:

Large writes: 1GB, written 1MB at a time.
(dd bs=1048576 count=1024 if=/dev/zero of=/t/afp/testfile)

CIFS: 97.6 sec (11.0 MB/sec) - OS X took 157.7 sec (6.8 MB/sec)


Small writes: 10MB, written 1KB at a time.
(dd bs=1024 count=10000 if=/dev/zero of=/t/afp/testfile)

CIFS: 6.1 sec (1.7 MB/sec) - OS X took 7.5 sec (1.4 MB/sec)

Small reads: 100MB, read 1KB at a time.
(dd bs=1024 count=100000 of=/dev/null if=/t/afp/testfile)

CIFS: 5.2 sec (19.7 MB/sec) - OS X took 64.0 sec (1.6 MB/sec). If repeated, takes 0.3 sec -- obviously cached better than OSX<>ReadyNAS (which took the same amount of time if invoked over and over).


Large reads: 1GB, read 1MB at a time.
(dd bs=1048576 count=1024 of=/dev/null if=/t/afp/testfile)

CIFS: 55.6 sec (19.3 MB/sec) - OS X took 78.6 sec (13.6 MB/sec).


Conclusion:

Windows is much better at CIFS against a ReadyNAS NV than Mac OS X is.
Message 7 of 19
Retired_Member
Not applicable

choice of file protocol (AFP vs CIFS vs NFS)

A very interesting compare - thanks for the input!

I get similar results here -- and used cygwin from CIFS.XP .

You did not, by chance, do a CIFS.XP (client) against CIFS.MacOS-X (server) ?
Message 8 of 19
magi1
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

Nah, I didn't try that. That would be interesting; another interesting comparison point would be between two Windows machines.
Message 9 of 19
ginigma
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

Nice testing and interesting conclusions.
Do you know if there's an AFP protocol for Windows?
Message 10 of 19
pprior
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

I have CIFS and AFP enabled as I have both Windows and Mac boxes accessing.

Lets assume that I want to use ONLY AFP from my Mac machine. What makes the decision? Mac OS? the NAS? Some setting I don't know about?

thanks.
Message 11 of 19
Flash1
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

Nice stuff there Magi, thanks for all the info.

pprior, OS X will default to AFP if it is available. If you want to force it just use the "afp://" precursor in your "Apple K" Connect to Server dialog.
For example: afp://192.168.1.100/media
Message 12 of 19
pprior
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

thanks for that tip. I'm still a new Mac user and learning after 20 years of PC use.
Message 13 of 19
remowylliams
Aspirant

CIF bugged on Linksys switches

Hi I just thought I'd throw my 2 cents worth in and previous
experience by mentioning that routinely I would get between
750 Kb (and that is bit) - 2.5 Mb/s through my linksys switches
using CIFS/SMBFS.

It didn't matter if I was using 2 Windows boxes, or a Samba
box to another Samba box, or Windows to Samba. They all
had horrendous network behavior. Watching the data stream
I'd see peaks and valleys of activity that the best I could say
was the switch turning off the network port like it was detecting
collisions or something.

I tried changing everything I could think of, half duplex, Full duplex,
polling. Nothing seemed to work.

Surprisingly, using nfs or ftp I'd get reasonably good performance
nothing stellar but not too bad. In all cases it was very much
like this. Everything but CIFS seemed to stream through the switch
just fine.

Finally I replaced my switches (a couple of 5 port and a 8 port) with
another maker of switch and suddenly all my problems are gone.
I have since sworn off the Linksys switches and anything that has
a linksys switch in it. Like the WRT-54G because of the same behavior.

I wrote time and time again to Linksys asking for help but never
received a sentient reply. (Hurrah auto-reply) If you could borrow
a different switch. I think you will find the same change.

Please let me know if you find a fix for this linksys weirdness.

Best wishes.

Remowylliams
Message 14 of 19
bhoar
Aspirant

Re: CIF bugged on Linksys switches

remowylliams wrote:
between 750 Mb (and that is bit) - 2.5 Mb/s t


? 750Mb to 2.5Mb?

-brendan
Message 15 of 19
remowylliams
Aspirant

Re: CIF bugged on Linksys switches

bhoar wrote:
remowylliams wrote:
between 750 Mb (and that is bit) - 2.5 Mb/s t


? 750Mb to 2.5Mb?

-brendan


Thanks for point that out. It's fixed now.

Remo
Message 16 of 19
owen_pg
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

I know this topic is stale but since the "4.x" firmware of the ReadyNAS now supports NFS over TCP, I wonder if the relative throughputs have been changed.

Ditto with the ReadyNAS Add-On for AFP . . .
http://www.readynas.com/download/addons ... rt_1.2.bin
"Update_AFP_Support - This add-on fixes the Meta data and time stamps not being carried over properly."
http://www.readynas.com/?page_id=617
Message 17 of 19
Scaevola
Aspirant

Re: choice of file protocol (AFP vs CIFS vs NFS)

I'm curious or maybe it's a dumb question.

How does UPNP play into the equation? Can anyone clarify?
Message 18 of 19
sphardy1
Apprentice

Re: choice of file protocol (AFP vs CIFS vs NFS)

UPnP is a discovery protocol, not a file transfer protocol like AFP/NFS/CIFS: http://en.wikipedia.org/wiki/Universal_Plug_and_Play
Message 19 of 19
Top Contributors
Discussion stats
  • 18 replies
  • 34449 views
  • 0 kudos
  • 10 in conversation
Announcements