NFS Tunning? - some benchmarks

I ran some benchmarks to compare various modes of file sharing, and got some surprising results. My intution was that NFS would be the best option, but its performance was off (see below).

So does anyone have any suggestions about tuning NFS (on either the server or the client). I'm operating between OSX client and ReayNAS Ultra2+. Alternatively, do folks think that AFP or CIFS is better solution then NFS? Their performance wasn't the greatest either, and I've read many posts from people having long delays with large directories.

I've included also below results from an FTP test, local drive test and iSCSI too. With the exception of FTP, all tests performed with the OSX XBench tool against ReadyNAS Ultra2+ 4.2.20, gigabit ethernet through Netgear router.

Local Drive Baseline

Sequential
Uncached Write     95.59 MB/sec [4K blocks]
                   86.67 MB/sec [256K blocks]
Uncached Read      26.79 MB/sec [4K blocks]
                   88.63 MB/sec [256K blocks]
Random
Uncached Write      1.14 MB/sec [4K blocks]
                   34.16 MB/sec [256K blocks]
Uncached Read       0.55 MB/sec [4K blocks]
                   25.66 MB/sec [256K blocks]

Notes - Why Random 4k blocks is so poor performance on local drive?

FTP with ReadyNAS

Sustained Download    79 MB/sec
Sustained Upload      71 MB/sec

iSCSI, HFS+ formatted between OSX 10.6.8 and ReadyNAS

Sequential
Uncached Write     82.55 MB/sec [4K blocks]
                   56.29 MB/sec [256K blocks]
Uncached Read      10.85 MB/sec [4K blocks]
                   82.25 MB/sec [256K blocks]
Random
Uncached Write     35.99 MB/sec [4K blocks]
                   65.80 MB/sec [256K blocks]
Uncached Read      10.85 MB/sec [4K blocks]
                   83.04 MB/sec [256K blocks]

Note - Is iSCSI performance really better than local drive?!?!?! Wow.

NFS - 2 NFS threads

Sequential
Uncached Write      2.48 MB/sec [4K blocks]
                   98.60 MB/sec [256K blocks]
Uncached Read       5.65 MB/sec [4K blocks]
                   90.75 MB/sec [256K blocks]
Random
Uncached Write     37.41 MB/sec [4K blocks]
                   98.46 MB/sec [256K blocks]
Uncached Read       5.55 MB/sec [4K blocks]
                   90.55 MB/sec [256K blocks]

Note - Similar to Local drive, poor 4k performance, but also poor for sequential

AFP

Sequential
Uncached Write     10.95 MB/sec [4K blocks]
                   69.41 MB/sec [256K blocks]
Uncached Read      10.28 MB/sec [4K blocks]
                   89.73 MB/sec [256K blocks]
Random
Uncached Write     10.77 MB/sec [4K blocks]
                   84.60 MB/sec [256K blocks]
Uncached Read      10.05 MB/sec [4K blocks]
                   89.61 MB/sec [256K blocks]

Note - doesn't have 4k probblem as severe as NFS (or local?)

CIFS - SMB

Sequential
Uncached Write     10.05 MB/sec [4K blocks]
                   48.05 MB/sec [256K blocks]
Uncached Read       8.15 MB/sec [4K blocks]
                   53.22 MB/sec [256K blocks]
Random
Uncached Write     10.58 MB/sec [4K blocks]
                   50.75 MB/sec [256K blocks]
Uncached Read       9.54 MB/sec [4K blocks]
                   53.27 MB/sec [256K blocks]

Note - AFP vs SMB, AFP wins across the board by 20% margin I guess.

Summary

• I was rather shocked to see such poor 4k block performance on my local drive, I need to understand why


• I was equally shocked that iSCSI out performed the local drive overall


• Between NFS, AFP, SMB;... NFS had fastest throughput with 256k blocks, but worst performance for 4k blocks. Can that be tuned?


• Between AFP and SMB, AFP wins


• In real use I do seem to experience lags when pulling up folders in Finder from AFP and SMB. Not sure what this delay is about. NFS seems more snappy in that regard. Just my impression, I don't have real data about that


• What is the implication of poor 4k performance, is that normal or does my system need tuning in some way


• I have net.inet.tcp.delayed_ack=2 set on Mac. This is supposed to help reduce delays on small packets. I will try a value of 0 later. Default is 3.


• NFS also has security requirement that both machines share same owner/group directory