NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
NFS Tunning? - some benchmarks
I ran some benchmarks to compare various modes of file sharing, and got some surprising results. My intution was that NFS would be the best option, but its performance was off (see below).
So does anyone have any suggestions about tuning NFS (on either the server or the client). I'm operating between OSX client and ReayNAS Ultra2+. Alternatively, do folks think that AFP or CIFS is better solution then NFS? Their performance wasn't the greatest either, and I've read many posts from people having long delays with large directories.
I've included also below results from an FTP test, local drive test and iSCSI too. With the exception of FTP, all tests performed with the OSX XBench tool against ReadyNAS Ultra2+ 4.2.20, gigabit ethernet through Netgear router.
Local Drive Baseline
FTP with ReadyNAS
iSCSI, HFS+ formatted between OSX 10.6.8 and ReadyNAS
NFS - 2 NFS threads
AFP
CIFS - SMB
Summary
So does anyone have any suggestions about tuning NFS (on either the server or the client). I'm operating between OSX client and ReayNAS Ultra2+. Alternatively, do folks think that AFP or CIFS is better solution then NFS? Their performance wasn't the greatest either, and I've read many posts from people having long delays with large directories.
I've included also below results from an FTP test, local drive test and iSCSI too. With the exception of FTP, all tests performed with the OSX XBench tool against ReadyNAS Ultra2+ 4.2.20, gigabit ethernet through Netgear router.
Local Drive Baseline
Notes - Why Random 4k blocks is so poor performance on local drive?
Sequential
Uncached Write 95.59 MB/sec [4K blocks]
86.67 MB/sec [256K blocks]
Uncached Read 26.79 MB/sec [4K blocks]
88.63 MB/sec [256K blocks]
Random
Uncached Write 1.14 MB/sec [4K blocks]
34.16 MB/sec [256K blocks]
Uncached Read 0.55 MB/sec [4K blocks]
25.66 MB/sec [256K blocks]
FTP with ReadyNAS
Sustained Download 79 MB/sec
Sustained Upload 71 MB/sec
iSCSI, HFS+ formatted between OSX 10.6.8 and ReadyNAS
Note - Is iSCSI performance really better than local drive?!?!?! Wow.
Sequential
Uncached Write 82.55 MB/sec [4K blocks]
56.29 MB/sec [256K blocks]
Uncached Read 10.85 MB/sec [4K blocks]
82.25 MB/sec [256K blocks]
Random
Uncached Write 35.99 MB/sec [4K blocks]
65.80 MB/sec [256K blocks]
Uncached Read 10.85 MB/sec [4K blocks]
83.04 MB/sec [256K blocks]
NFS - 2 NFS threads
Note - Similar to Local drive, poor 4k performance, but also poor for sequential
Sequential
Uncached Write 2.48 MB/sec [4K blocks]
98.60 MB/sec [256K blocks]
Uncached Read 5.65 MB/sec [4K blocks]
90.75 MB/sec [256K blocks]
Random
Uncached Write 37.41 MB/sec [4K blocks]
98.46 MB/sec [256K blocks]
Uncached Read 5.55 MB/sec [4K blocks]
90.55 MB/sec [256K blocks]
AFP
Note - doesn't have 4k probblem as severe as NFS (or local?)
Sequential
Uncached Write 10.95 MB/sec [4K blocks]
69.41 MB/sec [256K blocks]
Uncached Read 10.28 MB/sec [4K blocks]
89.73 MB/sec [256K blocks]
Random
Uncached Write 10.77 MB/sec [4K blocks]
84.60 MB/sec [256K blocks]
Uncached Read 10.05 MB/sec [4K blocks]
89.61 MB/sec [256K blocks]
CIFS - SMB
Note - AFP vs SMB, AFP wins across the board by 20% margin I guess.
Sequential
Uncached Write 10.05 MB/sec [4K blocks]
48.05 MB/sec [256K blocks]
Uncached Read 8.15 MB/sec [4K blocks]
53.22 MB/sec [256K blocks]
Random
Uncached Write 10.58 MB/sec [4K blocks]
50.75 MB/sec [256K blocks]
Uncached Read 9.54 MB/sec [4K blocks]
53.27 MB/sec [256K blocks]
Summary
- I was rather shocked to see such poor 4k block performance on my local drive, I need to understand why
- I was equally shocked that iSCSI out performed the local drive overall
- Between NFS, AFP, SMB;... NFS had fastest throughput with 256k blocks, but worst performance for 4k blocks. Can that be tuned?
- Between AFP and SMB, AFP wins
- In real use I do seem to experience lags when pulling up folders in Finder from AFP and SMB. Not sure what this delay is about. NFS seems more snappy in that regard. Just my impression, I don't have real data about that
- What is the implication of poor 4k performance, is that normal or does my system need tuning in some way
- I have net.inet.tcp.delayed_ack=2 set on Mac. This is supposed to help reduce delays on small packets. I will try a value of 0 later. Default is 3.
- NFS also has security requirement that both machines share same owner/group directory
11 Replies
Replies have been turned off for this discussion
- The local drive 4k results are most accurate as far as HDD performance. HDDs can only seek to a file a hundred or so times per second, because it takes around 10ms each time. 552KB/sec / 4KB = 138 IOPS. (around 100) Sequentially is another story, but that's what you get with random access.
The other results are all higher because of caching - your NAS is acting as a huge read/write buffer. With 1GB of memory, about 800MB of that would be available to act as read/write cache, inflating the results for benchmarks with smaller datasets. In a truly random scenario with a large enough dataset, you would get less than the local drive benchmarks.
Sequential performance is lower for some protocols because of the protocol overhead. iSCSI is clearly the most efficient protocol, and CIFs is clearly the least efficient. I wouldn't worry about low 4K speeds - very little is 4K in size. Go with whatever feels snappiest - NFS may be doing a better job caching directory trees, at the expense of its file cache. CIFS is probably doing the opposite. - Right Bike Helmet, well I guess the main thing I was wondering if someone has some insight about is tuning NFS.
The local drive did ok performance on 4K blocks sequentially. As you said, it was the random access that caused it to break down due to all the seeking. But NFS fell apart sequentially too.
iSCSI, did pretty well, but specifically uncached 4k, was kind of poor, which makes sense by what you just said. But it did better then local for random access, which I can't explain unless the NAS cashing is superior to the cacheing on my local drive. It could also be that the benchmarking software is able to force the local drive into true random access and not able to do so as well for mounted file server volumes, not sure.
iSCSI would be great, but its of little use for most real world situations to me. iSCSI would work great for a thin client workstation.
NFS I think would be fastest, if the random access of 4k sequential access could be improved some how.
Random sequential access? Don't think you meant that.Dewdman42 wrote: NFS I think would be fastest, if the random access of 4k sequential access could be improved some how.
Are you thinking a specific application where the random access is critical?
Also, how many drives are in the raid array? Read performance generally improves as you add more drives.- correct I didn't mean that. I meant the poor sequential 4k access on NFS. Any tuning suggestions?
I am not sure that tuning NFS will help much, and suspect it is already reasonably tuned for typical loads.Dewdman42 wrote: correct I didn't mean that. I meant the poor sequential 4k access on NFS. Any tuning suggestions?
As I mentioned, the number of drives in the array can make a huge difference here. So JBOD is probably not the best choice if your goal is to get the max read speed. What is your raid configuration, and how many drives are in the array you are testing?- two disks. But AFP and SMB both out performed NFS for 4k blocks. The protocol must be a limiting factor.
Dewdman42 wrote: iSCSI, did pretty well, but specifically uncached 4k, was kind of poor, which makes sense by what you just said. But it did better then local for random access, which I can't explain unless the NAS cashing is superior to the cacheing on my local drive. It could also be that the benchmarking software is able to force the local drive into true random access and not able to do so as well for mounted file server volumes, not sure.
That's what it is. Your NAS is acting as a huge read/write memory cache. Your computer can't do anything about that.Dewdman42 wrote: iSCSI would be great, but its of little use for most real world situations to me. iSCSI would work great for a thin client workstation.
Yeah, the singular access eliminates most usage. No good for anything that would typically use "shares".Dewdman42 wrote: NFS I think would be fastest, if the random access of 4k sequential access could be improved some how.
This might be a situation where "less is more".
Among SSDs, it has been noted that firmware make a major difference in how a controller performs. Over the years several times SSDs have been tweaked for responsiveness in later firmware revisions - they do that by slowing them down.
On slow internet connections (lets say 5mbit DSL), you can make them more responsive by implementing QOS - throttling incoming and outgoing connections and then prioritizing certain ports/IPs/etc.; it makes your connection seem faster, even though you just dropped your usable bandwidth to 3-4mbit. Weird, right?
You said it feels faster - just tossing possible explanations out there.- hmm, no what I mean to say is that the benchmark results from NFS were better than AFP and SMB in terms of large block size, but worse in terms of 4k blocks. If I can tune how the smaller blocks are handled while also keeping the nice high numbers of the large blocks, then it would out perform both AFP and SMB.
AFP and SMB also seem to have some delay overhead related to browsing directories. This is undoubtedly because they are writing odd ball hidden files related to resource forks, trash cans and such things. AFP always has a long delay before opening a folder for the first time Finder, but then after that its faster. In the past I have experienced similar delays with SMB, even from windows clients. When I was using windows clients before, I used to use FTP for large transfers because it would always out perform SMB by a long shot. Not sure if Windows explorer was causing the bottleneck or SMB.
when I use AFP to access the NAS, I see deskop and trash can files being created and/or maintained. Not sure what is in there, but I'm assuming its related to resource forks, icons, etc. That means overhead, so feels slower. I believe NFS avoids that, but granted, resource forks are probably lost to NFS shares. For Mac users, care has to be exercised to avoid losing resource forks from mac files with NFS.
In any case, NFS has a buzzillion options for tuning this or that, so just wondering if anyone has some suggestions along those lines. Otherwise I will sick with AFP and just endure the long load time for browsing directories
Adding more disks speeds things up for all protocols. Going from 2 to 4 disks on my Pro increased sequential read speed by almost 3x (from about 110 MB/s to 319 MB/s). This was the internal speed (measured with Bonnie). Random access also improved, but I didn't save the numbers for that.Dewdman42 wrote: two disks. But AFP and SMB both out performed NFS for 4k blocks. The protocol must be a limiting factor.
I'm not saying that protocol has no effect. Just that adding disks can have a huge effect.
The main advice I've seen on tuning NFS for random access is to disable read ahead (nfs_server_clread). I suspect that will create poor performance for sequential applications. Being more aggressive with NFS threads might help (why not try 16, and maybe some intermediate values).
I found an old paper here: http://www.columbia.edu/~ra2028/USENIX/ ... ix2004.pdf it might have some useful ideas. Note the claim that many performance issues are due to the NFS client - not the server(I've seen that claim about NFS in other papers).- I will check all that and absolutely I am asking about NFS client tuning as much if not more so than server side tuning.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
