NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.

Forum Discussion

SuperJT's avatar
SuperJT
Aspirant
Aug 02, 2016

NFS Performance issues

I work for Netapp in the NFS support area.

I puchased this device almost a year ago and for the most part have been happy to have it just as a storage device.

Recently I tried to put a VM on the device.  Mounted nfsv3, and default mount parameters.  The mount is fine, however performance is absolutely terrible.  I took a packet trace to see if it potentially could be network loss.  There were a few packets that appear to be loss, but VERY little and they do not correspond with the latency that i'm seeing.

 

I am seeing upwards of 20 seconds for a single write call.  I can send the trace out to you, but I dont see a way to attach it to this post.  I downloaded gcc5.4 and attempted to untar.  The untar process took 3min 54s to complete.  The file was stored on the filer, so this would be a read and write.  This was intentional to gauge performance. 

 

I need assistance to see what I can do to get better performance out of this.  I dont expect the world, but I do expect to be able to install updates on a windows vm in under 10 hours.  (Local disk it is ~15 minutes).  If there is anything / logs / traces / additional tools to run, please let me know.

8 Replies

Replies have been turned off for this discussion
  • Apart from what mdgm mentioned (definitely worth a read), can you please let us know whether you have NFS async enabled or disabled?

    Go to the settings page of the share --> Network Access --> NFS --> Advanced.

     

    This can have a very significant performance impact - running with or without async.

     

     

    Thanks

    • SuperJT's avatar
      SuperJT
      Aspirant

      Hi all, 

       

      Thanks for the response.  I looked over the KB's listed.

      1) I am running 6.5.0 (1 minor version below current i believe)

      2) ASYNC is not enabled (I'll play with this to see if this improves performance)

      3) Bit-Rot Protection is not enabled.

      4) Snapshots are enabled, set for 'smart' for management and are set to daily.

       

      edit:

      Additional outputs:

      mount options

      jtownsen@superjt:~$ mount | grep vm
      192.168.1.79:/data/vm_store on /var/lib/kimchi/nfs_mount/vm_store type nfs (rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.79,mountvers=3,mountport=59523,mountproto=udp,local_lock=none,addr=192.168.1.79)

      top.

       

      Tasks: 212 total,   1 running, 211 sleeping,   0 stopped,   0 zombie
      %Cpu(s):  0.2 us,  0.8 sy,  0.0 ni, 85.3 id, 13.7 wa,  0.0 hi,  0.1 si,  0.0 st
      KiB Mem:   3931280 total,  3245088 used,   686192 free,      312 buffers
      KiB Swap:  2094844 total,        0 used,  2094844 free,  1827016 cached
      
        PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                       
       3656 root      20   0     0    0    0 D   1.7  0.0   0:23.35 nfsd                                                                                          
       1504 root      20   0     0    0    0 S   1.3  0.0  49:32.66 md127_raid5                                                                                   
       3655 root      20   0     0    0    0 S   0.7  0.0   0:12.35 nfsd                                                                                          
       1443 root       0 -20     0    0    0 S   0.3  0.0   1:28.83 kworker/2:1H                                                                                  
       2835 root       0 -20 2228m  17m 3120 S   0.3  0.5  16:10.14 vpnserver                                                                                     
       2954 root      35  15 1830m  71m 6596 S   0.3  1.9   6:21.11 Plex Script Hos                                                                               
       2957 admin     20   0 2363m 137m 5188 S   0.3  3.6  14:43.74 sabnzbd                                                                                       
       3543 admin     20   0  130m 7376 3084 S   0.3  0.2   0:02.97 apache2                                                                                       
      14297 root      20   0     0    0    0 S   0.3  0.0   0:01.15 kworker/u8:5                                                                                  
      14549 root      20   0     0    0    0 S   0.3  0.0   0:00.13 kworker/2:1                                                                                   
          1 root      20   0 45720 5052 2488 S   0.0  0.1   0:06.58 systemd    

       

      • SuperJT's avatar
        SuperJT
        Aspirant

        I enabled async on the volume.. Huge perf increase. 95 minutes to 4 minutes. I feel like this is indicating my disks are a problem.

         

        root@superjt:/var/lib/kimchi/nfs_mount/vm_store# time tar xf gcc-5.3.0.tar.gz 
        
        real	95m11.482s
        user	0m9.828s
        sys	0m48.948s
        root@superjt:/var/lib/kimchi/nfs_mount/vm_store# rm -rf gcc-5.3.0/
        root@superjt:/var/lib/kimchi/nfs_mount/vm_store# time tar xf gcc-5.3.0.tar.gz 
        
        real	4m7.290s
        user	0m8.152s
        sys	0m30.148s
        root@superjt:/var/lib/kimchi/nfs_mount/vm_store# 
        
        
    • StephenB's avatar
      StephenB
      Guru - Experienced User

      SuperJT wrote:
      I have 6x 3tb WD red drives

      They seem unlikely to be the botteneck.   With single redundancy they should be giving you 100 MB/sec for both reads and writes.

       

      You could try nastester, that will take NFS out of the equation: http://www.808.dk/?code-csharp-nas-performance

       

      Maybe download the logs and double-check the smart stats.

      • mdgm-ntgr's avatar
        mdgm-ntgr
        NETGEAR Employee Retired

        Did StephenB's post resolve your problem. If not, could you send me your logs (see the Sending Logs link in my sig)?

NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology! 

Join Us!

ProSupport for Business

Comprehensive support plans for maximum network uptime and business peace of mind.

 

Learn More