× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

RN104 6.9.1 poor SMB read/write performance

SeeD
Tutor

RN104 6.9.1 poor SMB read/write performance

Hello,
I have RN104 with OS 6.9.1 on it. Since 6.9.0 i have really poor read/write performance from SMB shares.
Setup:
4 drives WD RED 3TB WD30EFRX - 40% free space
Link bound 2x1Gb uplinks to Netgear switch - both negotiated 1Gb (MTU 1500, static IP, no dropped frames/errors detected on switch links)
AV disabled
SMB3 encryption disabled globally

I have run NAStest1.7 from 3 different Windows 7 laptops - doesnt matter if connected through switch or directly to NAS - more or less same output:

 

Running a 400MB file write on (...) 5 times...
Iteration 1:      3,36 MB/sec
Iteration 2:      3,04 MB/sec
Iteration 3:      3,11 MB/sec
Iteration 4:      3,03 MB/sec
Iteration 5:      3,11 MB/sec
-----------------------------
Average (W):      3,13 MB/sec
-----------------------------
Running a 400MB file read on (...) 5 times...
Iteration 1:     14,42 MB/sec
Iteration 2:     14,99 MB/sec
Iteration 3:     15,36 MB/sec
Iteration 4:     15,24 MB/sec
Iteration 5:     14,84 MB/sec
-----------------------------
Average (R):     14,97 MB/sec
-----------------------------

Top command results during test:

 PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 4734 root      20   0       0      0      0 R 27.2  0.0   0:53.31 kworker/u2:6
        6 root      20   0       0      0      0 R 26.5  0.0   1:40.00 kworker/u2:0
 4731 root      20   0       0      0      0 D 26.5  0.0   0:42.04 kworker/u2:3
 5616 user      20   0   48832  15020  13700 D  4.8  3.0   0:12.86 smbd
 1018 root       0 -20       0      0      0 S  4.2  0.0   0:25.46 kworker/0:1H
 1112 root      20   0       0      0      0 S  3.2  0.0   0:23.50 md127_raid5
 4068 root      20   0    5736   2656   2160 R  1.0  0.5   0:10.30 top

I have no clue what can I check more... Anyone had simmilar issue?

 

SeeD

Model: RN104|ReadyNAS 100 Series
Message 1 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance

Do you get the same speeds if you break the bond (that is, disconnect one of the two ethernet cables)?

Message 2 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

Hello,

Yes, it's the same story when only one link is active.

I forgot to mention that IPv6 is disabled, and disks run in RAID5.

 

Regards

SeeD

Model: RN104|ReadyNAS 100 Series
Message 3 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance

It doesn't appear to be CPU bound (based on your TOP results), and a network issue seems unlikely since the direct connects are giving you the same results, and IPv6 is disabled.

 

Have you looked at the SMART stats for the disks (download the log zip, and look in disk_info.log)?

You could also try a balance and maybe a defrag (retesting when both complete).

 

Also, if ReadyCloud is installed on all the PCs, you might want to uninstall it on one, just to make sure the system isn't routing through the VPN for some reason.

Message 4 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

Thank you for the tip - I will take a look at SMART statisticks and perform both defrag and rebalance. I will post an update when it will be finished. I'm not using ReadyCloud. 

Message 5 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

Hello,

I checked the SMART stats, no errors found:


Device:             sdd
Controller:         0
Channel:            0
Model:              WDC WD30EFRX-68EUZN0
Firmware:           80.00A80
Class:              SATA
RPM:                5400
Sectors:            5860533168
Pool:               data
PoolType:           RAID 5
PoolState:          1
PoolHostId:         e362f16
Health data
  ATA Error Count:                0
  Reallocated Sectors:            0
  Reallocation Events:            0
  Spin Retry Count:               0
  Current Pending Sector Count:   0
  Uncorrectable Sector Count:     0
  Temperature:                    29
  Start/Stop Count:               4452
  Power-On Hours:                 10051
  Power Cycle Count:              675
  Load Cycle Count:               4413


Simmilar output for the rest of drives.

I also completed defragmentation and after that rebalance process. I have re-run the test with one link in bound active. Unfortunatelly no improvement in NAStester1.7 output:

Running a 400MB file write on (...) 5 times...
Iteration 1:      3,06 MB/sec
Iteration 2:      2,60 MB/sec
Iteration 3:      2,76 MB/sec
Iteration 4:      2,91 MB/sec
Iteration 5:      2,89 MB/sec
-----------------------------
Average (W):      2,84 MB/sec
-----------------------------
Running a 400MB file read on (...) 5 times...
Iteration 1:     13,02 MB/sec
Iteration 2:     13,70 MB/sec
Iteration 3:     12,21 MB/sec
Iteration 4:     14,24 MB/sec
Iteration 5:     13,31 MB/sec
-----------------------------
Average (R):     13,30 MB/sec
-----------------------------

Regards
SeeD

 

 

Model: RN104|ReadyNAS 100 Series
Message 6 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance

Can you try a direct-connect test, and see if that gives you better results?

 

You'd connect the ReadyNAS NIC directly to the PC.  To do this you'd need to configure a static address on the NAS port (for instance 192.168.100.100, subnet mask 255.255.255.0, gateway 192.168.100.1), with a compatible static address on the PC (for instance, 192.168.100.101, with the same subnet mask and gateway).  Then run NAStester using the ReadyNAS static address.

Message 7 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

I have tried that. No change. Still the same poor performance...

I'm not sure what could I check more. It's definitely something wrong with NAS, but I do not get any errors in logs. As I mentioned this started after 6.9.0 upgrade, however downgrade to version prior 6.9.0 is not recommended by vendor.

 

Regards

SeeD

Message 8 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance

Since you have SSH enabled, you can also measure the data volume performance.  That might also provide some clues (or rule some things out).  https://www.cyberciti.biz/faq/howto-linux-unix-test-disk-performance-with-dd-command/

 

It is also pretty simple to install iperf on the NAS and the windows system, and measure the network throughput (separately from the disk performance).

 

Note also that 6.9.2 is released - though I don't know of any reason that would resolve your issue, it is something you could try.

Message 9 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

Ok, additional tests are done:

 

Data volume performance

root@NAS:/tmp# dd if=/dev/zero of=/tmp/test1.img bs=200M count=1 oflag=dsync
1+0 records in
1+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 3.08846 s, 67.9 MB/s

root@NAS:/tmp# dd if=/dev/zero of=/tmp/test2.img bs=512 count=500 oflag=dsync
500+0 records in
500+0 records out
256000 bytes (256 kB, 250 KiB) copied, 20.0716 s, 12.8 kB/s

IPERF from PC to NAS:

 

TCP window size: 85.3 KByte (default)
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   708 MBytes   593 Mbits/sec

 

You're right 6.9.2 is released, however for now my NAS is not showing this in automatic update yet. So probably only manual installation is possible.

Thank you for your help so far.

 

Regards

SeeD

Message 10 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance


@SeeD wrote:

 

Data volume performance

root@NAS:/tmp# dd if=/dev/zero of=/tmp/test1.img bs=200M count=1 oflag=dsync


Please retry these tests, with the output going to the data volume (/data would be the default path).

 

Of course delete the temporary .img files when done.

Message 11 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

Ok, done:

root@NAS:/# dd if=/dev/zero of=/data/test1.img bs=200M count=1 oflag=dsync
1+0 records in
1+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 5.75487 s, 36.4 MB/s

 

root@NAS:/# dd if=/dev/zero of=/data/test2.img bs=512 count=500 oflag=dsync
500+0 records in
500+0 records out
256000 bytes (256 kB, 250 KiB) copied, 40.377 s, 6.3 kB/s

 

However one strange thing when I have change te file size to 400M:

root@NAS:/# dd if=/dev/zero of=/data/test1.img bs=400M count=1 oflag=dsync
1+0 records in
1+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 114.464 s, 3.7 MB/s

 

This is the value more or less the same I'm getting during file write to SMB. However I'm not sure if this is related. While idle I have about half memory free so about 256MB. File 200MB fits the memory, file 400MB does not. When I run the top during this dd process in second session I can see that NAS is swapping memory:

 

KiB Mem:    508280 total,   486472 used,    21808 free,       76 buffers
KiB Swap:  1046524 total,   523244 used,   523280 free.     4440 cached Mem

 

I do not see the swapping process during Nas performance tester. When I run tester with filesize 100MB I get the same result:

Running a 100MB file write on (...) 5 times...
Iteration 1:      3,99 MB/sec

...


Regards

SeeD

Message 12 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance


@SeeD wrote:

 

root@NAS:/# dd if=/dev/zero of=/data/test1.img bs=200M count=1 oflag=dsync
1+0 records in
1+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 5.75487 s, 36.4 MB/s

  

However one strange thing when I have change te file size to 400M:

root@NAS:/# dd if=/dev/zero of=/data/test1.img bs=400M count=1 oflag=dsync
1+0 records in
1+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 114.464 s, 3.7 MB/s

 

This is the value more or less the same I'm getting during file write to SMB. However I'm not sure if this is related. While idle I have about half memory free so about 256MB. File 200MB fits the memory, file 400MB does not. When I run the top during this dd process in second session I can see that NAS is swapping memory:

 

It does make me wonder if the NAS is starved for memory though.  Does NAStester give a faster result with 200 MB/sec?

 

What file sharing protocols are enabled?  In particular, if NFS is enabled, try disabling it (or reducing the number of user threads to 4).

Message 13 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

I'm using only SMB and ReadyDLNA protocols. During normal operation I do not see memory swapping.

NAStester is giving the same results no matter how big is the file 100, 200 or 400MB.

 

Regards

SeeD

Message 14 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance

Well, it doesn't seem to be the network (since the iperf tests were fine), and the internal disk speed seemed fine as long as you didn't trigger the swapping.

 

Do you have any scratch disks? If you do, you could try powering down and removing the current disks (labeling by slot) and then do a factory install with only the scratch installed.  Then test the speeds with a fresh install.

Message 15 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

Unfortunately I do not have scratch disk at the moment...

What about Readynas OS re-installation? I know, it is hard to say what can be the result without any root cause... but is it good and safe idea?

 

Regards

SeeD

Message 16 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance

Of course, something can always go wrong.  But OS reinstalls are designed to be non-destructive, and generally are quite safe.  They will reset the IP configuration to DHCP (likely it already is), reset the admin password to password, and disable quotas on the data volume. 

 

So you'd need to reset the admin password for sure.  Disabling quotas might actually improve the performance, so you might check to see if they are enabled now (look in the settings wheel on the volume tab).  

 

Apart from the quota possibility, the OS-reinstall seems unlikely to improve performance.  If your data volume is very old, then a factory reset likely will improve it.

 

It's early days with 6.9.3-beta, but it's safe mode will provide another way to get at the data when a firmware install or OS reinstall goes bad.  Of course, installing beta releases has it's own risks.

Message 17 of 22
bedlam1
Prodigy

Re: RN104 6.9.1 poor SMB read/write performance

@StephenB Where is the "Safe Mode" option on 6.9.3

On the Boot Menu?

Message 18 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

OK, I have removed all quotas on all folders - no change on performance.

However I have noticed strange thing. When I just for another test run NASTester on folder with SMB and DLNA protocols I get these results:

Running a 400MB file write on \\(Folder 1) 5 times...
Iteration 1:     35,51 MB/sec
Iteration 2:     34,42 MB/sec
Iteration 3:     33,43 MB/sec
Iteration 4:     35,46 MB/sec
Iteration 5:     34,10 MB/sec
-----------------------------
Average (W):     34,58 MB/sec
-----------------------------
Running a 400MB file read on \\(Folder 1) 5 times...
Iteration 1:     56,85 MB/sec
Iteration 2:     60,76 MB/sec
Iteration 3:     57,11 MB/sec
Iteration 4:     57,62 MB/sec
Iteration 5:     64,59 MB/sec
-----------------------------
Average (R):     59,39 MB/sec
-----------------------------

 

When I run it on the folder I run it before:

 

Running a 400MB file write on \\(Folder 2) 5 times...
Iteration 1:      2,98 MB/sec
Iteration 2:      2,63 MB/sec

...

 

Is there any difference between them?

Folder 1 - SMB, DLNA, shared for everyone, quota removed, Bit Rot protection removed

Folder 2 - SMB only, hidden folder, shared just for specific users, quota removed, Bit Rot protection removed

 

So what am I doing wrong? 🙂

 

Regards

SeeD

Message 19 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance


@bedlam1 wrote:

@StephenB Where is the "Safe Mode" option on 6.9.3

On the Boot Menu?


I don't know - hopefully @mdgm-ntgr will chime in.

 

 

Message 20 of 22
StephenB
Guru

Re: RN104 6.9.1 poor SMB read/write performance


@SeeD wrote:

OK, I have removed all quotas on all folders - no change on performance.

 

Try also tuning quotas off on the settings wheel in the volume tab.

 


@SeeD wrote:

 

Running a 400MB file write on \\(Folder 1) 5 times...
Iteration 1:     35,51 MB/sec
Iteration 2:     34,42 MB/sec

... 

When I run it on the folder I run it before:

 

Running a 400MB file write on \\(Folder 2) 5 times...
Iteration 1:      2,98 MB/sec
Iteration 2:      2,63 MB/sec

...

 

 


There's a clue here for sure.

 

Can you try creating a fresh test share that matches the configuration of folder 2?, and measure the speed in the test share?  

Message 21 of 22
SeeD
Tutor

Re: RN104 6.9.1 poor SMB read/write performance

A small update in this topic.

I created a new folder with the same settings. After the test I'm getting ~30MB/s - so it's good.

I copied almost all the files from old folder to the new one and it was still ok. When I copied the biggest archive file which is currently ~1,5TB in size the performance for this folder dropped to ~3MB/s. In other words it looks like the presence of big file (even if the file is not used by anyone during the test) kills the NAS performance on folder level. Strange thing is that prior the 6.9.0 upgrade there were no issues with that.

Now I know what is the reason of this performance drop but I do not know if anything can be done with this, as removal of this file is not an option.

Thank you for your help.

 

Regards

SeeD

Message 22 of 22
Top Contributors
Discussion stats
  • 21 replies
  • 2464 views
  • 1 kudo
  • 3 in conversation
Announcements