Re: SMB over RN528X not working

jimk1963 · ‎2020-05-09

Setup:

- Core i7 PC, Win10 Home x64, with Intel X550-T2 dual 10GbE NIC card, and EVO 970 Plus SSD hard drive

- XS716E 10GbE switch

- RN528X with dual 10GbE ports

- RN212 with dual 1GbE ports

- Qty 6 Cat6A ETH cables, 3' each

- PC/RN528X/RN212 all connected to switch

- Switch backhauls to Orbi WiFi6 router, which administers DHCP for entire network - no Static IP's other than a printer

- "SMB Plus" app is loaded on both NAS boxes; all values are left as default (experimented with encryption disable/enable, and SMB version min/max, none of these made a difference)

Using NAS Performance Tester 1.7, or ATTO, or Cyrstal Disk Benchmark, all return similar results.

PC-to-RN528X: 1 GB/s read, 600 MB/s write (NAS Performance Tester 1.7 shows 900 MB/s write, it's the only discrepancy)

PC-to-RN212: 117 MB/s read, 117 MB/s write (NAS PT 1.7 agrees in this case)

Using Task Manager/Performance Monitor, I can see that only one ETH is active on the PC, with transfers to/from either NAS. From YouTube videos I see others are observing both ETH ports sharing the load equally. So from my point of view, SMB is not working.

Expecting RN212 to run at over 200MB/s when SMB is working. Not sure if to expect anything more from the RN528X as I may be running up against its limits. However in Netgear's "NAS Performance Guide" it does show the RN528X capable of 2+ GB/s. My PC has qty 2 970 Evo Plus that I've tested and they can easily move data well above 1GB/s in either direction, so that's not the limitation.

First things first. Are there any WINDOWS 10 HOME settings I need to modify? Looked at "Turn Features On/Off" and saw that SMB1.0 is not checked. Don't believe that's an issue since that's a very old protocol and I understand ReadyNAS boxes support SMB3.1.1 natively.

Checked NIC card settings. RSS is enabled - saw on an L1 Tech YouTube vidwo that he disabled RSS. Tried that, didn't help.

Spent a lot of time on this, any help greatly appreciated.

Sandshark · ‎2020-05-09

If you just have two independent ports from your PC connected to your router/switch, that's the way it works. Your PC is only initiating the exchange on one port, so the NAS can only respond on that one. And, of course, all connections from the NAS to the PC must be capable of the speed you are wanting.

If you do have the two PC ports joined with link aggregation and all links in the pathway are capable of the faster speed, what aggregation method are you using?

jimk1963 · ‎2020-05-09

@Sandshark wrote:
If you just have two independent ports from your PC connected to your router/switch, that's the way it works. Your PC is only initiating the exchange on one port, so the NAS can only respond on that one. And, of course, all connections from the NAS to the PC must be capable of the speed you are wanting.

If you do have the two PC ports joined with link aggregation and all links in the pathway are capable of the faster speed, what aggregation method are you using?

Thanks Sandshark. Already went down the Link Aggregation path - resulted in zero speed improvement. Link aggregation only helps when you're dealing with multiple clients. My goal here is different - just want faster NAS reads/writes to one PC. Since you asked, for Link Aggregation I did the following: (1) configured X550-T2 for StaticLinkAggregation using Powershell, and verified the combined link was listed by Windows as 20 Gbps, (2) set up the XS716E for static LAG (it doesn't support LACP), with LAG1 on Ports 1&2 (RN528X), LAG2 on ports 3&4 (RN212), and LAG3 on ports 7&8 (PC), and (3) configured NAS NIC's as Bonded using RoundRobin option. Transfers were identical to just one ETH port, as expected since I only have one PC running.

The promise of SMB multichannel is much different. When properly configured, Windows automatically bonds DHCP-assigned ports between devices and automatically shares bandwidth over those ports, whether 2, 3, 4 or more (in my case, 2 ports on each device). I've watched numerous YouTube demos of this working between a single PC and a NAS, and have read multiple articles explaining same. In most cases, the PC used by these guys is running a Windows Server flavor (2012 or higher). Windows Server is "plug and play" for SMB multichannel - you can literally just plug in two cables into the NAS and get double the bandwidth instantly (again, only limited by your PC's and NAS's HW). But there's precious little info out there for Windows 10 users. I've confirmed through Powershell that SMB2.0 is "True" and so is SMB Multichannel. So in theory Windows is set up and should just "work". I'm looking for examples from anyone who has actually made this work over pedestrial Windows, as opposed to Windows Server.

StephenB · ‎2020-05-09

@jimk1963 wrote:

The promise of SMB multichannel is much different.

AFAIK SMB multichannel isn't enabled on the NAS.

jimk1963 · ‎2020-05-09

@StephenB wrote:
@jimk1963 wrote:
The promise of SMB multichannel is much different.
AFAIK SMB multichannel isn't enabled on the NAS.

Thanks Stephen, thought I read that it was supported somewhere but am looking again and all I see is SMB3 support, but SMB Multichannel is "experimental" per Samba and not enabled on the NAS. This is same story as QNAP NAS. With QNAP, there are several good videos showing how to SSH into the NAS and change the Multichannel to "True". Do you know if that's possible on ReadyNAS? I've used Putty to SSH into the RN528X but don't know the syntax once I'm in there and don't want to mess anything up.

I did run Powershell's "get-smbserverconfiguration" command on my PC and "EnableMultiChannel" is True in the list, as are EnableSMB1Protocol and EnableSMB2Protocol. There's no entry for SMB3 in the list. I don't know run the equivalent command with Putty on the NAS.

StephenB · ‎2020-05-10

@jimk1963 wrote:
I've used Putty to SSH into the RN528X but don't know the syntax once I'm in there and don't want to mess anything up.

You could try enabling the setting in /etc/frontview/samba/addons/addons.conf as described here: https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Samba-manually-modify-etc-frontview...

You might also want to try disabling strict sync in the NAS share you are using for testing. There is also a sync setting you can turn off in SMB plus.

Note the NAS is running Samba 4.8.0

FWIW, a static lag on the switch with round robin on the NAS will in fact increase the throughput with a single client. But in the case of the RN528x that also depends on the PC being able to use both of it's NICs.

But in your case, that would maxmize the performance with the RN212 since either NIC in the PC can carry the traffic easily. Note that the round-robin only applies in packets the NAS is delivering. The packet transmission policy in the switch determines the loading of the two NICs in the switch->NAS direction. So it might only help reading from the RN212, and not writing to it.

jimk1963 · ‎2020-05-10

@StephenB wrote:
@jimk1963 wrote:
I've used Putty to SSH into the RN528X but don't know the syntax once I'm in there and don't want to mess anything up.

You could try enabling the setting in /etc/frontview/samba/addons/addons.conf as described here: https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Samba-manually-modify-etc-frontview...

You might also want to try disabling strict sync in the NAS share you are using for testing. There is also a sync setting you can turn off in SMB plus.

Note the NAS is running Samba 4.8.0

FWIW, a static lag on the switch with round robin on the NAS will in fact increase the throughput with a single client. But in the case of the RN528x that also depends on the PC being able to use both of it's NICs.

But in your case, that would maxmize the performance with the RN212 since either NIC in the PC can carry the traffic easily. Note that the round-robin only applies in packets the NAS is delivering. The packet transmission policy in the switch determines the loading of the two NICs in the switch->NAS direction. So it might only help reading from the RN212, and not writing to it.

StephenB you are a great contributor, thanks for the link and info. Some info/feedback:

- Quickly figured out how to hack my way through SSH. Without benefit of your link, I manually opened dozens of files in the etc/frontview/samba directory and subdirectories including addons, and couldn't find a single entry with "multichannel" so gave up. I'll go back now with the info you provided and try again

- Believe my PC has ample capability to utilize both 10Gbps NIC outputs, with two 970 Evo Plus SSD drives, 64GB RAM, etc. It's only Core i7-6700 but I'm confident it's not a bottleneck, nor is the Intel X550-T2. I can shuttle data between EVO's at 2-3 GB/s, and when one ETH port caps at almost exactly 10Gbps I'm pretty sure there's more gas left in the PC's tank

- With everything configured properly (per my earlier post on LAG), I never saw increased speed in either direction. And others reported similar results. Without multiple clients, there's no joy

- However, you hit on something very important. Yesterday, by accident, I found a way to achieve higher read speeds. I had torn down my LAG config (unbonded the NAS ETH ports, disabled LAGs in the switch, tore down the StaticLinkAggregation config of X550-T2), putting everything back to individual DHCP-assigned IP addresses but leaving all the dual cabling connected. Later, I turned on my RN314 (4x1TB drives, ancient 2013 box) and realized it was still configured for Bonding with RoundRobin. Did a quick ATTO test - and what do you know, read speeds were over 180 MB/s (it has 2x1GbE ports). Write speeds - still stuck around 100 MB/s. What the heck???

- So I thought, OK... let's re-do the ETH Bonding on RN212. No other changes. Sure enough, ATTO shows 210 MB/s read, 113 MB/s write. Appears to be following the behavior you described, to a tee.

- But two confusing/troubling issues. First, although ATTO and NAS Performance Tester 1.7 show these faster read speeds, when I do a simple file transfer from NAS-to-PC, and then PC-to-NAS, I see only about 110MB/s in either direction. Never do I see any speed exceeding a single 1 Gbps ETH link. Why is that?? Second thing is, with everything implemented (Static LAG's on switch, Intel X550-T2 static link-aggregated), I don't see any speed improvement at all, whether ATTO or file transfers.

- Ultimate goal is, move large blocks of data back and forth at maximum speed. The speed test apps are cool, but they don't seem to mirror an actual file transfer.

Feels like I'm making at least some progress now, thanks again for your great insights!

StephenB · ‎2020-05-10

@jimk1963 wrote:

First, although ATTO and NAS Performance Tester 1.7 show these faster read speeds, when I do a simple file transfer from NAS-to-PC, and then PC-to-NAS, I see only about 110MB/s in either direction. Never do I see any speed exceeding a single 1 Gbps ETH link. Why is that??

NAStester is simply timing file reads and file writes to the NAS share, so you should see similar speeds for large file transfers That is assuming that the RN212 is connected to the XS716 switch, and that you are transfering to/from the SSDs in your PC.

I don't know you aren't seeing that. Also, I don't know why you aren't seeing the gain with the LAG enabled on the switch.

One thing you could try is toggling the flow control setting on the RN212 ports (done on the switch) and see if that makes a difference.

jimk1963 · ‎2020-05-10

Thanks StephenB. Looking at the link you provided earlier, it's mainly about putting commands in a directory that stays persistent. Regarding the actual SMB Multichannel commands required, still searching. QNAP only requires a single change to their SMB.conf file, but appears Netgear's folder structure is completely different, even though it's also using Samba. More research required.

Re: RN212, yes it is connected directly to XS716E switch, same as RN528, RN314 and PC. All are connected directly to switch with 3' Cat7 cable pairs, total of 8 ports. All are working, each assigned its own IP address (unless/until a pair gets bonded of course).

Ran another puzzling test this morning: re-enabled bonding on RN528X (dual-10GbE), with RoundRobin and MTU=9014. Unbelievably, and unlike the RN212 and RN314, ATTO (and file transfer) speeds dropped to 573MB/s read, 250MB/s write - barely over half of what I was getting without bonding. Rebooted both the NAS and PC after first runs because I couldn't believe it. Same result after reboots. Again, config is PC w/ 2 independent 10GbE ETH ports (each assigned its own IP address over DHCP, no bonding), the XS716E 10GbE switch with no LAGs configured, and the RN528X as described above w/ bonding. The RN212/314 units have 1GbE, I'm thinking that bonding these is somehow OK for the PC but bonding to 10GbE ports on the RN528X is not OK for the PC. But I don't (yet) understand why. Since I've transferred at 10Gbps over each of the PC ETH ports previously, I know the X550-T2 NIC is OK. It just doesn't like to see a bonded pair of 10GbE ports on the NAS. Maybe it's a combo of the switch/PC/NAS, will experiment.

Sandshark · ‎2020-05-11

Both are basic Linux so as long as the Samba version in the ReadyNAS supports the command, whatever works for QNAP should work for ReadyNAS. ReadyNAS just separates the file into several based in the include statements at the end so share operations (add, delete, change protocols), OS updates, apps installs, and the like do not overwrite things they should not . One of the included files is tha addons.conf, which, as best I can tell, is overwritten by nothing in the OS itself. Anything you put in it is "in" smb.conf via the include statement.

jimk1963 · ‎2020-06-21

You could try enabling the setting in /etc/frontview/samba/addons/addons.conf as described here: https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Samba-manually-modify-etc-frontview...

Hi Guys, after some hiatus am back examining SMB multichannel. Picture attached is SSH view of RN212. In the etc/Frontview/Samba directory, I do not see any subdirectory named "addons" per the link provided earlier. The only "addons" directory I see is under etc/Frontview/Addons and that directory has but one file, addons.conf which has a "bin" designation after it so it appears to be a non-editable binary file. When I type "cat addons.conf" it returns nothing. I'm a newbie with SSH syntax, I used "ls -alr" to list out all hidden directories, etc., just to make sure I didn't miss something.

Basically, I'm stuck at this point. Not sure where the appropriate file is located to add smb multichannel commands.

Sandshark · ‎2020-06-21

If no apps have created /etc/frontview/samba/addons and it's addons.conf file, you have to create them yourself. If you look at /etc/samba/smb.conf, you'll see it is referenced in an include statement.

jimk1963 · ‎2020-06-21

Thanks @Sandshark ! I see the include statment "include = /etc/frontview/sama/addons/addons.conf" and understand I must now do the following:

1) Create subdirectory "/etc/frontview/samba/addons"

2) Create a text file in the above subdirectory called "addons.conf" with the desired SMB multichannel commands

Found the link below. Not sure if I'm fully clear on this, but believe I just need to add the 3 lines below to the "addons.conf" file to enable SMB multi-channel - sound right?

https://blog.chaospixel.com/linux/2016/09/samba-enable-smb-multichannel-support-on-linux.html

Enable multi-channel in smb.conf

This is really simple, just put:

server multi channel support = yes
aio read size = 1
aio write size = 1

in your smb.conf.

Sandshark · ‎2020-06-21

You need an applicability header for it in the file, too. Most likely [global].

jimk1963 · ‎2020-06-21

Thanks @Sandshark . I've created the directory using "mkdir addons", and have created the file using "touch addons.conf". So now I see the file. But how to edit addons.conf?? I've tried:

1) vim addons.conf --> command not found

2) nano addons.conf --> command not found

Searching the internet for solutions, all I can find are recommendations to add the above two commands somehow (also not clear, and am thinking I shouldn't need to do this).

jimk1963 · ‎2020-06-21

Sorry bad info from another site. The command "vi addons.conf" works for editing the file.

It now looks like this:

admin@xxx:/etc/frontview/samba/addons$ cat addons.conf
[global]
server multi channel support = yes
aio read size = 1
aio write size = 1
admin@Kirkpatrick2016:/etc/frontview/samba/addons$

I added a space between every word, and between the equals signs, because that's how it looks on the website I referenced. Hopefully that's correct. I assume I need to reboot the NAS so will do that now and see what happens.

jimk1963 · ‎2020-06-21

After all that, am seeing no better performance than before, on either the 10GbE RN528x or the 1GbE RN212. Config:

1) Intel X550-T2 NIC is in Static Lag mode, Windows shows the team as a 20 Gbps link

2) XS716E switch configured with static LAGs

Ports 1/2 = PC (X550-T2)

Ports 3/4 = RN212 (2 x 1 GbE)

Ports 5/6 = RN314 (2 x 1 GbE)

Ports 7/8 = RN528 (2 x 10 GbE)

3) RN212, 314, 528 all set identically:

Bonded Ports

MTU = 9014

Round Robin

Using NAS Performance Tester 1.7:

RN528: 770 MB/s Write 737 MB/s Read

RN212: 116 MB/s Write 242 MB/s Read

RN314: 122 MB/s Write 232 MB/s Read

Running actual file transfers over Windows Explorer, from PC-to-RN528 is ~ 550 MB/s and from NAS-to-PC is ~625 MB/s. Between RN528-RN212, transfers are capped at ~125 MB/s in either direction.

Basically, I have yet to successfully write to either of the 2x1GbE NAS's at 2 Gbps. Could be the commands I entered are insufficient for SMB Multichannel to work properly, or maybe I need Windows Server, or not sure what else to try. I'ts been educational but pretty frustrating as well.

StephenB · ‎2020-06-22

@jimk1963 wrote:

After all that, am seeing no better performance than before, on either the 10GbE RN528x or the 1GbE RN212. Config:

Using NAS Performance Tester 1.7:

RN528: 770 MB/s Write 737 MB/s Read

RN212: 116 MB/s Write 242 MB/s Read

RN314: 122 MB/s Write 232 MB/s Read

Running actual file transfers over Windows Explorer, from PC-to-RN528 is ~ 550 MB/s and from NAS-to-PC is ~625 MB/s. Between RN528-RN212, transfers are capped at ~125 MB/s in either direction.

Basically, I have yet to successfully write to either of the 2x1GbE NAS's at 2 Gbps. Could be the commands I entered are insufficient for SMB Multichannel to work properly, or maybe I need Windows Server, or not sure what else to try. I'ts been educational but pretty frustrating as well.

Maybe take a look here to double-check the SMB configuration on the PC: https://docs.microsoft.com/en-us/windows-server/storage/file-server/troubleshoot/smb-multichannel-tr...

Have you checked the end-to-end MTU with ping? In some Netgear switches the MTU needs to be set for both the port and the LAG.

jimk1963 · ‎2020-06-22

Thanks @StephenB , I've taken snapshots of the commands in your reference as I'm not sure how to interpret the info. The bonded X550-T2 ETH adapter on the PC is "Ethernet 10", based on the two NICs Ethernet 7/8. For Ethernet 10 (Interface 20), SMBClient /SMBServer both say TRUE under "RSS Capable" and FALSE under "RDMA Capable". Under Net Adapter Binding, most of the Eth 10 Component ID's say TRUE but two are FALSE. I don't see a specific Component ID listed for SMB but am not familiar with these bindings so may be missing it.

jimk1963 · ‎2020-06-22

Also @StephenB regarding your MTU question, the XS716E doesn't have a configurable MTU from what I can see in the management console, for either the port or the LAG. It's auto-populated at MTU=9198 and not editable.

Pinging any of the 3 NAS boxes from the PC, max payload without fragmenting is 8972 bytes so +28 gives MTU=9000 optimally if I understand correctly. The PC NIC's are set for jumbo frames (9014 bytes, not tune-able) and the NAS's are each set to MTU=9014. I can drop the NAS MTU's down to 9000 without affecting max payload but any lower and I start to see fragmenting. Curiously, setting NAS to MTU=8992 still results in max unfragmented payload (MUP) = 8972 bytes, but MTU=8991 or any lower number results in a byte-for-byte reduction in max payload. For example, NAS MTU = 8980 results in MUP = 8960. I thought the byte overhead was 28 bytes but I guess that's 20 bytes for TCP and 8 bytes for UDP... so I guess the generic ping command doesn't include the UDP header? Not sure about that. Anyway seems I'm getting max jumbo frames from the pings (8972 byte MUP).

schumaku · ‎2020-06-22

Please allow two comments:

@jimk1963 wrote:
After all that, am seeing no better performance than before, on either the 10GbE RN528x ...

Using NAS Performance Tester 1.7:
RN528: 770 MB/s Write 737 MB/s Read

Running actual file transfers over Windows Explorer, from PC-to-RN528 is ~ 550 MB/s and from NAS-to-PC is ~625 MB/s.

The RN528 performance number in the 700 MB/s range are ways below of what a 10 GbE interface can achieve. Unless you are near to some 1250 MB/s minus some protocol overhead ... let's say at least in the range around 1100 MB/s you can't achieve more bandwidth by using a LAG of two 10G interfaces or using individual interfaces in an SMB Multipath config. The achievable throughput depends on the storage blocks in the RN528[X]. On an all-SATA-Flash equipped RN628X with a X-RAID (RAID6) config, the performance does run in the higher range wher bonding (or an SMB Multipath) config might start to make some sense. On mechanical HDD I doubt you will be able to achieve much higher numbers.
Either use a LAG _or_ use SMB Multipath. SMB Multipath does always work over multiple individual network interfaces, each with a dedicated link and IP address.

jimk1963 · ‎2020-06-22

Thanks @schumaku for your inputs.

Please reference: https://www.downloads.netgear.com/files/GDC/READYNAS-100/ReadyNAS_OS_Performance_Guide.pdf

In this report, they use pedestrian Seagate HDD's. Their setup is different than mine - I'm using just one PC for testing, while in the report they use 4 PC's to push/pull data from the 2 10GbE ports via switches that sit in-between. As you can see, with the RN528X they can achieve 981 MB/s (just under 8 Gbps) writes and 2027 MB/s (16.2 Gbps) reads. Using HDD's. My box came with Toshiba MGO3ACA400 HDD's which are similarly spec'd.

With LAG, I've observed that multiple simultaneous PC transfers can in fact push the write speeds higher. However in this case, what I'm trying to accomplish is a faster lilnk to the 10GbE NAS from just a single PC. That's the entire goal of this thread and why I explored both (A) LAG and (B) SMB MultiChannel.

In this video for example, the engineer was able to demonstrate how SMB Multichannel did in fact improve speed over a single PC connected to a NAS: https://youtu.be/UjdPrCWiYwY ("SMB Multichannel - What is it? & QNAP NAS Setup")

In this case, the engineer is using Windows Server 2012 if memory serves. Windows Server has built-in, automatic SMB Multichannel capability. Windows 10 Home 64-bit can allegedly also support SMB MultiChannel, but again that's what I'm trying to prove out here. How do you get it working, and what is the benefit?

Regarding your observation to configure either (1) LAG or (2) SMB MultiChannel, I'm unclear on this point but will try out my newly NAS-implemented SMB Multichannel commands with the LAGs torn down to compare. It's certainly worth a try.

schumaku · ‎2020-06-22

Hope you understand why experimenting with LAG or Multipath does not make any sense if the set-up does not make high enough numbers to come near to the 10G performance.

Are your ~7xx MB/s 10G numbers coming from a real single threaded copy from/to your PC hardware, including read/write from the local storage?

Or are these pure benchmark numbers using /dev/null as source or destination, or a whatever benchmarking tool?

jimk1963 · ‎2020-06-22

Thanks @schumaku . If you're asking, am I transferring real files back and forth between the NAS and PC, the answer is yes. I'm transferring a number of multi-GB zip files. As the thread indicates, I'm also testing with NAS Performance Tester 1.7, Atto, Black Magic, and one or two others.

Regarding speed, and my understanding of it, I just explained the justification for higher speed expectation, namely, a reference to a Netgear test report which documents the numbers I just posted, using my same NAS model RN528X. I am not achieving these numbers and am looking to get closer to these. Additionally, if you followed the thread, you'd see I also have 2 NAS's with dual 1GbE ports. These are well below the bandwidth of my system (also well documented in the thread) and are also failing to achieve full 2 Gbps bidirectionally. These are also set up with LAG and now, with SMB MultiChannel.

Here's another example of an engineer making LAG work with a single PC, in this case 4 x 1 GbE and he's achieving 400 MB/s bidirectionally with pedestrian HDD's. I want to know how to enable this same performance but with 2 x 1 GbE. This was actually the reason I started the thread in the first place, as my 10GbE RN528X is great already. It's these old 1GbE NAS boxes I want to pep-up by taking advantage of their dual 1GbE NIC's. It's working in the Read direction, but not at all in the Write direction.

https://youtu.be/cCygGB2-6Uw

schumaku · ‎2020-06-22

> RN212: 116 MB/s Write 242 MB/s Read

> RN314: 122 MB/s Write 232 MB/s Read

Write: You are putting data sourcing the 10G interface (or a LAG'ed 2*10G) computer not applying any load balancing distribution mode (policy), or applying address hash policy. All data does source from one MAC address, and the switch does again forward everything from the same MAC source to one MAC on the destination.

A switch with a static LAG does not apply any Tx policy - it will ensure the same source MAC will be sent to the one destination MAC. This makes only one 1G channel is used from the switch to the NAS.

Read: The NAS does apply some Tx policy, e.g. in balance-rr (round robin) on each 1G interface so flooding both interface almost equally. The switch will put this to the one and only MAC of one 10G interface.

That makes the difference in your set-up ... or the reason why I think it is that way.