SMB over RN528X not working

[StephenB]

---

@schumaku wrote:

A switch with a static LAG does not apply any Tx policy - it will ensure the same source MAC will be sent to the one destination MAC. This makes only one 1G channel is used from the switch to the NAS.

Read: The NAS does apply some Tx policy, e.g. in balance-rr (round robin) on each 1G interface so flooding both interface almost equally. The switch will put this to the one and only MAC of one 10G interface.

---

I agree the switch policy is part of the problem (although technically the switch is using a policy, since it does have to decide what NIC to use to send each packet). Likely it is using an XOR hash of the source and destination MAC.  If that is the case, even if fhe PC is using both NICs for writes, there is a 50-50 chance the switch will undo that, and send all the traffic for the flow to one NIC. 

FWIW, a direct connect of the PC to the NAS would allow @jimk1963 to take the switch policy of of the equation.

This article (though older) might be more informing than the one I sent earlier:  https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn...

@jimk1963  does seem to have RSS enabled on the 20 GB NIC.

I also suggest switching the MTU back to 1500 on both the NAS and the PC, and see what difference that makes.  There shouldn't be any packet fragmentation, and if I am reading the earlier posts correctly, there is some.

Overall, I also am not saturating a single 10GBASE-T link with my own RN526x.  I haven't attempted to duplicate Netgear's performance numbers (as my current speeds are enough for my purpose).  But I wouldn't expect much gain from multichannel if the overall setup can't saturate a single connection.

[jimk1963]

Thanks @schumaku  and @StephenB , great info. I'm an RF guy attempting to better understand network theory, your tutorials are much appreciated. Next up:

1) try direct PC-to-NAS connection to eliminate any switch policy/configuration issues 

2) try MTU=1500... not sure why this will help, I have jumbo frames enabled all the way through and don't see fragmentation with packets up to 8972 bytes. But I'll try it anyway

3) Per @schumaku , I removed LAG and kept SMB Multichannel... so now there are two PC NIC outputs with individual DHCP-assigned IP addresses, same for all 3 NAS boxes. The SMB Multichanmel commands I noted earlier remain in the /addons subfolder of all 3 NAS. With this configuration, NAS Tester read performance is now cut in half on RN212/314 (120 MB/s vs 230-240 MB/s) and the RN528X is having an even worse issue with reads below 100 MB/s. Rebooted everything, no improvement. Actual PC-to-RN528X file transfers are in fact showing this sub-100 MB/s speed so something is definitely wrong. The other two NAS transfer files at very close to the 1GbE limit so seems something is wrong with the RN528X itself. Will debug.

[schumaku]

Well, SMB Multichannel is not intended to provide a single session high-speed-trunk - this is in place for serving multiple clients.

[StephenB]

---

@jimk1963 wrote:

3) Per @schumaku , I removed LAG and kept SMB Multichannel... so now there are two PC NIC outputs with individual DHCP-assigned IP addresses, same for all 3 NAS boxes. 

---

You could have kept the LAGs on the NAS, and only dropped it in the PC.

In Windows, getting more performance with multichannel with a single session requires RSS on each link.  If you have that, then it is supposed to use one channel per CPU core even if you are using teaming. Also, looking at the load balancing modes available in Windows - you should have the static lag configured to use "Dynamic"   Though I think the switch will undo that.

On the NAS side, RSS might not be set up at all (or might not be set up correctly).  That's not something I've ever played with though.

On the MTU size - it might not make a difference.  But when you are transferring large files, the system will be using the full MTU size.  And packet fragmentation will slow down the transfers.  The receiving system has to defragment, and that does add latency.  If the MTU size is set correctly, you won't see any packet fragmentation in the NAS or the PC.  You likely will see them on other devices.

Also, there are many deployments where jumbo frames slow down performance, and don't speed it up.  The performance gain is largely because the PC and the NAS aren't processing as many packets per second.  If the edge devices can keep up with the packets per second rate, then jumbo frames don't speed anything up.   In your case, you want to keep both links loaded.  More packets per second might actually help with that.

Generally my advice here is that if jumbo frames aren't significantly improving your performance, then you should turn them off.  FWIW, they don't improve my own performance enough to be worth the bother.

[schumaku]

RSS is between the adapter driver, the TCP offload engine, and the processing on the host and comes into the game where more CPU performance is as available from a single core. 

Correct is that the obvious advantage of Jumbo Frames in added throughput is much less obvious then it was back in the times of 200 MHz Inhell processors or 800 MHz ARM single cores serving a NAS.  Powerful servers and NAS with the "right" interfaces make use of TCP offloading to the network adapter. This was originally done to have more CPU ticks available on the CPU for doing it's job. Similar what RSS does so more than one core can work on an interface.

Appears there are a lot of strange ideas on how Jumbo Frames (JF) are handled in today's networking equipment. Packet fragmentation e.g. for TCP sessions does rarely happen on the data path - the PMTUD does always negotiate the complete path. All IPv4 end points should, all IPv6 end points must support this handshake. Permitting the interfaces are working correct and routers are properly handling it (some consumer router garbage doesn't) the stack does handle the MTU for each connection correctly. Old legacy routers might do some hard TCP fragmenting - in reality this should not happen anymore today. And this does no happen on modern L2 networks where most users are operating NAS. The small PMTUD three way handshake during the session establishment does not kill performance. The days where we had to tell people that the MTU eg. the lowest MTU support on it's stack is dictating the network maximum MTU are history - everything works perfectly transparent nowadays.

[StephenB]

---

@schumaku wrote:

Appears there are a lot of strange ideas on how Jumbo Frames (JF) are handled in today's networking equipment. 

---

@jimk1963 is seeing some fragmentation, so something isn't negotiating quite right on his equipment.

---

@schumaku wrote:

This was originally done to have more CPU ticks available on the CPU for doing it's job. Similar what RSS does so more than one core can work on an interface.

Right.  Also- one cost of using Jumbo Frames is that you either need to have more memory for packet buffering, or you need to buffer fewer packets.

With my own equipment, I've sometimes seen that JFs have actually reduced my throughput.  I don't think I've seen a substantial gain in anything (though it's not something I've measured recently).

[jimk1963]

Thanks @StephenB  and @schumaku  for your inputs, I've run some tests that you may find of interest.

General comments:

1) I wasn't reporting earlier that the system suffered from fragmentation. Rather, I just ran tests at different MTU's to force fragmentation so I could confirm those boundaries. With MTU=9014 on both Intel X550 NIC and on the NAS, I don't see any fragmentation

2) I'm not convinced at all that Jumbo Frames are immaterial to performance in modern systems. My test data shows quite the opposite, maybe you can enlighten me as to why

3) By far - the best performance I can achieve is 1 ETH connection directly to the NAS, where the NAS is in Bonded Mode (I ran all tests with NAS in bonded mode so cannot comment on single NAS ETH connection). In this mode, I achieved writes in the mid-800 MB/s range and reads in the 1100-1200 MB/s range using NAS Tester 1.7. Using actual file transfers I saw writes in the 700 MB/s range and reads in the 900-1000 MB/s range. Far better than anything I achieved with a switch in series or with Static LAG.

4) With the switch included, one PC ETH and NAS Bonding with all Jumbo Frames (last run in table), I'm at 400MB/s reads and 900 MB/s writes, back to where I started.

What I don't understand is, compariing Run 1 to Run 12, why adding the switch in series is causing so much degradation in Reads. The switch doesn't offer much configuration, mainly static LAG (which is disabled for those runs). 

I'm sure some of these runs are considered "invalid" by network experts, for example I'm not sure setting up the PC with static LAG with a direct-to-NAS connection makes any sense. From what I've read a switch or router should be in-between but anyway, the data is there for analysis.

Also, the few ATTO/Black Magic runs don't agree well with the NAS Tester 1.7 numbers. Guessing they do things quite differently and probably ATTO and BM are more realistic (I guess). 

[StephenB]

---

@jimk1963 wrote:

2) I'm not convinced at all that Jumbo Frames are immaterial to performance in modern systems. My test data shows quite the opposite, maybe you can enlighten me as to why

I think we already did explain that.  The effect of JF on the ethernet itself is immaterial.  The improvement in performance (when there is one) is only because the CPUs in the systems (NAS and PC in this case) are processing fewer packets per second.  In some cases that can help (and it seems to be doing that in your system).  In other cases, the offload processing of the NIC cards offset the CPU gain (achieving a similar result in a different way).

---

@jimk1963 wrote:

What I don't understand is, compariing Run 1 to Run 12, why adding the switch in series is causing so much degradation in Reads. The switch doesn't offer much configuration, mainly static LAG (which is disabled for those runs). 

---

Maybe look at the packet statistics on the switch before and after the test (looking for errors).

Also, is flow control enabled on the switch ports?

You could repeat test 12, but remove one of the ethernet connections from the NAS to the switch.

[jimk1963]

Thanks @StephenB :

1) Re: JF, I was replying to @schumaku 's comment that modern systems don't have this bandwidth limitation. Appears that it does show up again in the world of 10Gbps ETH file transfers...  The Core i7-6700 PC (2016 vintage) is loaded with EVO 970 Plus cards (C and D drives) and 64GB DDR4-2166 RAM (slow by today's standards but still tons of bandwidth). File transfers over PCIe between EVO 970 SSD's easily hit 3GB/s. The Intel X550-T2 has offloading capability as well. Not much more I can do there but I did take your point to heart so ran some JF tests (more below). I also have a Threadripper 3970 machine that is definitely not bottlenecked, will try that with/without JF's in a future test.

2) XS716E switch reports CRC Packet Error Statistics as shown below. If this is what you're referring to, I'm not seeing any errors.

3) Repeating Test 12 with one NAS ETH disconnected - tried this with (a) NAS ETH Bonding left enabled, and (b) NAS ETH Bonding torn down. Same result. A single NAS cable is the best config, producing over 800 MB/s write and well over 1100 MB/s read as shown below. Config is:  PC = 1 ETH to XS716E, JF=9014, 1 ETH disabled; Switch = No LAG; RN528X = 1 ETH to XS716E, JF=9014.

4) Experimented with different MTU sizes. JF on only one side (either PC or NAS) with 1500 on the other side gives poor results, as expected. Didn't include those numbers but they are basically the same as Column 3 below (PC/NAS both at MTU=1500). Setting the NAS to a fixed MTU=9014, and stepping up the PC MTU (first MTU=1500 in Column 3, then MTU=4088 in Column 2, then MTU=9014 in Column 1), the reads and writes continue to climb with highest JF packet size. This trend correlated perfectly with file transfers between PC-NAS using File Explorer. So I guess this confirms your hypothesis that the PC may be struggling with smaller packet size. Monitored CPU loading using Task Manager, and did observe that when PC MTU=1500 the CPU was peaking at 100% during NAS reads (even though the reported average never climbed above 37%). With MTU=9014 the peaks never get above 80%. Pic of each below.

Through all this, I've left those added SMB Multichannel commands in the NAS. Guessing they are harmless, and useless, at this point but just for completeness, all tests in these tables do have the commands in the \addons.conf file.

First pic:  PC MTU=1500, NAS MTU=9014.  Second pic:  PC & NAS MTU = 9014

[jimk1963]

Final configuration:

PC:

- MTU=9014

- Static Link Aggregation Enabled

XS716E Switch:

- Ports 1/2 (RN528X):  no Static LAG

- Ports 3/4 (RN212):  Static LAG Enabled

- Ports 5/6  (RN14):  Static LAG Enabled

- Ports 7/8 (PC):  Static LAG Enabled

RN212 and RN314:

- ETH Bonding Enabled

- MTU=9014

- SMB Multichannel "enabled" (if I did it right)

RN528X:

- No ETH Bonding

- One ETH port active, the other disabled

- SMB Multichannel "enabled" 

With this configuration, I can achieve read/write file transfer speeds around:

RN528X:  810/560 MB/s  (1170/850 on NAS Tester 1.7)

RN212:     135/90  MB/s ,  205/135 MB/s simultaneous multi-file transfers,  (235/110 on NAS Tester 1.7)

RN314:     166/101 MB/s ,  same with simultaneous multi-file transfers, (236/108 on NAS Tester 1.7)

This configuration gives me the fastest 10GbE performance with the RN528X, while also enabling faster reads on both the RN212 and RN314. Interestingly, the RN314 with the 4 new WD Red 4TB drives can read up to around 170MB/s, not bad for slow-spinning HDD's. They are dead quiet so I'm happy. Reliability.. we'll see. Also interestingly, the RN212 with its original Toshiba 7200RPM HDD's (circa 2014) does great when copying multiple groups of files simultaneously (over 200 MB/s) but can't quite keep up with the RN314 with just a single block of files. Of course, the RN314 does have 4 disks compared to only 2 in the RN212... 

Thanks @StephenB , @schumaku  and @Sandshark  for all your insight and course corrections. System is tuned about as good as I'm going to get it.