M4300 LACP/LAGG only works when incorrectly configured
I have video proof this time, although I'm not sure if I can post it.
I had a misconfigured M4300 where one of the hosts had a VHID that overlapped the LACP ports. The LAGG functioned correctly and delivered 38gbs.
When fixed, I could no longer achieve the higher speeds- only 10gbs. However if I deleted the 4x ports that were attached to the LAGG, the transfer speed via iperf3 would jump to 40gbs, and stay there until the ARP table refreshed and the connection was lost.
Honestly at my wits end on configuring this thing.
For reference: two Chelsio cards connected via FS.com breakout 40gbe to 4x10gbe sfp+. The LAG is configure to use 4x adjacent ports (1/2/3/4) and 13/14/15/16. When I edit the LAG memberships by unchecking the individual ports, the speed will jump to 40gbe speeds- until it stops working. If I turn them back on, speed drops to 10gbe.
Suggestions? Debugging options?
Model: XSM4348S | M4300-24X24FStackable Managed Switch with 48x10G including 24x10GBASE-T and 24xSFP+ Layer 3
Re: M4300 LACP/LAGG only works when incorrectly configured
Indeed; they are configured/flashed as 4x10gb (that was one of the first issues- vendor shipped wrong).
The cards ONLY will talk at 40gbe when one of the interfaces is going up or down. Attached is a picture showing 40gbs bandwidth. I got that 'throughput' by unchecking the 4x ports assigned to the LAG, and hitting apply- then for the next 30 seconds or so iperf3 would show the requisite speeds.
The LAG is set to
Admin: Enable
Link Trap: Disable
STP: Enable
Static: Disable
I believe if I set it to 'static', the LAG isn't working with Chelsio- I believe this quote is what I'm following:
• Static Mode indicates whether or not LACP is used, i.e. Static Mode set to Disable means LACP is used.
Re: M4300 LACP/LAGG only works when incorrectly configured
(The inactive port is weird as it was working fine until this pull)
Anyone in tech?
Having cleaned out everything, can tech support provide a working LACP example script?
*************** show port status all ***************
Media STP Physical Physical Link Loop Partner Flow Intf Type Mode Mode Status Status Status Control ------ ----------- ------ ---------- ---------- -------- -------- ------------ 1/0/1 DAC DIS 10G Full 10G Full Up NA 1/0/2 DAC DIS 10G Full 10G Full Up NA 1/0/3 DAC DIS 10G Full 10G Full Up NA 1/0/4 DAC FWD 10G Full 10G Full Up NA 1/0/5 DAC DIS 10G Full 10G Full Up NA 1/0/6 DAC DIS 10G Full 10G Full Up NA 1/0/7 DAC DIS 10G Full 10G Full Up NA 1/0/8 DAC DIS 10G Full 10G Full Up NA 1/0/9 DAC FWD 10G Full 10G Full Up NA 1/0/10 DIS 10G Full Down NA 1/0/11 DIS 10G Full Down NA 1/0/12 DIS 10G Full Down NA 1/0/13 DAC FWD 10G Full 10G Full Up NA 1/0/14 DAC FWD 10G Full 10G Full Up NA 1/0/15 DAC FWD 10G Full 10G Full Up NA 1/0/16 DAC FWD 10G Full 10G Full Up NA 1/0/17 DAC DIS 10G Full 10G Full Up NA 1/0/18 DAC DIS 10G Full 10G Full Up NA 1/0/19 DAC DIS 10G Full 10G Full Up NA 1/0/20 DAC DIS 10G Full 10G Full Up NA 1/0/21 DIS 10G Full Down NA 1/0/22 DIS 10G Full Down NA 1/0/23 DIS 10G Full Down NA 1/0/24 DAC FWD 10G Full 10G Full Up NA
*************** show port-channel all ***************
Log. Channel Admin Mbr Device/ Port Port Intf Name Min Link Mode Type Ports Timeout Speed Active ------- --------------- ---- ------ ------ ---- ------- ------------- --------- ------- lag 1 NAS15x50 1 Up En. Dyn. 1/0/1 actor/long 10G Full True partner/long 1/0/2 actor/long 10G Full True partner/long 1/0/3 actor/long 10G Full True partner/long 1/0/4 actor/long 10G Full False partner/long lag 2 NAS16x51 1 Up En. Dyn. 1/0/5 actor/long 10G Full True partner/long 1/0/6 actor/long 10G Full True partner/long 1/0/7 actor/long 10G Full True partner/long 1/0/8 actor/long 10G Full True partner/long lag 3 NAS11x60 1 Up En. Dyn. 1/0/17 actor/long 10G Full True partner/long 1/0/18 actor/long 10G Full True partner/long 1/0/19 actor/long 10G Full True partner/long 1/0/20 actor/long 10G Full True partner/long
I have video proof this time, although I'm not sure if I can post it.
If you have a google account, you can post the video in YouTube. Then, post the YouTube video link here.
Having cleaned out everything, can tech support provide a working LACP example script?
I think NETGEAR Tech Support people are not able to monitor the NETGEAR community from time-to-time since they are more focused on the support tickets opened by NETGEAR clients. It would be best if you open a chat or online support ticket with NETGEAR Support at anytime with regard to your concern. Let us know how it goes.
Re: M4300 LACP/LAGG only works when incorrectly configured
Well, the OP isn't able to provide any indication (links, documentation, whatever, ...) that these 40G cards or the host should be workable on x10GbE links, and wht configuration is required. In my understanding - unless there is something specifically configured on the card and/or the related host - with all due respect, any support attempt is a waste of time. We only get shown whats on the switch, nothing from the peer.
The switch was reset to default configuration and ports were added to each of the LAGGs- 4x on each LAGG, and L3 hash was chosen.
The 'static' mode is disabled- cards will not work with that- and TrueNAS is set to LACP.
Running iperf3 forward with 8 threads shows 10gbE bandwidth. The crazy part- IF I uncheck the ports associated with the LAGG of the initiating machine (so, on 1/2/3/4 with machine 1), I can run iperf3 and see 40gb/s of transfer speed aggregate. This lasts for about 30 seconds, then the connection dies (since there are no ports associated with the LAGG anymore). Re-enabling all of the ports results in this running at 10gbE speeds again.
Even more crazy if I run iperf3 in reverse (using the -R) on machine 1, and uncheck the ports associated with machine 2, it'll run at 40gbe speeds until, again, the connection dies.
The only other time everything worked correctly was when an invalid port was assigned to the LAGG due to an ARP conflict. It's as if the switch simply can't accept a correctly working LACP setup.
Which brings me back to- if I've followed the directions to set up LACP correctly on the M4300, why does it only work during transitioning states?
I'm not a video guy so I have no idea how to enlarge portions of it to show you better, but at HD you should be able to watch the speed suddenly increase once I uncheck the ports associated with the LAG membership, and then decrease when I add the ports back in.
