NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
M4300 LACP/LAGG only works when incorrectly configured
I have video proof this time, although I'm not sure if I can post it.
I had a misconfigured M4300 where one of the hosts had a VHID that overlapped the LACP ports. The LAGG functioned correctly and delivered 38gbs.
When fixed, I could no longer achieve the higher speeds- only 10gbs. However if I deleted the 4x ports that were attached to the LAGG, the transfer speed via iperf3 would jump to 40gbs, and stay there until the ARP table refreshed and the connection was lost.
Honestly at my wits end on configuring this thing.
For reference: two Chelsio cards connected via FS.com breakout 40gbe to 4x10gbe sfp+. The LAG is configure to use 4x adjacent ports (1/2/3/4) and 13/14/15/16. When I edit the LAG memberships by unchecking the individual ports, the speed will jump to 40gbe speeds- until it stops working. If I turn them back on, speed drops to 10gbe.
Suggestions? Debugging options?
8 Replies
There is a single 40G interface (QSFP) on the Chelsio - are the four channels really working together as dynamic LACP LAGs?
Does the switch see and show a 802.1ad LACP LAG which is correctly up?
You might talk to Chelsio when it comes to the support of break-out interfaces (they list similar stuff on Chelsio Hardware Compatibility Guide (Cable configs July 2012) and the required configuration of a multi-10G based LAG.
Without practical experience with such fan-out ... I would guess a static LAC is the way to go.
Indeed; they are configured/flashed as 4x10gb (that was one of the first issues- vendor shipped wrong).
The cards ONLY will talk at 40gbe when one of the interfaces is going up or down. Attached is a picture showing 40gbs bandwidth. I got that 'throughput' by unchecking the 4x ports assigned to the LAG, and hitting apply- then for the next 30 seconds or so iperf3 would show the requisite speeds.
The LAG is set to
Admin: Enable
Link Trap: Disable
STP: Enable
Static: Disable
I believe if I set it to 'static', the LAG isn't working with Chelsio- I believe this quote is what I'm following:
• Static Mode indicates whether or not LACP is used, i.e. Static Mode set to Disable means LACP is used.
(The inactive port is weird as it was working fine until this pull)
Anyone in tech?
Having cleaned out everything, can tech support provide a working LACP example script?
*************** show port status all ***************
Media STP Physical Physical Link Loop Partner Flow
Intf Type Mode Mode Status Status Status Control
------ ----------- ------ ---------- ---------- -------- -------- ------------
1/0/1 DAC DIS 10G Full 10G Full Up NA
1/0/2 DAC DIS 10G Full 10G Full Up NA
1/0/3 DAC DIS 10G Full 10G Full Up NA
1/0/4 DAC FWD 10G Full 10G Full Up NA
1/0/5 DAC DIS 10G Full 10G Full Up NA
1/0/6 DAC DIS 10G Full 10G Full Up NA
1/0/7 DAC DIS 10G Full 10G Full Up NA
1/0/8 DAC DIS 10G Full 10G Full Up NA
1/0/9 DAC FWD 10G Full 10G Full Up NA
1/0/10 DIS 10G Full Down NA
1/0/11 DIS 10G Full Down NA
1/0/12 DIS 10G Full Down NA
1/0/13 DAC FWD 10G Full 10G Full Up NA
1/0/14 DAC FWD 10G Full 10G Full Up NA
1/0/15 DAC FWD 10G Full 10G Full Up NA
1/0/16 DAC FWD 10G Full 10G Full Up NA
1/0/17 DAC DIS 10G Full 10G Full Up NA
1/0/18 DAC DIS 10G Full 10G Full Up NA
1/0/19 DAC DIS 10G Full 10G Full Up NA
1/0/20 DAC DIS 10G Full 10G Full Up NA
1/0/21 DIS 10G Full Down NA
1/0/22 DIS 10G Full Down NA
1/0/23 DIS 10G Full Down NA
1/0/24 DAC FWD 10G Full 10G Full Up NA*************** show port-channel all ***************
Log. Channel Admin Mbr Device/ Port Port
Intf Name Min Link Mode Type Ports Timeout Speed Active
------- --------------- ---- ------ ------ ---- ------- ------------- --------- -------
lag 1 NAS15x50 1 Up En. Dyn. 1/0/1 actor/long 10G Full True
partner/long
1/0/2 actor/long 10G Full True
partner/long
1/0/3 actor/long 10G Full True
partner/long
1/0/4 actor/long 10G Full False
partner/long
lag 2 NAS16x51 1 Up En. Dyn. 1/0/5 actor/long 10G Full True
partner/long
1/0/6 actor/long 10G Full True
partner/long
1/0/7 actor/long 10G Full True
partner/long
1/0/8 actor/long 10G Full True
partner/long
lag 3 NAS11x60 1 Up En. Dyn. 1/0/17 actor/long 10G Full True
partner/long
1/0/18 actor/long 10G Full True
partner/long
1/0/19 actor/long 10G Full True
partner/long
1/0/20 actor/long 10G Full True
partner/longI have video proof this time, although I'm not sure if I can post it.
If you have a google account, you can post the video in YouTube. Then, post the YouTube video link here.
Having cleaned out everything, can tech support provide a working LACP example script?
I think NETGEAR Tech Support people are not able to monitor the NETGEAR community from time-to-time since they are more focused on the support tickets opened by NETGEAR clients. It would be best if you open a chat or online support ticket with NETGEAR Support at anytime with regard to your concern. Let us know how it goes.
Regards,
DaneA
NETGEAR Community Team
Well, the OP isn't able to provide any indication (links, documentation, whatever, ...) that these 40G cards or the host should be workable on x10GbE links, and wht configuration is required. In my understanding - unless there is something specifically configured on the card and/or the related host - with all due respect, any support attempt is a waste of time. We only get shown whats on the switch, nothing from the peer.
Well, hadn't realized that was what was required- but thanks for that hint!
The cards are Chelsio T580s.
https://www.chelsio.com/nic/40gigabit-ethernet-40gbe-adapters/
They are configured in 4x10gb mode, and quite capable of pushing 4x10gbE in LACP mode- in a Cisco switch.
https://github.com/freenas/freenas/blob/master/src/freenas/usr/local/bin/chelsio_adapter_config
The switch was reset to default configuration and ports were added to each of the LAGGs- 4x on each LAGG, and L3 hash was chosen.
The 'static' mode is disabled- cards will not work with that- and TrueNAS is set to LACP.
Running iperf3 forward with 8 threads shows 10gbE bandwidth. The crazy part- IF I uncheck the ports associated with the LAGG of the initiating machine (so, on 1/2/3/4 with machine 1), I can run iperf3 and see 40gb/s of transfer speed aggregate. This lasts for about 30 seconds, then the connection dies (since there are no ports associated with the LAGG anymore). Re-enabling all of the ports results in this running at 10gbE speeds again.
Even more crazy if I run iperf3 in reverse (using the -R) on machine 1, and uncheck the ports associated with machine 2, it'll run at 40gbe speeds until, again, the connection dies.
The only other time everything worked correctly was when an invalid port was assigned to the LAGG due to an ARP conflict. It's as if the switch simply can't accept a correctly working LACP setup.
Which brings me back to- if I've followed the directions to set up LACP correctly on the M4300, why does it only work during transitioning states?
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
