Discussion stats
  • 38 replies
  • 7487 views
  • 4 kudos
  • 9 in conversation
Announcements

Top Contributors
Reply
Highlighted
Tutor

STP Leak using Broadcast packet 01:80:c2:00:00:1c

Netgear Support,

 

Please take note that the community and several customers have discovered a possible flaw in your STP implementation that causes CPU panic and circular broadcasts that disable switches and some other vendor gear.

 

The  effect happens when the packet 01:80:c2:00:00:1c is broadcast from a Ubiquity AP.  The scenario is as follows.

If you have two or more switches connected with lags and redundant links in a circular fashion, with STP and MSTP on.  the redundant links.

The redundant links show that they are disgarding packets as normal.

However this packet is 01:80:c2:00:00:1c is getting past the stp disgard and continues to circulate around the links in perpetuity and is being constistently reflected by the switches.

The only respit is to remove the redundant link or put a MAC ACL on the lags to prevent the panic.

 

Please look into this issue.

I am runing the latest FW 6.3.1.16

 

The issue is well documented on the UBNT forums here, and we are asking both Netgear and UBNT to figure out what is going on and how to repair this ...

 

https://community.ubnt.com/t5/UniFi-Wireless/Firmware-3-7-x-and-NetGear-Switches-is-completely-broke...

 

Thanks

Rich

Model: GS716Tv3|ProSAFE 16-port Gigabit Smart Switch
Message 1 of 39

Accepted Solutions
Highlighted
NETGEAR Moderator

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

@LarryV and to others, 

 

Since a firmware fix is not yet available, the MAC ACL workaround can be used as described on the article below:

 

Adding a MAC ACL on Fully and Smart Managed switches to prevent STP leak of multicast packet from 3r...

 

 

Regards,

 

DaneA

NETGEAR Communtiy Team

View solution in original post

Message 38 of 39

All Replies
Highlighted
Tutor

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Has anyone from Netgear seen this and can comment?

Message 2 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Please comment someone

 

 

Message 3 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Hi

 

Thank you for your messages. Please understand that the Community would really need, at very least, the Model Number of the switch presenting the issue. I'm sure this would trigger faster response rate.

 

But yes, to answer your questions, several key people in NETGEAR Community as well as several NETGEAR employees at Tech Support, Engineering Q&A and PLM heard about this new issue.

 

I am not sure which switches you have, but in case these are Smart Managed Switches (GS1xxTxx, GS5xxTxx, GS7xxTxx, XS7xxTxx) or above, please use the solution provided in this thread. Please enable Storm Control first. If not sufficient, you can use a MAC Access Control List (ACL) to drop packets with a target hardware address of 01:80:c2:00:00:1c. You can attach this ACL to your ports and your LAGs between your switches.

original.png

 

Please let us know how it goes, while we are investigating on our side.

 

Regards,

Laurent Masia
NETGEAR Senior Product Line Manager, Managed Switches
Message 4 of 39
Highlighted
Tutor

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Thanks for the consideration and looking into it.

I currently have the MAC acl on and that prevents the packets.  But that is not a long term fix.

I have two GS716Tv3 and one GS110TP all running the latest firmware.

The three switches run 5 vlans and STP (MSTP) is running.

All interconncected.

 

Hope that helps diagnose it.

Model: GS716Tv3|ProSAFE 16-port Gigabit Smart Switch
Message 5 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Thank you XDRich, yes it helps.

 

We can reproduce the issue now. The offending packet is a multicast packet with MAC destination address of “01:80:c2:00:00:01c” that was generated by Ubiquiti’s new wifi AP software 3.7.37 release. The switch would react negatively as you’d reported when it received such packet, but there is no long term impact to the switch.

 

We’ll test all Netgear switches to ensure if any other models might be affected to this same issue and for affected series, we will provide permanent fix without requiring filtering out these packets using MAC ACL.

 

We will regularly update the team here on our progress.

 

Regards, 

Laurent Masia
NETGEAR Senior Product Line Manager, Smart Switches & Managed Switches

Laurent Masia
NETGEAR Senior Product Line Manager, Managed Switches
Message 6 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Having same issue with Unifi AC-Pro access points with fw 3.7 and Netgeat switches in spanning tree (rstp).
Netgear switches in spaning tree: GS728TXS, GS752TXS, M7100-24x, GSM7248V2

Message 7 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Hi elderone

 

Thanks for your feedback. We are adding these switches to our tests. We started troubleshooting operations simultaneously.

 

In the meantime, I strongly advise MAC ACL as described above in this post, so that you temporarily alleviate the issue in your network.

 

Regards,

Laurent Masia
NETGEAR Senior Product Line Manager, Managed Switches
Message 8 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Hi elderone

Welcome to NETGEAR community.

Can you kindly provide us the detailed GS728TXS/GS752TX/M7100-24X topology when you saw the same issue with Unifi AC-Pro access points?

BR,
Netgear Employee
Message 9 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Here's network layout:
network.png

RSTP keeps link open between GS728TXS and GSM7248v2.


Tried also connecting AP-s to GS728TXS, at the same time I did block MAC 01:80:c2:00:00:1c on GS728TXS uplink ports. "My PC" did get access to Unifi AP-s fine, but most of the time Unifi controller didn't get data from AP-s - controller showed that AP-s missing heartbeats, disconnected and occasionally connected.

Message 10 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Same problem here with the new Ubiquiti UAP-AC-HD with 3x GS752TXS in a stack and 3x XS712T's hanging off of the strack in a LACP trunk.

Model: GS752TXS|NETGEAR 48-port Stackable Smart Switch with 10G, XS712T|ProSAFE 12-port Smart Switch with 10G fiber|EOL
Message 11 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

For the workaround is it sufficent to just apply the MAC ACL to the port that the AP is plugged into? or do I need to apply it to all ports across all switches?

Message 12 of 39
Highlighted
Tutor

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Hi,  Welcome Red Techie.

I am patient zero on this even.

 

Yes putting the mac acl will work on the port, but that will kill the purpose of the packet, which is quick notify to other AP's.

The problem is the STP leak of this packet on the LAGS.

 

Pick one of the lags that create your circular network and add it there.  That prevents the reflection in the network while allowing the packet to get out and be used.

 

Netgear ... Whats the update to this issue?  When can we expect a FW fix? I suspect you will be getting more users figuring out this is a problem.

 

-Rich

Message 13 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Please be sure Engineering team is working on this issue across our portfolio. So MAC ACL is temporary workaround only. But very efficient.

Yes please apply the MAC ACL discarding these packets, on all ports which are connecting the Access Points. It is not necessary to bind the ACL with other ports.

Regards,
Laurent Masia
NETGEAR Senior Product Line Manager, Managed Switches
Message 14 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

I have a few LAGs and the native stacking functioanlity of the GS752TXS's enabled. See Image.

LAGs.PNG

 

Do I need to add it to all of them (including the stacked links)?

 

 

Also the "quick notify" is that the zero-handoff config? As I am not using that:
zero-handoff.PNG

 

Or is quick notify something completely different with the UBNT hardware?

Message 15 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

We have the same issue.

 

Our Netgear switch setup looks like this:

 

  • Level One: Switch Stack of 2 x NetGear GS752TS Switches
  • Level Two: Switch Stack of 5 x NetGear GS752TS Switches

These stacks are connected via a 6 port LAG. Could we please add the GS752TS to the list of switches needing a FW fix for this issue.

 

Message 16 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

@XDRich Could you further explain "Pick one of the lags that create your circular network and add it there.  That prevents the reflection in the network while allowing the packet to get out and be used."? I just recently bought a second UAP-AC-HD and would like to take advantage of the quick roaming (802.11r) feature but not have my network melt at the same time Smiley Happy (See attached image for switch topology) What link should I enable the ACL on? Just one of the Stack links, multiple? what about the LACP trunks?

 

Switch Topology.png

 

 

@LaurentMa Any update on the pending firmware fix from engineering? 

 

 

Thanks!

Message 17 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Hi @RedTechie and @elderone,

 

We didn't reproduce the issue locally with 1* M7100 (v11.0.0.28) and 2* GS728TXS (v6.1.0.35), 2 GS728TXS are in stacking, and connect to M7100 through LACP, enable RSTP on all ports and LAGs.

Then use two IXIA ports to simulate broadcast packet 01:80:c2:00:00:1c (1Mbps), all 3 DUT Web GUI can be accessable without hanging.

So appreciate if you could help confirm below?

1> Did you DUTs upgrade to the latest FW? If no, please download center to get

2> What about throughput of broadcast packet 01:80:c2:00:00:1c (generated by AP)?

3> Yous issue is DUT hang without Web access?

4> If possbile, can you help send back config of all DUTs for our further debugging?

     How to get config for smart switch GS752TXS and GS728TXS - http://kb.netgear.com/31438/How-do-I-send-diagnostic-files-from-my-Smart-Switch-to-NETGEAR-community...

    How to get config for M7100 - http://kb.netgear.com/31439/How-do-I-send-diagnostic-files-from-my-Managed-Switch-to-NETGEAR-communi...

BR,
Netgear Employee
Message 18 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

I do have latest firmware on GS728TXS and GSM7248v2 switches (where AP-s where connected). Other 3 switches are not with the latest firmware.
I don't have LAGs or LACP in use.

During that time when Unifi AP-s where on 3.7 firmware and inaccessible, there was constant 110 Kbit/s traffic from switch port where AP was connected.
Also somewhere in than timeframe my M7100 switch WEB GUI stopped working. Otherwise switching works, snmp monitoring and remote loggin also works. Only web gui won't respond.
Downgraded Unifi firmware back to 3.4 for now.

network.png

Message 19 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

@Bruce_G, I'm not sure what DUTs are? I'm assuming thats a technical name for a switch? I'm confused as I thought a previous post mentioned that Engineering was able to reproduce the issue?

 

 

1. My three GS752TXS's are on FW v6.1.0.27 and boot vB6.1.0.1. My XS712T's are all at FW v6.1.0.34 and boot vB6.1.0.3

 

2. I don't know... I noticed ALL status lights on all the switches blanking WAY faster than they do normally and the fact that the AP was only responding to a single ping every 1min with constant heartbeat signal misses to the controller. Other than that everything else was working fine (including Switch GUIs). I only found the issue when I turned on wireshark after noticing the abnormally fast blinking lights.

 

3. No, my switche's Web GUI always worked even durring the issue.

 

4. When I follow those steps to get the logs on any of my switches I get an Error 404 when i hit apply to download (upload, is what its called in the Gui) the logs.

Message 20 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Hi @RedTechie and @elderone,

 

Very appreciate for the information providing, our development team are working on it and will provide the fixing soon.

BR,
Netgear Employee
Message 21 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

@Bruce_G / @LaurentMa

 

It's been a number of weeks. Any update on this bug? And the firmware release ETA?

Message 22 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Hi RedTechie

 

We have worked on this issue with top priority, warm thanks to all contributors (tech support file exports, configuration file exports etc..) on this post.

 

Yes we have internal new software releases for M4100, M5300, M6100, M7100 Managed Switches and for FS728TPv2, GS728TS/TPS, GS752TS/TPS, S3300 and XS708T/XS716T Smart Switches that are in the process of heavy validation testing before web posting. We will come back here asap to deliver more precise expected public release ETA on the above.  

 

For M4200, M4300 Managed Switches and for all other impacted Smart Switches, we are working hard on the fix as I speak.

 

Regards,

Laurent Masia
NETGEAR Senior Product Line Manager, Managed Switches
Message 23 of 39
Highlighted
Aspirant

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

@LaurentMa Updates?

Message 24 of 39
Highlighted
NETGEAR Expert

Re: STP Leak using Broadcast packet 01:80:c2:00:00:1c

Our initial firmware rollup fixing this issue is mostly done, after testing/validation and internal processes for publishing online. Please download new firmware and plan for maintenance window / upgrade at your convenience. 

 

For instance GS752TXS firmware version 6.1.0.36  and XS712T same version 6.1.0.36

 

Thanks for all your inputs regarding this issue, it was instrumental for NETGEAR team.

 

Regards,

 

 

Laurent Masia
NETGEAR Senior Product Line Manager, Managed Switches
Message 25 of 39