Reply

Re: ORBI RBR/RBS Broadcast Storm Problem / Ethernet Backhaul creates network loop & kills networ

Damien_
Initiate

ORBI RBR/RBS Broadcast Storm Problem / Ethernet Backhaul creates network loop & kills networks

WARNING!!: This may or may not help you, and can be a risky, you may need to factory reset if you break your config.

 

IMPORTANT: This relates specifically for Orbi when you are using an Ethernet backhaul (That means an ethernet cable connects the router with the satellites).  Note I only use Orbi as a mesh Wifi AP solution, not as a router.  This was based on firmware version: v2.5.1.8

 

The most noticable way this affected me originally was my router would shutdown it's LAN interface, and required a reboot to recover.  This can get worse in some networks where the entire network is brought down.

 

I am hoping to provide some others a hint of hope, as I have been battling with this problem for a very long time through many firmware versions.  I even bought a network enabled power outlet (with inbuilt monitoring) to monitor my router and reboot as needed due to this problem.  Recently after buying a new fully managed switch to try and give me tools to fight this it actually had a bigger impact to my network, but this allowed me to do more troubleshooting and verify my suspicions.

 

Netgears response has been absolutely unacceptable.  This is an obvious problem that has been raised previously, this problem has existed for years.  This is a software design fault: it's a startup race condition.  This is one that can be easily fixed with a patch, but I have only seen comments blaming people of misconfiguration.

 

It goes like this:

 

When the Orbi devices startup, they initially connect to each other via a wireless backhaul, and then later will check for an ethernet backhaul and switch to that instead.  While this seems reasonable, this actually creates a period of time where both the wifi backhaul and the ethernet backhaul are connected at the same time.  During this time orbi creates a network loop that continuously retransmits broadcst packets throughout the network.  This is very bad!

 

I could go into the various ways I have configured my network/orbi to try and deal with this, but basically this is 2 orbi devices connected directly together, with only the router being connected to my LAN (configued in AP mode).  One other way to work around this would be to segregate my network, and put Orbi (Wifi) in a separate isolated subnet and route the traffic back to my LAN as this would prevent the broadcast storm from affecting the rest of my LAN.  This is a painful solution.

 

Anyway...

 

The best solution would be for Orbi to test for Ethernet backhaul before trying to bring up the WiFi backhaul. (Are you listening Netgear??)

 

I did some research in trying to find a way to disable the wifi backhaul, but people claim this is not possible, I did already know how to get debug access and telnet in, and then I stumbled on a forum post talking about disabling various wifi AP radio interfaces for turning off 2g & 5g wifi via CLi.

 

Putting this all together, I was able to find how to disable the 5g backhaul (2g seems to partly exist also, but doesn't seem to cause an issue).

 

Again: disclaimer: Be careful, you need to know what you are doing / be willing to factory reset

To be "safe", I only made this change on the Satellite, NOT on the Router, as we should only need one side of the connection to be disabled.

 

Getting telnet access, requires turning this on in debug.htm

http://<orbi ip>/debug.htm

 

### This was my original config

root@RBS50:/# config show | grep endis_wl
endis_wlg_wireless_isolation=0
endis_wlg_sta_wps=1
endis_wla_guest_wireless_isolation=0
endis_wla_wireless_isolation=0
endis_wla_wmm=1
endis_wla_2nd_sta_wps=1
endis_wla_wps=1
endis_wla_radio=1
endis_wl_wmm=1
endis_wla_2nd_radio=1
endis_wlg_arlo_wireless_isolation=1
endis_wl_radio=1
endis_wl_wps=1
endis_wlg_guest_wireless_isolation=0

root@RBS50:/# config show | grep AP
lbd_LowRSSIAPSteerThreshold_CAP_W2=35
lbd_APSteerToRootMinRSSIIncThreshold=10
lbd_LowRSSIAPSteerThreshold_CAP_W5=20
wl2g_BACKHAUL_AP=ath01
wl5g_BACKHAUL_AP=ath2
lbd_LowRSSIAPSteerThreshold_RE_W2=35
lbd_APSteerToPeerMinRSSIIncThreshold=10
lbd_LowRSSIAPSteerThreshold_RE_W5=20
lbd_APSteerToLeafMinRSSIIncThreshold=10
ManageVAPInd=1
wl2g_GUEST_AP=ath03
wl5g_GUEST_AP=ath11
wl2g_NORMAL_AP=ath0
wl5g_NORMAL_AP=ath1

### Putting this together I disabled the 2nd radio

root@RBS50:/# config set endis_wla_2nd_radio=0

### Based on the other forum post I also set ath2 down, but I don't think this is critical
root@RBS50:/# ifconfig ath2 down

### I did a config commit to ensure it was persistent across reboots.
root@RBS50:/# config commit

### Physically power off and on the Satellite (not reboot: why, because when I did a soft reboot it didn't come back initially, I expect it had trouble finding the backhaul, but powering off and on resolved it, and I have tested this a couple times).

Power OFF & ON (not reboot)

### After reboot I reviewed the settings:
### It was persistent and note that the 5g BACKHAUL disappeared.

root@RBS50:/# config show | grep AP
lbd_LowRSSIAPSteerThreshold_CAP_W2=35
lbd_APSteerToRootMinRSSIIncThreshold=10
lbd_LowRSSIAPSteerThreshold_CAP_W5=20
wl2g_BACKHAUL_AP=ath01
lbd_LowRSSIAPSteerThreshold_RE_W2=35
lbd_APSteerToPeerMinRSSIIncThreshold=10
lbd_LowRSSIAPSteerThreshold_RE_W5=20
lbd_APSteerToLeafMinRSSIIncThreshold=10
ManageVAPInd=1
wl2g_GUEST_AP=ath03
wl5g_GUEST_AP=ath11
wl2g_NORMAL_AP=ath0
wl5g_NORMAL_AP=ath1


root@RBS50:/# config show | grep endis_wl
endis_wlg_wireless_isolation=0
endis_wlg_sta_wps=1
endis_wla_guest_wireless_isolation=0
endis_wla_wireless_isolation=0
endis_wla_wmm=1
endis_wla_2nd_sta_wps=1
endis_wla_wps=1
endis_wla_radio=1
endis_wl_wmm=1
endis_wla_2nd_radio=0
endis_wlg_arlo_wireless_isolation=1
endis_wl_radio=1
endis_wl_wps=1
endis_wlg_guest_wireless_isolation=0

 

After all this I gathered stats on broadcast packets coming from the router, and noticed a significant drop on startup now I only see hundreds of broadcasts, not millions, and my network survives.

 

It's still early days, but I expect this has resolved it for me, hopefully this will help someone else.

 

Good Luck!

 

Damien.

Model: RBR50|Orbi AC3000 Tri-band WiFi Router
Message 1 of 4
FURRYe38
Guru

Re: ORBI RBR/RBS Broadcast Storm Problem / Ethernet Backhaul creates network loop & kills networ

I have not seen any broadcast storms for a long time now. I have a RBR connected to a RBS that has 4 Gb LAN switches in between that are daisy chained. Have not seen any problems like this in a long time. I know Did see this once and I fould the RBS and my VoIP devices were in IP address conflict at the time. Once I got the VoIP device on a IP address reservation away from the RBS IP address, all was good. 

 

How do you have RBS ethernet connected?

This is NG recommendation:

https://kb.netgear.com/000051205/What-is-Ethernet-backhaul-and-how-do-I-set-it-up-on-my-Orbi-WiFi-Sy...

 

" Recently after buying a new fully managed switch to try and give me tools to fight this it actually had a bigger impact to my network, but this allowed me to do more troubleshooting and verify my suspicions." What is the mfr and model# of the switch being used? If this is a managed switch, you need to disable ALL IGMP protcols on this switch if you use it between the RBR and RBS when ethernet connected. Possible your switch your using maybe a cause as well. 

https://community.netgear.com/t5/Orbi/Orbi-MR-2-1-Update-3-23-18/m-p/1553749/highlight/true#M30673
https://community.netgear.com/t5/Orbi/Using-Router-Configured-as-Access-Point-Cannot-see-Satellites-...
https://community.netgear.com/t5/Orbi/Orbi-Ethernet-Backhaul-with-Managed-Switches/m-p/1613897#M3812...

 

Also disable any green ethernet features on this switch if possible. Some switches you might not be able to so you'll need a non managed non green ethernet switch to use with Orbi. 

https://community.netgear.com/t5/Orbi/switching-requirements-for-ethernet-backhaul/m-p/1653718/highl...
https://community.netgear.com/t5/Orbi/Ready-to-chuck-this-system-through-the-window/m-p/1658470/high...
https://community.netgear.com/t5/Orbi/Orbi-AC3000-with-Cisco-SG112-24-Switch/m-p/1700690/highlight/t...
https://community.netgear.com/t5/Orbi/Orbi-Netgear-GS908E-100PES-Backhaul-Connection/m-p/1784425/hig...

 

Its always best to connect the RBS wirelessly FIRST to make sure it connects well to the RBR, then once this gets established, connect the RBS to the RBR via ethernet directly to the RBR using a good quality LAN cable. CAT is is recommended. Once the RBS is connected and everything is working here, then place the RBS out at the remote location and connect it there. 

My Setup ISP SparkLight | Internet Cable 1000↓/50↑ CM2000 Modem | Wifi Router RBK752(v.5) | Switches NG GS105/8 and XS505M | 

Additional NG HW: C7800/CAX80/CM1100/CM1200/CM2000, Orbi: CBK40, CBK752, RBK50, RBK853, RBK752, RBK953, SXK30 | NightHawk: R7000, R7800, R7960P, R8000, R8500, RAXE500, RAX50, XR450, EX7500/EX7700, GS308v3

Message 2 of 4
Damien_
Initiate

Re: ORBI RBR/RBS Broadcast Storm Problem / Ethernet Backhaul creates network loop & kills networ

Thanks for trying, but I wasn't looking for more community answers, I am satisifed with my diagnosis.  My intention is to provide some more information to those who might be having this problem, as I hadn't found any other real solutions through my search, and I had also found other long running threads that did not end with a resolution.

 

The issue is that Orbi creates it's own network loop between the router and satellite.  This is independent of any other connected devices or topology.  This is independent of whether you connect to the rest of the LAN via the RBR WAN or LAN port, and regardless of directly connecting orbi's together with no other devices connected to Orbi's ethernet ports.

 

It's probably not likely that someone would notice/encounter this problem unless running in AP mode with Ethernet backhaul.  Also as this only occurs on device boot, many may not even notice it.  Before I obtained my current switch I had suspected something else in my network.  But it's obvious when looking at the stats for huge numbers of broadcast packets all flowing into the switch interface connected to the Orbi router.

 

<all other devices> -> [lan switch] <- [Orbi RBR] <- [Orbi RBS]

 

Simply having an ethernet backhaul connected causes Orbi to create the network loop on boot.  Yes Orbi later prioritises Ethernet backhaul traffic, and the broadcasts stop, but the network loop still exists on boot where both the Wifi Backhaul and the Ethernet backhaul are active for a time until it completes initialisation.  It's possible to see evidence of this in the "Statistics" on the device when viewing WLAN backhaul packet numbers immediately after boot, but it was much more clear when I connected it to a decent switch to see the numbers.

 

Thanks again,

 

Damien.

Message 3 of 4
FURRYe38
Guru

Re: ORBI RBR/RBS Broadcast Storm Problem / Ethernet Backhaul creates network loop & kills networ

Ive had both Orbi AC and AX in AP mode. Haven't seen a storm problem. Switches in the middel as well. I would know. I watch my switch activity for any constant use. 

 

I would suspect how you have your system configured or a devices on your network or even a bad LAN cable could be a cause. IF you have gone thru many FW updates and still see this, possible that your system is just faulty as well. Something to see if a different system does the same thing or not.

 

There hasn't been any reports of someone have a storm in a long time. I would not assume that others are not seeing this either or assume that everyone may have this issue. This would cause problems and some would be postings about it. Just haven't seen any in a while now. You seem to be the only one currently. There are reasons why we ask about configurations and switches. Help to vet some of these issues out to find where the real problem is. 

 

Good Luck in your endeavors. 

 

 

My Setup ISP SparkLight | Internet Cable 1000↓/50↑ CM2000 Modem | Wifi Router RBK752(v.5) | Switches NG GS105/8 and XS505M | 

Additional NG HW: C7800/CAX80/CM1100/CM1200/CM2000, Orbi: CBK40, CBK752, RBK50, RBK853, RBK752, RBK953, SXK30 | NightHawk: R7000, R7800, R7960P, R8000, R8500, RAXE500, RAX50, XR450, EX7500/EX7700, GS308v3

Message 4 of 4
Top Contributors
Discussion stats
  • 3 replies
  • 1017 views
  • 5 kudos
  • 2 in conversation
Announcements

Orbi WiFi 6E