Orbi system - Faulty LAN / switching design - Will NG ever fix it?

[superczar]

Issue - Orbi system with 2 satellites (R1, S1, S2)  will randomly drop LAN communication between segments.

e.g. devices connected to S1 (both wired/wireless) won't be able to ping devices on S2 (or vice versa)

or sometimes devices on S2 won't be able to ping devices on R1 and vice versa

All devices will continue to have Internet access though.

4 years and counting yet NG haven't fixed this issue which is so blatantly bad

Here are 4 threads detailing the issue 

https://community.netgear.com/t5/Orbi/Terrible-Bug-Adding-a-RBS20-satellite-to-RBK-50-system-breaks/...

https://community.netgear.com/t5/Orbi/Serious-Satellite-Connectivity-Bug/td-p/1303604

https://community.netgear.com/t5/Orbi/Orbi-loosing-part-of-the-network-NOT-Internet/m-p/1298723

https://community.netgear.com/t5/Orbi/Devices-attached-to-satellites-unable-to-talk-to-each-other/td...

I personally spent days trying to resolve it. Eventually called it a day by keeping all devices on 2.1.4.16 on which the onset of thsi issue is relatively delayed.

After another recurrence few days ago, decided to upgrade to 2.5.1.16 in the hope it would be fixed by now.

Guess what, the issue started to recur within 30 mins of the upgrade.

Other points to note:

- System Restart fixes the issue temporarily

- After some time, 15 mins to a few hours, the LAN comm will breakdown and require a restart again

- Occurs in both AP and router mode

- 2.1.4.16 is relatively stable 

My guess - this has something to do with messed up / out of sync ARP tables between R1, S1, S2

Such a shame actually  - Orbi is a brilliant system otherwise.

It's just appalling though that a fundamental networking bug with LAN switching would not be resolved in production firmware even after years

[superczar · Accepted solution]

@MS , @FURRYe38 

It's been more than a week since i loaded Voxel and the system hasnt needed a restart.

I will take that as a win!

[CrimpOn]

This topic appears to expose my lack of sophistication in IPv4 networking.  My understanding is that every device on a LAN maintains its own ARP table.  When one device wants to ping another device, it looks in its own ARP table to find the MAC address, and the device receiving the ICMP request likewise looks in its own ARP table to find the MAC address that it should respond to.  I do not see how any switches are involved in the process.

If the phenomenon exists when Orbi is in both router and access point mode, there must be another router present.

My first step would be to use the debug page to capture LAN traffic and see what it shows in terms of ARP and ICMP between these devices.

[superczar]

I may not be a certified network engineer but I can certainly confirm that there are no other routers on the network when the Orbi is setup in router mode 🙂

With the orbi in AP mode, I use an Edgerouter ER-X as the router (although I also tried a TP-Link TL-470T+ as router when troubleshooting ) 

Also switches do maintain either an ARP table (managed switches) or a forwarding table (unmanaged switches) so that the switch knows where to send a packet if the packet is addressed to an entity on the LAN

I don't know if the switch is a managed or unmanaged one (br0 is the internal naming convention on Orbi) but it does have a crucial role to play in routing packets correctly

Let's take an example of the Orbi in AP mode with

Satellite 1 - Sat1

Satellite 2 - Sat 2

Orbi Router (In AP mode) - Orbi1

Router (upstream from Orbi1) - Router

e.g. if I have a wifi bulb (say LIFX_1) connected to Sat1 wifi and a phone (say iP_1) connected to Sat 2 wifi.

The LIFX app will try to send and receive packets from the bulb so I think the route packets will take are as follows:

LAN route (i.e the one prone to failure)

iP_1 -> Sat2 br0 -> Orbi1 br0 -> Sat1 br0 -> LIFX_1  (and vice versa)

while if I try open a webpage on ip1 then:

WAN route (works fine)

ip_1 -> Sat2 br0 -> Orbi1 br0 -> Router

The latter works fine but the former  is what is prone to issues 

To extend further, if I have another phone (say ip_2) connected to Sat2, i think the path would be 

LAN route (works fine)

iP_2 -> Sat2 br0 ->  LIFX_1  (and vice versa)

WAN route (works fine)

ip_2 -> Sat2 br0 -> Orbi1 br0 -> Router

In this scenario, both work fine

Just to add, the route in Red works fine after a restart

It's just that at some point in time, it will start to fail - the time duration could be minutes or hours (with the new firmware ) or it could be days to weeks (with 2.1.4.16) - but it will fail

What's funny is that this won't happen if I were to replace the Orbis with $20 equivalent routers setup as APs.

PS: The debug page will allow me to enable capture for WAN-LAN traffic (which works fine at all time )

The issue is with LAN-LAN traffic

Also adding another thread detailing the same issue

https://community.netgear.com/t5/Orbi/Random-ARP-Problems-w-WiFi-nodes/m-p/1799406

and the rest:

https://community.netgear.com/t5/Orbi/Terrible-Bug-Adding-a-RBS20-satellite-to-RBK-50-system-breaks/...

https://community.netgear.com/t5/Orbi/Serious-Satellite-Connectivity-Bug/td-p/1303604

https://community.netgear.com/t5/Orbi/Orbi-loosing-part-of-the-network-NOT-Internet/m-p/1298723

https://community.netgear.com/t5/Orbi/Devices-attached-to-satellites-unable-to-talk-to-each-other/td...

[CrimpOn]

Thanks for the thorough explanation.  It was the role of the other router that eluded me.  ("Here" vs. "Absent")

I agree that every device with a brain maintains an ARP table, including the router and satellites, and that unmanaged switches (having no "brain") maintain only a MAC address forwarding table.

Thanks also for the tip about the LAN/WAN packet capture.  Looking more closely, it appears that the Orbi captures packets that appear at the LAN or WAN interface, so a packet that remains entirely within the Orbi LAN does not.  Any packet that contains the Orbi MAC address (either source or destination) or a broadcast address is captured.  This makes sense.  If the switch module has directed a packet out a specific LAN port, the actual "Orbi LAN" will never see it.

Seems to identify the Orbi LAN module as the critical piece in this.  If the flaw is in the hardware module itself, it might not be "fixable".

My first thought was, "Aha.  The MAC forwarding table is overflowing." but that just seems ridiculous. We had problems 25 years ago when our corporate network outgrew the tables in our early switches, but today?  Surely whatever module this is can handle the number of MAC addresses in a typical residential installation.

The two satellites are linked to the router over WiFi or etherent?

[FURRYe38]

What is the Mfr and model# of the ethernet switch in the configuration?

[superczar]

So another update on this perpetual problem: 

What works mostly stable - Leaving all devices on 2.1.4.16 with updates / Internet access blocked at the firewall

It's usually weeks (or a power outage) after which this situation recurs .

At each major firmware iteration, I update to see if it solves the issue but alas, it only makes it worse.

Anyway , After the latest firmware (2.5.1.32 (?) ) , I had to take the RESET route to to do a clean revert to 2.1.4.16

A few days later, I realized the guest network was left on its default off setting so I enabled the guest network and right after, the LAN segment on the RBS50 stopped responding (Internet working as expected on all devices)

So I disabled guest network and the RBS50 devices re-appeared.

I am going to leave the guest network off to see if it truly helps or if the above was just an aberration.

@FURRYe38 

The switches are unmanaged TP-Link and D-Link.

they are too dumb to be the culprit

/in any case, I have run tests with temporary CAT6 running between them (which I so wish but unfortunately cannot keep)

[superczar]

The satellites are on the wifi backhaul.

My house conduits are rather tightly packed and drawing a CAT6 unfortunately will be a rather expensive and time consuming exercise .

I really doubt though that the issue is with the switching hardware. Network Switches are a very stable technology by now.

Also, I had no issues as long as I was on a 2 device setup.

I do believe that the LAN routing tables on the Orbi hardware get confused with 3 or more devices -

In a normal wired AP setup (say ubiquiti or even a basic cheap TP link), if i had 3 APs in a star formation with the central switch (say S1), a packet from a device on AP1 going to a device on AP2 would go : AP1 ->  S1 -> AP2

I suspect the orbi stack maybe trying to optimize by routing packet directly :

i,.e. RBS20 -> RBS50 which obviously fails

instead of RBS20 -> RBR50 -> RBS50 

[Thw0rted]

Hi @superczar , it looks like this is the most recent thread detailing your LAN / ARP issues.  Are your devices still losing the ability to see each other?  I'm shopping for a mesh kit and if I go with Orbi, my setup would probably look a lot like yours.  I'm very worried about buying a kit that seems to be fine at first, but starts exhibiting this sort of behavior once the return period has elapsed.  LAN-to-LAN traffic is very important to me, and I don't want to be stuck spending hours debugging what sounds like a longstanding firmware problem.

[Mstrbig]

---

---

@superczar I wanted to address your concerns.

I have been installing and servicing Wireless mesh systems since they came out. Comparing the Netgear Orbi to the competitors, I have found the Orbi systems are more powerful, stable and seem to give the best WIFI speeds over the competitors. And not to insult anyone, as it is not my intention, but 90% of wireless mesh problems are not the fault of the system, but rather, not in any order, connections, install and configuration, wiring, ISP device, switches, placement, interference, purchased used without testing, and more. To add to it, their are users that should just hire someone to install their system. I'm a big "Do It Yourselfer", but I would pay rather someone, for many things I don't understand. 

Regarding the Orbi brand, I installed a competitor's AC2200 5000+ sq ft 3 device system in my home. It was great and covered quite well. I was very happy, but it never gave me excellent WIFI speeds. They were good but I knew they could be better. Then I installed an Orbi system, in a huge warehouse, and could not believe the coverage. So I replaced my 3 device system with an Orbi RBK50 3 device system. The speed and coverage difference was enormous. I could not be happier. The only complaint, that I don't use, is their support is as terible as the competitors. And to be honest, you get better support from the forums than you ever get from the manufacturer. Speaking for the Orbi forum, there are so many knowedgeable people participating, that you can see most issues get resolved. 

[Thw0rted]

Thanks, that's a helpful perspective.  I'm sure that 99% of users have no problems, they just aren't coming here to tell us about it.  I don't mind "do it yourself" that much, but I get resentful when I spend hundreds of dollars on gear, set it up "as intended", then it still breaks through no fault of my own and I'm stuck spending my limited free time troubleshooting.  I get plenty of that in my day job, so I want something that "just works" at home.  In this case, I think "just works" is not so much a matter of having somebody set it up right, it's down to what the vendor is shipping.

[Mstrbig]

I tell my Do It Yourself customers to follow the instructions Verbatum. The issue they have that comes up quite often is the connection from the ISP to the Orbi router. This is the most important starting point. The other frequent problem is when they buy used, and the seller never reset the system. They just unplugged and sold it.

[GMoGoody8]

@superczar , Furry was asking about the unmanaged switches because there are known firmware issues on the Orbi's with switches supporting 802.3az aka Green Ethernet. As of now i have only seen reports when wired ethernet backhaul is used in conjunction with 802.3az. This might be worth investigating for your issue too. 

This is My setup and I can ping through my LAN with no issue. Wireless Backhaul 

Sonos play 1 [Ethernet] --> Netgear gs108v3 --> RBR50 --> RBS50 --> Windows 10 PC [WiFi 5.0]

Synology server [Ethernet] -->  Netgear gs108v3 --> RBR50 --> RBS50 --> Windows 10 PC [WiFi 5.0]

Epson Printer [WiFi 2.4] --> RBR50 --> RBS50 --> Windows 10 PC [WiFi]

Netgear gs108v3 does not support 802.3az

[Mstrbig]

I find the best switches to use, unless some other features are required, are the inexpensive Netgear ProSafe Gigabit switches. They are plug and play and require no setup. They just work flawlessly.

[GMoGoody8]

@Mstrbig , All of my switches are either the Netgear ProSafe Gigabit gs108 or gs105. I have never had issues with them. I am moving later this year and was investigating wired backhaul since the new place will allow me to setup that way now. Nice to tell builders where to run Ethernet cables. In my research I came across Furry's and others posts and experiments in these threads about 802.3az with wired backhaul. 

I told Furry once I move I will report back on the newer revisions of the gs108v4 and gs105NA since the newer revisions of Netgear ProSafe Gigabit all support 802.11az now and see if Orbi has the issues others have reported.

[Mstrbig]

I can say wired backhaul is a whole new and better advantage for mesh systems. some think it defeats the purpose of mesh and why not go separate routers or extenders. I've done them all, and the mesh system wins out every time. Good luck with the move.

[Thw0rted]

I mean, it's still a "mesh" system even with wired backhaul, right?  The key thing is communication between the nodes that facilitates smoother handoff when the client is moving around.  This is the main reason I'm looking at mesh as an upgrade.  I currently have a router from the ISP with one SSID for both bands, then a powerline AP and a wired AP (Nighthawk in AP mode) both of which use the same SSIDs, one each for 2.4GHz and 5GHz bands.  Since all 3 devices are from different manufacturers, they don't do any sort of coordination as clients move around and they tend to "stick" to a crappy signal until you go in and manually pick a better network.  I just want my network set up so that the mesh "knows" that it has a better receiver available and quietly boots the client over on its own, like a cellular network.

[GMoGoody8]

@Thw0rted that's the dream right. I find the Orbi is pretty good with roaming clients. I have Fast Roaming Enabled. Just be advised the clients also need to play their part. They have to support all three of 802.11k, 802.11r, and 802.11v.  These specifications are over 7 years old so hopefully all clients efficiently support them by now. 

Apple put together a good explanation here: https://support.apple.com/en-us/HT202628

[Thw0rted]

I did a bit of reading about this, and I think 11r only applies if you use Enterprise network security ("RADIUS", maybe?).  The other two are supposed to be "hints" to the client, right?  Anyway, the clients I'd be most concerned about are modern iOS, Android, Mac, and Windows devices.  At the very least, I'm pretty sure they can't do worse than they are right now...

[GMoGoody8]

You are correct 802.11r predominantly benefits enterprise solutions, WPA2-Enterprise with Radius, but the specification also included enhancements to WPA2-Personal with efficiencies in establishing a new encryption key. 

For both WPA2 Enterprise and WPA2 Personal re-associations. In both cases, the eight messages passed between an AP and a client device for authentication, association, and the four-way handshake are reduced to four messages. 

Another thing to note is for all three of these specifications if you have mixed clients where some support them and other don't it is recommended to disable fast roaming since the older clients don't understand. 

[Mstrbig]

---

@Thw0rted wrote:

I mean, it's still a "mesh" system even with wired backhaul, right?  The key thing is communication between the nodes that facilitates smoother handoff when the client is moving around.  This is the main reason I'm looking at mesh as an upgrade.  I currently have a router from the ISP with one SSID for both bands, then a powerline AP and a wired AP (Nighthawk in AP mode) both of which use the same SSIDs, one each for 2.4GHz and 5GHz bands.  Since all 3 devices are from different manufacturers, they don't do any sort of coordination as clients move around and they tend to "stick" to a crappy signal until you go in and manually pick a better network.  I just want my network set up so that the mesh "knows" that it has a better receiver available and quietly boots the client over on its own, like a cellular network.

---

I had 3 powerful routers in my home all brodcasting the same WIFI name. They worked quite well, but nowhere near as good as a mesh system. The backhaul makes all the difference, whether wired or wireless. The handoffs are way smoother. And wired backhaul, if possible, eliminates virtually all interference between the router and satellites. My system is ISP modem > orbi RBR50 > 12 port netgear switch > 2 Orbi RBS50 satellites and various hard wired devices > 8 port switch on each satellite, connecting various wired devices in each location. The system is super fast with no issues. WIFI speeds, from my phone, in the same room as the nodes are 478 down, 54 up (ISP plan 500/50). Of course the speed drops the further away I get from a node. But that is normal. 

[FURRYe38]

Im still interested in your findings and report. 

@superczar @Thw0rted I'm for one am reporting that I have had ZERO issues with my RBK50 system wired with NON managed switches. I have tested both D-Link DGS105/108 series and HP Pro Curve 1400/1800 series switches with no issues. I do have a D-Link DGS-1100 managed switch that also supports Green ethernet which can be disabled. However I usually don't use that switch normally as I prefer to use the back RBR ports as much as possible and keep switch use to a minimum. 

Right now have a 24 pt HP procurve that services the main home network and different rooms in the home a long with NAS and general wired PC and printer connections. One another segment I have a DISH hopper system that connects to a DGS108 behind the RBR. Joeys, xbox, microcell and 2 RBS are connected behind this switch. One RBS in the basement is connected behind two DGS-105s daisy chained behind the DGS-108. These are my two switch segments behind the RBR. Have had no issues with RBS not connecting, causing ARP storms or Obi app reporting incorrectly. 

---

@GMoGoody8 wrote:

I told Furry once I move I will report back on the newer revisions of the gs108v4 and gs105NA since the newer revisions of Netgear ProSafe Gigabit all support 802.11az now and see if Orbi has the issues others have reported.

---

[GMoGoody8]

Definitely will let you know. Construction hasn’t started yet on the home. Closing is set for the winter. Since it’s how I want my network setup it will be the first thing I try.

Maybe you won’t have to wait as long with Netgear pulling a miracle and fixing the issue in their firmware before I close. 😂

[FURRYe38]

Well we can hope. Seems that Orbi is continuing to not work great with these green ethernet switches. Kind of a mixed bag I think. Seen some say it works for them, others have posted that changing out for a non green ethernet switch resolved there problems. Not sure why Orbi and NG haven't made the FW more compatible. Possible green ethernet introduces problems NG doesn't wan't to play with. 

Hoping the Close and build goes well for you. Always a good feeling when things are done. Mine was completed 2015. 

---

@GMoGoody8 wrote:
Definitely will let you know. Construction hasn’t started yet on the home. Closing is set for the winter. Since it’s how I want my network setup it will be the first thing I try.

Maybe you won’t have to wait as long with Netgear pulling a miracle and fixing the issue in their firmware before I close. 😂

---

[superczar]

WIth the workaround mentioned (sticking to FW 2.1.4.16), the system has been reasonably stable of late.

Regarding the other suggestions on the switches, I am not using any energy savings settings on the switches.

All devices were brought new and they have gone through multiple full resets for any old setting to have any detrimental impact.

I have no complaints regarding the speed, roaming behavior and performance.

The system does indeed work better than any other router/AP I have used when it works.

However I do maintain the firmware has some deep rooted flaws.

If I hadn't plonked so much money on this system already and if I could start again, I would have looked at something else.

Most people who have a reasonably simple network will not notice issues - but if your use case includes larger than usual number of devices or a lot of LAN traffic, there is a good likelihood you will run into problems.

[FURRYe38]

So to recap, which mfr and model # switches are you using with the system now? Helpful to know this for anyone else looking or having problems. Not all switches are made the same I believe. There maybe differences between brands as well. I try to keep to one brand of switch through out the configuration to maintain some sort of uniformitiy. Have a mixed brand of switches in play may also be causing odd problems as well. I have tested two brands and keep these separate when I decide to swap brands. 

As far as I an tell, having lots of devices online doesn't present a problem for me, though I have about 30 devices. RBS are all wired behind switches. v16 has been the most stable to date IMO. 

---

@superczar wrote:

WIth the workaround mentioned (sticking to FW 2.1.4.16), the system has been reasonably stable of late.

Regarding the other suggestions on the switches, I am not using any energy savings settings on the switches.

All devices were brought new and they have gone through multiple full resets for any old setting to have any detrimental impact.

I have no complaints regarding the speed, roaming behavior and performance.

The system does indeed work better than any other router/AP I have used when it works.

However I do maintain the firmware has some deep rooted flaws.

If I hadn't plonked so much money on this system already and if I could start again, I would have looked at something else.

Most people who have a reasonably simple network will not notice issues - but if your use case includes larger than usual number of devices or a lot of LAN traffic, there is a good likelihood you will run into problems.

---