Re: Orbi RBR850 unstable/DHCP assignment issues after reboot

rdouma · ‎2024-05-30

Post concerns a RBR850 and 2 RBS850 satellites, all running firmware version V7.2.6.31_5.0.24.

Summary: I have to turn off all WiFi devices near the router in order to get a successful reboot where devices get an IP-address via DHCP. If I don't, devices don't get a DHCP address and get a self-assigned DHCP-address instead.

Background: I have a lot of WiFi-enabled devices. I suspect this was introduced mainly by the latest firmware that was probably installed automatically (I had auto-update on before). I cannot say for sure because as far as I know there is no log or notification when firmware gets updated, and the Release Notes of the firmware are of no help either because they don't mention the release date. My experience is: I had been adding several Wiz lightbulbs in the last week of April. Everything was working fine when I left for a trip on May 2. On May 3 I notice my house was offline in Apple Home. I asked a relative to reset the router and the Apple TV that is the HomeKit hub, but nothing helped: it stayed dark. When I came back on May 11 I started this journey to figure out what is going on.

Coming back I could not get things to work AT ALL. I discovered I had to basically turn off ALL devices in the house that had WiFi, restart the router, wait for a minute or 10 and then slowly turn devices back on. If I did that too quickly devices would not connect (I suspect not get a valid IP) and subsequent devices would then also not connect reliably. Absolute nightmare that had me at the end of my wits for 3 days. I then decided to factory reset it all and set it up anew in a more organized way.

Short summary from the top of my head:

Scanned networks with WiFi-explorer and made sure I chose channels that are not having too much competition. This is a residential area with relatively large distances between the houses (next house is about 20 meters from my place). Tried to play all kinds of combinations of 2.4 and 5Ghz channels, but didn't make a big difference. Played with all kinds of tweaks on the router, no big difference.
Hardwired first satellite to router. Made router connect quicker but no success neither.
Named devices for ALL devices that should be online (Sonos, lights, etc.). I gave DHCP range 150-254 and organized the IoT devices below that.
No big difference, so decided to set up the IoT network. Moved most devices over. Not a huge difference but at least it's organized.
Decided to make IoT network 2.4Ghz only. Regretfully I cannot make the regular network 5Ghz only or assign it a different channel. seems sub optimal but not sure; this surpasses my skillset; I'm a "software guy".

Finally I discovered the ONLY way to reboot successfully in about 80-90% of the cases (which I had to do countless times because assigning a device a fixed IP only works when you apply it and reboot it) is:

Turn off ALL wifi-devices near the router (computers, lamps).
Restart the router and wait for about 5 minutes or at least until the final satellite in the chain says the backhaul is "good".
Then turn on back my wifi-devices in my office.

Sometimes I need to do that 2 or 3 times but then it works fine. Sometimes for a few days on end. Now going 3 days already without a reboot.

I also noticed it makes a huge difference if I move the router a few centimeters (!). I'm still figuring out the optimal position now. I currently am trying to find the best position for the router. I moved the RBR850 to another room with long cables to allow me to try and find the optimum spot. By the way; I had it running fine for YEARS with the router in my office and a wireless backhaul to the satellites.

TL/DR: I suspect the latest firmware update made the router unstable and it seems to have trouble assigning DHCP addresses if wifi-devices are turned on before it finalizes its booting sequence. Assigning them IP-addresses via named devices doesn't seem to help; the devices need to be OFF. Not doing so makes the entire house fail in DHCP-assignments.

FURRYe38 · ‎2024-05-30

So with all these devices online, and setup IP address reservations, does the system work ok?

I see some devices that you might consider setting a Static IP address ON that would be outside of the suggested default DHCP IP address pool range.

You have lots of plugs and lights. Wondering if some of these could be causing issues. Wondering if the system is bogging down.

What channel configurations are currently set on the router?

What is the size of your home? Sq Ft?
What is the distance between the router 📡 and satellite(s)🛰️? 30 feet or more is recommended in between RBR 📡 and RBS 🛰️ to begin with depending upon building materials when wired or wirelessly connected.
https://kb.netgear.com/31029/Where-should-I-place-my-Orbi-satellite ‌‌🛰

What is the Mfr and model# of the Internet Service Providers modem/ONT the NG router is connected too?
Be sure your using a good quality LAN cable between the modem and router. CAT6A STP is recommended.

What FW version was loaded prior to v.31?

Prior FW working?

Has a factory reset and setup from scratch been performed since last FW update?

rdouma · ‎2024-05-30

Channel configurations:

The house is about 250m2 (2690 ft2). About 10 of these devices are outside (garden, front area), total property is about 1100m2 (11840 ft2).
Distance router to first router is now about maybe 15m in a straight line (it's now in a staircase, walls between it) but is connected via ethernet (I also have a Gigabit switch that is taking care of the cabling, so 1 cable from modem to Orbi and another one running back to the switch). Distance to 2nd satellite to 3rd is about 10 meters (32ft). One brick wall between them.
It's a Zxel ZXHN F670. Why is this relevant, curious? The rest of the house is oblivious to it and it's here for years. The Orbi doesn't have an issue with it. WiFi is turned off on that modem.
Sure, I use good cables. I've also tested them with an ethernet tester. There is no issue with the wired connections. Wired devices do get a DHCP address correctly.
I don't know what version was loaded prior to it. I just had it running without paying attention to it.
Yes, I had to do factory reset as it was already upgraded to the latest version.

FURRYe38 · ‎2024-05-30

Reason for asking about ISP modem or ONT is to see if there is any router and built in wifi here. I see this unit you have does have a router and wifi. Additional WiFi noise can cause interferences. Additional router can be problematic as well with having a double NAT condition. Just wanna rule out this as well. Can you let us know what configuration this ONT is in? Please disable any wifi radios on the ONT to help check this if you haven't already.

For users with built in routers and wifi at the ISP level, what is recommended:

Your ISP Modem already has a built in router and wifi. This would be a double NAT (two router) condition which isn't recommended. https://kb.netgear.com/30186/What-is-Double-NAT
https://kb.netgear.com/30187/How-to-fix-issues-with-Double-NAT
Couple of options,
1. Configure the modem for transparent bridge or modem only mode. Then use the Orbi router in router mode. You'll need to contact the ISP for help and information in regards to the modem being bridged correctly.
2. If you can't bridge the modem, disable ALL wifi radios on the modem, configure the modems DMZ/ExposedHost or IP Pass-Through for the IP address the Orbi router gets from the modem. Then you can use the Orbi router in Router mode.
3. Or disable all wifi radios on the modem and connect the Orbi router to the modem, configure AP mode on the Orbi router.
https://community.netgear.com/t5/Orbi-WIFI-6-AX-AND-Wi-Fi-6E-AXE/Procedure-for-Factory-reset-of-RBS8...
https://kb.netgear.com/31218/How-do-I-configure-my-Orbi-router-to-act-as-an-access-point and https://www.youtube.com/watch?v=H7LOcJ8GdDo&app=desktop
Try option #2 first...

What is the brand and model of the LAN switch in between the RBR and the RBS?

So of wired devices get good IPs and wireless don't, possible wifi issue on handling all those devices that are wireless and the system maybe having an issue or in this new FW version.

Something to review, might consider downgrading back to v.21 on the RBS first then RBR lastly and after all is downgraded, power cycle the system OFF for 1 minute then back ON with the RBR first, then about 1 minute later, power ON the RBS. See what happens.

TC_in_Montana · ‎2024-05-31

@rdouma

I see a few things that could be in play - not guarantee that they are, but possible.

1) It takes a while for the satellites to connect and sync to the router (unfortunately this is something across all Orbi lines). So when you reboot the router, all of your devices try to connect to the router - because that is all that is seen for a few minutes. If they don't have a great connection, they still try to connect and may pop on/drop off, pop on, drop off, etc. A lot of older devices, once connected, don't try to reconnect to a better signal once they have connected. Remember, it is the device that requests connection - not the router/satellite asking to connect to the device. That is why after your system (router and satellites come up), turning off and turning on your devices seem to make a better connection - they can connect to the closest and best signal.

2) On the flip side - if you have too much coverage - that do try to reconnect, may ping pong between router and satellite - thus providing crappy connectivity. Check your router logs and see if you have a device or multiple devices doing lots of DHCP requests - this is a sign of ping-ponging. If this is the case, you may want to drop the power level on your system (if that is still available on the 850 - it was previously). You can choose 100%, 50%, 25%. It has been seen previously that dropping the transmit power helps eliminate ping-ponging.

3) There are two methods to insure devices get the same IP address every time - Reserved Addresses - set on the router - and Static Addresses - set on the device. Make sure you are not assigning static IP addresses (on the device) within the DCHP Pool (you have set an address pool, I just don't remember what it is now). So - devices assigned by the router will only get addresses within that pool range, devices with static IPs will get outside of that range.

I use reserved addresses for 90% of my devices - and they always get that address without fail. The only time they don't is if I forget to turn off MAC ID randomization on the device and it doesn't recognize the device. When you set your reserved addresses, you may need to turn the router off for 5 minutes or so to insure that all of your devices see that they are disconnected (sometimes this is slow) - or the other way to do this is to power off off and power back on the device to insure they forget. Some devices are just stubborn sometimes.

Unfortunately, I don't use HomeKit for anything and don't know much about it - but if Homekit can assign or manage IP addresses - make sure you are not double booking IP addresses between static, reserved, and Homekit.

raven_au · ‎2024-06-02

@rdouma wrote:
TL/DR: I suspect the latest firmware update made the router unstable and it seems to have trouble assigning DHCP addresses if wifi-devices are turned on before it finalizes its booting sequence. Assigning them IP-addresses via named devices doesn't seem to help; the devices need to be OFF. Not doing so makes the entire house fail in DHCP-assignments.

I don't think so, I have had this same problem for a long time.

I was excited to read that, for you, not making any DHCP reservations resolved the problem.

I reinstated my 850's to see if that would work and to my surprise removing all my DHCP reservations did allow the system to reboot and continue issuing addresses, unfortunately, after less than 24 hours one of my satellites started showing magenta light which I took to mean it could not renew its IP address and sure enough my Android phone could not renew its address either.

A couple of things to note:

1) This is on V7.2.6.21_5.0.20 and I have seen it on earlier firmware versions.

2) Some time after I first saw this I tried my RBK50 kit and saw the same problem leading me to think it is related to some sort of a race in the DHCP server when the number of devices is large, possibly more than about 50 but more than 80 is when I really started having a problem.

3) Following a factory reset the system, which generally works to get the system going again, will sometimes reboot successfully a couple of times before the same problem occurs again.

I have no idea why this happens but once it starts it doesn't seem to stop, and I have no reliable work around, the only thing I can do is disable the router DHCP server and use my NAS for DHCP instead.

This has been a problem for years and Netgear not paying any attention to the forums is just not good enough, but then they give only three months support on new purchases, so ...

Ian

rdouma · ‎2024-06-02

I was excited to read that, for you, not making any DHCP reservations resolved the problem.

Eh... where did I say that? I surely didn't intend to say that; one of the steps I used to get back to a workable solution was the opposite of that: I basically have all known devices in my house as a named device now.

Some time after I first saw this I tried my RBK50 kit and saw the same problem leading me to think it is related to some sort of a race in the DHCP server when the number of devices is large, possibly more than about 50 but more than 80 is when I really started having a problem.

Very well possible it's related to the amount of devices. I have a bunch of them and a large amount was added over the last months when installing Wiz light bulbs.

This has been a problem for years and Netgear not paying any attention to the forums is just not good enough, but then they give only three months support on new purchases, so ...

Yeah, with you on that one. I would expect that you would never want to restrict users to be able to easily submit bug reports but no. Luckily there's this forum and some power users here that seem to know their way around this but in my opinion NetGear should be welcoming bug reports instead of trying to make it hard to submit one.

raven_au · ‎2024-06-02

@rdouma wrote:

I was excited to read that, for you, not making any DHCP reservations resolved the problem.
Eh... where did I say that? I surely didn't intend to say that; one of the steps I used to get back to a workable solution was the opposite of that: I basically have all known devices in my house as a named device now.

Umm ... reading your post again I think I misunderstood, sorry.

Nevertheless, the dhcp server is probably single threaded and applications like this usually spawn a sub-process for each request to increase throughput. So doing that means they need to record information about the address assignments in a file somewhere because sub-processes are themselves isolated from one another. If there are multiple concurrent requests (a lot of devices wanting to register at once) and the locking of that file isn't right or isn't even done then you can easily get corruption in the file.

I've seen that before in Linux, for years we had a lot of trouble with /etc/mtab (mounted mounts list), in spite of quite a sophisticated locking algorithm it frequently got corrupted during heavy mount activity. It wasn't until /etc/mtab was made a symbolic to the proc file system mount table it was resolved (the proc file system mount tables are files generated on the fly from kernel data structure, they are not written to).

So, it is easily possible the dhcp server is broken, of course the problem is that Netgear hasn't done anything about it or aren't competent enough to fix it if they have tried. The problem is no secret either, there have been a number of reports of it over the years.

For my part I have never been in a position to log a call with support because of the support policy and even if I was within the twelve weeks period at some time I'm not at all sure I would be willing to subject myself to the pain and suffering of what logging a support call involves.

Ian

rdouma · ‎2024-06-07

Sorry for my late response. I agree it looks suspiciously like a the DHCP server not being thread safe. I'm going to experiment with running a docker with a DHCP server and see how a network restart behaves with that.

Timtech · ‎2024-06-08

Any chance you have a MyQ garage door opener? I moved mine to the guest network and dhcp is working well ever since.

rdouma · ‎2024-06-10

Well, I can definitely confirm this is the problem. I have spent countless hours trying to figure out what the issue was, attributing it to network interference but no. I am now running ISC DHCP on Ubuntu and DHCP works flawlessly. Quite unbelievable if I'm being honest that such a core feature of the router is clearly broken. I'll log an issue with NetGear.

rdouma · ‎2024-06-10

Sorry, had missed your message! No, I don't have that.

@Timtech wrote:
Any chance you have a MyQ garage door opener? I moved mine to the guest network and dhcp is working well ever since.

rdouma · ‎2024-06-13

For everyone suffering from this issue: my network has been working flawlessly and I have been able to reboot effortlessly over 10 times when testing. Doesn't skip a beat anymore. I submitted this to NetGear:

RBR850 + 2 RBS850. I have ~100 devices. Upon reboot multiple devices won't get an IP. Only way to get it reliably to work is to turn of all devices in the house, reboot the Orbi, wait ~5 minutes and then slowly turning things on. I solved the issue by running ISC DHCP on a Linux machine in my network and turning it off on the Orbi.
Suspicion is that DHCP in Orbi is not thread safe, and upon reboot it cannot handle the simultaneous DHCP requests, **even with all devices being named**.
Firmware version V7.2.6.31_5.0.24

Hope this helps my fellow Orbi users that suffer from this issue. I have literally spent over 100 hours to narrow it down to this, thinking it was related to network interference, experimenting with router/AP placement etc. They probably just forgot to make the IP assigmnent method thread safe and things have clearly not been tested with large amounts of devices.

FURRYe38 · ‎2024-06-13

Had you seen this behavior on FW v.21 as well?
Since v.31 has been out, though were seeing more issues with 750 series and some on the 960 series, not really confident that v.31 is a stable FW as v.21 is.

TC_in_Montana · ‎2024-06-13

@rdouma wrote:

Hope this helps my fellow Orbi users that suffer from this issue. I have literally spent over 100 hours to narrow it down to this, thinking it was related to network interference, experimenting with router/AP placement etc. They probably just forgot to make the IP assigmnent method thread safe and things have clearly not been tested with large amounts of devices.

At approximately 100 devices you are definitely at the published limit for the number of concurrent devices listed for the 8 series and probably the user with the highest number of connected devices I've seen on the forum.

Great research and testing - hopefully someone at NG is taking this in and verifying/resolving.

rdouma · ‎2024-06-13

@FURRYe38 wrote:
Had you seen this behavior on FW v.21 as well?
Since v.31 has been out, though were seeing more issues with 750 series and some on the 960 series, not really confident that v.31 is a stable FW as v.21 is.

I didn't know there was an update, I will try. To be honest I'm quite weary by now of upgrading firmware as it was the latest upgrade that introduced me to a world of pain... while I was outside of the country my house went dark. 😬 But I have the previous firmware version here and a backup; I should be good with that. Just have to find some motivation. Sort of enjoying the situation where I'm not losing sleep trying to fix my network 😂

rdouma · ‎2024-06-13

At approximately 100 devices you are definitely at the published limit for the number of concurrent devices listed for the 8 series and probably the user with the highest number of connected devices I've seen on the forum.

Great research and testing - hopefully someone at NG is taking this in and verifying/resolving.

Thanks for the feedback, appreciate it.

As for the limit to concurrent devices... I wasn't aware there was a maximum other than the number imposed by a class C network. I could imagine there could be a performance issue if these would all be high throughput devices, but the majority are IoT things (light bulbs & smart plugs) that have almost no traffic. But I do concede I'm not your average user. Just would never expect (what I assessed to be) a high-end consumer router to cause me a problem on something as basic as DHCP assignments.

By now I'm actually happy with my setup and I consider it even better, as I can now assign a named device a specific IP and just restart the DHCP server instead of having to reboot the entire router for it to work. I don't think I'll be using the Orbi's DHCP anymore for that reason alone.

FURRYe38 · ‎2024-06-13

v.21 is a downgrade. FYI, v.21 was beta tested and lots of users helped out on that BEFORE it was released so after testing, it had mostly a good release and good user experiences. We didn't get a chance to beta test v.31 so I've been seeing posts over past month in regards to users not having great experiences with it. So been recommending for users having problems on v.31, go back to v.21. 🙄

Completely understand Sir. If it's working, run it and relax man. 😉

rdouma · ‎2024-06-13

Yeah... I'm a big fan of "if it ain't broke, don't fix it" 🙂

FURRYe38 · ‎2024-06-13

Ya, that too. 👍

TC_in_Montana · ‎2024-06-14

@rdouma wrote:

Thanks for the feedback, appreciate it.

As for the limit to concurrent devices... I wasn't aware there was a maximum other than the number imposed by a class C network. I could imagine there could be a performance issue if these would all be high throughput devices, but the majority are IoT things (light bulbs & smart plugs) that have almost no traffic. But I do concede I'm not your average user. Just would never expect (what I assessed to be) a high-end consumer router to cause me a problem on something as basic as DHCP assignments.

By now I'm actually happy with my setup and I consider it even better, as I can now assign a named device a specific IP and just restart the DHCP server instead of having to reboot the entire router for it to work. I don't think I'll be using the Orbi's DHCP anymore for that reason alone.

I wonder if this issue/limitation is part of the OpenWrt Chaos Calmer 15.05.1 code base that the router firmware is built upon or part of the Netgear personalization that sits on top of it.

Either way, it definitely is something that SHOULD work correctly!

raven_au · ‎2024-06-14

@FURRYe38 wrote:
Had you seen this behavior on FW v.21 as well?
Since v.31 has been out, though were seeing more issues with 750 series and some on the 960 series, not really confident that v.31 is a stable FW as v.21 is.

As you may have seen I have reported this problem more than once on different firmware versions.

v21 is the highest version I have seen it on, didn't go to v31.

raven_au · ‎2024-07-07

@raven_au wrote:
@FURRYe38 wrote:
Had you seen this behavior on FW v.21 as well?
Since v.31 has been out, though were seeing more issues with 750 series and some on the 960 series, not really confident that v.31 is a stable FW as v.21 is.
As you may have seen I have reported this problem more than once on different firmware versions.
v21 is the highest version I have seen it on, didn't go to v31.

And here we are, years later and the DHCP problem remains on models prior to the WiFi 7 routers.

Yes, that';s right, the RBE973 kit appears to have a working DHCP server, all it took was a move away from Broadcom to Qualcomm, a different sdk, so it's unlikely Netgear actually fixed anything themselves.

And another bug that has been reported and continued to go unfixed for years is the Orbi app seems unable to see iOS and iPadOS devices that have Armor installed always reporting them as not having Armor. The status of these devices gets recorded on armor.netgear.com AFAICS but the Oribi app can't see it. This can (and does) make the security score meaningless which, IMO, is useful. It's a shame that a feature that, IMHO, is quite good has remained buggy for so long as to make people that would appreciate it not want to use it.

While I'm at it I have to say that the TP-Link be85 works much better than this router. The hardware these two devices use is very similar and yet Netgear managed to mess up the WiFi, it's less stable and it's slower. I am persevering though because I believe the security on the Orbi is actually fairly good although still plagued with bugs too.