Re: Orbi Showing Wireless Devices as Wired in Attached Devices List

perrynmyers · ‎2020-09-27

Hi,

Issue:

I have found that on RBR50 deployments where there are two satellites connected via Ethernet Backhaul, that the Attached Devices List will incorrectly show which Orbi device a network client is connected to, and if it gets that wrong, will often show Wireless devices as Wired in the list.

Details:

I have two different physical locations (houses) with exactly the same router setup. Each site has an RBR50 acting as the primary Router and Orbi. Each site also has an RBR50S and an RBR40V (the Harmon Kardon Orbi Speaker). So, I have 3 Orbi devices at each physical site. The two sites are not connected at all. I am mentioning this because I am able to reproduce the behavior I am seeing at two different sites and installations completely.

The two satellites at each location are connected via Ethernet Backhaul and the connection status on each is Good. I am not using 5g Backhaul at all.

The behavior I am seeing is that some of the devices on each network, that are wireless only, such as Amazon Echo Dots, iPhones and other Android phones, are showing up as "Wired" in the Attached Devices List. This is, of course, impossible, since none of these devices even have Ethernet ports on them.

Not all of the Wireless devices show up incorrectly this way. In my observation, it depends on whether or not the device is listed as associated to the correct Orbi access point. Let me give a concrete example.

In site 1, I have an RBR50 in the basement, the RBR40V is on the first floor of the house connected to the RBR50 via Ethernet Backhaul. I have an Ethernet cable also running from the basement out to a detached garage (the distance is about 100 yards). The garage has an RBR50S connected via Ethernet Backhaul as well.

I have a generator with Wireless capability (2.4g only). The range on it is such that it SHOULD only be able to connect to the RBR50S in the garage. When I see the generator in the Attached Devices list associated to the RBR50S in the garage, it correctly shows up as connected via 2.4g. However, at other times I will see the generator listed as attached to the RBR50 and sometimes the RBR50S. When it shows up on the Attached Devices list associated to those other Orbi's, the generator shows up as Wired.

My hypothesis is that the algorithm for determining which Orbi access point "owns" a particular client device is broken. If the client is associated as attached to the same Orbi that it is using as a wireless access point, then it will properly show up as 2.4g or 5g (as appropriate). If however, the "wrong" Orbi picks it up as attached to it, it will show as Wired, because from the perspective of that Orbi device, it is NOT connected to it via Wireless (because it is connected to some other Orbi as Wireless).

This may be an issue with Ethernet Backhaul deployments. I have not tested this using 5g Backhaul to see if the issue persists. But I am able to reproduce it reliably with a random subset of wireless devices at each of the two physical deployments (houses) that I have.

Because one of my RBR50 devices is still under warranty for Technical Support, I did file a support case on Aug 30, 2020. Over the last several weeks I have provided screen shots and additional details to the technical support representative but as of yet no assistance has been provided (or ideas about root cause and resolution). I am presently waiting for a callback from their Technical Support engineers to discuss the issue further (but as they have called during times when I have not been available, I have been unable to speak to them as of yet).

If anyone else is seeing this behavior, please do weigh in on this thread and let me know what your specific details are. For example, if someone can reproduce this with 5g Backhaul or with only 2 access points (instead of 3, like in my setup) it would be useful to further narrow down what the root cause configuration might be.

I will continue to work with Technical Support, but I thought that posting the issue here to crowdsource some additional help would make sense.

Thanks!

Perry

perrynmyers · ‎2021-01-20

Hi,

I just closed out my support case with Netgear on this issue. It took 4 months (opened in August, finally closed in January) and I spent much of that time being passed back and forth between support associates. But in the end, Netgear finally admitted to me the following about this particular issue.

From Support:

"I have confirmed with our engineers that the satellite has a limitation with reporting connected devices while its in ethernet backhaul as the router identifies that the wired backhaul connection of the satellite rather than the actual connection of the device to the satellite which are engineers a indicated that it is a limitation in the design of the system.

I apologize for the inconvenience. Let me know if you have questions or concerns."

I then asked:

"Is it unfixable and therefore a product flaw that just needs to be lived with?"

And was told:

"As I have confirmed that this is a limitation as per our engineers. We were only informed of this upon consultation with our engineers and I sincerely apologize if it took awhile to get answers for you."

(Note: This answer from Engineering took 4 months to get to.)

So in short, the way the Orbi system is designed, it is impossible for them to report properly whether a device is Wired or Wireless if any of the satellites are using Ethernet Backhaul.

In my investigation into how the Orbi works (which under the covers appears to use technology provided by Fing to do device discovery), here is what I summarized based on a discussion with technical support at Netgear:

"After noting that all of the devices incorrectly labeled as Wired were coming from the RBS40V, you suggested that we remove the Ethernet backhaul from that specific satellite to see if it would correct the problem. I did this, and checked the Attached Devices list. I found that the RBS40V no longer reported any Wired devices at all. I also found that the RBR50 Attached Devices list was now 100% correct, and no longer incorrectly listed any wireless devices as Wired. You also mentioned that the Orbi's use Fing's software to do device collection. Perhaps the way the Fing utility is being run on the RBS40V is causing it to collect Wired devices instead of only reporting back the Wireless ones. Again, thank you for the call and helping to do some additional troubleshooting on this. I hope the above notes are helpful."

I also provided a potential software solution that could likely fix the problem:

"Each device (RBR50, RBS50, RBS40V) collects a list of devices it "sees". Each device knows which clients are connected wirelessly to it, and each device can see ARP requests for devices not connected to it and it detects those devices as "Wired". So, every device is both reporting "Wireless" devices accurately, and reporting "Wired" devices sometimes accurately, and sometimes inaccurately.

The right way to fix this would be for the RBR50 to have logic that preferred reporting Wireless if it gets conflicting information. For example: RBS40V reports a device as 2.4G, RBR50 reports same device as Wired. The RBR50 should "prefer" the Wireless reporting, since that is more accurate than the Wired reporting, and therefore should display the device as 2.4G. Right now, it appears to be a "random" determination as to whether the RBR50 will pick the Wired or Wireless designation for a device that is reported by multiple Orbi devices.

Further, Satellites (like the RBS40V and RBS50) can report Wired devices they see to the RBR50, but they should never list those as "locally connected". There's no way to determine if a wired device is connected to one Orbi device or another, so the default should be to always say that Wired devices are solely connected to the primary router (i.e. RBR50)

If the above logic was put into place, I believe this problem would be resolved."

Given that a user would expect to see Wired and Wireless devices reported accurately, and there is a potential software solution to fixing this issue, I believe this absolutely is classified as a bug.

Netgear has chosen to not fix this issue, and that is of course their perrogative. Not all bugs are of equal priority, and this one being purely cosmetic perhaps does not rise to the level of importance to justify investment in it being fixed. And it could also be that the fix for this would be more invasive than is warranted for an existing deployed system. But, I did suggest to Netgear support that perhaps they could implement what I described above in future generations of Orbi systems if that was indeed a concern.

In the end, the answer I got indicated that it is a limitation of the system. But in a sense, any bug can be waved away as a limitation of the system. A bug is a bug, even if the software vendor chooses not to fix it. I am content with the response I got from Netgear, because they acknowledged this was an issue (by claiming it was a limitation of the design) and eventually letting me know that they would not address it. (I only wish that it didn't require countless hours of my time and literally 4 months to get there...)

On the other hand, this is a bug, and I would hope that in future generations of Orbi software that they fix it, and I believe that the mechanism I suggested above could do that.

If the software was Open Source, I likely could fix this myself... But I could only go so far in trying to fix the issue given that much of the Orbi software are composed of binary blobs without accompanying source code.

Best regards,

Perry

View solution in original post

FURRYe38 · ‎2020-09-27

If you have RBS that are ethernet connected to the RBR, then any wifi devices connected at the RBS will get reported as wired connected devices as the line of status and detection are over the wired ethernet connection between the RBR and RBS, that the RBR uses to report the connection status. If you disconnect the RBS from the wire and let it wirelessly connect, then the status will change from wired to wireless for those devices connected at the RBS. This is normal operation.

perrynmyers · ‎2020-09-27

That answer doesn't make sense given the explanation I gave in the original post.

Take the wireless connected Generator I mentioned. Sometimes it is detected as being attached vis 2.4g to the Garage Orbi (RBR50S). And sometimes it is detected as being attached via Wired to the Main Orbi (RBR50). On any given day, it'll show up in one state or the other, and toggle later.

If this is normal operation, then why does it "toggle back and forth" between the two states?

Shouldn't it either:

1) Be detected as Wirelessly connected to the Garage Orbi and show as associated with that access point via 2.4g?

OR

2) Be detected by the Main Orbi and always show up as Wired, because it is not connected wirelessly to the Main Orbi (as it is connected via 2.4g to the Garage Orbi)

To me, it seems like there is a race condition here, and depending on the outcome of the race, one of the Orbis "wins" and gets to associate the device with it. If Main Orbi wins, it shows up as Wired (associated with that Orbi). If Garage Orbi wins, it shows up as 2.4g (associated with that Orbi).

This may be working "as coded" but I consider this to be a user experience deficiency that should be fixed. It's a bug.

Thanks,

Perry

FURRYe38 · ‎2020-09-27

Is the system working and is the device working other than the status changing?

Possible that the device is connecting to the RBR and when this happens then yes it will report as a wireless device instead of wired.

It's up to devices to pick and choose where they connect too. So this one device seems to be changing the connection location from the RBS which are wired to the RBR which would report and wireless.

This is not a bug.

If you feel other wise, then please contact NG support. There would be nothing we can do to effect any FW changes here in the forum.

Good Luck.

perrynmyers · ‎2020-09-27

The devices are working normally. It's purely a cosmetic issue in the Attached Devices reporting, as far as I can tell. That said, cosmetic issues affect the user experience, and so they should be fixed (but I agree at a lower priority than a functional issue that prevents usage).

The device is not connecting to the RBR (Main Orbi), because it is far enough away that there is zero signal at the device from the RBR. The RBR is in the basement of the house and also several hundred feet away, and a signal analysis from my phone doesn't even detect the RBR signal from the device's location at the Garage.

This is in fact why I am using Ethernet Backhaul. I am unable to do 5g backhaul (or get 2.4g signal) from the RBS (Garage Orbi) to the RBR (Main Orbi). Therefore, it is not the case that the device is connecting to the RBR, as it physcially cannot do so. Devices can pick and choose where they connect to, but only within the limits of the range of the access points.

I have contacted NG support, so far they have taken a month or so to... do pretty much nothing. I took this to the forum mainly to let others know about the problem, and to see if anyone else with a similar setup sees the same issue. Sometimes community forums are about sharing information and not just resolving the issue. 🙂

FURRYe38 · ‎2020-09-27

Are the RBS directly connected to the RBR or are there any kind of LAN switches in be ween the RBR and RBS?

IF this is just one device, then user experience maybe effected however honestly, how often does the status need to be checked. If you know the devices is connected to the RBS and the RBS is wired, then for the part, the status will be wired.Something to keep in contact with NG support about to see if they can do anything. I kind of doubt NG will be doing anything to correcxt the issue since this seems to be a corner case issue. Takes time and money to effect changes in FW development. Many people don't understand this.

perrynmyers · ‎2020-09-27

Hi,

There is a Netgear gigabit Ethernet switch between the two Orbi devices. It is a dumb switch however (no advanced filtering or other trafic shaping going on on that switch, just basic Ethernet frame switching).

It's multiple devices. I'd say about 1/3rd of the Wireless devices show up incorrectly on my network as Wired, attached to the wrong Orbi device in the Attached Devices list.

In terms of how often the status needs to be checked, perhaps not often. However, I expect to be shown accurate information in user interfaces and this seems like a place where inaccurate information is being shown. I brought this issue up with support and in the forum because I was trying to point out a bug that I thought could be worth fixing. I'm not trying to be the typical "angry user, fix my issue now!" person. I'm trying to participate in a community by providing my feedback and observations, and sharing information.

I've worked in open source software development for 20+ years and feedback like this is often considered very useful from a product and project development perspective. Maybe the issue can't be fixed in this iteration of the hardware, but maybe there is a design issue that could be addressed in future generations of the hardware.

As a software developer (in the past) and now managing several hundred software developers, I appreciate this sort of input, so long as it is given politely, and with an understanding that not everything can be fixed right away. I certainly don't expect this to be fixed immediately, but again, I posted the issue here so that others who might also be seeing this bug could learn and perhaps contribute to a broader understanding of it. Is that not appropriate for a community forum like this?

Thanks,

Perry

FURRYe38 · ‎2020-09-27

What is the model# of this switch?

Any green ethernet features?

FURRYe38 · ‎2020-09-27

And What FW version are you using?

perrynmyers · ‎2020-09-27

Hi,

Actually, I misspoke... I forgot that I had wired it so the Orbi Satellites were all direct Ethernet connected to the main RBR50 via the 3 Ethernet ports on the back of it. So my network configuration is... Main RBR50 connected to Cable Modem via WAN port, RBS50 and RBS40V are both connected via Ethernet directly to two of the RBR50 Ethernet ports. The third Ethernet port on the RBR50 is connected to the Netgear switch, which has the remainder of the actual Wired devices hanging off of it. (There are actually zero wired devices hanging directly off the two satellites)

In any case, the Netgear switch is this one: https://www.amazon.com/gp/product/B00M1C0186/

Since it's not between the RBR50 and the satellites, it shouldn't be affecting anything but... if you have any ideas/thoughts about something that could be potentially interfering let me know.

Lastly re: Firmware... The RBR50 and RBS50 are running latest: V2.5.1.16

The RBS40V is running V2.5.1.6 which is also the latest available for that model satellite.

Thanks!

Perry

FURRYe38 · ‎2020-09-27

Ok, Reason for asking is that some managed switches, like yours can cause problems between the RBR and RBS when the RBS is ethernet connected. However in your case, I would not believe it would. However you might test this out by disconnecting this switch from the RBR to see if anything changes.

I would also track down the RBS where this one device connects to and swap it out to with the other RBS to see if anything changes or follows. I would check when you notice when the device is on 2.4Ghz, go in and look at all the RBS connected devices and check the RBS status pages to see if anything appears to you. Possible that there could be a loose connection in the ethernet cable as well. However if the RBS appears as wired all the time in the RBR, connection should be stable however there is a lag time seen on the RBR status page if a connection changes. Its not quite immediate in the status page.

FURRYe38 · ‎2020-09-27

This switch also has green ethernet features as well. Something that shouldn't effect any downstream devices, hwoever could posibly do something on the upstream or connected to the RBR. Try disabling this feature if you can on the switch to test.

perrynmyers · ‎2020-09-27

Good idea on disconnecting the switch to rule it out. Always good to work reproducers down to bare minimum. I can also try to reproduce the issue with just a single satellite as well vs. having two. Maybe that will tell us something.

The Wired (Ethernet Backhaul) between the RBR and the two RBS's is very stable. I don't think I've yet seen it flake out or toggle back to 5G Backhaul. The other thing is that I see this behavior with devices connected to all of the devices, not just one. So I don't think it's related to a specific RBS or RBR.

I'm not at the location physically right now, but I will attempt some of the troubleshooting you've suggested to see if I can eliminate some variables but still reproduce the problem. Thanks for the advice (It may be a few days or a week before I can get back to this location to try out your suggestions though).

Thanks!

FURRYe38 · ‎2020-09-27

perrynmyers · ‎2020-11-03

My support case with Netgear to troubleshoot this really has made zero progress. Mainly because they keep asking me for times for their level 2 support engineers to call me, and then when the agreed upon time comes, no one calls because there is always a scheduling conflict on their end. I've had 6+ scheduled calls go this way over the last month and a half. So, because support seems unwilling to actually assist, I've done some debugging on my own and have figured out a few things.

(Note: I did end up removing the Ethernet switches between the RBS40V/RBS50 and the RBR50. So at this point, the RBS40V is directly connected via Ethernet cable to the RBR50. And similarly, the RBS50 is directly connected via Ethernet cable to the RBR50. The Ethernet switches are no longer in between any Orbi Devices)

1) I don't think Voxel's firmware is going to help, because ultiamtely Voxel is taking the stock Netgear firmware and updating the bits that can be updated (Open Source code, utilities, etc). But, the bits of Netgear's firmware that are proprietary can't be maintained or updated by Voxel. I think he's just using those as is. (For example the attached-devices/net-scan binaries)

2) I've observed by doing dumps of Attached Devices on the RBR50, RBS50 and RBS40V that the only devices that are misrepresented as Wired are devices that show up on the RBS40V as Wired. The RBS50 never reports Wired devices attached to it. It seems like the RBS40V is sending it's list of Attached Devices to the RBR50, and this contains a mix of attached 2.4G/5G devices as well as devices incorrectly determined to be Wired.

3) This likely causes the RBR50 to contain an attached devices list with duplicate MACs in it. For example, RBS50 reports Foo is connected to it's 2.4G interface. RBS40V incorrectly determines that Foo is Wired (because it is not attached locally) and reports that back as well. Now, when the RBR50 is generating the DEV_show_devices.htm page, whatever row comes up first wins. If the row submitted by the RBS50 is first, the Foo will show correctly as 2.4G and associated with the RBS50. But, if the table has the row from the RBS40V first, it likely results in the device being incorrectly shown as Wired and associated with the RBS40V. I think this is the case because the RBS40V (via looking at its DEV_device.htm page) always shows more Wired devices attached to it than the RBR50 shows as being associated to the RBS40V). And also, the set of Devices that the RBS40V reports incorrectly as Wired seems to be fairly static, but the set of devices the RBR50 ultimately reports incorrectly as Wired is a subset of those that varies from query to query over time.

4) Looking at the source code for the RBR50 (https://www.downloads.netgear.com/files/GPL/orbi-desktop-V2.7.0.70_gpl_src.zip) I was able to figure out that the souce for the DEV_show_devices.htm page come from files in /tmp/netscan. These files are generated via net-scan binary that in the source zip come from a proprietary attached-devices binary.

So, in short... I think if I was able to see the source for attached-devices/net-scan binary, so that I could see how the files in /tmp/netscan are generated, I'd likely be able to propose a fix for this bug. It could be as simple as "If a Satellite reports a Wired device attached to it, but the Router or another Satellite says that that same MAC is Wireless 2.4G/5G, then trust the device that reports the device as Wireless over the device that reports the device as Wired"

Unfortunately, since the attached-devices/net-scan binary is proprietary I can't proceed further with debugging. I have asked Netgear support if they can confirm my hypothesis or give me access to the source code so that I can debug further.

FURRYe38 · ‎2020-11-03

Ok.

ottopaulsen · ‎2021-01-20

How can you say that it is not a bug, that wireless devices are shown as wired? Of course it is a bug. You must see this from a users perspective. There is no way that those devices are wired.

For me using the UI it is very misleading that wireless devices are shown as wired.

Netgear should fix this!

FURRYe38 · ‎2021-01-20

It's NOT a bug. Again, when RBS are ETHERNET (wired) connected to the RBR, and any wireless device is the connected to the RBS, the detected connection will be wireled, not wireless incoming to the RBR since the RBS is ETHERNET(wired) connected to the RBR. If the RBS was wireleslly connected to the RBR, the the wireless device will be reported as wireless, not wired.

You can of course contact NG support to express your concerns over this as there would be nothing we can do about it here in the forums.

Good Luck.

@ottopaulsen wrote:
How can you say that it is not a bug, that wireless devices are shown as wired? Of course it is a bug. You must see this from a users perspective. There is no way that those devices are wired.

For me using the UI it is very misleading that wireless devices are shown as wired.

Netgear should fix this!

perrynmyers · ‎2021-01-20