Reply
Highlighted
Master

Re: Massive packet loss over Ethernet backhaul

@Mooose Good troubleshooting and interesting observations. It's clear that the packet loss is directly related to the wired backhaul.

I'm now running the ongoing beta software (2.3.0.23) and I can see that NG has done quite good improvements on wired device handling. Would you want to test this SW and see if you will see improvements? If yes, please PM to @ChristineT and ask to get the SW download links. 

My Setup Internet Fiber ONT 250↓/250↑ISP Telenor | Wifi Router Orbi RBK50 AC3000, Router Mode, Wired Backhaul / Orbi RBK852 AX6000, Router Mode, Wireless Backhaul | Switches Netgear GS208Time Zone CET (Sweden)

View solution in original post

Message 26 of 40
Highlighted
Luminary

Re: Massive packet loss over Ethernet backhaul

@FURRYe38 Something is definitely going on with the forum software. I have been awarded a badge for 25 posts even though obviously only 10 of them have been successful. Notepad++ isn't available for my OS of choice, but I am using BBEdit which should provide the same plain-text results.

 

I had moved the satellite to the same room and used a short cable for the tests described in my first post. After updating and resetting, and adding one thing at a time, I did move the satellite to the remote (long cable) location before changing the SSID, thus allowing the wifi devices back on, which caused the issue to resurface.

 

However, now I have gone fully nuclear, ripped fuses from the fusebox, removed backup batteries and pulled cables so that only two devices are connected. Results look good so far, still in the remote location.

 

190206-Nuke.gif

 

Pingplotter looks interesting, will have to try it.

 

My Orbi may very well be faulty but I doubt I can return it as I had to glue a plastic mount to the back of the router as that was the only way I could screw it to the wall where I wanted it. Also, I bought it over two years ago.

 

@ekhalil Does the 2.3.0.23 firmware seem stable to you? If so, I may just try it.

 

 

 

Message 27 of 40
Highlighted
Guru

Re: Massive packet loss over Ethernet backhaul

Wow, ok. 

 

So if your up and running with wired back haul and not seeing any PLs, graduate adding devcies then and see if the problem appears. 

Is the Switch still connected to the Orbi router? Just curious. 

 

You might give the beta a try as well. 

 

Ok about your warranty, I didn't know how long you have had it and made modifcations to the case. Smiley Wink


@Mooose wrote:

@FURRYe38 Something is definitely going on with the forum software. I have been awarded a badge for 25 posts even though obviously only 10 of them have been successful. Notepad++ isn't available for my OS of choice, but I am using BBEdit which should provide the same plain-text results.

 

I had moved the satellite to the same room and used a short cable for the tests described in my first post. After updating and resetting, and adding one thing at a time, I did move the satellite to the remote (long cable) location before changing the SSID, thus allowing the wifi devices back on, which caused the issue to resurface.

 

However, now I have gone fully nuclear, ripped fuses from the fusebox, removed backup batteries and pulled cables so that only two devices are connected. Results look good so far, still in the remote location.

 

190206-Nuke.gif

 

Pingplotter looks interesting, will have to try it.

 

My Orbi may very well be faulty but I doubt I can return it as I had to glue a plastic mount to the back of the router as that was the only way I could screw it to the wall where I wanted it. Also, I bought it over two years ago.

 

@ekhalil Does the 2.3.0.23 firmware seem stable to you? If so, I may just try it.

 

 

 


 


My Setup (Cable 900Mbps/50Mbps)>CAX80>RBK50 v2.7.0.70(AP Mode)
Additional NG HW: C7800/CM1100/CM1200/CM2000, Orbi CBK40, RBK853, R7800, R7960P,
EX7500/EX7700, XR450 and WNHDE111
Message 28 of 40
Highlighted
Master

Re: Massive packet loss over Ethernet backhaul


@Mooose wrote:

 

......

@ekhalil Does the 2.3.0.23 firmware seem stable to you? If so, I may just try it.

.....

 


@Mooose Yes, this beta seems very stable and I can see it solved many issues. I strongly recommend it! Smiley Happy

My Setup Internet Fiber ONT 250↓/250↑ISP Telenor | Wifi Router Orbi RBK50 AC3000, Router Mode, Wired Backhaul / Orbi RBK852 AX6000, Router Mode, Wireless Backhaul | Switches Netgear GS208Time Zone CET (Sweden)

Message 29 of 40
Highlighted
Luminary

Re: Massive packet loss over Ethernet backhaul

@FURRYe38 Neither the Nintendo Switch or the DGS-1005D switch have been allowed back on. However, all the Ring Floodlight Cams, the doorbell and the Z-wave gateways are back. (I needed to restore power to the garage freezer. Please don't ask.) Some phones, a tablet and one Airport Express are on as well, 12 devices in total.

 

Looking good so far, this is from a computer connected through wifi to the satellite to the router, satellite and another computer connected through wifi to the router.

 

--- 192.168.1.1 ping statistics ---

40000 packets transmitted, 40000 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.908/2.673/244.120/6.409 ms

 

--- 192.168.1.2 ping statistics ---

40000 packets transmitted, 40000 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.854/2.482/212.241/6.583 ms

 

--- 192.168.1.35 ping statistics ---

40000 packets transmitted, 39999 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 1.758/4.609/222.301/10.133 ms

 

I think the Apple TV and then the Drobo will be welcomed back next.

 

@ekhalil Thanks, I have put in my application for the beta community.

Message 30 of 40
Highlighted
Guru

Re: Massive packet loss over Ethernet backhaul

Good to hear. 

Seems like maybe it's not a FW issue. Smiley Wink

If you keep graduating devices online, hoping that something will reveal itself or possible switch issue. 

Keep us posted. 


@Mooose wrote:

@FURRYe38 Neither the Nintendo Switch or the DGS-1005D switch have been allowed back on. However, all the Ring Floodlight Cams, the doorbell and the Z-wave gateways are back. (I needed to restore power to the garage freezer. Please don't ask.) Some phones, a tablet and one Airport Express are on as well, 12 devices in total.

 

Looking good so far, this is from a computer connected through wifi to the satellite to the router, satellite and another computer connected through wifi to the router.

 

--- 192.168.1.1 ping statistics ---

40000 packets transmitted, 40000 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.908/2.673/244.120/6.409 ms

 

--- 192.168.1.2 ping statistics ---

40000 packets transmitted, 40000 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.854/2.482/212.241/6.583 ms

 

--- 192.168.1.35 ping statistics ---

40000 packets transmitted, 39999 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 1.758/4.609/222.301/10.133 ms

 

I think the Apple TV and then the Drobo will be welcomed back next.

 

 


 


My Setup (Cable 900Mbps/50Mbps)>CAX80>RBK50 v2.7.0.70(AP Mode)
Additional NG HW: C7800/CM1100/CM1200/CM2000, Orbi CBK40, RBK853, R7800, R7960P,
EX7500/EX7700, XR450 and WNHDE111
Message 31 of 40
Highlighted
Master

Re: Massive packet loss over Ethernet backhaul


@Mooose wrote:

 

.........

@ekhalil Thanks, I have put in my application for the beta community.


Great. I hope that this will help, as I know that the current software has issues handling wired devices and LAN port management in general.

My Setup Internet Fiber ONT 250↓/250↑ISP Telenor | Wifi Router Orbi RBK50 AC3000, Router Mode, Wired Backhaul / Orbi RBK852 AX6000, Router Mode, Wireless Backhaul | Switches Netgear GS208Time Zone CET (Sweden)

Message 32 of 40
Highlighted
Luminary

Re: Massive packet loss over Ethernet backhaul


@FURRYe38 wrote:

Seems like maybe it's not a FW issue. Smiley Wink

If you keep graduating devices online, hoping that something will reveal itself or possible switch issue. 

Keep us posted. 

I let the Apple TV back on (wired, and with wifi disabled) and the packet loss is back. (Although I thought for a moment that disabling 20/40 MHz Coexistence had solved the problem.)

 

I just disconnected Ethernet backhaul again and waited for the 5G to come up without changing anything else, looks fine so far. I will run a ping test overnight and see what happens.

 

If it's good I'll unplug the Apple TV, plug the Ethernet backhaul back in, run a test and then try plugging the Apple TV back in.

 

I just realized this was the first wired device I reconnected. Perhaps the triggering scenario is something (anything) wired to the LAN ports in combination with Ethernet backhaul and something on wifi? There are just too many variables to test conclusively considering the circumstances.

Message 33 of 40
Highlighted
Guru

Re: Massive packet loss over Ethernet backhaul

Great troubleshooting and narrowing it down. I was hoping one of your devices would reveal something. 

Is the ATV up to date?

Keep us posted. 


My Setup (Cable 900Mbps/50Mbps)>CAX80>RBK50 v2.7.0.70(AP Mode)
Additional NG HW: C7800/CM1100/CM1200/CM2000, Orbi CBK40, RBK853, R7800, R7960P,
EX7500/EX7700, XR450 and WNHDE111
Message 34 of 40
Highlighted
Luminary

Re: Massive packet loss over Ethernet backhaul

Feels like I'm back to square one. Had no packet loss all night. Reconnected Ethernet backhaul this morning, and ran 200000 pings from a  device at each end of the backhaul to five or six other devices on the network. It was fine until this afternoon, when the second test showed 8%.

 

I disconnected the Apple TV and connected the Drobo (attached to the router instead of the satellite) and ran a new test. The result was pretty much the same.

 

In both cases the pinging laptop at the same end (i.e. *.35 for the Drobo and *28 for the Apple TV) showed zero loss to the respective device, but the one at the other end of the backhaul did show loss in both cases.

 

I feel the only thing I have accomplished is proving that the packet loss is somewhat intermittent even though it is fairly consistent, which means that practically all my testing where the result was no packet loss is useless.

 

The only thing I know for sure is that I have only been able to measure packet loss exceeding 0.3% when the Ethernet backhaul is connected, and when there is packet loss I only see it between devices at opposite ends of the backhaul.

 

There are simply too many variables and testing is too time-consuming, so my only hope now is that a new firmware will resolve this.

Message 35 of 40
Highlighted
Luminary

Re: Massive packet loss over Ethernet backhaul

190208-Network.jpg

Message 36 of 40
Highlighted
Guru

Re: Massive packet loss over Ethernet backhaul

Can you reboot the router and satellite while wire connected and leave OFF the ATV and Drobo devices for the weekend and test the system here and there over the the weekend. It's possible that once the system gets in a bad state due to something on the network, it can't get back to a good state and needs a reboot. Seen this before with mine,especailly if the loop happens. Just curious if a longer duration with out those two devices being online if system will maintain or have the problem appear again. 

 

Hope you can get ahold of the beta FW as well and test that out. 


@Mooose wrote:

Feels like I'm back to square one. Had no packet loss all night. Reconnected Ethernet backhaul this morning, and ran 200000 pings from a  device at each end of the backhaul to five or six other devices on the network. It was fine until this afternoon, when the second test showed 8%.

 

I disconnected the Apple TV and connected the Drobo (attached to the router instead of the satellite) and ran a new test. The result was pretty much the same.

 

In both cases the pinging laptop at the same end (i.e. *.35 for the Drobo and *28 for the Apple TV) showed zero loss to the respective device, but the one at the other end of the backhaul did show loss in both cases.

 

I feel the only thing I have accomplished is proving that the packet loss is somewhat intermittent even though it is fairly consistent, which means that practically all my testing where the result was no packet loss is useless.

 

The only thing I know for sure is that I have only been able to measure packet loss exceeding 0.3% when the Ethernet backhaul is connected, and when there is packet loss I only see it between devices at opposite ends of the backhaul.

 

There are simply too many variables and testing is too time-consuming, so my only hope now is that a new firmware will resolve this.


 


My Setup (Cable 900Mbps/50Mbps)>CAX80>RBK50 v2.7.0.70(AP Mode)
Additional NG HW: C7800/CM1100/CM1200/CM2000, Orbi CBK40, RBK853, R7800, R7960P,
EX7500/EX7700, XR450 and WNHDE111
Message 37 of 40
Highlighted
Luminary

Re: Massive packet loss over Ethernet backhaul


@FURRYe38 wrote:

Can you reboot the router and satellite while wire connected and leave OFF the ATV and Drobo devices for the weekend and test the system here and there over the the weekend. It's possible that once the system gets in a bad state due to something on the network, it can't get back to a good state and needs a reboot. Seen this before with mine,especailly if the loop happens. Just curious if a longer duration with out those two devices being online if system will maintain or have the problem appear again. 

I'd love to do more testing and actually figure out what is going on, but the family needs their stuff up and running again, as do I. I have also lost about two weeks of productivity, and more testing will without a doubt lead me down more rabbit holes.

 

If I am offered a beta firmware I will try that, but other than that I won't be doing any more testing.

 

Even if I managed to track down the cause it seems likely that the network would be so fragile, requiring connecting and restarting devices in a specific way, that it would still not be practically usable.

 

Everything is back on the network now, about 35 devices in total. I just finished a run of 200000 pings from two devices at opposite ends of the (wireless) backhaul to six different destinations each with a result of 0.0% packet loss. A few of them even managed every single ping, so obviously there is nothing wrong with any of my attached devices.

 

I also realized that I can hook the Nintendo Switch up to the backhaul Ethernet cable directly to the router and reduce its latency that way. It sucks that other devices will have to deal with unnecessary latency and wifi congestion, but it's still workable.

 

I really appreciate all the advice. Thanks, all!

Message 38 of 40
Highlighted
Master

Re: Massive packet loss over Ethernet backhaul


@Mooose wrote:

........

If I am offered a beta firmware I will try that, but other than that I won't be doing any more testing.

........


Have you sent a PM to @ChristineT regarding the beta participation?

My Setup Internet Fiber ONT 250↓/250↑ISP Telenor | Wifi Router Orbi RBK50 AC3000, Router Mode, Wired Backhaul / Orbi RBK852 AX6000, Router Mode, Wireless Backhaul | Switches Netgear GS208Time Zone CET (Sweden)

Message 39 of 40
Highlighted
Luminary

Re: Massive packet loss over Ethernet backhaul


@ekhalil wrote:

Have you sent a PM to @ChristineT regarding the beta participation?

Yes, I have both posted in her thread on the subject and sent a PM.

Message 40 of 40
Top Contributors
Discussion stats
  • 39 replies
  • 5226 views
  • 29 kudos
  • 5 in conversation
Announcements