Wired backhaul problem with MR60 + 2xMS60 (only when wiring both satellites)

[cjgrant]

I've looked through previous posts, and don't see the same issue I'm seeing.  I have a MR60 + 2 MS60 satellites.  When only one of the satellites is wired, everything is fine.  When I attempt to hard-wire the second, the network collapses.

Setup:

- Rogers cable modem (bridge mode) -> MR60 -> [Negear GS308 switch] -> [Both MS60's and several other things (PCs, Chromecast, TVs)]

- CAT6 everywhere

- Firmware updated via the Netgear Nighthawk app v1.1.7.134

- No issues with network speed when it's working; no issues with signal strength, AP spacing, etc.

- The 3 APs are spread around the corners of a 2400 sqft home on 3 levels (30 to 40 feet apart laterally, and through floors)

Issue:

- When I got this system, I wasn't ready to use the second satellite.  The MR+MS wired setup worked flawlessly for months.

- Then, I attempted to add the second satellite.  When wireless, the satellite is happy in the network.  But when wired, the entire network grinds nearly to a halt.  Wired PCs see ping time to the router increase to 100's of milliseconds, or completely drop the LAN connection.  Streaming anything is impossible, and various status LEDs on the MRs cycle between blue and orange.

- I've factory reset and re-set-up several times.  Doesn't help.

- Doesn't matter which of the 2 satellites is added to the initial pair (ie. it's not a defective single MS device)

- Removed other downstream unmanaged switches, just to simplify the setup for debugging

- The issue is not specific to a particular cable.  I see the same problem when centralizing everything and assembling a test setup.

Summarizing, our network is unusable, until I un-hardwire one of the satellites, and then it becomes a solid working network.  I'd keep it this way, except that I ran the wiring for a reason, and want this to function properly.

It's as if wiring the second satellite causes some sort of loop in the network, and packets start getting lost in space.

[plemans]

What happens if you put the rogers gateway in router mode (disable its wireless) and the MK63 system in access point mode?

Make sure you go rogers modem---->wan port on MR60---->lan port to switch and then to satellites

[FURRYe38]

Hmmm interesting...Something I've not seen when I had my MK63 online and both MS ethernet connected. I also have a GS308 and GS-105/108 swicthes.

Does the problem track or flow if swapping MS locations? or does it remain in one location? Depending on which MS you turn ON first? 

Does the network go down when both MS are wirelessly synced only? 

Did you first turn ON the MS and let them sync to the MR wirelessly first, then after they are synced, connect them one at a time via ethernet? 

Did you try testing with a different non managed switch? 

Does the LEDs on the GS switch remain always constantly blinking when you connect the 2nd MS via ethernet or do the LEDs blink randomly here and there? 

Seem like if a 2nd MS is connected, something in between maybe causing the problem. Bad cable, cable plug wall jack. In wall cabling...

For that size of home, only the MR and 1 MS maybe needed...depending...

[cjgrant]

> What happens if you put the rogers gateway in router mode (disable its wireless) and the MK63 system in access point mode?

I can give that a shot to help diagnose, but I'm in the business of keeping my ISP's involvement in my network to as little as possible.  I really wish it was still possible to buy "just a modem".  I'll report back if this changes anything for the better.

[cjgrant]

Thanks for those thought-out suggestions.  In-line...

> Does the problem track or flow if swapping MS locations? or does it remain in one location? Depending on which MS you turn ON first? 
The problem does not follow location.  It depends on which device is powered up first.

> Does the network go down when both MS are wirelessly synced only? 
No, this works fine.

> Did you first turn ON the MS and let them sync to the MR wirelessly first, then after they are synced, connect them one at a time via ethernet? 
No, but I can try it.  I've done the setup with them wired.  I think I did a wireless setup way back initially, but not since.  Worth a try.

> Did you try testing with a different non managed switch? 
Not yet, but I'm going to.  I have two other switches, a newer GS308 and something by DLink - worth a try!

> Does the LEDs on the GS switch remain always constantly blinking when you connect the 2nd MS via ethernet or do the LEDs blink randomly here and there? 
The LEDs settle on solid orange or blue, then seem to change over time depending on mood, cosmic rays, randomness, etc.  🙂

> Seem like if a 2nd MS is connected, something in between maybe causing the problem. Bad cable, cable plug wall jack. In wall cabling...
I'd have guessed that too, except that I can reproduce this with the APs layed out on a floor in front of me, with different cables.

> For that size of home, only the MR and 1 MS maybe needed...depending...
Yeah, it's true - we can live with one AP.  However, the coverage gets spotty in certain rooms, and with a newly finished basement, our 3-AP arrangement puts one AP at each end and corner of an L-shaped home.  I agree it may be slightly overkill, but not by much, and it was a conscious choice.  😄

[cjgrant]

A little update if interested:

During a time when our network was degraded, I happened to be near an 8-port network switch.  Every interface light was going berserk.  Highly suspicious.  I fired up Wireshark on a Linux box and did a capture - hundreds of thousands of UDP  SSDP (Simple Service Discovery) packets per second hitting the interface.

I traced the source MAC address and IP to a Google Home Mini.  Now the interesting part - we unplugged the mini, and the flood continued.  Zombie Mini?!?  Unplug one of the MS60's, and the flood stops.

I am not a network protocol expert, but this smells like the mesh network is getting confused about routing multicast packets.  It might be the case that our network is fine until (1) I plug in the second satellite, and (2) a certain type of packet enters the network, at which point, kaboom, our network is DOSing itself.

I'll leave that Mini unplugged for now, but it's one of 6 in the home, and I'd be a little surprised if it's actually doing something different than the other 5...   especially when, uhh, not powered.

[FURRYe38]

So after unplugging this mini device, does all MS and the backhaul now work correctly? 

[cjgrant]

Yes.  I've verified over the past week that our network has been fine.  Today, I re-tested by plugging the Home Mini back in, and within a few seconds, the network went back to getting slammed.  But again, unplugging the Mini doesn't resolve the issue - the traffic continues.  Why this happens with 2 satellites but not 1 is still a mystery.

- Is there a Spanning Tree Protocol bug in the Nighhawk firmware?

- Why does packet TTL not address this?

- Why does this one Mini cause problems when the other ~5 don't?  (yes, I know leaving it unplugged is a good idea 🙂

I need to do a bit more research on the packet structure, and spend some time in Wireshark.  Maybe I can identify more of what's happening.

[FURRYe38]

Possible that these mini devices are broadcasting something and having more than 4 online causes the system to get into a bad state. I would contact the Mfr of these devices and ask them about this. 

[cjgrant]

That's an option, but I'm betting this is not a Google Home problem.  There was no issue with our old router, or the two-node Netgear setup.  It's only with the 3rd node, when wired, that this problem starts, and then persists when the Google Home is removed.  Also, I don't have numbers, but I'm betting there are orders of magnitude more Home devices deployed than Nighthawk mesh systems.  These comments are no proof of anything, but I'm planning to leave the post unresolved until I've at least taken another crack at narrowing the issue further.

[FURRYe38]

So with all MS connected and all Mini devices disconnected or turned...the system works right? 

What happens when you start to one one mini device at a time...start with the last mini device that you turned on before that seems to cause this problem. Graduate adding or turning ON the mini devices until something happens. 

Google may know or have information regarding this issue. Might ask them how many devices they support on one network. Any known issues with having too many devices on one network. 

[cjgrant]

Another update - it turns out my week of having the network "working" wasn't actually the case.  The second MS60 was put on a switch with a TV, and that switch now appears dead to the world, so that MS60 wasn't actually in wired-backhaul state all week.  Maybe the traffic overheated the switch (partially joking).

After removing the switch (restoring wired backhaul), bam, problem's back.  I got a better look at a packet capture, and the flooding packets are evenly distributed from these devices:

fe80::22df:b9ff:fece:aa43 192.168.1.9 Google-Home-Max
fe80::d6f5:47ff:fe29:546c 192.168.1.17 Google-Nest-Mini
fe80::d6f5:47ff:fe2d:8473 192.168.1.11 Google-Nest-Mini
fe80::d6f5:47ff:febd:852d 192.168.1.10 Google-Nest-Mini

Ie, there was no problem with the original Mini I've since unplugged - I just saw its packets first and focused on that device.

I don't know if it's normal to paste packet captures into this forum, but the packets captured from each ot these devices are all identical, and coming in at 100's of thousands per second.  They're IPV6 multicast SSDP.  I'm convinced now that when I connect the second MS60, I'm creating a layer 2 loop somehow (eg. the wireless mesh backhaul path stays up, despite the wired path), and multicast packets fall into it.  I wonder how many MR60 + 2 x MS60 + all wired backhaul setups exist in the wild - ie. how well tested is this config...

[FURRYe38]

What was the brand and model of the failed switch and what is the brand and model of the new switch in place? 

Sounds like these devices are flooding the system when they are online and probably the system is collapsing when the 4th one gets online. The system is already flooded with 3 and the 4th just put's it over the edge. 

I'd contact Google about this and give then this info.

I've not need this kind of behavior with my MK63 when I had it online, though I don't have these specific devices or quantity either. 

[cjgrant]

Thanks for the thoughts FURRYe38...

The failed switch is irrelevant I think - the only effect of this was that it inadvertently cut off the wired backhaul, preventing the problem.  It's an 8-port D-Link unmanaged - very similar to the 8-port Netgears we're using.

Please don't take this the wrong way - I appreciate your thoughts!  But, you're suggesting that the Google Home devices are flooding the network, and I've proven that's not the case.  Once the flood starts, it continues after removing those devices.  We systematically unplugged/disconnected nearly every device in our home, and the flood continued until we broke the wired backhaul connection to one of the MS-60's.  The packets seen in the Wireshark capture are identical per source device.  These symptoms *exactly* fit a layer 2 routing loop.  Ie, the Netgear nodes are receiving packets on one interface (eg. wired backhaul), and forwarding them out another (eg. wireless backhaul), without detecting there's a loop in the spanning tree.  When we disconnect wired backhaul, the loop is broken, and there's no problem.

If the Google Homes were an issue, I'd see a stream of SSDP packets at all times - there isn't.

Debugging this would be a beast of a problem, AFAIK, without a collection of time-synced network analyzers watching various links, to see how the flood starts.  I've considered trying to synthesize the problem by writing a little tool to send out an IPV6 multicast SSDP UDP packet manually - ie, try and induce the problem by creating one specific packet, and watching the network grind to a halt.  Maybe on a rainy day.

Another option would be a Netgear config option to disable wireless backhaul...  ie, reduce the system to a router and 3 total access points, with only wired connections between the APs (no loops possible).  I feel like maybe this is possible, but in skimming the UI options, looks like I'd lose the routing capabilities too, which are needed in our setup.

Getting back to the previous post - is ther any evidence here that the issue is *not* the Netgear system?

[FURRYe38]

If the problem starts with the Google devices, and is caused by the devices, and causes the MK system to get into this bad state, the MK system may not have the ability to stop it on it's own as you say that you have to break the wired backhaul to stop it, or probably power cycle the MS, switch or MR to stop the bad behavior. I have seen this in other systems when a ARP storm happens. Systems can't stop it and have to be power cycled or break the connection to make it stop. Home systems are not robust enough like enterprise systems where they may have features to stop this kind of bad behavior. 

Again, this all starts with these devices and there flooding fo the network seems to point to the root cause of making the MK system and network un-stable when you added the 4th or more devices. I would again, recommend that you contact Google about this and ask about these broadcasts from there devices and what are there expectations of having 3 or more of these devices on the same network. I can only presume that maybe this could happen on other MESH systems with ethernet backhauls.