Orbi WiFi 7 RBE973
Reply

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

Pokemaniac
Aspirant

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I've been having these issues with RCN, as well, so it is clearly not something entirely specific to Comcast.

Message 26 of 151
CGASTON
Guide

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I dont have an issue. So believe me or not.
Message 27 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

CGASTON:  It's not that we don't believe you; it's just that without any metrics to go over with the ISP to indicate the problem is on their side, it's hard to justify calling and asking them to do something about it.  Since the issue seems to be across multiple ISPs and the fact that disabling link aggregation resolves the issue, it would make more sense to be related to the CM1200 rather than the upstream provider.

 

Having said that, since you're no longer having the issue, can you provide screenshots showing that link aggregation is indeed still enabled, the firmware version, and showing the system up time is over 48 hours? 

Message 28 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

And since I hit the issue again this morning, I wanted to add the increased latency metrics that I see when the issue occurs.  As you can see, the issue is local to the cable modem.

 

$ ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1): 56 data bytes
64 bytes from 192.168.100.1: icmp_seq=0 ttl=63 time=255.084 ms
64 bytes from 192.168.100.1: icmp_seq=1 ttl=63 time=281.300 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=63 time=295.809 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=63 time=308.160 ms
64 bytes from 192.168.100.1: icmp_seq=4 ttl=63 time=282.951 ms
64 bytes from 192.168.100.1: icmp_seq=5 ttl=63 time=296.976 ms
64 bytes from 192.168.100.1: icmp_seq=6 ttl=63 time=314.000 ms
64 bytes from 192.168.100.1: icmp_seq=7 ttl=63 time=308.621 ms
64 bytes from 192.168.100.1: icmp_seq=8 ttl=63 time=211.340 ms
64 bytes from 192.168.100.1: icmp_seq=9 ttl=63 time=229.270 ms
64 bytes from 192.168.100.1: icmp_seq=10 ttl=63 time=315.650 ms
64 bytes from 192.168.100.1: icmp_seq=11 ttl=63 time=357.060 ms
64 bytes from 192.168.100.1: icmp_seq=12 ttl=63 time=265.974 ms
64 bytes from 192.168.100.1: icmp_seq=13 ttl=63 time=326.605 ms
64 bytes from 192.168.100.1: icmp_seq=14 ttl=63 time=268.339 ms
64 bytes from 192.168.100.1: icmp_seq=15 ttl=63 time=290.238 ms
64 bytes from 192.168.100.1: icmp_seq=16 ttl=63 time=336.371 ms
64 bytes from 192.168.100.1: icmp_seq=17 ttl=63 time=347.420 ms
64 bytes from 192.168.100.1: icmp_seq=18 ttl=63 time=369.044 ms
64 bytes from 192.168.100.1: icmp_seq=19 ttl=63 time=262.031 ms
64 bytes from 192.168.100.1: icmp_seq=20 ttl=63 time=278.381 ms
64 bytes from 192.168.100.1: icmp_seq=21 ttl=63 time=284.568 ms
^C--- 192.168.100.1 ping statistics ---
22 packets transmitted, 22 packets received, 0% packet loss
round-trip min/avg/max/stddev = 211.340/294.781/369.044/38.279 ms

 

$ traceroute 192.168.100.1
traceroute to 192.168.100.1 (192.168.100.1), 64 hops max
1 10.0.0.1 0.141ms 0.165ms 0.122ms
2 192.168.100.1 276.441ms 245.408ms 259.397ms

 

Normally this is sub-1ms.  Attaching screenshots showing the statistics to the CMTS are unchanged, my uptime is over two days, link aggregation is enabled and active, and I'm running the latest available firmware.

 

I have opened a NETGEAR support ticket for this issue.

 

Thanks,

Kenny

Message 29 of 151
cursixx
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I'm having the same issue with my CM1200 on 1Gbps service (Charter/Spectrum) and LACP, after 8-10 hours the speeds drops to sub 100Mbps and more like 20-30Mbps. I'm running Pfsense and I have found if I 1. restart the modem it works for awhile or 2. just resetting the LACP interface will restore normal performance. I had a moto MB8600 before this and never had a problem with the LACP link on it. Is there a firmware fix in the works or already out? has it been pushed to ISP for testing? Looks like I might be going back to the MB8600 and returning the CM1200  

Message 30 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

Oh, great catch.  Confirmed that if I bounce (shut / no shut) the physical interfaces, performance returns to normal.  This should be easy to script to run every 12 hours or so.  I'm going to give NETGEAR until near the end of my return period to try to resolve this one, otherwise I'll be swapping for a MB8600 myself.

Message 31 of 151
cursixx
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I'm going to open a ticket as well and see what happens but i'm sure even if they fix it via a firmware update it will take 6 months before Spectrum pushes it out. I know Comast is a little faster with firmware updates. Nothing like a $200 modem not working out of the box correctly but its netgear so I should know better by now.

Message 32 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

Well, NETGEAR support got back to me and said I had to call in, which I did, as this was not an issue they could resolve via email support.  Phone support said there was no way to report this as a bug and insists that it's either a problem that will be resolved by RMA'ing the modem or had to be a problem with the switch, which conveniently was also a NETGEAR device.  They said they had no method of reporting bugs to backline engineering, which is a bummer.  I guess I have no choice but to return the unit and get a Motorola or Arris. I hate that, since I'm sure engineering could resolve this if I had a method of reporting it.

Message 33 of 151
cursixx
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I switched back to my MB8600 last night and the issue is gone and honestly the MB8600 is just as fast if not better. So I'll be returning my CM1200 this weekend as well. Sad really because its a nice modem and like I said before even if/when they do fix the problem it would take close to a year if I was lucky before the update was pushed out by Spectrum. Better testing, quility control and reporting Netgear. 

Message 34 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I ended up returning my CM1200 and replaced it with a Motorola MB8600 and everything is working great now.  If you're looking to do LACP, I just can't recommend anyone purchase the CM1200 despite it being an advertised feature.  It's obvious that this was untested and NETGEAR support is completely unwilling to be accountable for this.  I'm tired of arguing about it... and to be honest, I'm not sure why I tried so hard anyhow to get someone to listen since I knew deep down that it would be a complete waste of time and nothing but blame shifting.

Message 35 of 151
CPLMayo
Guide

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I didn't do a search and made a similar post here Duplicate Post.

 

I am having the same problem on Spectrum; I have the 1Gbps service and initially connected to my pfSense router via LACP. My findings where similar in that if I reset the connection my speeds would return to normal. However my time frame for the reduced throughput was significantly lower than reported by others.

 

Initial speeds would be 800Mbps - 1Gbps but after four to five hours the speeds would drop to ~20Mbps with high latency. I found that resetting the LACP link would return speeds to normal. This lead to me to wondering if it was a firmware issue, maybe the hashing used on the CM1200? So I put a switch between the CM1200 and my pfSense router. This would extend the time frame between slow downs but inevitably they would return.

 

At this point I am have removed the LACP and I am down to one port. While my speeds are horrendous they are much better than 20Mbps; I attribute this Spectrum over provisioning or some other shenanigans.

 

I am attach screen shots of speed tests and a pdf that shows my signal levels.

 

I would like to think that Spectrum is not playing any silly games with my connection but I have no proof that they are not; my main goal is to try and figure out what is going on with this. If anyone has any recommended troubleshooting steps please let me know. I have a good amount of networking knowledge and want to get this fixed.

 

@vkdelta- Any information on possible fixes? If I have to call Spectrum and give them hell I can but don't want to take that step until I know it is them that is the problem.

 

8/5/2018 - Screenshots

Model: CM1200|Nighthawk Multi-Gig Speed Cable Modem
Message 36 of 151
cursixx
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

The issue is not with Spectrum. I'm on Spectrum and have been using LAGG with my MB8600 with no issues. I can hit 1.22Gbps with my pfsense build. I have also tried a SB8200 without this issue. The Netgear CM1200 LACP is broken and must be out of ieee standard because if you take one of the members down the interface will no longer pass traffic. That is one of the most basic fuctions of 802.3ad "redundancy of link if a members connection is lost"

 

I dont think its hash related because I tested that as well using my Cisco 3850 switch. Same results with the CM1200 but I would interrested in hearing from someone with the a multigig Netgear router with the CM1200 but I wouldn't be surprised if the results are the same.

 

If you want multigig with Spectrum or any other provider look at the MB8600 or SB8200 and I guess maybe the CM1100 but I have not tested that model.

Message 37 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I can confirm that your ISP is not the issue.  After several days of bouncing around with NETGEAR support (I kept getting bounced between the switch and cable modem teams since both devices are NETGEAR) and hitting up NETGEAR on Twitter, it was clear that they don't consider this a high priority issue.  Even IF they do eventually fix this problem, it may take months to roll out a firmware update since your ISP has to push that to your cable modem.  Long story short, if you're going to do LACP don't even bother with the CM1200.  The fact that a NETGEAR employee hasn't even replied to this thread in a long time confirms to me that they just don't care.  I've had great luck with the Motorola MB8600 and I've heard good things about the SB8200 as well.  I'm getting around 1.2Gbps with my MB8600 and it's been up so far for almost 10 days.  See attached screenshot for evidence.

 

The thing that gets me is that I assume the CM1200 is just running Linux; the Linux bonding driver has supported 802.3ad link aggregation / LACP for a very long time and is very, very mature.  I'm not sure what NETGEAR has done with this firmware to introduce this issue.

 

- Kenny

Model: CM1200|Nighthawk Multi-Gig Speed Cable Modem
Message 38 of 151
CPLMayo
Guide

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation


@kvanals wrote:

I can confirm that your ISP is not the issue.  After several days of bouncing around with NETGEAR support (I kept getting bounced between the switch and cable modem teams since both devices are NETGEAR) and hitting up NETGEAR on Twitter, it was clear that they don't consider this a high priority issue.  Even IF they do eventually fix this problem, it may take months to roll out a firmware update since your ISP has to push that to your cable modem.  Long story short, if you're going to do LACP don't even bother with the CM1200.  The fact that a NETGEAR employee hasn't even replied to this thread in a long time confirms to me that they just don't care.  I've had great luck with the Motorola MB8600 and I've heard good things about the SB8200 as well.  I'm getting around 1.2Gbps with my MB8600 and it's been up so far for almost 10 days.  See attached screenshot for evidence.

 

The thing that gets me is that I assume the CM1200 is just running Linux; the Linux bonding driver has supported 802.3ad link aggregation / LACP for a very long time and is very, very mature.  I'm not sure what NETGEAR has done with this firmware to introduce this issue.

 

- Kenny


Good to know!

 

I'm not writing off the CM1200 just yet; if nothing else I would like to help figure out what the problem is.

 

It is probably running a paired down version of the linux kernel but just because the driver is mature in x86/x64 does not mean that it is simple plug and play with their hardware. I am sure there is a significant amount of optimization that is required to get it to run on their hardware. I would venture a guess that their CM is using an ARM processor or it is running a RTOS (Real-Time OS) that while it does use a lot of the code they can't just dump the entire linux kernel and drivers on the CM.

Message 39 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

Well, after two support tickets (since both my switch and modem are/were under warranty and had online and email support still) and getting bounced around, I lost faith in trying to solve the issue with NETGEAR.  And honestly, that's your only hope since you can't fetch the firmware binary blobs yourself and even if you could AND had the means to unpack them, there's no way to load the image onto the device yourself if you or someone else was able to figure out what the issue is.  As far as the OS on the device, it is certainly only conjecture at this point, but if it indeed is Linux I'm not sure what permutation of hardware configuration would lead them to develop their own LACP driver, even on a non-x86/x86_64 architecture like ARM/AARCH64/MIPS/etc.

 

To be honest, the only reason I'm still replying to this thread is that I'm salty that nobody at NETGEAR has even acknowledged that this is clearly an issue with an advertised feature and I'm hoping that keeping the thread alive will get it noticed. (Which would definitely help you if you're staying on the CM1200 bandwagon.)

 

- Kenny

Model: CM1200|Nighthawk Multi-Gig Speed Cable Modem
Message 40 of 151
CGASTON
Guide

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

You know this thread is pretty funny. I gave my suggestion and what to do to get the situation fixed and nobody has yet to reach out and try mine solution. I've done it my ship works I get one gig download speed constant no problems ask in no issues or resetting every 12 hours or every day. Quit thinkin it's your modem and router and try the simple fix I did simple enough

Message 41 of 151
kvanals
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

CGASTON:  The simple fix of calling Comcast and yelling at them for no reason?  You've yet to provide at least a screenshot showing the uptime of >48 hours, the firmware version is what everyone else is on, and showing that LACP is indeed active.  It's not even clear what Comcast actually did for you.  With multiple ISPs across multiple locations at play, I'm not sure why you're so quick to blame the ISP and not hold NETGEAR accountable.  Trust me, I would have liked to have actually fixed the issue myself rather than swap hardware, which resolved my issue finally.

 

- Kenny

Model: CM1200|Nighthawk Multi-Gig Speed Cable Modem
Message 42 of 151
CPLMayo
Guide

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation


@CGASTON wrote:
You know this thread is pretty funny. I gave my suggestion and what to do to get the situation fixed and nobody has yet to reach out and try mine solution. I've done it my ship works I get one gig download speed constant no problems ask in no issues or resetting every 12 hours or every day. Quit thinkin it's your modem and router and try the simple fix I did simple enough


Your solution was basically yell at the ISP until they fix it.

 

If I had something to point to and yell about I would do that but calling and saying "Bump my speed peasant" doesn't seem like the solution that you make it out to be.

 

  • Did they change the firmware?
  • Change the provisioning file?
  • What did they actually do?

Details are important and words have meaning. Both of which are what would allow others to follow what you laid out.

Message 43 of 151
cursixx
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation


@CGASTON wrote:
You know this thread is pretty funny. I gave my suggestion and what to do to get the situation fixed and nobody has yet to reach out and try mine solution. I've done it my ship works I get one gig download speed constant no problems ask in no issues or resetting every 12 hours or every day. Quit thinkin it's your modem and router and try the simple fix I did simple enough


Ignorance is bliss and I'm glad that is working for you but clearly you are not using LACP with the CM1200. I encourage you to do more research on the subject before contributing further with useless information.

Message 44 of 151
retate
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I have been experiencing similar problems to what others have reported in this thread so thought I would provide some additional perspective.
 
My configuration is a CM1200 (firmware 1.02.02) connected to a Netgear XS716T switch (10 gigabit switch).  The CM1200 connects to Xfinity/Comcast Gigabit cable service.  Cable connection data (SNR and Power) look very similar when performance is degraded to what they are when performance is good.
 
Before performance degradation is encountered I can repeatedly measure download performance between 1100 to 1200 megabits and upload performance between 39 to 42 megabits.  Performance is being measured via speedtest.xfinity.com using Chrome as the browser.  The CM1200 connects via LACP to the XS716T.  The traffic then passes through an OPNSense firewall with 10 Gigabit LAN and WAN interfaces connected to the XS716T switch.  The desktop running the test also connects to the XS716T switch with a 10 Gigabit interface.  If I run the performance test via either IE or Edge the results are less than with Chrome which I have not taken the time to understand.
 
When speeds degrade typical results are 22 Megabits down and 16 to 22 Megabits up.  The degradation occurs after a few hours.  A modem factory reset, modem power cycle, or disabling and enabling the LACP/lag interface on the switch restores performance.  I did notice that when I disabled and enabled the lag interface the first speed test I ran afterwards performance was only somewhat restored (139/19).  When I tested again after a couple of minutes speed was 1189/41.
 
I have not seen the degraded ping times to the CM1200 that are mentioned in this thread.  I believe I have seen the LACP problem not failing over to a single connection that is mentioned in this thread, but I have not investigated.
 
When I have checked the switch or the modem during degraded speeds link aggregation has appeared to be functioning normally.  I have just started looking at port statistics on the switch, and there may be a correlation with "Total Transmit Errors" / "Dropped Transmit Frames" on the switch ports the CM1200 is connected to occurring when speed is degraded.
Model: CM1200|Nighthawk Multi-Gig Speed Cable Modem
Message 45 of 151
cursixx
Star

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

So recently I talked to someone with an enginering sample of the CM1200 and they confirmed Netgear is aware of this problem and have been working on it. From my understaning this problem was known by Netgear before the public product release but here we are with a broken modem on retail shelves. Don't get me wrong the CM1200 is a nice modem but with the LAGG issue is a deal breaker. Hopfuly a GBASE-T or SPF+ modem is released by a manufacturer soon. SFP+ would be awesome HINT HINT NETGEAR

Message 46 of 151

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation


@cursixx wrote:

So recently I talked to someone with an enginering sample of the CM1200 and they confirmed Netgear is aware of this problem and have been working on it. From my understaning this problem was known by Netgear before the public product release but here we are with a broken modem on retail shelves. Don't get me wrong the CM1200 is a nice modem but with the LAGG issue is a deal breaker. Hopfuly a GBASE-T or SPF+ modem is released by a manufacturer soon. SFP+ would be awesome HINT HINT NETGEAR


Well since they are all probably based on the same chipset the cm1150v also has the same problem, when I enable LAG on mine the speed degrades in 12 to 24 hours so once they fix this for one modem it should fix it for all of the modems having this issue.

Message 47 of 151
CivHunter
Tutor

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I recently purchased a CM1150v and an ASUS rt-ax88u router. I enabled link aggregation on both the cable modems LAN ports (1&2) and the routers WAN ports (It uses WAN port and LAN 4 for WAN aggregation). Both the router and the cable modem showed that link aggregation was functional. But as a few people have mentioned here my internet speed went from 1000/35 to about 20/30 (down/up Mbps) after 12 hours or less. Also the log page in the router was getting flooded with Net_ratelimit errors because of rapid fire errors being generated. Rebooting the modem and router would re-establish normal internet speeds but the errors in the router log came right back. I've disabled aggregation in both devices for now and speed has not dropped off and error messages no longer appear. I realise my download speed (as rated by my 1Gb package) can be handled by a single network port, but sharing the load on two lines sounds and maybe I'd see higher downloads if it was working properly. Guess I'll keep an eye open on the modem firmware to see if netgear/xfinity ever flash a patch that fixes this issue.
Model: CM1150V|DOCSIS 3.1 Nighthawk® Multi-Gig Speed Cable Modem for XFINITY® Voice
Message 48 of 151
Dolvey
Tutor

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I just got off of the phone with support about my new CM1200 connected to a RAX80 router and was told that the additional ports on the CM1200 are useless and cannot be used if port aggregation is used. I am getting the same slowdown plus if you do not have multi-gig web service that the port aggregation will not work and that it is acting properly in choking off the internet speed because it is cheating to try to get a higher speed than for what you paid.

So Netgear WAN aggregation from router to modem is useless.

Model: CM1200|Nighthawk Multi-Gig Speed Cable Modem
Message 49 of 151
Retired_Member
Not applicable

Re: CM1200 slows down after 24-48 hours of uptime when using link aggregation

I really like this modem but with my ISP being Suddenlink I am not sure they will ever have an update for this modem.  I will go get the Moto 8600 as it will do the aggrigation without issues.

I wish that you could update the firmware as we buy these our self.

Message 50 of 151
Top Contributors
Discussion stats
Announcements

Orbi WiFi 7