Hey everyone, I know that I'm not the only one with problems on the Nighthawk WiFi 6 devices, but thought I'd post my most recent headache. My hardware consists of a Motorola MB8600 DOCS 3.1 modem w/ 1Gbps service from my ISP, the modem links a RAX120 router via a single 3' CAT6 (i know, overkill), and I have a single 100' CAT5E connecting the RAX120 to a RAX80 in AP mode on the other side of my property. I have a dynamic IP address provided to my modem from my ISP, the modem provides a single address to the RAX120 WAN connection via DHCP, DHCP is configured on the RAX120 to provide a separate NAT for all clients, and the RAX80 has a static IP address set within that NAT subnet. When the problem began today the RAX120 was running FW 1.0.1.114 and the RAX80 is running V1.0.1.70_1.0.30. Nothing in my environment has changed for a month and this issue arose out of nowhere.
Anyway, this morning I noticed that none of my TV streaming devices were able to connect to their streaming services. The streaming device in one side of the house is connected via WiFi (802.11ac) and the device on the other side is connected directly to the RAX80. I found that I was unable to ping any web sites (www.google.com) from my laptop when connected via 5Ghz or 2.4GHz on the router or AP, but I was able to ping the IP addresses for these sites (172.217.0.36). I could ping DNS server IP addresses for my ISP and other public DNS server addresses, yet I couldn't query DNS running on the RAX120 to get a non-authoritative response. However, I did get an authoritative response when I ran an nslookup and specified an external DNS server. After logging into the RAX120 I could see that it was pulling the DNS server IP addresses from my modem as expected, which were some of the same that I was able to ping without issue. I checked the logs and nothing abnormal popped up. Just the standard logins of MAC addresses and some BS DoS from known services. So after kicking the RAX120, everything began functioning again and I was able to query the router get DNS answers.
Issue solved with a simple kick of the router right? Wrong… 20 minutes later the issue reoccurred. This time I took it a step further and sequentially powered down and up the hardware. Modem, router, and AP. Again, the issue was resolved for a short time, but then randomly came back. I dug into the RAX80 logs to see if it was causing the pain and saw the following.
[Time synchronized with NTP server] Sunday, Mar 08,2020 19:50:31
[Internet connected] IP address: X.X.X.X, Sunday, Mar 08,2020 19:50:20
[Initialized, firmware version: V1.0.1.70_1.0.30] Sunday, Mar 08,2020 19:50:20
[Time synchronized with NTP server] Sunday, Mar 08,2020 19:47:26
[Internet connected] IP address: X.X.X.X, Sunday, Mar 08,2020 19:45:57
[Initialized, firmware version: V1.0.1.70_1.0.30] Sunday, Mar 08,2020 19:45:57
[Time synchronized with NTP server] Sunday, Mar 08,2020 18:54:28
[Internet connected] IP address: X.X.X.X, Sunday, Mar 08,2020 18:54:08
[Initialized, firmware version: V1.0.1.70_1.0.30] Sunday, Mar 08,2020 18:54:08
[Time synchronized with NTP server] Sunday, Mar 08,2020 14:27:44
[Internet connected] IP address: X.X.X.X, Sunday, Mar 08,2020 14:27:04
[Initialized, firmware version: V1.0.1.70_1.0.30] Sunday, Mar 08,2020 14:27:04
The FW reinitializing without me intervening with a reboot is a bad sign, so I isolated it out for a factory reset and rebuild. As an aside, the RAX80 has been problematic in the past with it basically corrupting it’s ARP table and not forwarding L2 traffic, so I thought it definitely could be horked once again. Nevertheless, after a rebuild and a few hours later the problem resurfaced.
The next troubleshooting step I took was to upgrade the RAX120 FW to the latest security patch 1.0.1.118, which still didn’t resolve the issue permanently. Now I’ve downgraded the firmware back to 1.0.1.108, and stepped back to .114 and .118 hoping it won’t resurface. It’s been about an hour, so my fingers are crossed. I don’t want to go down the path of a factory reset of the RAX120, cuz it’s a major PITA to redo all my settings with ACLs and port forwarding. I could just restore from the backed-up firmware after a reset though, which is the next step I’ll take if this crops up again.
Thoughts? Let me know if you guys have seen oddities like this and if Netgear is doing anything to actually solve customer problems. I liked the RAX120 cuz I have a system connected via the 5GbE port and NAS via LACP and the performance is great. But this is only one of many problems that I’ve had with these devices and I’m on the verge of giving Netgear the finger to go with a 10GbE switch and some Ubiquiti APs… and thrown psfense back in while I’m at it.
The router stayed up all night for about 8 hours and then the same symptoms occured once I logged into my laptop and started checking emails for about 5 minutes. I attempted to login to the RAX120 and it would not accept my username and password... I attemtped a number of times and I was not fat fingering, because I was pasting the password from my PW database. I logged into the RAX80 just fine without any issue. Nonetheless, I just reset my router back to factory and the issue still occured after it came online with a default config. WTH?!?!? Is my modem hosed and causing me the problem? The logs in the modem have nothing telling and only a couple status messages. Possibly it's a coincidence and I wasn't being patient. Eitherway, I reapplied my RAX120 config from a backup and the connections are up again.
Anyone out there have any ideas? Are these Nighthawks trash like the community has been evangalizing over the past year? Or am I just a fool? Even though I've managed numerous other network devices from Cisco, Brocade, Linksys, TP-Link, Apple, and even Extreme Networks without ever seeing an issue like this...
Welcome to the community! Thank you for taking the time to write a detailed report of your experience. We are currently running a beta firmware for the RAX120 that may help resolve your issues. Please let me know if you'd like access to the beta firmware.
Just an update. After re-applying my configuration this morning the network has been stable since. However, I did disable all device MAC addresses, except for my laptop, via the security ACLs as well. About an hour back I enabled 3x IP cameras, a security NVR, and my alarm system gateway. So far so good. Possibly something on my network is being a bad neighbor after an automagic FW update the night before. I'll find out sooner or later.
Could I get access to the beta firmware? I just loaded .118 and it caused my router to completely die on me. It randomly reset its settings to the default SSID, etc. and took nearly an hour of resetting over and over again before I could finally get it cleared out and re-setup with the new.
As an aside, two things that are missing that are super frustrating for such an expensive router:
* more DDNS provider support (or, alteratively, just custom settings for DDNS - it's not that hard); and
* more VPN provider support (or, alternatively custom settings - again, not that hard to use standard protocols to connect to VPN).
Any thoughts? I bought this thing expecting the firmware to mature, but it doesn't really seem to be getting better. If anything, less stable. I took the update today hoping it would make it improved, and ended up losing more than an hour of my life.
The issue is back! The modem ran well all day yesterday, so at about 10pm I decided to allow a few more devices. I allowed an Amazon cloud cam, a thermostat, 5 light bulbs, and 4 smoke detectors. Everything ran until about 1am and the issue resurfuced. Here's the interesting part. If I connect my phone via LTE, I am able to communicate with my alarm system that is physically connected via a CAT5e cable and to the light bulbs that are on the 2.4GHz guest network. However, the thermostat and smoke detectors that are on the non-guest WiFi network are not online. I connected my laptop once again to the non-guest WiFi and I can ping external IP addresses, but I cant resolve hostnames/URLs. The same occurs on my phone or other devices connected to that network WiFi network. I connected my laptop to the guest 2.4GHz network, but I still couldn't resolve anything to the routers DNS. The next time it happens I'll have to connect my laptop via a physical cable to see if I can get DNS resolution, but I'm not optimistic since my streaming TV device didn't work. So here's a few questions.
Why are the light bulbs accessible from outside, inbound to the network when they are on the internal 2.4GHz guest network?
Why is the security system accessible from outside, inbound to the network when it's physically plugged with a CAT5e cable network?
This is getting very tiresome though... Everything functioned wthout an issue for a few months and no changes were made. Then on Sunday morning, it all went to hell in a handbasket.
@Christian_R What is Netgear doing for customers? After 90 days you are stuck with no support and a paperwieght cuz of poor FW code.
Rax120_rax80, did you have an opportunity to install the RAX120 beta firmware (see message #3 above) – and, did it improve the situation you are experiencing?
Per the site, the only version in the Security Hotfixes (Beta Firmware) is 1.0.1.118 which is also in the Firmware and Software Downloads section as well. I upgraded to 1.0.1.118 after having the issue occur on 1.0.1.114 and I have been testing with .118 since. Unless there is another beta FW I'm not aware of, I'm running the latest.
Here's another update in the saga. The issue occured again about 9am this morning after I reset the RAX120 a little before 8am. The only things allowed network access were the devices running all day yesterday with no issue (WiFI IP security cams, physically connected security NVR and alarm system gateway, laptop via WiFi) and the WiFI bulbs that were functional this morning. However, I did enable the Amazon Cloud cam to swap it to the guest network for testing when the RAX120 went bonkers. It could be a coincidence, but it could be a problematic device. It's currently blocked via MAC on the router, but I'll test re-enabling that one device later if I have some stability for the entire day.
Anyway, I tested/verified the following when the RAX120 was in it's zombie state.
* Connected laptop directly to the RAX120 via CAT5e, DNS lookups succeeded when choosing external DNS servers (75.75.75.75, 8.8.8.8, etc), pings to IP addresses internally within the NAT and externally on the web succeeded, DNS lookups when choosing the RAX120 as a DNS server failed, pings to external hostsname on the web failed (as expected since DNS on the RAX120 is failing)
* ARP table on my laptop took an extrememly long time to populate, but it did list all the device MACs allowed access on the RAX120.
* Netstat from the laptop wasn't showing any socket connections, which was expected.
Rax120_rax80, note that the “beta firmware” referenced in message #3 is a non-public release in development, distinct from the “Firmware Version 1.0.1.118 - Hot Fix” on the Netgear website. Please communicate with Christian_R (forum moderator) through a private message to request access.
I am having the EXACT same issue with my new RAX120.
I went through all of the same troubleshooting as you, but below are my findings in a nutshell.
The issue is with the RAX120 providing name resolution....period. I thought it could be bad actors on the network as well, but have rulled that out as you did by only allowing 1 device at a time. (I generally have 50+ devices of all sorts like yourself).
I have Spectrum cable hooked up to a netgear nighthawk cable modem CM1200 (working fine with a ASUS RT-AC3200 running asus-merlin for 8+ months).
My RAX120 receives my dynamic IP from spectrum, but I always specify my DNS after running DNSBench. When my "internet" went down on my laptop, I noticed that web pages wouldn't load, my exchange server connection failed, etc, but services which use IP addresses were still connected (PuTTY, pidgen, etc). So I pinged my old DNS server 4.2.2.2...no issues. Pinged google.com...unable to resolve.
I logged into the RAX120, and changed my DNS servers (I was using a new set based on my DNSBench run so wasn't confident in them). Applied them and was off and running again....for 20 minutes.
Same thing happened...changed the DNS to new ones....applied....20 min later, no name resolution.
As you noted, if you manually add the SAME DNS servers to your IPv4 adapter in your laptop/pc/etc, everything works fine.
I did this for a while and tried other things to get the RAX120 to work with no avail, and then called support (I have never done this in my life (I'm 41) given my line of work, but figured WTH). Needless to say we tried nothing I hadn't already tried, but he did have me change my DNS servers to 1.1.1.1 and 1.0.0.1.
I generally do like these servers for the non-logging aspect, but since they aren't the fastest I don't use them. Well low and behold, I went 16 hours straight with no issues.
I went back into the RAX120 and changed my DNS from 1.1.1.1 and 1.0.0.1 to some of my other ones, and AGAIN within 20 min, I was "down". Changed them back to 1.1.1.1 and 1.0.0.1, and here I am typing this 3 hours after the fact.
I don't know why these seem to be working and others do not. I wish I could just have the RAX120 pass the DNS servers to my devices who request it and not do the resolution itself.
Its hard to come from managed devices and devices that run Merlin to something as "consumer grade" as this. It may be going back to Amazon if Netgear can't fix this asap...
Let me know what else you find, and I'll do the same.
Bahahahaha....I should have known I'd jink myself. After I posed my reply, I noticed an email from netgear and when I clicked on it, none of the images loaded. Sure enough...no name resolution again.
Here is one funny thing though. I logged into the device, went to Internet under the basic tab, and click the bottom Test button. It went to the "Updating" page (not sure why as it should be testing...), but when it finished and popped up the product registration box (I assume this is the test). It's working again. No change to the DNS needed.
So all you have to do is log into the admin interface every 20 min, and "Test" the internet setup....mark this one as resolved...you're welcome!
Good info Hanz. Maybe we can right an API call to test every 10 minutes. 🙂 I actually did figure out how to make a call to reboot either the RAX120 or RAX80 (I dont remember which one) once a day when I had some crashing issues last year, but that's a pretty crappy solution when they should have a reboot schedule built into the OS. *Hint, Hint @netgear give the users more command and control*
My network has been up for almost 5 hours now, but I know it's only a matter of time. I've been given access to the .115 beta so I'll give that a after work tonight and report back.
Ha...I may have to. I work from home and this could end up being a major problem for me.
Ah, scheduled reboot...yet another thing missing from this console. The list of things is growing that I'm accustomed to having. Jeez...this thing may be going back. The Advanced section isn't very advanced. I sent a PM to the person in this thread to try and get the beta as well. Hopefully they respond soon...and hopefully it works.
I can deal with software short-comings as long as the device is reliable.
I moved from an Apple Airport Extreme Base station and a TP link AP to two Nighthawks for better performance. What I have gained in performance I've lost in stability. My old switches would run for a year with no reboot and never miss a beat...The last time I saw a switch with stability problems this bad was a Brocade/Extreme VDX 6740. But at least those had an excuse, cuz the switch would eat the CF card due to a bug that literally wrote it to death. >_<
Netgear needs to do something for their customer base. $400 per device with only 90 days of support, but code that makes support a necessity, is unnacceptable. Nighthawk is the flag ship prosumer device, but they are plauged with problems. A never ending upgrade path to unstable beta firmware isn't a good solution either. Better software engineering w/ Agile and real QA seems to be the lacking here as well. I wish I was in my return period still, cuz I'd send these back to Amazon in a heartbeat.
I've also replaced the entire RAX120 with Amazon (thinking it was a defective unit). No luck.
I thought I had a solution by turning AX off, but it appears the problem still persists even after turning that off (turned it off last night, whole router lost WAN connection again today upon new WiFi 6 device connecting to WAP).
(Specific Setting: Enable AX - This WiFi mode will enable AX features such as OFDMA)
However I have pinpointed the trigger to WiFi 6 capable devices (Macbook Pro, latest iPhone, etc) when they connect to the Wireless Router, it'll drop every connection (even ethernet) via WAN. The only solution is a router reboot which will allow my Wifi 6 capable devices to connect too (within a brief period). However any new connections after ~5-10 min or so will result in the complete data disconnect again.
I'm literally to the point of thinking of using one RAX120 for Wireless and another for a Router. This is crazy.
This seems like a different issue than Hanz and I are facing. We are seeing the RAX120 randomly fail to resolve DNS requests from connected hosts, which bascally breaks connectivity cuz everything relys on DNS unless hard coded with IP addresses.
This seems like a different issue than Hanz and I are facing. We are seeing the RAX120 randomly fail to resolve DNS requests from connected hosts, which bascally breaks connectivity cuz everything relys on DNS unless hard coded with IP addresses.
Thank you. I noticed the mention of using CloudFlare (1.1.1.1) and Google (8.8.8.8) DNS servers.
I haven't actually tried using anything OTHER than these servers (I do not use what Comcast provides).
I will test further to see if I can send an ICMP ping directly to an IP when I have the next connection drop.
Of course I got the standard apply hotfix .118 code above which i did already to no avail. This morning I did a hardware device reset. My problems also seemed to start on Evening of March 8th. Wonder if it has anything to do with Daylight savings time and TTL with DNS or something wierd.
So far network stable for almost 12 hours now with 16 devices and 2 kids level work load after factory reset. Support is asking me to place device in AP only mode now which seems just like trial and error method.. Is there no logs or debug output we can get captured and analyzed? It doesn't matter if I use different DNS servers, I've tried 3 now, and rest of my network that hangs right off my provider router works just fine, so its definately DNS proxy issue on the RAX120.
I've had the .115 beta firmware in for 3.5 hrs now and so far so ok. I'll let you guys know if/when this craps out.
I may just be sending this back to amazon. I don't have any wifi6 devices yet and was really just trying to get some more distance (and future proof). I understand the 1watt limit on these devices, but I was thinking this antenna aray would set it apart.
My ASUS RT-AC3200 has been solid, and espicially useful with the Merlin 3rd party firmware. I don't think I'm getting any better distance, but it's all down to measuring bars since this device doesn't tell me the dBm of each wifi client like my ASUS did. I may trade this for a TP-Link product...
This DNS issue doesn't seem to be isolated to the RAX120, but is a common problem over the years with other netgear devices.
Thanks everyone for all the input and experiences. I'd like to think that we will get a solution with more people posting and submitting cases, but I'm kinda pessimistic in regard to Netgear support.
Here’s a little update on my issue. I had no problems as of 6pm tonight (9 hours after the last reboot), but my wife complained that her system wasn’t functioning. She had all of the same issues as my laptop during the other failures, but my laptop was still functioning normally. I was scratching my head until I realized that I was VPN’d into work. Once I disconnected the VPN session, my machine experienced the RAX120 DNS failure problem. My company’s VPN profile forces my machine to use their DNS servers when I’m VPN’d in to their network. Hence, I did not notice any failures until my wife mentioned something.
This is definitely a DNS server issue on the RAX120. It’s like the DNS server service dies and fails to restart. Also, I saw another poster had the issue occur the morning of March 8th as well and suggested the daylight savings change. @netgear is there a bug with the daylight savings time change? Does NTP not function as it should and somehow hoses DNS? A time change should and NTP should not be related to internal DNS whatsoever, but who the heck knows at this point.
The last thing I can try at this point is update to .115. I’ll have that done in about 15 minutes. Crossing my fingers with lacking optimism…
24hrs stable now after Factory Reset on V1.0.1.118. At least 3 diagnosed DNS outages from my RAX120 connected devices within 2 day period on v114 and v118 code . I suspect would have been more but I left it off at one point for a while and used my Spectrum WiFi. I have no WIFI-6 devices. After factory reset outside of changing password and SSID config, I changed the timezone back to ESTDT ~12hours ago. Will be getting used heavily next 12 hours so will update if anythign pops up.
As a point of (hopefully helpful) information for this thread, I have been using the CleanBrowsing DNS service (185.228.168.10 and 185.228.169.11) on an RAX120 without experiencing any “random disconnects” to-date.
It installed ok, and I poked around the interface not changing anything else just making sure all settings were ok/same.
I logged back in last night to check attached devices....no issues.
Logged in this morning to do the same...no issues.
Logged in again a bit ago 10:30am and got authentication errors....weird. I then had to use the default password and create a new one (this time creating a strict password with one cap, one lower, one numeric and one symbol). I'm not sure why this just now occurred, but also after the password save....MY DNS STOPPED RESOLVING AGAIN!
I changed my DNS servers to ones I want to use, and am rebooting. If the DNS fails to resolve again on .115 as of now (11:23am CST), this thing is going back to amazon.
