Re: RBR850 frequently issuing DNS REFUSED responses

tantrum · ‎2020-06-05

Had the RBK852 now for just over a month, and have an issue I'll open a support ticket for. Wondered though if others have been experiencing anything similar.

Basically and since day 1 after putting this new router in to replace the old one, every machine on the network has complained periodically about being unable to resolve / connect to a host.

The problem has been tracked down to the DNS proxy software in the router that just seems too easily overloaded and returning REFUSED response flags for DNS requests, even to hosts that recently resolved fine, and who resolve fine on the next attempt.

A number of things have been ruled out. It is not, for example, because:

1. The ISP's DNS servers are flaky. Whether I use those or Google's, the result is the same. Also configuring the machines to use those DNS servers directly and bypassing the proxy has no issues.

2. The router is not that busy at the times; there may be (say) a dozen or so DNS requests issued in short period of a second or two sometimes, such as when opening an ad-and-cdn-heavy browser page, but still the volume and packets are relatively small all things considered and occurs even at quiet times of the night when there's probably less than 10Mb/s being pulled in either direction through the router.

3. Not a machine issue; for as long as the DNS results are in the cache (before TTL expires), there are no problems, and there are no problems with DNS resolution when using the servers directly by IP instead of DHCP/proxy.

4. Wireshark confirms absolutely that the problem is a REFUSED ("by policy, etc.") situation to resolve the DNS, not because it can't be done or there are extenuating issues such as network/backbone outages. E.g. a wireshark filter of "dns.flags eq 0x8185" is enough to see that every problem occurring corresponds with this exact single response to the DNS request.

5. Packet analysis of the Wireshark data shows the DNS requests made and the responses received are all correctly formed and the network is not suffering from any issues relating to TCP retransmissions, dropped packets, etc.

6. In that past month the machines have undergone full shutdown and restarts, I've fully reconfigured the network properties, and the router itself has been subject to at least 3 and maybe more firmware updates and full reboots, all seemingly having zero effect.

I haven't ruled out that this might be some odd incompatibility issue between the router and the ISP/modem, but then I can't other than to show that without using the DNS proxy in the router and everything else being the same, the problem doesn't happen even one time.

I suspect, although I'm loathe to do it, that I'll be asked to do a full system reset of the router. I also suspect it'll do zero to address the issue despite having heard on the forums that this has seemingly resolved some other issues before. So as it is very inconveniencing to have to reset it I would rather not given the evidence doesn't suggest there's a good cause for it.

Also note that this router doesn't have Circle and I have never used/activated (and never will use/activate) Armor. I've seen that some non-DNS issues with connecting to sites can occur as a result of these systems blocking access for example, but that doesn't apply to me.

Has anybody else been getting these kinds of issues with the DNS proxy? Any solutions that worked for you besides manually configuring all your devices to use another DNS server than the proxy (and/or setting up a replacement DHCP service to do so more easily)?

I'd rather not introduce additional links into the chain, but as best I can tell we don't really have ways of touching the DNS proxy configuration (e.g. perhaps increase its concurrent request capacity or timeout levels) in order to see if they would help improve things or not.

P.s. This issue and the previous ports one that got fixed in a recent FW release are really the only big issues I have had with the system along with a couple of other minor gripes that were resolvable/bearable; I'm not unhappy with it overall and it has had quite a few good points going for it, but the DNS issue as it stands is not something that can be lived with. I am a heavy user and would estimate that I easily get over 50 occurrences of this issue every day that the DNS proxy is being used. That drives me nuts, especially when my family get on my case about it too!

FURRYe38 · ‎2021-07-29

Good Luck. This issue is resolved by the beta thats available.

View solution in original post

tantrum · ‎2020-06-05

Also not having the issue with (or any issue with) the devices losing connectivity from the network or such, as I'd seen in a couple of other threads. Everything stays happily connected and other concurrent requests continue streaming uninterrupted, just individual and intermittent host lookups that fail.

kildare · ‎2020-06-06

If I were you I'd do a settings backup and then a full reset.

After that I'd restore the settings and see if DNS lookup problems come back.

AFAIK no others have posted this problem before.

tantrum · ‎2020-06-06

Routers with Circle have reported the same issues, but this router doesn't have that yet. The same thing Circle is activating may be getting leverated here though too.

Although I haven't yet seen users report it with this router, so perhaps it will have to come to that.

The telnet debug option isn't available on this router either currently, or else I may be able to have a little more control (e.g. restart dnsmasq etc.).

Mikey94025 · ‎2020-06-06

@tantrum wrote:
Has anybody else been getting these kinds of issues with the DNS proxy? Any solutions that worked for you besides manually configuring all your devices to use another DNS server than the proxy (and/or setting up a replacement DHCP service to do so more easily)?

I do think I've seen this. As I browse websites and click links I've intermittently seen my Windows laptop browser briefly display "cannot connect to site XXX", then redisplay and succeed. The frequency is pretty low, maybe once or twice a day.

I've seen this for the months I've owned my Orbi AX system and it didn't occur with my older Orbi AC system. I'm guessing it's this same DNS proxy issue you're describing because it would explain what I'm seeing. Like you, I've tried switching to Google DNS servers but that did not stop this intermittent lookup. I am not using Armour or Circle and my Orbi router settings are very basic/default.

Thank you for the deep dive and investigating with Netgear.

FURRYe38 · ‎2020-06-06

I also have not see this aswell. Would point to either a configuration issue or symtopm upstream. A factory reset should be performed on the RBR if one hasn't been done since last FW update. Revert back to v11.2 as well to see if the problem follows..

Possible issue would be at your ISP services, ISP DNS or ISP Modem. Seen others with your particular modem having problems with Orbi systems historically. Though its just a modem only.

tantrum · ‎2020-06-06

Yep @Mikey94025 that's one of the key ways you'll observe it, sometimes the brower window will even self-refresh and get over it, and for a while I had been putting up with that.

Unfortunately lots of other apps such as dev tools, auto-updaters, etc., are less forgiving and I often have to restart them or spam retry several times.

Occassionally impacts things like disney+ on appletv as well, if left long enough for the dns cache entries to expire. Restarting the app will solve it for that instance.

I'm hoping to get it resolved permanently though.

You'll also get it more frequently if wanting to test it, if you ipconfig /flushdns (or equivalent on non-windows OS) as it will now not be able to rely on the cached entries and it will attempt another lookup with a potential to fail.

DNS entries can be cached for 2, 5, or 10 minutes, and in some cases even hours or days (for very static configs), so depending on the individual sites it can also vary - the flushdns evens that playing field though.

FURRYe38 · ‎2020-06-07

Have NG help you capture the logs from/debug.htm

Retired_Member · ‎2020-06-08

I see the same things here. I have Armor activated but the problem has been there before activation.

Logs from dnsmasq on the rbr850 when I try to access http://bugs.debian.org (when not in DNS cache)

--

dnsmasq: query[A] bugs.debian.org from 192.168.1.32
dnsmasq: forwarded bugs.debian.org to 92.220.228.70
dnsmasq: forwarded bugs.debian.org to 109.247.114.4
dnsmasq: query[A] bugs.debian.org from 192.168.1.32
dnsmasq: query[A] nav.smartscreen.microsoft.com from 192.168.1.32
dnsmasq: forwarded nav.smartscreen.microsoft.com to 109.247.114.4
dnsmasq: forwarded nav.smartscreen.microsoft.com to 92.220.228.70
dnsmasq: reply nav.smartscreen.microsoft.com is <CNAME>
dnsmasq: reply wd-prod-ss.trafficmanager.net is <CNAME>
dnsmasq: reply wd-prod-ss-eu-north-1-fe.northeurope.cloudapp.azure.com is 23.102.47.40
dnsmasq: query[A] google.com from 192.168.1.32
dnsmasq: forwarded google.com to 109.247.114.4
dnsmasq: forwarded google.com to 92.220.228.70
dnsmasq: reply google.com is 216.58.207.238
dnsmasq: query[A] bugs.debian.org from 192.168.1.32
dnsmasq: forwarded bugs.debian.org to 92.220.228.70
dnsmasq: forwarded bugs.debian.org to 109.247.114.4
dnsmasq: reply bugs.debian.org is 140.211.166.212
dnsmasq: reply bugs.debian.org is 140.211.166.201
dnsmasq: reply bugs.debian.org is 209.87.16.39

--

As you can see it need to forward the request 3 times before we get a dns replay, and this makes the browser show "page not found"

Mikey94025 · ‎2020-06-08

To narrow and workaround this DNS issue, can I change my Windows connection's TCP/IP properties for DNS and choose directly Google DNS (8.8.8.8 and 8.8.4.4)? By doing that and not using the default DNS server address (i.e., my Orbi router's IP address) then should we expect my intermittent browser DNS episodes to stop? I'm trying it now to see if I notice a difference today.

tantrum · ‎2020-06-08

Yes that's right.

Setting them on the router wouldn't help workaround it (but does help confirm it's not an ISP DNS service issue but a dnsmasq one).

Setting them directly on your host (e.g. windows) machine IP properties instead of relying on DHCP picking up the router's address for DNS should instantly** prevent further occurences.

** At least in something modern, i.e. Windows 10, which doesn't require you to restart from DNS IP changes.

And to exacerbate / test harder on purpose, issue frequent "ipconfig /flushdns" calls inbetween attempts to resolve a hostname, either with the router/DHCP address for DNS, or whether the directly entered google ones into your connection's IP properties, and you should see the frequency of the issue increase with the router/DHCP address, and still no problems with the directly entered google ones to the machine.

tantrum · ‎2020-06-08

@Retired_Member

May be a silly question, but how are you seeing the dnsmasq logs on the router please?

Is it from settings done on the debug.htm page, or some log settings on another one like advanced > logs?

The recent logging issue in the firmware means I see very little entries in the advanced > logs screen now.

FURRYe38 · ‎2020-06-08

What Firmware version is currently loaded?
What is the Mfr and model# of the Internet Service Providers modem/ONT the NG router is connected too?

What browser are you using?

Can you try a factor reset on the RBR and setup from scratch and this time, do not enable Armor...

@Retired_Member wrote:
I see the same things here. I have Armor activated but the problem has been there before activation.

Logs from dnsmasq on the rbr850 when I try to access http://bugs.debian.org (when not in DNS cache)
--
dnsmasq: query[A] bugs.debian.org from 192.168.1.32
dnsmasq: forwarded bugs.debian.org to 92.220.228.70
dnsmasq: forwarded bugs.debian.org to 109.247.114.4
dnsmasq: query[A] bugs.debian.org from 192.168.1.32
dnsmasq: query[A] nav.smartscreen.microsoft.com from 192.168.1.32
dnsmasq: forwarded nav.smartscreen.microsoft.com to 109.247.114.4
dnsmasq: forwarded nav.smartscreen.microsoft.com to 92.220.228.70
dnsmasq: reply nav.smartscreen.microsoft.com is <CNAME>
dnsmasq: reply wd-prod-ss.trafficmanager.net is <CNAME>
dnsmasq: reply wd-prod-ss-eu-north-1-fe.northeurope.cloudapp.azure.com is 23.102.47.40
dnsmasq: query[A] google.com from 192.168.1.32
dnsmasq: forwarded google.com to 109.247.114.4
dnsmasq: forwarded google.com to 92.220.228.70
dnsmasq: reply google.com is 216.58.207.238
dnsmasq: query[A] bugs.debian.org from 192.168.1.32
dnsmasq: forwarded bugs.debian.org to 92.220.228.70
dnsmasq: forwarded bugs.debian.org to 109.247.114.4
dnsmasq: reply bugs.debian.org is 140.211.166.212
dnsmasq: reply bugs.debian.org is 140.211.166.201
dnsmasq: reply bugs.debian.org is 209.87.16.39
--

As you can see it need to forward the request 3 times before we get a dns replay, and this makes the browser show "page not found"

Retired_Member · ‎2020-06-10

@tantrum wrote:
@Retired_Member

May be a silly question, but how are you seeing the dnsmasq logs on the router please?

Is it from settings done on the debug.htm page, or some log settings on another one like advanced > logs?

The recent logging issue in the firmware means I see very little entries in the advanced > logs screen now.

Use telnet to get access to the router, then look for a process called dnsmasq

You can kill this running process and start again with logging ex.

dnsmasq -d -k -q -h -n -c 0 -N -i br* -r /tmp/resolv.conf -u root

tantrum · ‎2020-06-10

@Retired_Member Are you running a beta / early firmware for the router?

The telnet option doesn't display in the debug form for me, and even if I try and force it, either by using the browser dev tools and forcing the css to show the form block, or manually issuing the form post with the correct parameters using a curl request, the option doesn't stick on the debug page and the router doesn't respond when I try to telnet (port 23) to it.

Retired_Member · ‎2020-06-10

@tantrum wrote:
@Retired_Member Are you running a beta / early firmware for the router?

The telnet option doesn't display in the debug form for me, and even if I try and force it, either by using the browser dev tools and forcing the css to show the form block, or manually issuing the form post with the correct parameters using a curl request, the option doesn't stick on the debug page and the router doesn't respond when I try to telnet (port 23) to it.

I am on current firmware V3.2.15.25_1.3.15 and telnet has been enabled by default on my router and sats.

Maybe it has something to do with my account being enrolled for betatesting? Maybe some other betatesters can answere this?

Reg

Arild

Retired_Member · ‎2020-06-10

@Retired_Member wrote:
@tantrum wrote:
@Retired_Member Are you running a beta / early firmware for the router?

The telnet option doesn't display in the debug form for me, and even if I try and force it, either by using the browser dev tools and forcing the css to show the form block, or manually issuing the form post with the correct parameters using a curl request, the option doesn't stick on the debug page and the router doesn't respond when I try to telnet (port 23) to it.
I am on current firmware V3.2.15.25_1.3.15 and telnet has been enabled by default on my router and sats.
Maybe it has something to do with my account being enrolled for betatesting? Maybe some other betatesters can answere this?

Reg
Arild

Telnet stoped working after I did a factory reset. But got it up again using this utlity

https://github.com/insanid/NetgearTelnetEnable

FURRYe38 · ‎2020-06-10

I believe telnet was removed in some recent FW updates. Even for beta testers.

Glad the work around helps.

arlomike · ‎2020-07-29

I too have experienced DNS timeouts...not frequent but enough to notice its a resolver issue. The routers recursive server is buggy. Wonder if I can telnet in and install powerdns recursive instead. It sucks that the we can't set what DNS servers DHCP hands out. I'm not about to manually set DNS servers on all my devices.

FURRYe38 · ‎2020-07-30

What version of FW do you have loaded?

I'm using v32 and have no seen any DNS issues thus far. Using Auto DNS detect on the RBR which sees my ISP DNS.

You might try setting manual DNS addresses to something else like 1.1.1.1 and 9.9.9.9 on the RBR to see if this maybe a ISP DNS issue. Not th first time ISP DNS has causes problems.

@arlomike wrote:
I too have experienced DNS timeouts...not frequent but enough to notice its a resolver issue. The routers recursive server is buggy. Wonder if I can telnet in and install powerdns recursive instead. It sucks that the we can't set what DNS servers DHCP hands out. I'm not about to manually set DNS servers on all my devices.

arlomike · ‎2020-07-30

I'm on .25 version. My RBR is set to use quad 1 and quad 8 already. This is something that has not happened or noticed before i switched to Orbi. The orbi is definitely suspect.

gb777 · ‎2020-07-31

The telnet debug option isn't available on this router either currently, or else I may be able to have a little more control (e.g. restart dnsmasq etc.).

tantrum · ‎2020-07-31

Yes, @Retired_Member pointed that out some weeks ago. I've used it now to get telnet access, thanks.

It hasn't yet helped me to fix dnsmasq's (configuration?) shortcomings though. Since the past few weeks the issues haven't been recurring so much on the apple tv's (and I don't think I hard coded their dns entries), and I did hard code the entries on my computers so don't notice it there.

I'll have to get back to retest it soon; I've been hellaciously busy since November, and it's not the only issue this mesh router has that needs to be sorted out.

FURRYe38 · ‎2020-07-31

I would contact NG support and ask them for help and information about this. Possible some thing in FW that may be faulty with particular ISP services. Which I may not be seeing with my ISP.

@arlomike wrote:
I'm on .25 version. My RBR is set to use quad 1 and quad 8 already. This is something that has not happened or noticed before i switched to Orbi. The orbi is definitely suspect.

amenic · ‎2020-09-18

Maybe I should buy another router (and return it) just to get the amazing 90 day support for a product that isn't working properly and then I can contact NG for support to actually (and properly) resolve this?