Orbi doesn't fall over to 2nd or 3rd DNS

[Southpaw32]

I have a couple of Raspberry Pi's running PiHole for my DNS. The setup works great, but the other day the first PiHole machine got unplugged, and my devices weren't able to reach the network, despite having a second and third DNS.

Why didn't the Orbi rollover to the other DNS?

[Southpaw32 · Accepted solution]

So I think I figured things out.

One of my RPi is setup as my DHCP server, and when I looked the DNSMasq .conf file it was only passing on the IP address of the DNS servers for that RPi, not the addresses for the RPi PiHole severs.

I edited the .conf file, and replaced 8.8.8.8, etc with the local pihole IPs, and for the first time ever I have all three of my RPis showing blocked traffic!

[CrimpOn]

---

@Southpaw32 wrote:

I have a couple of Raspberry Pi's running PiHole for my DNS. The setup works great, but the other day the first PiHole machine got unplugged, and my devices weren't able to reach the network, despite having a second and third DNS.

Why didn't the Orbi rollover to the other DNS?

---

I was going to suggest filing a Bug Report (https://bugcrowd.com/netgear ), but that seems to be aimed at vulnerabilities rather than products simply not working correctly.  Certainly posting in the Idea Exchange would be a positive step (although I have serious doubts that anyone at Netgear ever reads these posts). https://community.netgear.com/t5/Idea-Exchange-For-Home/idb-p/idea-exchange-for-home 

I have a dim memory from the (very old) DOS Days that IBM PC's would take multiple DNS entries, but after they got a response from one, they would never try any of the others again.  The only recourse was the Big Red Switch (turn it off).  In Windows 10, it appears that queries are sent to every DNS server and the first response is acted on.  (Seems highly wasteful to me, except that most LAN devices have only one DNS server entry <the router>.)

With multiple Pi-holes, it is obvious from the Console that only one of them is being queried? (not "round robin")

[Southpaw32]

Yes, I can see live stats on the query numbers, and for sure it doesn't move on to the next DNS. 

I've had this happen on a few occasions now. (I was testing some overclocking for awhile)

[CrimpOn]

Makes a person wonder if the Orbi DNS code came from (a) the OpenWRT that Orbi is based on, or (b) was ported from the Nighthawk line.  Would be a hoot to see if the Nighthawk line has the same 'disappointment'.  Will take me the weekend to set up a second Pihole and dig out a Nighthawk to compare with Orbi.

[Southpaw32]

Anything I can do to help test?

[CrimpOn]

Have two Pi-holes running now.  Am hoping that "pihole disable 10m" means to turn off everything, rather than "quit blocking and resolve everything."  Validating your results on Orbi will be quick.  Have to dig in my "Big Box of Stuff" to find my Nighthawk R7000 and Archer (might as well test it, too.)

[Southpaw32]

My understanding is the “disable for X mins” is simply to turn off the content blocking.

[CrimpOn]

Well, this is not working out as I anticipated.  I have two Raspberry Pi's running Pi-hole.

My "test" Orbi is connected to my regular Orbi and configured as a router.

It is set to use the two Pi's as DNS servers. (192.168.1.27 and 192.168.1.30).

I have a Windows PC connected to the test Orbi (ethernet). When I open a web browser, what seems to happen is that all DNS requests are sent by the Orbi to both Pi-holes. What appears to be happening is that the Orbi behaves as Windows 10 seems to behave. If DNS requests cannot be satisfied from the Pi-hole cache, they are sent to all DNS resolvers.

I will perform another test tomorrow where DNS server #1 is taken off-line to see if DNS requests still go to DNS server #2.

Not sure what to make of all this.  So far, it is not confirming your experience.

More tomorrow.....

[CrimpOn]

There is another way to verify Orbi DNS behavior: capture WAN traffic.

If Orbi has two defined  DNS servers (perhaps Google and Open DNS), a WAN traffic capture should show whether Orbi queries one or both of them.  Will try that tomorrow as well.

[CrimpOn]

Another puzzle.  Used the debug feature to capture WAN/LAN traffic while I opened a series of web pages that had not been opened in a while (avoid Windows and Orbi cache).  Test Orbi set to use 1.1.1.1 and two Pi-holes.  The WAN packet capture shows the Orbi sending queries to all three up-stream DNS servers at the same time, with all three responding.

i.e.

DNS 1.1.1.1

Internet

Production Orbi Router (LAN 192.168.1.1)

Pi-hole1 (192.168.1.27)

Pi-hole2 (192.168.1.30)

Test Orbi Server (192.168.1.81) (LAN side 10.0.0.1, configured to use 3 DNS servers)

Test Windows PC (10.0.0.2) Set to use DHCP provided DNS server, which is 10.0.0.1

The goal was to reproduce and document the Orbi DNS failure. These results are not encouraging.  There seem to be two additional avenues to explore, but I am not certain how to go about it:

1. I had thought that "losing" one of the DNS servers would cause it to be "marked" somehow and forgotten (no longer used). I shut off one of the Pi's, waited a few minutes, and then turned it back on.  New debug log shows the Orbi continuing to query all three servers.
2. Perhaps the Orbi treats DNS servers attached to the LAN side differently than on the WAN side.  Exploring this is much more complicated as it involves several production Orbi restarts, which the family will not enjoy.

This exercise has made me realize that using Pi-hole to filter DNS queries works only if every DNS server the Orbi uses is a Pi-hole.  Servers are not primary, failover1, failover2.  They are all equal and all used every time.

[Southpaw32]

Here is a look at my Pi setup. I have three now. (it's kind of an addiction.)

They are listed in order of how they appear in the DNS in my Orbi.

I recently added the RPi4, and made it the main DNS, demoting RPi3 to second.

Only my NAS, Orbi and the RPi4 are hitting the RPi3.

Nothing is using the ZeroPi.

I can't imagine that if all three were being hit every time, that the other two wouldn't show some queries or blocks.

[CrimpOn]

The "in-line" image will not appear until a forum moderator approves it.  (Using the "Browse" button in the lower left make images available instantly.)

Since I was able to capture WAN packets of the Orbi doing simultaneous DNS queries on all DNS servers, the only remaining possibility is that DNS servers on the LAN side of the Orbi may be treated differently.  After stringing ethernet cables around the room to move two Pi-holes from the 192.168 Orbi to the 10.0 Test Orbi, I realized that these Pi's have static IP's for eth0.  Have to stop and research now (a) did I set these Pi's up with static IP's or did Pi-hole? and (b) how do I move the damn things from one IP subnet to another (and back). 

Or, I can wait until people are asleep and fiddle with the family Orbi

• reset the DNS servers to local Pi-hole (I have been using Pi-hole only for "my" devices; not for everybody)
• set up packet capture
• run some tests
• save the debug file
• put everything back the way it was

[CrimpOn]

This is awkward.  Further experiments confirm  that Orbi sends DNS requests to every DNS server, every time.

Reconfigured my Orbi (everyone left for an hour).

Orbi had three DNS Servers:

Pi3 - 192.168.1.27

Pi4 - 192.168.1.30

CloudFlare 1.1.1.1

Pi3 had two DNS Servers:

CloudFlare

Google

Pi4 had two DNS Servers:

OpenDNS

Level3

PC gets DNS from DHCP (Orbi router at 192.168.1.1).

Started Packet Capture;

Pinged 15-20 sites: harvard.edu, ford.com, dmv.ca.gov, etc. etc. etc.

Saved the debug log.

Opened LAN.pcap with Wireshark.

Shows PC asking Orbi for DNS. Orbi asking Pi3 and Pi4. P3 and P4 responding. Orbi responding to PC.

Opened WAN.pcap with Wireshark.

Shows Orbi asking Cloudflare. Shows queries to Google, Cloudflare, OpenDNS, Level3.  On WAN.pcap, all packets come from the Orbi public IP, but those queries must be coming from Pi3 and Pi4.  Shows responses coming back from the DNS servers.

Logged into the management consols of Pi3 and Pi4. See the same queries. (I will go through the Query Log line-by-line).

Sorry for the long post. My "bottom line" is that my Orbi queries all three DNS servers, just as Pi-hole queries all DNS servers.

[Southpaw32 · Accepted solution]

So I think I figured things out.

One of my RPi is setup as my DHCP server, and when I looked the DNSMasq .conf file it was only passing on the IP address of the DNS servers for that RPi, not the addresses for the RPi PiHole severs.

I edited the .conf file, and replaced 8.8.8.8, etc with the local pihole IPs, and for the first time ever I have all three of my RPis showing blocked traffic!

[CrimpOn]

Thanks for the information.  Glad it's resolved happily.

[Southpaw32]

Some more info from the folks at the PiHole discord channel:

From user Bucking_Horn:

"To have your Pi-hole on your RPi 4 distribute several DNS servers to your DHCP clients, you could create a custom configuration for dnsmasq (Pi-hole's embedded DNS/DHCP server), e.g.:

sudo nano /etc/dnsmasq.d/42-multi-dhcp-dns.conf

and add the following line:

dhcp-option=option:dns-server,0.0.0.0,ip.of.pihole.2,ip.of.pihole.3

0.0.0.0 will make the Pi-hole on your RPi 4 distribute its own address, so you'd have to replace only the remaining two items with your other Pi-holes' correct IPs.

Verify your configuration is still valid:

pihole-FTL dnsmasq-test

If it doesn't come back OK, check the file content for typos.

Then apply the settings to your Pi-hole by running:

pihole restartdns

Note that my first answer still applies:
Your clients may pick any of the three Pi-holes, but likely they'd tend to prefer the first on their list."

This has my setup running great!

I know it's a little specific, but if anyone else is running PiHole, I hope this helps!

[CrimpOn]

---

@Southpaw32 wrote:

Note that my first answer still applies:
Your clients may pick any of the three Pi-holes, but likely they'd tend to prefer the first on their list."

---

My experiments show that Windows 10 and Orbi will query every DNS server at the same time.