Re: Internal Server Error when hostname used

linutic · ‎2023-08-30

I have configured my RBR750 as an access point with the following IP addresses set as static DHCP addresses in my OpenWRT router.

router: "mesh" 192.168.49.4

satellite: "mesh1" 192.168.49.5

satellite: "mesh2" 192.168.49.6

I can access all these devices with no difficulty from Chrome and Firefox from my Ubuntu 22.04 system, so long as I enter their IP addresses in the browser URL field. I can also access the two satellites as http://mesh1 and http://mesh2. However when I attempt to acess the router (remember it is configured as an AP) as http://mesh I get:

ERROR 500: Internal Server Error

This is a new problem. It used to work perfectly. I cannot remember configuring anything new in many months. Of course I have power cycled everything, no joy. My current firmware is V4.6.14.3.

When accessing the router with chrome and firefox, there are slightly different fields in the headers, depend on whether the hostname or IP address is entered in the URL. So it is difficult to find just one field different. So for this email, I used "wget" to access the router, and monitored it with Wireshark. I have attached the output of both queries.

Here is the TCP trace from:

% wget http://192.168.49.4

GET / HTTP/1.1
Host: 192.168.49.4
User-Agent: Wget/1.21.2
Accept: */*
Accept-Encoding: identity
Connection: Keep-Alive


HTTP/1.1 200 OK
Content-length: 2221
Content-type: text/html; charset="UTF-8"
x-frame-options: SAMEORIGIN
Cache-Control: no-store
Pragma: no-cache
Date: Wed, 30 Aug 2023 17:04:54 GMT
<html><head>
<META name="description" content="RBR750">
<META http-equiv="Content-Type" content="text/html; charset=utf-8">
<META http-equiv="Content-Style-Type" content="text/css">
<META http-equiv="Pragma" content="no-cache">
<META HTTP-equiv="Cache-Control" content="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="0">
<meta http-equiv="x-ua-compatible" content="IE=EDGE">
<!--<meta name="viewport" content="width=device-width, minimum-
scale=1.0, maximum-scale=1.0, user-scalable=no">-->
<title>NETGEAR Router Orbi Index</title>
<script language="javascript" type="text/javascript">

########### The rest deleted for brevity

Here is the TCP trace with:

% wget http://mesh

GET / HTTP/1.1
Host: mesh
User-Agent: Wget/1.21.2
Accept: */*
Accept-Encoding: identity
Connection: Keep-Alive


HTTP/1.1 500 Internal Server Error
Content-Type: text/html
Content-Length: 365
Date: Wed, 30 Aug 2023 17:03:49 GMT
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>500 Internal Server Error</title>
</head>
<body>
<h1>500 Internal Server Error</h1>
</body>
</html>

As you can see above, the two HTTP queries are identical, except that the sucessful query has the field:

Host: 192.168.49.4

While the failed query changes that field to:

Host: mesh

This looks like a newly introduced error in a recent firmware update. Can anyone think of another explanation?

CrimpOn · ‎2023-08-30

It appears that the DNS server in Open WRT is not resolving "mesh" to the correct IP address.

What happens if you do the command line command

ping mesh

linutic · ‎2023-08-30

Good thought, but the IP address in the query is correct. Here's what "wget" outputs when typed from the command line:

% wget mesh
--2023-08-30 12:09:08-- http://mesh/
Resolving mesh (mesh)... 192.168.49.4
Connecting to mesh (mesh)|192.168.49.4|:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-08-30 12:09:08 ERROR 500: Internal Server Error.

% ping mesh

PING mesh.lan (192.168.49.4) 56(84) bytes of data.
64 bytes from mesh.lan (192.168.49.4): icmp_seq=1 ttl=64 time=0.322 ms
64 bytes from mesh.lan (192.168.49.4): icmp_seq=2 ttl=64 time=0.353 ms

linutic · ‎2023-08-30

Here's some additional behavior. The satellites have no problem with wget queries.

% wget http://mesh1
--2023-08-30 14:02:14-- http://mesh1/
Resolving mesh1 (mesh1)... 192.168.49.5
Connecting to mesh1 (mesh1)|192.168.49.5|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 839 [text/html]
Saving to: ‘index.html.4’

% wget http://mesh2
--2023-08-30 14:02:17-- http://mesh2/
Resolving mesh2 (mesh2)... 192.168.49.6
Connecting to mesh2 (mesh2)|192.168.49.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 839 [text/html]
Saving to: ‘index.html.5’

CrimpOn · ‎2023-08-30

Thanks for providing the recording showing the DNS resolution.

Version V4.6.14.3 appears to have been out since 12/1/2022 (almost nine months). Unless this system was just recently updated, the firmware seems unlikely to be the cause of the change in behavior.

My Orbi requires authentication. (user, password) Do you have those parameters in a configuration file, rather than on the command line? It appears that more parameters are required to get the output in your post. All I get is this:

C:\Users\Dick>wget.exe 192.168.1.1 --user=admin --password=xxxxxxxxx
--2023-08-30 16:13:45--  http://192.168.1.1/
Connecting to 192.168.1.1:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="NETGEAR Orbi"
Connecting to 192.168.1.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 531 [text/html]
Saving to: 'index.html'

index.html                    100%[=================================================>]     531  --.-KB/s    in 0s

2023-08-30 16:13:46 (7.90 MB/s) - 'index.html' saved [531/531]

One of the 'missing features' in Orbi is the ability to resolve local DNS entries, so I am unable to duplicate the entire situation. (Also testing on an RBR50 in router mode) It is interesting to note that on my Windows computer, wget connects to the IPv6 address of the web site when Orbi resolves http://orbilogin.net :

C:\Users\Dick>wget.exe http://orbilogin.net --user=admin --password=xxxxxxxxxx
--2023-08-30 16:25:52--  http://orbilogin.net/
Resolving orbilogin.net (orbilogin.net)... 2603:8000:403:bd7c:a204:60ff:fe1c:239, 192.168.1.1
Connecting to orbilogin.net (orbilogin.net)|2603:8000:403:bd7c:a204:60ff:fe1c:239|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="NETGEAR Orbi"
Connecting to orbilogin.net (orbilogin.net)|2603:8000:403:bd7c:a204:60ff:fe1c:239|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 531 [text/html]
Saving to: 'index.html.1'

index.html.1                  100%[=================================================>]     531  --.-KB/s    in 0s

2023-08-30 16:25:52 (5.56 MB/s) - 'index.html.1' saved [531/531]

linutic · ‎2023-08-30

We get different results, but there are a couple things different here.

--- router mode vs AP mode.

--- Windows wget vs ubuntu wget.

I just looked at the index.html received by my ubuntu wget. It's mostly a pile of javascript that (at least sometimes) redirects the page from the implicit index.html to start.html. wget knows nothing about javascript, so fat dumb and happy, it just stores the javascript in index.html.

If I attempt to view the some_path/index.html file with chrome, I get a message saying my file could not be accessed. What?? However the URL line now shows that chrome has been redirected to somepath/start.html. That's the file it can't access. So apparently the javascript IS redirecting to start.html. Okay, so I tried manually following the redirect.

% wget 192.168.49.4/start.html
--2023-08-30 21:24:59-- http://192.168.49.4/start.html
Connecting to 192.168.49.4:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

Yup, now I get an authentication failure, just as you do. BUT that doesn't help understand the root problem because:

% wget mesh/start.html
--2023-08-30 21:29:41-- http://mesh/start.html
Resolving mesh (mesh)... 192.168.49.4
Connecting to mesh (mesh)|192.168.49.4|:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-08-30 21:29:41 ERROR 500: Internal Server Error.

So thanks for the great detective work, BUT, we are still on the wrong track.

linutic · ‎2023-08-30

You also mentioned that the firmware version is about 8 months old. Well it is sure possible I haven't tried accessing it by name for 8 months, or I did, and when it didn't work put in the IP Address and forgot about it. That day I might have been working on something else, and didn't want to do the context switch.

But I am absolutely sure it used to work.

I love my Orbi also.

CrimpOn · ‎2023-08-30

Sorry not to be more explicit. The firmware version on this router appears to be the version that was released on 12/1/2022 and there is no newer version. i.e. this router is "up to date." i.e. if the firmware has not been updated recently, then the firmware version is not likely to be the cause of the problem.

It is really frustrating that Error 500 provides so little information:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

i.e. the web server is unhappy, and cannot explain why.

linutic · ‎2023-08-31

Here is how to reproduce the problem in Windows without a router that supports DHCP IP assignment and DNS. You need:

A Windows computer.
A spare router (any type)

1) Configure your Orbi as an AP before removing it from service. You will immediately loose internet connectivity.

2) Remove your Orbi from service, and temporarily replace it with the spare router.

3) Plug in your Orbi (now configured as an AP) into the spare router.

4) Find the IP address of the Orbi on the spare router network. Most routers have a way of seeing the DHCP addresses the router assigned.

5) Enter the IP address of the Orbi AP and the name "win-mesh" in your windows file with the following name:

C:\Windows\System32\drivers\etc\hosts

If you don't know how to do this, see the article below:

https://www.howtogeek.com/27350/beginner-geek-how-to-edit-your-hosts-file

6) Optionally do the same with the Orbi's satellites. You can get their IP addresses from the Orbi AP. Call them "win-mesh1" and "win-mesh2".

7) Now try accessing the Orbi AP (and optionally its satellites) with Edge, Chrome, Firefox, wget etc at the following URL(s).

http://win-mesh

http://win-mesh1

http://win-mesh2

😎 Log the results.

Note: I get the same results with Windows as I do on Linux. There are many versions of wget that run on Windows. Results vary depending on the wget program selected. With the Power Shell version I get:

PS C:\Users\gene> wget http://win-mesh

wget : 500 Internal Server Error
At line:1 char:1
+ wget http://win-mesh
+ ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
eption
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

CrimpOn · ‎2023-08-31

With the RBR750 in Access Point (AP) mode, the situation remains the same:

Opening a web browser to the IP address allows normal login.
In the test setup, the RBR750 in AP mode has IP 192.16.1.84 on my LAN.
http://192.168.1.84 opens normally.
An entry has been made in C:\Windows\System32/\drivers\etc\hosts:
192.168.1.84 Test750
This correctly resolves Test750 to the IP address 192.168.1.84. The command
ping Test750
for example, results in the RBR750 responding to ping requests.
Opening a web browser to http://Test750 results in Error 500.
I wondered if the port makes a difference (WAN vs. LAN) and it does not.

(In router mode, the RBR750 should not accept a login through the WAN port - although I believe there was a discussion about Netgear allowing connection through the WAN port by mistake in one of the beta versions. In AP mode, I expect the 750 to accept web connections from anywhere.)

Connection to http://192.168.1.84 works normally on both WAN and LAN ports in AP mode, and connection to http://Test750 gives Error 500 on both WAN and LAN port.

I do not have a 750 satellite, and thus am not able to replicate the situation with regard to satellites.

Using Wireshark to capture the HTTP get statement, the only difference I see is the host: entry, i.e.

Host: test750\r\n fails, while

Host: 192.168.1.84\r\n works.

CrimpOn · ‎2023-08-31

This Hosts file thing is fun!

Dug out my spare RBR50 and connected it to the LAN. Created Hosts entry for it (192.168.1.80 Test50). When in AP mode, the RBR50 web server responds correctly to both:

http://192.168.1.80 and to

http://Test50

This it appears that this phenomenon exists on the RBR750, but not on the old RBR50. (Have no other Orbi routers to try.)

From the Wireshark capture, it appears that the only difference between an http connection by IP and and by host name is that the host name is included in the http packet as an entry:

Host: xxxxx\r\n

One of the goofy features included in Netgear routers is watching for those special URLs (routerlogin.com, routerlogin.net, orbilogin.com, orbilogin.net, etc.) and substituting the Orbi router IP for them.

I wonder if Netgear decided to improve the AX product by having the router check the "Host:" parameter in the http get statement to see if it matches one of those special URLs???

Would it be convenient to temporarily add another entry in the OpenWRT hosts file, perhaps

192.168.49.4 orbilogin.net

(it will take me a bit to change equipment around to test this myself.)

linutic · ‎2023-08-31

Exceptional effort. Wow. Seems you have reproduced the problem just as I described it.

Okay, following your clever suggestion, with the OpenWRT router firmware, I found I can assign any IP address to any legal internet name, and the DNS server in the router will honor it.

So I tried it. Well I'll be damned.

The Orbi, in AP mode appears to accept any nodename that is a prefix of either:

orbilogin.net

orbilogin.com

For example it accepts:

o

orbi

orbilogi

orbinlogin

orbilogin.n

orbilogin.c

But gets a 500 error with:

orbit

orbilogin.nett

orbilogin.comm

orbilogin.org

So thanks to your creative idea, we have characterized the behavior of the Orbi firmware. Question is: "Is this a bug or a feature"

Whilst we ponder this question, I am going to rename my Orbi AP and its satellites:

orbi, orbi1, orbi2

CrimpOn · ‎2023-08-31

Thanks for conducting the experiment. My guess is that Netgear engineers attempted to resolve an issue that affected Orbi routers when in Access Point (AP) mode. To wit:

When in router mode, users could reach the web interface with a browser using any of the special URLs (routerlogin.com, routerlogin.net, orbilogin.com, orbilogin.net, etc.), but
When in AP mode, the router did not perform DNS analysis and thus would forward connection attempts to those URLs "up the line" to the primary router, which would not be able to resolve them into IPs (because they are not really URLs).

I seem to recall posts on the forum from users unhappy that they had to enter the router IP address rather than the cool URL when in AP mode. Netgear may have fixed that problem by having the AX series router examine URLs even when in AP mode.

What is not clear (to me) is whether the AX series:

Examines http URLs (the "Host:" parameter) on every http connection attempt that passes through the router in AP mode, or
Examines http URLs ("Host:" parameter) only when the http connection is directed at the IP address of the router in AP mode.

I guess another experiment would be:

Have no hosts entry for the Orbi router.
Open a web browser to http://orbilogin.net from a device that is on the Orbi LAN (not on the WAN side of the Orbi)
See if the router in AP mode pops up the web interface.

I'm still voting for "bug".

CrimpOn · ‎2023-09-01

Just to put lipstick on this pig.....

Reinstalled the RBR750 in AP mode.
Opened a web browser on a laptop wired to the RBR750.
The Windows host file does not contain any entry for orbilogin.net
Connected to http://orbilogin.net

It opened the web interface on the RBR750 - not the primary Orbi router.

This leads to the conclusion that Netgear traps every http connection that passes through the AX router when in AP mode to see if the "Host:" might be the Orbi itself.

Thus users who are accustomed to their Orbi being "orbilogin.net" (or equivalent) can reach the device even when it is in AP mode.

linutic · ‎2023-09-01

Here's a simplified explanation of what happens with a home router (including an Orbi router) when you enter a URL into your browser.

The browser does a DNS request to the router. The request effectively says: "What is the IP address associated with this name?"
The router gets the request, and checks if the name corresponds to a node on the local network, or is otherwise configured as a special case. If either is true, it sends the corresponding IP address back to the requesting node.
Oftherwise, the name is not local or special. The routers forwards the request to the internet DNS servers and waits for a response. If the response is successful, the received IP is sent to the browser. If there is an error, or the request times out, an error response is sent to the browser.

With an Orbi home router, running the factory firmware, the name "orbilogin.net" is recogtnized as a special name, and a special case. When received, the router sends back its own IP address on the local network. By default, this is 192.168.1.1, but you can change that.

When the browser gets the IP address, it sends an HTTP request to whatever IP address is received. The HTTP request includes the originally requested name. The name is used if there are multiple virtual web servers with the same IP address. This is commonly done where a single computer handles multiple different customers. For example "landscape-spcialities.com" might be handled by the same computer, at the same IP address as "bills-appliances.com".

In the case where the IP address is the address of the router on the local network, it goes to a web server in the router. The multiple customer case does not apply. Everything received by the Orbi goes to the same UI, or should, no matter whether the name is "orbirouter.com" or "hows.your.sister" or just an IP address.

You can make an argument that an orbi router might check for "routerlogin.com" to detect some sort of error. It's dubious why you would want that. For example my internet router has been called "gw" (gateway) for 20+ years, although many different routers from different manufactuers have served that function. Checking for "routerlogin.com" is just an inconvenient pain-in-the-butt, as far as I am concerned. My chosen name "gw" would always be rejected.

Now checking for "orbirouter.com" on an AP is even worse. Suppose I use an orbi AP and a seconds Netgear router to get to the internet. I actually did that for awhile. The second Netgear router was positioned where the internet came into the house, which was in a far corner in the basement. A lousy place for a WiFi server. So I ran an ethernet cable from there to my Orbi AP in the center of the house. The perfect place place for a WiFi server.

What possible good could it have done to have both of these devices named "orbirouter.com". Why would you want that? How could that ever work?

So this convention in the firmware is not sensible. It's irritating in the router, and downright stupid in the AP. It it is deliberate, it indicates that Netgear doesn't understand home network configuration. If it is a bug, well then all software has bugs, so I can understand that.

This is the kind of thing that comes out of Marketing requirments. Some idiot in the Marketing department (having never configured a network) thinks it is a cool idea, and insists on it in a bug fix meeting. The software mainteance engineer gets tired of arguing with Marketing, and puts it in. He forgets to turn it off in AP mode, and that results in this discussion. Maybe the Marketing guy is smarter this time, and tells the software engineeto take it out everwhere. Like it was in the first place. Like it should be. Or maybe the Marketing guy is just a stupid **bleep**.

This sort of thing allowed Scott Adams to make his fortune with Dilbert.

linutic · ‎2023-09-01

Using "routerlogin.com" to get to the router is a bad idea in the first place. It is too easy to misspell. Common misspellings include "ruterlogin.com", "routerlogon.com" and so forth.

Suppose a hacker wants to break into your system. He registers a few common misspellings of "routerlogin.com". On each he runs a web page that looks *exactly* like the Netgear login page.

You aren't paying attention one day, and you misspell routerlogin.com. You go to the hacker web page. You see exactly what you expect to see, so you type in your username and password. The web server records them, together with your IP address. Then it redirects you to the correct spelling of "routerlogin.com", which sends you to your router web page. You figure you misspelled the password, so you enter the information again. This time it works. You don't suspect a thing.

However now the hacker has everything he needs to break into your router. From there he can run barefoot through your home network.

I never used "routerlogin.com". I was afraid I would misspell it when I wasn't paying attention. I used the IP address, or my own simple name for the router.

This is a variation on an old trick that dates back at least to 1970s time sharing systems. It worked them. It still works.

CrimpOn · ‎2023-09-01

I fear that the AX series Orbi does not follow the typical procedure when in AP mode. (God knows what it does in router mode, and she's not telling us!)

In my test:

The RBR750 is AP mode
A laptop is connected (Ethernet) to the RBR750.
The laptop web browser attempted to connect to http://orbilogin.net
The laptop hosts file did not have an entry for http://orbilogin.net
If a DNS request had been sent through the RBR750 to the primary Orbi router (RBR50), it would have responded with the primary router IP address (192.168.1.1)
This would cause the web browser to attempt to connect to the primary Orbi router web site.
However, the web connection was made to the RBR750 web server (in AP mode).

Thus, it appears that in AP mode, the AX router series (not the original Orbi products) intercepts and redirects web connections to http://orbilogin.net

p.s. I am SO DISAPPOINTED that Scott Adams lost his cool and Dilbert is no longer available in my daily paper.

kotfu · ‎2023-09-26

Joining this thread after having been directed here from https://community.netgear.com/t5/Orbi-WIFI-6-AX-AND-Wi-Fi-6E-AXE/New-RBR850-RBS850-Firmware-Version-... by @CrimpOn .

I also have this same issue, my original description is posted at https://community.netgear.com/t5/Orbi-WIFI-6-AX-AND-Wi-Fi-6E-AXE/New-RBR850-RBS850-Firmware-Version-...

Today I did some packet captures to try and better understand this issue. For context, here's the key parts of my setup:

I have a RBR850 and 3 RBS850s. I run the RBR850 in AP mode and it has a single ethernet connection to my network switch. There are no other ethernet cables connected to the RBR850.
My RBR850 had an IP address of 192.168.13.11, and a mac address that ends in c7:02
My internal network is 192.168.13.0/24.
I have a pair of firewall/router machines that run OpenBSD (which is not relevant for this discussion)
I run my own DNS servers (i.e. not running on the RBR850): they have an authoritative zone for internal hosts, and resolve all other requests via the root name servers.
My internal DNS server resolves ap1.kotfu.net to 192.168.13.11.
I know this works because I can ping ap1.kotfu.net and I get responses from 192.168.13.11
For this experiment, I accessed the web admin from a computer with an IP address of 192.168.13.25
For this experiment, 192.168.13.25 was connected via ethernet only, so there were no relevant packets in this capture transmitted via WiFi, they were all over hardline.

Here's the capture of me typing "http://ap1.kotfu.net" into my browser address bar and hitting return:

Here's the capture of me typing "http://192.168.13.11" into my browser address bar and hitting return:

Now a couple of observations and a theory.

1. Note that there are no DNS requests made by 192.168.13.11 in any of these packet captures. That's not surprising to me. It means that the RBR either doesn't care about the hostname for the IP address, or it already has it cached. In Access Point mode, the only reason for the RBR to want/need the hostname for an IP address would be to log the hostname in a log file. In the logs available to us, entries include the IP address, not the hostname (i.e. the log entry when you log in to the web console).

2. The only material difference initial packets in the two different captures, is the "Host" header in the HTTP request. When the "Host" header contains an IP address, the web admin pages are served to the client. When the "Host" header contains my hostname "ap1.kotfu.net", the HTTP error 500 is generated.

Here's my theory. I don't think it's a DNS issue. My RBR850 can resolve hostnames on my local network, because I have it using an internal time server, and that works fine. Also, the "Host" header is not a DNS thing, it's a HTTP protocol thing. When the RBR850 is in Router mode, two things must happen in order for "orbilogin.net" and "routerlogin.net" to work properly: 1) DNS queries for those domains must resolve to the IP address of the RBR850, and 2) the web server running on that IP address must properly respond based on the contents of the "Host" header. In this case, the RBR850 should serve the same web content regardless of what's in the "Host" header. In my setup, where the RBR850 is in AP mode and I run my own DNS servers, I can already assure that DNS queries for "ap1.kotfu.net" resolve to "192.168.13.11", the IP address of my RBR850. The packet captures above confirm this is true.

That means there must be a problem with the second condition. My guess is that when you switch to AP mode, the RBR850 puts a different web server configuration file in place (or applies some edits in place to the config file already there). The web server configuration in AP mode seems to have an error in how it handles the "default server". This configuration error causes the web server to return HTTP code 500.

EDIT: further experimentation reinforces this theory. By tinkering with the hosts file on the machine running your web browser, you can make several entries which point to the IP address of the RBR850. I added:

192.168.13.11	someserver.com
192.168.13.11	routerlogin.net
192.168.13.11 	orbilogin.net

When I use the hostnames that we know are configured in the web server on the RBR850, ie "routerlogin.net" and "orbilogin.net", the web page loads fine. When I use "someserver.com", I get the HTTP 500 error.