GS110EMX - HTTP management web interface connections rejected
I just found a workaround for a fairly obnoxious problem with my GS110EMX switches, running firmware 188.8.131.52. Hopefully this is helpful to someone, and I really hope NETGEAR can fix this issue in future firmware revisions. I should note that this problem did not affect any of my other NETGEAR switches (models GS116Ev2 and MS510TX). It only affected my GS110EMX switches.
I was frustrated to discover that I was unable to connect from a Linux host to the HTTP web interface on the switches. Windows clients could connect to port 80 on the switches without any issue.
It took me a while to figure out what was wrong.
nmap showed the port was open on a SYN scan:
# nmap -sS -p 80 ulrack05 ulrack06 Starting Nmap 7.70 ( https://nmap.org ) at 2018-11-04 17:58 PST Nmap scan report for ulrack05 (192.168.32.14) Host is up (0.0020s latency). PORT STATE SERVICE 80/tcp open http MAC Address: CC:400:xx:xx:xx (Unknown) Nmap scan report for ulrack06 (192.168.32.15) Host is up (0.0017s latency). PORT STATE SERVICE 80/tcp open http MAC Address: CC:400:xx:xx:xy (Unknown) Nmap done: 2 IP addresses (2 hosts up) scanned in 0.17 seconds
But not on a TCP connect scan:
# nmap -sT -p 80 ulrack05 ulrack06 Starting Nmap 7.70 ( https://nmap.org ) at 2018-11-04 17:58 PST Nmap scan report for ulrack05 (192.168.32.14) Host is up (0.0019s latency). PORT STATE SERVICE 80/tcp closed http MAC Address: CC:400:xx:xx:xx (Unknown) Nmap scan report for ulrack06 (192.168.32.15) Host is up (0.0018s latency). PORT STATE SERVICE 80/tcp closed http MAC Address: CC:400:xx:xx:xy (Unknown) Nmap done: 2 IP addresses (2 hosts up) scanned in 0.11 seconds
I looked at a tcpdump capture from my Windows and Linux hosts in Wireshark to determine what was happening.
On the Windows client, I saw the typical TCP connection flow: SYN, SYN-ACK, ACK.
On Linux, I saw SYN, RST-ACK. So the switch was rejecting the TCP connection immediately for some reason.
I noticed the TCP SYN packet from my Linux host differed from the SYN packet from my Windows host -- there were two TCP header flags set that were unset on the Windows host: ECN (Explicit Congestion Notification) and CWR (Congestion Window Reduced). Apparently having these flags set is enough to upset the TCP stack on these switches.
On all my Linux hosts, I have the sysctl net.ipv4.tcp_ecn set to 1 (on). If I set it to either 0 (off) or 2 (enabled if incoming connection requests it), then I can connect to the web interface with no issues.
I'm not sure what these switches are running in the firmware -- I would guess some kind of embedded Linux kernel. If so, chances are future firmware revisions just need to set net.ipv4.tcp_ecn=2 and they'd be able to accept TCP with SYN+ECN+CWR.
Re: GS110EMX - HTTP management web interface connections rejected
Also, it appears that the net.ipv4.tcp_ecn_fallback=1 flag doesn't kick in on the client side because the switch explicitly *rejected* the SYN packet with an RST-ACK. If the switch had *dropped* the packet instead, then the client would fall back to a non-ECN SYN and things would be fine.
It's interesting that this *exact* issue (handling of unrecognized ECN bit) is mentioned on page 5 of RFC 3360, which is titled "Inappropriate TCP Resets Considered Harmful".
I implemented an iptables-based workaround from the host which will be connecting to the management web interfaces, so that I could leave net.ipv4.tcp_ecn=1:
-A OUTPUT -d 192.168.32.14/31 -p tcp -m ecn --ecn-tcp-cwr -j DROP
This makes the kernel drop outbound connections to these two switches if the ECN flag is set. This also allows the net.ipv4.tcp_ecn_fallback=1 sysctl to behave as intended.
Alternatively, if I strip out the ECN bits from the TCP header, the connection works as intended:
-t mangle -A OUTPUT -d 192.168.32.14/31 -p tcp -j ECN --ecn-tcp-remove