MR2100 - Need to Manually set MTU (due to PMTUD failure)

[tom-]

I've been trying out the 3 big LTE networks in the UK for comparison (EE, Three, Vodafone), I'm sticking with Vodafone which has the best reception in my area, and this is where the MTU problem starts.

The MR2100 appears to do some kind of PMTUD for the LTE connection (Path MTU Discovery) and then relays this for new wifi connections to ensure connected devices fragment packets at the appropriate size for the network, which is great, and seemingly necessary as each LTE network appear to have fairly arbitrary and low MTUs.

It correctly determines the MTU for EE and Three, e.g it uses 1380 bytes for Three which is correct, however It uses 1500 bytes for Vodafone which is incorrect. Now I'm pretty sure I know why this is - because the initial connection on Vodafone _is_ in fact 1500, the problem is with Vodafone the effective MTU _always_ lowers over time, eventually settling on 1464 bytes - I can only speculate as to why, perhaps cell aggregation causes it to settle on a different path - Anyway, the MR2100 does not detect this, it only appears to attempt PMTUD upon the initial connection, and so of course all the connected devices will continue attempting to send 1500 for large enough packet... and you can probably guess what happens, things get flakey, website loading gets a bit annoying, HTTP requests that had a particularly big header will just vanish... uploading becomes almost unworkable.

It's possible to manually work around this by first using "PMTU black hole discovery", i.e bisect the MTU size by manually pinging different sizes starting with `ping -M do -s 1472 1.1.1.1` (-28 byte header), this is how I found the correct Vodafone MTU. Now I can set this for every - single - device... which works, but is incredibly annoying.

So my question is: How can I manually set the MTU on the MR2100 to centralise this, this used to be an ordinary setting for the modem on old routers, but there doesn't appear to be any setting for this in the MR2100 user interface (192.168.1.1). I'm happy to fish around in the telnet console but don't know where to start or if it's even possible.

[tom-]

So i'm guessing this would require an AT command if it's possible, and I've now spent longer than I care to say trying to find an AT command reference for the "Qualcomm SDX24 LTE Modem" in this thing, these baseband people seem to keep this info under lock and key... Any hints for the AT command setting IP MTU?

[schumaku]

---

@tom- wrote:

Anyway, the MR2100 does not detect this, it only appears to attempt PMTUD upon the initial connection, and so of course all the connected devices will continue attempting to send 1500 for large enough packet... and you can probably guess what happens, things get flakey, website loading gets a bit annoying, HTTP requests that had a particularly big header will just vanish... uploading becomes almost unworkable.

---

This is clearly a probem with the (always ongoing) PMTUD: The last system, resp the IP stack on the Web server or the proxy/load balancer in the front of the Web server must realize that the packet is incomplete, ...  Also http headers can be fragmented.

[tom-]

> This is clearly a probem with the (always ongoing) PMTUD: The last system, resp the IP stack on the Web server or the proxy/load balancer in the front of the Web server must realize that the packet is incomplete, ... Also http headers can be fragmented.

No, this dissapears at the first few hops irrespective of the address, i.e inside of Vodafone's network (I've verified this with multiple addresses including one of my own servers to double check, where I can see the MTU for the public interface). As I said it's pretty easy to bisect the size with `ping` (using any well behaved server that will bounce a sizable ICMP), and then also see where the packet dissapears with MTU +1 byte using something like MTR.

I'm using "PMTUD" here losely and perhaps incorrectly because I don't know what actual mechanism the modem uses to determine the appropriate IP MTU, it's more likely it's determined based on something explicitly negotiated in the LTE layer... but these modems aren't exactly very open and I'm not an LTE expert so I'm saying "PMTUD" to mean "auto IP MTU based on whatever the LTE modem says"... from what i've read on other better documented modems there tends to be an AT packet size command to either set an explicit IP MTU or "auto" mode which I presume is determined by the LTE modem.

[schumaku]

Nothing magically disappears. If your server does operate on the industry standard MTU, the IP stack must recognize that an incomplete packet arrived, and it has to inform the sender by returning an ICMP Fragmentation Needed (Type 3, Code 4). The same must happen by any L3-aware device in the data path. If these are any kind of L2 bridges with some reduced frame size, probably because your data is flowing over different data paths wtjh potentially different MTU, the next device in the data path has to do it, worst case it's the server at the end.

To answer your original question why there is no max MTU config on the MR2100 and the likes? Simply not required these days*.Or have you spotted an ability to configure the max MTU on your mobile phone, and then probably for the WiFi and the WWAN interface?

*The problem existed e.g. on connections like xDSL on two wire, where instead of transparent IP tunneling like PPPoE or PPTP, IP-stacks where substandard, crazy ideas of blocking any ICMP were common.

There are still some IP stacks in the field (Netgear introduced a problem as part of some security fixes on some Plus switch models recently) which have issues on connections with MTU below of the standard MTU, e.g. over VPNs - thus not fully standards compliant.

Last but not least, the mentioned PMTUD is not a "one-off" process, it's an ongoing thing for each session. 

[tom-]

> Nothing magically disappears. If your server does operate on the industry standard MTU, the IP stack must recognize that an incomplete packet arrived, and it has to inform the sender by returning an ICMP Fragmentation Needed (Type 3, Code 4). The same must happen by any L3-aware device in the data path. If these are any kind of L2 bridges with some reduced frame size, probably because your data is flowing over different data paths wtjh potentially different MTU, the next device in the data path has to do it, worst case it's the server at the end.

Unfortunately not everything on the internet will bother returning an ICMP fragmentation needed packet, which is why "PMTU black hole discovery" is a thing, which is effectively what I'm doing manually here but for the LTE network. To be clear, this is _not_ for a particular server or website. For _all websites_ fragmented HTTP requests will dissapear _inside of Vodafone's network_. As I said, I have verified this by pinging multiple servers with different MTUs to determine the max (including for example 1.1.1.1 which definately accepts 1500 which you can verify yourself: `ping -M do -s 1472 1.1.1.1`), then ran a traceroot (MTR) (again to multiple servers) consistently revealing a black hole on the first few hops (i.e in Vodafone's network)... until I lower the MTU to 1464.

> To answer your original question why there is no max MTU config on the MR2100 and the likes? Simply not required these days*.

I understand that's the intention, that it be automatic, and that is necessary considering the vastly different MTU requirements of different LTE networks... and it's true it mostly works in my experience with _other_ LTE networks (which is apparent only through the negotiated MTU of the connected wifi and ethernet devices). But it's not working in this case.

> Or have you spotted an ability to configure the max MTU on your mobile phone, and then probably for the WiFi and the WWAN interface?

I must set the network interface MTU from each device... i.e I'm literally having to `ip link set wifi mtu 1464` on each device or equivalent, otherwise the MR2100 will negotiate 1500 for the wifi or ethernet connection. Unfortunately there is no MTU setting in the MR2100 user interface for either the modem or the wifi or ethernet to override this centrally.

> Last but not least, the mentioned PMTUD is not a "one-off" process, it's an ongoing thing for each session.

For the LTE modem at least (or whatever mechanism it's using to determine the appropriate MTU), either it's not ongoing or not working at all for Vodafone. It does select the correct MTU initially (1500) but that could be by chance since 1500 is the standard - either way, for Vodafone the effective MTU will lower to 1464 _after_ it starts using carrier aggregation for any length of time, and the MR2100 does not notice this... for comparison, when using Three it will correctly start off with 1380 and stay at 1380 which the MR2100 has no problem determining (because Three's network doesn't seem to have a dynamic effective MTU).

The TL;DR is, 1464 is needed for this network, for whatever reason the MR2100 does not detect this... and so I would obviously prefer to be able to override this centrally than have to configure every new device's wifi connection... alternately it would be great if the MR2100 could detect the dynamic MTU and force the wifi devices to reconnect, but I suspect that's part of the SDX24 firmware and not under Negear's control.

I think the best I can hope for is for someone to point me in the right direction for the appropriate AT command.