Don't trust the Orbi app's speed test with 2Gbps service!

I think I have discovered that with a 2GB download plan with Comcast, Orbi App is completely inaccurate in its download speed tests.

Background

For several years I have had Comcast's 1200/35 plan - which in real world was 1440/42. Through the Orbi app, which allegedly tests router to an unknown/undiscoverable server, I got in the 1400s download, and 42-44 upload. All fine.

But a couple of weeks ago, I found I could move up to the 2GB download plan, for $10 less a month - sold! That plan is 2000/200, with over provisioning it is 2400/240.

The Problem

So I noticed that 90% of the time with the Orbi app, while I never had a problem with upload speeds (new mid-split neighborhood) getting 245-250 Mbps, the download speeds were frequently well below 2000 Mbps. So I got to thinking, why do I trust this app? So I set up a test!

My configuration

For this test, I have Comcast cable -> Cat 6a cable to -> Netgear CM3000 modem -> Cat 6a cable to -> Orbi RBR960 router. I plugged my M2 MacBook Pro into the router on its 2.5Gbps port, using a Cat6a cable plugged into a Belkin USB Type C to 2.5 Gb Ethernet Adapter (so it plugs into the Mac via a USB-C/Thunderbolt 4 port).

The test

I was 2 feet from the router, MacBook Pro plugged in for direct 2.5GB connection. I had my iPhone 15 Pro iPhone next to the Mac, to run the Orbi app. I ran 6 speed tests - first a normal Ookla Speedtest from the app on the Mac - it was using the default closest server, a Comcast server, 0 miles away according to the app.

The instant it finished, I ran the speedtest on the Orbi app on my iPhone (no idea what server it is using, not displayed or configurable). when it finished, I immediately ran another speed test on the MacBook Pro.

I did this 6 times, the tests were so close together I thought any network congestion would be noticeable and not an influence to the test.

As a note, Ping results were very close throughout - they ranged from 8 ms - 13 ms.

The results (speeds in Mbps)

Test # Direct connected Mac to Router Orbi App from iPhone
1 2291 2193.90
2 2081 1669.94
3 2276 1278.89
4 2279 1505.36
5 2288 1444.57
6 2265 2031.29

Interpretation of results

1) The speediest from the direct connection is remarkably consistent
2) The Orbi app speed is mostly well below where it should be, and wildly inconsistent
3) The Orbi test results are what I have experienced over the last few weeks. While the 1GB Orbi speed tests were exactly what they should have been for my service, the 2Gbps were not.
4) I suspect the Orbi app is using some server not optimized for my location, or it could be a server that can't handle 2Gbps speeds very well.

Conclusion - don't use or trust the Orbi app for speed tests if you have a 2 Gbps plan!