Had this with the RAXE450 and worked with ProSupport for hours on end, for days, with no luck

The RAX450/RAX500 are pretty much the same thing. Netgear is knowns for selling a "slightly" lower spec device to costco so they can sell it cheaper. But hardware wise, they're basically the same. See the R7960P and R8000P

One thing I ran into with my RAX200 was that it would occasionally have issues with 500+mbps speeds. I worked with engineering quite a bit on it and would only ever be told "it must be something with my setup".  I never did find a resolution with current firmware for it so pulled my RAX out of service for a different netgear router. Some of the older firmwares worked fine while the newer ones broke it. 

One thing to try though that i didn't see was to go XB8 modem/router---->RAXE (in access point mode).  

See what speeds that gives. With the 500-600mbps that you're getting, it makes me curious if the RAXE isn't doing some traffic inspection.

Also, check what your link speed is on the device you're testing. I'm curious what link speed its connecting at as actual throughput tends to be around 55-65% of link speed.