5GHz wi-fi only works with no security

I recently purchased a used R8900 Nighthawk x10 on eBay, and I've tried just about everything I can think of to get the 5GHz wifi working. But, it will only connect successfully if I turn off the WPA/WPA2 security. The 2.4GHz connection works just fine with security, but anytime I attempt to connect to the 5GHz wifi (with security enabled), I get an error that "Wi-Fi does not have an IP address and cannot connect to the Internet" followed by "Wi-Fi has the self-assigned IP address 169.254.xxx.xxx and will not be able to connect to the Internet." When I look at the IP information it has the wrong subnet mask and no router information.

If I manually assign a correct IP, subnet mask, router, and DNS, everything appears to be connected, but there is still no internet connection, i.e. no webpages will load, can't connect to any router administration pages, etc.

I've tried reseting the router to factory defaults multiple times. I've tried reinstalling, downgrading, and even uploading custom third party firmwares (1.0.5.24, 1.0.5.18, and Voxel's 1.0.4.45.2HF firmware). I've also tried adjusting channel options, but nothing seems to allow me to connect to the 5GHz wifi connection if WPA/WPA2 security is enabled.

I've tested it on both my home and work networks with the same results. The router is in router mode behind an AT&T gateway. Any advice or additional things to try/test would be greatly appreciated. Or, if anyone knows that this is for sure a hardware issue instead of a software/settings issue, then I could at least contact eBay to try to get my money back for receiving defective hardware.

Thanks in advance!