My XR500 has packet loss and lag spikes during games

While gaming, it's pretty stable but then it will have these little episodes of high packet loss and big ping spikes. These episodes will happen about 2 times during a 12 minute game.

It usually recovers and the game continues but every once in a while it will completely lag out of the game. 

Old Netgear router had higher ping but none of these episodes. Was always stable.

Firmware is all up to date. QOS is currently on for my Xbox (I've tried it off as well - same thing).