Slow Ping on Wired Backhaul?

I've been using the Orbi for 6 months now and have them upgraded to the latest firmware V2.3.1.44.  In my home I have it pre-wired with CAT5e cable to every room.  I've set up the Orbi Router in my media closet on the second floor and the internet from our ISP (Fiber) is coming from the modem directly into the Orbi router.  From there it goes to two DLink unmanaged switches (DGS-1024D) that then run to each room in the house.  

 

I've connected the two satelites to the main Ori router using the ethernet ports in two other rooms in the house.  All of the routers/satelities are on the same firmware and the Orbi admin tool shows that the two satelities are connected by wired connections.  

 

However, in the two rooms where the satelites are located I'm having streaming issues.  Both satelites have wireless connections to two separate FireTVs and I'm running into buffering and pixeling issues on both sets.  The same is NOT true of the TV connected wirelessly to the main Orbi router upstairs.  The issues happen both on public streaming services (Hulu, Playstation Vue) as well as streaming from my NAS which is connected to the DLink switch.

 

I've been trying to figure out what's happening and been monitoring the performance of each satelite through a simple monitoring program (EMCO Ping Monitor).  Ping on average is 4.6ms but every 3-4 hours the ping spikes to up to over 400ms with the peak at 458ms.  I have in the past also have complete internet drop outs randomly for devices connected to the satelites.

 

Any thoughts on what might be going on here that's degrading my service?

 

For clarity here's how the system is configured:

 

Guest Network : Off

Daisy Chain : Off

MU-MIMO : Off

Beamforming : Off

Fast Roaming : Off