First - Netflix has a playback setting that you can set on their webpage that controls the video "quality", and through that the bandwidth required - 0.3 Gb/hour, 0.7 GB/hour or the highest 3 Gb/hour for HD & 7 Gb/hour for UltraHD - take a look at it, it may solve your problem.
Second - I'm not certain how effective downstream QoS will be - your router is at the "downstream" end of the ISP connection, so by the time you can control what's passing through that link, it is already on the link - you can however control upstream QoS, by changing the "queuing order" of the data being transmitted.
Also worthy of consideration - your upstream link speed is usually significantly lower than your downstream link speed, meaning that you're more likely to need QoS there, and what MAY be affecting your downstream speeds is congestion on the upstream link, the data coming down in most cases has to be acknowledge as received, if not, it will be assumed lost and retransmitted, slow acknowledgement leads to reduced transfer rates.
