× Get free training on Switching for AV over IP and receive AVIXA Credits. Sign up at NETGEAR.academy
Reply

[NDI] Multicast UDP Packet Loss with Multiple Streams on Windows PC

nmarques
Guide

[NDI] Multicast UDP Packet Loss with Multiple Streams on Windows PC

The following is a version of an email I sent to Netgear's ProAV Team already. Already had a good conversaion with @NeilS and Netgear is going to look to do some internal testing. Note that I don't think these issues are necessarily a Netgear issue, but we're all just trying to understand what might be going on. 

 

Tl;Dr: A Windows 10 PC is dropping frames while receiving 4x full bandwidth NDI sources in multicast UDP mode on it's gigabit NIC. Ethernet bandwidth doesn't exceed 550Mbps and PC resources are not maxed out. Switching source devices to unicast TCP solves the problem and bandwidth doesn't change. Why?
 
I'm an AV integrator and I have a site where we deployed a system with some NDI (full bandwidth) cameras, encoders, and decoders using a M4200 (edge switch for cameras, providing PoE) and a M4300 (connected to M4200 by 10Gig copper). I know the M4200 isn't "certified" for NDI, but it met the requirements for the application and I didn't see anything wrong with chosing it. It's quiet, has PoE+, low port count, and 10Gig uplink. I liked the NBASE-T for future too. 
 
Out of the gate, I configured everything for multicast because it's more bandwidth efficient. The M4200 and M4300 multicast settings are all fine and that part of the system is operating normally. The switches are all configured according to the NewTek NDI deployment guide, with flow control enabled, jumbo frames and QoS disabled, etc. @LaurentMa had previously reviewed the configuration. 
 
They have a computer (Core i7-9700, 16GB RAM, GTX 1650, NVMe SSD) running BirdDog Multiview program to display 4x NDI sources in a 2x2 grid. So, this PC is configured to receive 4x full bandwidth (not proxy) NDI streams. All streams are 1080p30 but we've tried 1080p60.
 
I've been troubleshooting freezing video, lost frames, etc for quite some time. The PC is not maxed out on resources. CPU sits at around 60% and GPU maybe 25%. Ethernet bandwidth is generally around 550 Mbps or less. 
 
I've also tried to use 4x instances of NewTek NDI Studio Monitor instead of the BirdDog program and performance is slightly better. However, the BirdDog program outputs NDI of the multiview which is used on other destinations on campus. 
 
I finally had a revelation about using multicast the other day -- the packets are all UDP with multicast, and so there's no acknowledgement of packets and I believe I'm just losing frames as a result. I suppose the frames I do get may be more timely? 
 
To my surprise, I switched the cameras and encoders all back to unicast TCP instead of multicast and performance was immediately flawless. Ethernet bandwidth on the PC didn't change. 
 
The frame loss was never visible on a Panasonic AV-HLC100 switcher (which is basically a NewTek TriCaster product). However, I think that the unit only receives 1-2 full bandwidth NDI streams (preview and program bus sources maybe?) and the thumbnail previews are all proxy.
 
For this installation, no NDI source is routed to more than 2x or 3x destinations, so I think I'm OK using unicast for now.
 
I am more familiar with other AVoIP solutions such as AMX SVSI, Crestron NVX, Wyrestorm, etc... and all of those use multicast for video as far as I know. One difference though, is a single decoder only receives one stream.
 
It seems to me the problem lies with one of these things:
  • Is there something about a single computer receiving multiple streams that is causing problems?
  • Does the Ethernet stack or NIC have a UDP buffer that is likely getting full?
  • Or is the NDI library not processing the buffer fast enough to keep up? 
  • Or does Windows or the NIC prioritize TCP traffic over UDP? 
I realize in general that UDP is unreliable but no one ever mentions this in regards to multicast NDI. In fact, it's been recommended to me to deploy NDI in multicast since I have switches that support it, I know how it works and how to deploy it, and I am sending sources to 2 or more destinations... so it made sense to do multicast. If you're unfamiliar with NDI, it is NOT multicast video by default. In fact, all of the "plug and play" bit about NDI is more related to doing a low number of sources or destinations and just running it as it ships. You have to manually switch sources to multicast. If you have managed switches, multicast config probably does help for the mDNS traffic. 
 
I know NDI has the ability to "load balance" across multiple NICs without the use of link aggregation. Would that help this problem? How does this actually work since we don't specifically bond an NDI stream to a particular NIC? I am so not sure how this would work in multicast unless both NICs join the same multicast groups and it figures out which packets are missing to stitch the streams back together.
 
I can try to run a iperf3 test at some point, but given that the UDP streams are flawless on other devices on the same switch, and the streams are flawless when viewing 1 or 2 at a time on this PC, it doesn't appear to be a network issue. 
 
Open to any ideas or experiences with similar setups. 
 
Thanks!
Model: GSM4328S|M4300-28G - Stackable Managed Switch with 24x1G and 4x10G including 2x10GBASE-T and 2xSFP+ Layer 3
Message 1 of 1
Discussion stats
  • 0 replies
  • 3063 views
  • 0 kudos
  • 1 in conversation
Announcements

AV over IP Switches by NETGEAR