UDP firewall rule goes to sleep after a few minutes and NTP traffic dies

Sorry for taking so long to reply - that tiny corona virus is surprisingly successful at throwing monkey wrenches into things. The bad news is that the standby/replacement firewall I ordered a week ago still hasn't arrived, which means I cannot risk bricking the firewall by trying to flash new firmware. The good news is that flashing new firmware no longer seems necessary.

I finally got my mitts on a pair of switches with port mirroring capability (by raiding my home network) and so I could sniff the complete traffic on either side of the firewall. The result is that the lost packets do make it out onto the wire on the near (intranet) side of the firewall, i.e. they aren't eaten by the network stack of the originating machine after having been seen by the Wireshark running there. The lost packets appear neither on the wire on the far (WAN) side of the firewall nor in the dropped packets log.

I also found out that both sides of the firewall are in the same broadcast domain because another programmer had installed a bypass to keep a legacy server functioning until a replacement can be rolled out. Obviously, this negates the security effect of the firewall but at least it keeps both kinds of clients running until the replacement is ready, i.e. the newer clients who are aware of the firewall and the old ones who need direct access.

After removing the bypass and rebooting the firewall the problem was completely gone, so it must have been caused by the fact that both ends of the firewall were connected to the same wire (broadcast domain). Hence the only thing I need to do is kick the programmer of the replacement server to make him work faster, in order to remove the need for the bypass.

The strange thing is that the sleepy firewall syndrome seems to affect only Windows 7 clients whereas packets from Windows 10 clients go through on the first try almost always, even after ages of inactivity. I.e. I haven't been able to provoke the glitch on Windows 10 clients at all. The few instances I observed on W10 clients were probably due to the boxen just having been plugged into different switches (meaning the switching mesh had to re-learn a different location for their MACs).

Inspecting the packets with Wireshark gave no indication of an appreciable difference between the Windows 7 packets and the Windows 10 ones - the packets were exactly the same on all levels, except for any contained sequence numbers and source IP/MAC naturally. Hence the difference probably lies in the things Wireshark cannot show, i.e. Ethernet framing and timings.

Anyway, the firewall works perfectly once the bypass has been removed, and so the bug hunt ends here. ;-)