Pelco VideoExpert Enterprise

It's hard to say what the issues are with this set-up, but I suspect some are a Windows issue or associated with the software you are running, at least in part, not strictly problems with the NAS, per se.  But here are some things to look at:

For the case where video is recorded locally and archived on the NAS (my situation for my home surveillance system, and likely your best option):

1. Insure that the share is visible to the application.  If you are using a mapped drive, that mapping has to be for the same "user" the application runs under.  If it's a service task (which is very typical for a surveillance app), that can be especially problematic.  you either need to change the "run as" parameter to a normal user or wrap everything in a helper app that also mounts the share.  depending on the app, those can have undesirable consequences of their own.  (NOTE:  In my BlueIris surveillance, I installed it as not a service task, but it has that built-in option.)  You may also need to put the NAS user name and password in the Windows Credentials Manager.
2. Insure the NAS shows sufficient available space to the app.  If the app is old (this one doesn't seem to be, but you never know if there are vestiges of old code remaining), it may not be designed to properly recognize a large space pool like on your NAS, so it may not be archiving because it thinks the NAS is full.  Using a smaller, dedicated share or a fixed size iSCSI LUN can get around that, or at least help in the diagnostic, but also limits how much you can archive.
3. Likewise, insure that whatever event is supposed to trigger the archival is also observed by the application.  (I just use time, and insure my local space is sufficient for even continuous video through that period.
4. Insure that the app separates the files into smaller pieces, as an actively recording segment is likely not going to get moved.
5. You could have a separate background archival task run from Windows Task Manager, but that could mess up any recording database the app has.

For the case where video is recorded directly to the NAS:

1. Unless the application has sufficient buffering, usually meaning it was designed with the idea of streaming to a server, this is probably not going to work well.  Using iSCSI instead of SMB may help, but will likely limit the amount of space you can use.  QoS and Windows network priority settings may also help.   A dedicated network may be necessary if there is a sufficiently large amount of other traffic on the network and/or no QoS settings available.  But even using these, and based on your observance thus far, NAS events like a scrub or balance are likely going to add too large a delay and create an associated data loss.

While the Pelco site says it runs under Windows, it does not give a lot of information as to how much control you have on the Windows system if you purchased pre-loaded hardware.  If it uses a Windows CE instance, it may be limited, though WinCE will also typically make the app run more smoothly since there are fewer background tasks "stealing" CPU time.  WinCE apps almost always run as a system task, which may be hard to change due to the limitations Microsoft puts on WinCE (or, at least did.  I have not used a WinCE instance since Win7.)