This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
We are attempting (and failing) to incorporate two RR4360S' into our CCTV system for archival. The VMS is Pelco VideoExpert Enterprise and we are using both virtual machines on a Cisco Enterprise Chassis as well as Pelco branded physical servers. So far, this is what we have attempted and how are system is configured:
1. We have configured both a physical Pelco server and a virtual server to record to a local drive and archive older files to their own dedicated share on the NAS, however the cameras will record until the local drive fills and then stop recording rather than archiving the older files to the NAS.
2. We have also configured the system to record video directly to a dedicated share on the NAS. This results in recorded video however it shows extensive gaps and loss of data.
3. Both NAS units are connected to a dedicated 10G network and our entire system is currently operating in a unicast mode.
4. We have multiple user consoles (Win 10) and get the same result regardless of the user account level or console used.
5. Captures made with Wireshark do not show any unusual traffic and are consistant with that seen in our test-bench system.
We created a second system on our test-bench using a Cisco chassis and virtual servers, connected multiple cameras through multiple switches to simulate our actual system and are able to record flawlessly to a local drive and archive to a share created on a seperate physical server (MS Server 2016 with SMB share). However when we attempt the same with the NAS if fails.
We suspect the issue is in the way the NAS is receiving and processing the data but would like to confirm that. We would also appreciate any asistance or suggestions this group can offer to try and resolve this challenge.
It's hard to say what the issues are with this set-up, but I suspect some are a Windows issue or associated with the software you are running, at least in part, not strictly problems with the NAS, per se. But here are some things to look at:
For the case where video is recorded locally and archived on the NAS (my situation for my home surveillance system, and likely your best option):
Insure that the share is visible to the application. If you are using a mapped drive, that mapping has to be for the same "user" the application runs under. If it's a service task (which is very typical for a surveillance app), that can be especially problematic. you either need to change the "run as" parameter to a normal user or wrap everything in a helper app that also mounts the share. depending on the app, those can have undesirable consequences of their own. (NOTE: In my BlueIris surveillance, I installed it as not a service task, but it has that built-in option.) You may also need to put the NAS user name and password in the Windows Credentials Manager.
Insure the NAS shows sufficient available space to the app. If the app is old (this one doesn't seem to be, but you never know if there are vestiges of old code remaining), it may not be designed to properly recognize a large space pool like on your NAS, so it may not be archiving because it thinks the NAS is full. Using a smaller, dedicated share or a fixed size iSCSI LUN can get around that, or at least help in the diagnostic, but also limits how much you can archive.
Likewise, insure that whatever event is supposed to trigger the archival is also observed by the application. (I just use time, and insure my local space is sufficient for even continuous video through that period.
Insure that the app separates the files into smaller pieces, as an actively recording segment is likely not going to get moved.
You could have a separate background archival task run from Windows Task Manager, but that could mess up any recording database the app has.
For the case where video is recorded directly to the NAS:
Unless the application has sufficient buffering, usually meaning it was designed with the idea of streaming to a server, this is probably not going to work well. Using iSCSI instead of SMB may help, but will likely limit the amount of space you can use. QoS and Windows network priority settings may also help. A dedicated network may be necessary if there is a sufficiently large amount of other traffic on the network and/or no QoS settings available. But even using these, and based on your observance thus far, NAS events like a scrub or balance are likely going to add too large a delay and create an associated data loss.
While the Pelco site says it runs under Windows, it does not give a lot of information as to how much control you have on the Windows system if you purchased pre-loaded hardware. If it uses a Windows CE instance, it may be limited, though WinCE will also typically make the app run more smoothly since there are fewer background tasks "stealing" CPU time. WinCE apps almost always run as a system task, which may be hard to change due to the limitations Microsoft puts on WinCE (or, at least did. I have not used a WinCE instance since Win7.)
It's hard to say what the issues are with this set-up, but I suspect some are a Windows issue or associated with the software you are running, at least in part, not strictly problems with the NAS, per se. But here are some things to look at:
For the case where video is recorded locally and archived on the NAS (my situation for my home surveillance system, and likely your best option):
Insure that the share is visible to the application. If you are using a mapped drive, that mapping has to be for the same "user" the application runs under. If it's a service task (which is very typical for a surveillance app), that can be especially problematic. you either need to change the "run as" parameter to a normal user or wrap everything in a helper app that also mounts the share. depending on the app, those can have undesirable consequences of their own. (NOTE: In my BlueIris surveillance, I installed it as not a service task, but it has that built-in option.) You may also need to put the NAS user name and password in the Windows Credentials Manager.
Insure the NAS shows sufficient available space to the app. If the app is old (this one doesn't seem to be, but you never know if there are vestiges of old code remaining), it may not be designed to properly recognize a large space pool like on your NAS, so it may not be archiving because it thinks the NAS is full. Using a smaller, dedicated share or a fixed size iSCSI LUN can get around that, or at least help in the diagnostic, but also limits how much you can archive.
Likewise, insure that whatever event is supposed to trigger the archival is also observed by the application. (I just use time, and insure my local space is sufficient for even continuous video through that period.
Insure that the app separates the files into smaller pieces, as an actively recording segment is likely not going to get moved.
You could have a separate background archival task run from Windows Task Manager, but that could mess up any recording database the app has.
For the case where video is recorded directly to the NAS:
Unless the application has sufficient buffering, usually meaning it was designed with the idea of streaming to a server, this is probably not going to work well. Using iSCSI instead of SMB may help, but will likely limit the amount of space you can use. QoS and Windows network priority settings may also help. A dedicated network may be necessary if there is a sufficiently large amount of other traffic on the network and/or no QoS settings available. But even using these, and based on your observance thus far, NAS events like a scrub or balance are likely going to add too large a delay and create an associated data loss.
While the Pelco site says it runs under Windows, it does not give a lot of information as to how much control you have on the Windows system if you purchased pre-loaded hardware. If it uses a Windows CE instance, it may be limited, though WinCE will also typically make the app run more smoothly since there are fewer background tasks "stealing" CPU time. WinCE apps almost always run as a system task, which may be hard to change due to the limitations Microsoft puts on WinCE (or, at least did. I have not used a WinCE instance since Win7.)
I'm also thinking that trying iSCSI would be a reasonable next step.
If you want to keep the video in a share, you could also try testing with NFS (adding the windows NFS client to the VMS). You might also look at the SMB encryption settings (and maybe explore the settings exposed in SMB Plus).
I'd also try turning auto-defrag and snapshots off in the share. SSD metadata tiering might be worth a try, especially if you have a lot of video files in the share.
Thanks for the quick reply and detailed explanation. I will address your questions and maybe give you a better picture of our system:
Pelco VX runs as an application on MS Server (we are using both 2012 & 2016).
Our system consists of a VX Core (the server is the brain), a Media Gateway (this server handles requests for video from the clients, and Storage Servers (these servers processes the storage of video data)
The Storage Servers are limited to 300 MB of throughput, which is why we have multiple servers filling this role.
When configuring the system, we go into the Core and assign individual cameras to the appropriate storage server.
Concerning the storage server, we can identify where the data is stored, for example, E:\\Video Data\ (in the case of a local drive) or \\10.10.10.100\Video_Data\ (in the case of a NAS share). We also can specify a buffer size on the drive we choose and to reserve bandwidth for that drive. Lastly, in the case of archiving, we identify a local drive for storage “E:\\Video Data\” and then a second drive, which we designate an archival-drive such as \\10.10.10.100\Video_Data\.
As I mentioned previously, some of our servers are configured to record to a local drive and archive off older files while others are set to record directly to our NAS.
The only “users” we have on any of our servers our the admins (myself and my coworker) The clients talk to the Core, and we build user accounts in the application.
Lastly, we set all of our shares on the NAS for archiving to “Annonymous access.”
I am working on clearing up the old data on the NAS today and will reconfigure the shares using a LUN configuration, as you suggested, and let you know how that works. Thanks again.
We do not currently encrypt any of our files as it is a stand-alone and very controlled network. We also do not use the auto-defrag or snapshots in the NAS.
I will need to get smarter on metadata tiering; however, the files stored are not typical .MOV, .MPJPG, or another standard video format. Pelco creates a proprietary data file and requires its viewer to see it (I've attached a screen shot).
For the archival drives, we currently have ten 10 TB drives in a RAID 50 configuration.
We do not currently encrypt any of our files as it is a stand-alone and very controlled network.
Yes, but the SMB 3 protocol itself is both encrypted and authenticated (though you should be able to turn that off). Though I expect your NAS does have hardware acceleration for AES (newer X86 ReadyNAS do), it still is just extra work in your setup, and might be affecting the performance.
Another thought - are your switches using ethernet flow control? How about link aggregation?
I will need to get smarter on metadata tiering; however, the files stored are not typical .MOV, .MPJPG, or another standard video format. .
If you look at performance measurements, when you use mechanical disks than folder browsing in network shares is always much slower than large file transfers. SSD metadata tiering addresses that (and improves that performance for any file type). There is also SSD data tiering, which can improve overall performance, especially if there are files being written simultaneously. Though you'd really need to benchmark the data tiering - I haven't seen much written up about that, and of course your use case is particularly demanding. Still, if there is disk-seeking going on, then that will reduce the throughput.
Thanks to you and StephenB for the great information. Ultimately our issue was a combination of configuration challenges however the iSCSI solution is proving to provide us with the best performance. It took me a little while as I had to get smart on iSCSI and ended up deleting my entire configuration and starting over which was actually a good think. Thanks again for the help.
