We installed a ReadyNas RR4312X with 12x 6TB Seagate SkyHawk Surveillance drives in RAID-50 for Avigilon security camera storage. We started out with 55 cameras and it worked flawlessly but as we are now approcahing 110 cameras it is starting to become sluggish and have issues reading data simultaneous to writing it. It is not a bandwidth issue as we have one of the 10Gb links dedicated to incomming traffic and the other 10Gb to outgoing. Respectively they are only showing 750MBps into the unit and 400MBps out. I am leaning towards this being a disk I/O issue however each drive has a data transfer rate of 600MBps and an internal rate of 195MBps.
So the question is am I on the correct path of troubleshooting or should I be looking at a configuration issue? All ideas are appreciated.
Respectively they are only showing 750MBps into the unit and 400MBps out.
I am wondering what the needed peak write throughput is, and also the amount of data being collected per day.
Using multiple volumes might help, as seeks would be independent. If you are writing to shares (as opposed to LUNs) then using SSDs for metadata tiering and possibly data caching might also help. Either approach would require using bigger physical disks to keep the same capacity.
You might also look at the CPU load in the performance graphics. RAID-10 would likely reduce the CPU load, since no parity computations would need to be done. It would also reduce the amount of disk I/O needed for writes.
We are a school so the money simply isnt there for SSD drives. Currently we are maxing out the storage with the desired retension rate so changing to RAID 10 would further limit retension unless like you stated we went to larger drives.
I didn't see a CPU performance section. Unfortunately we have been closed for the last 5 weeks (spring break and COVID-19) so the 30-day chart is not represenative of average usage and I am unable to pull specific numbers.
I didn't see a CPU performance section. Unfortunately we have been closed for the last 5 weeks (spring break and COVID-19) so the 30-day chart is not represenative of average usage and I am unable to pull specific numbers.
Do you have ssh access remotely? If you do, then you can just run top.
Re: ReadyNas 4312x surveillance video storage issue
The NAS is just storage, not running the surveillance app itself, correct?
With 16GB of RAM standard, I'd think that was enough, but you could try adding more. While it would require completely re-configuring the NAS, I wonder if it would help to have multiple volumes and the cameras spread out as to on which each puts it's data.
Are the cameras motion triggered, which means the demand fluctuates a lot, or continuous? If motion triggerred, have you noticed the problem worse during class changes when a lot are going to trigger at a time? Of course, modern codecs used by most security software compress unchanging data much smaller than changing data. So even if continuous monitoring is used, the data volume may increase with lots of motion all over campus.
Re: ReadyNas 4312x surveillance video storage issue
The NAS is just the storage location and the application itself runs on another server. All cameras are set to record on motion. Time with many cameras recording on motion seems to make things a bit more sluggish.
The way that Avigilon writes to its storage is by populating the entire drive (as seen by Windows) with folders that it randomly writes to. I am unable to define which cameras write to which drives. To create multiple volumes would decrease the usable storage availabe and decrease retension.
The way that Avigilon writes to its storage is by populating the entire drive (as seen by Windows) with folders that it randomly writes to. I am unable to define which cameras write to which drives. To create multiple volumes would decrease the usable storage availabe and decrease retension.
I do understand that. The multiple volume idea was to reduce the disk I/O and perhaps improve seek performance. But thinking about it more, you already have the benefit of that with RAID-50. Another thought I had was that the CPU was getting stressed by the parity computations. That could be a factor.
I wondering about your workload though. You say you have a 780 MB/s write stream from 110 cameras. That's an extremely high data rate (56 megabits per camera), and you are saying they aren't continuously recording on top of that.
Is that correct??? What resolution and compression are you using that requires such a high data rate? 1080p Motion JPEG???
Re: ReadyNas 4312x surveillance video storage issue
Yes, if you can't do load leveling between windows "drives", then you risk reducing retention times on the most-used ones. And it sounds like that is your situation.
Do you happen to have audit enabled on the NAS? That can be a real throughput killer.and isn't really going to help a lot unless you also have users manually logging in.
Using iSCSI instead of SMB to access the NAS will cut down some on the NAS overhead. Do you have the space to run an experiment to see if it helps?
Yes, if you can't do load leveling between windows "drives", then you risk reducing retention times on the most-used ones. And it sounds like that is your situation.
If @Doblerm actually has a 780 MB/s write workload, then his best strategy is to switch to H.264 and reduce the bitrates.
Every block written to the RAID50 volume requires I/O to several disks in parallel. For example, if you have three RAID-5 arrays of 4 disks each, organized in a single RAID-0 volume, then writing one data block requires reading the two other data blocks in the RAID-5 stripe, computing the parity block, and then writing the RAID-5 parity block. That turns into 4 disk accesses per written block.
If the 780 MB/s write stream and your 400 MB read stream is sustained, then this is what I see for total disk I/O for the RAID-50 array above:
With 780 MB/s coming in to be written, you'll need 3120 MB/s of disk I/O (half reads and half writes). Add in the 400 MB/s read stream, and you have 3520 MB/s total disk I/O. Divide by 12 to get the I/O per disk, and you get 300 MB/s per disk - which is physically not possible with your mechanical disks even if the I/O is all sequential (and it isn't). The only way it works is if a lot of the data is already in the cache (NAS RAM and the on-disk caches).
Using four RAID-5 arrays of 3 disks each would work out better, but would require one more parity disk. The output stream requires 2340 MB/s of disk I/O, so the total disk I/O needed drops to 2740 MB/s., and you end up with ~230MB per disk. That's still over your measured disk speed, and you won't get that measured speed over the entire disk. So it still depends on the cache to work.
So if my analysis here is correct, you're reaching the limits of what you can do with RAID-50 if the workload you are seeing is sustained.
Now, RAID-10 will reduce your capacity a lot for sure. But each block coming in is simply mirrored. So the I/O becomes 1560 MB for the write stream, and 1960 total - about 160 MB/second per disk altogether.
Abandoning RAID redundancy altogether gives you maximum capacity and only requires 98 MB/sec per disk. But that is very fragile.
Reducing the video bitrate down to 16 mbit/s H.264 would still give transparent quality for 1080p, and I think could be handled by the current RAID50 setup.
Re: ReadyNas 4312x surveillance video storage issue
All cameras are running H.264. About 40% of them are 1080p and 60% are 4k. 3 of the cameras are also multi-head(4 lenses) so technically you could say that there are 122 cameras. The 4k cameras record about 10-14MBps when there is high motion.
After reading over your calculations it seems like my only choice is to back down the framerate thus lowering the overall bandwidth and I/Os. Another option that we are looking at is a new Dell R540 server with 12x Seagte Exos SAS 12G drives in RAID-10. The investment will be worth it as our current server is reaching EOL. Then we could use the Netgear storage as an archive location.
All cameras are running H.264. About 40% of them are 1080p and 60% are 4k. 3 of the cameras are also multi-head(4 lenses) so technically you could say that there are 122 cameras. The 4k cameras record about 10-14MBps when there is high motion.
4K would account for it I guess. Still, Netflix does 4K at 16 mbits, not 80-92 mbits. They are using H.265, so that would translate to 32 mbits with H.264.
High-delay encoding is more efficient than real time, but it shouldn't be a factor of 2-3x. Are you sure the only setting you have is to reduce the framerate?
Re: ReadyNas 4312x surveillance video storage issue
Either I lower framerate or drop the resolution to 2k but that kinda defeats the purpose of the camera. Most of our cameras are not Avigilon so we are unable to bump them to H.265 for better compression. At the moment only Avigilon cameras are able to run H.265 with the Avigilon ACC software.
Either I lower framerate or drop the resolution to 2k but that kinda defeats the purpose of the camera. Most of our cameras are not Avigilon so we are unable to bump them to H.265 for better compression. At the moment only Avigilon cameras are able to run H.265 with the Avigilon ACC software.
Often there are other knobs (a quality level or bitrate setting). Lowering framerate will help some, but often not as much as you'd think. The faster the frame rate, the less change there is between frames - so the bits per frame will go down. Cutting the frame rate in half won't cut the bitrate in half.
The impact of lowering resolution depends on what the camera is looking at. For instance, if you want to capture license plates, you probably want to keep it at full resolution. But if the area of interest is fairly close to the camera, and you aren't needing to capture relatively small text, then lower resolution might be enough. Also, some cameras can zoom into the moving area that triggers the motion. If that feature works well, then it does let you lower the resolution and still capture a lot of detail.
The practical problem here for you is that you have a lot of cameras, so figuring out the needed resolution for each one would be a pretty big job.
