For years, we've been running eight custom built servers running ZM 1.28.1 on Ubuntu server 12.04. Each server has 14-20 cameras on each recording at 720p or 1080p 5 fps in MJPEG and this gets us at our required 30 days of retention for our ~130 Axis IP cameras; and this has served us quite well but the 56 WD Red NAS 6 TB drives we have are starting to have problems and the servers are about 5-7 years old now.
So I've just been tasked with making a massive Dell PowerVault ME5084 SAN with 700 TB usable storage space work for replacing our current NVR server and storage solution, virtualizing the NVR's, and doubling retention to 60 days. With total storage across all our current NVR servers being just under 300 TB, in theory all I need is 600 TB to double number of days keeping the same resolutions and framerates. They are fine with the current quality, they just want 60 days-worth of it.
We have an existing VMware vSphere 7 virtual infrastructure, with all hosts running ESXi 7. I could just try to create eight similar spec'd VM's like our current solution, but i'd like to do this as efficiently as possible and not waste resources since our virtual environment is already running typical business servers. We only use about 10-15% CPU and 65% RAM across our virtual infrastructure, and now am only using about 3% storage of our new storage for it- so I think this is very do-able.
My real question here is how can I most efficiently virtualize our NVR's? I'm thinking of running FreeBSD and a newer version of ZM inside just one or two VM's with big compute resources, but VMware has a virtual disk size limitation of 62 TB. With so much I/O from ~130 cameras I think I should probably spread it across multiple LUNs and possibly even VM's. I am thinking in 62 TB I can probably get about 8-12 cameras and their 60-days-worth of storage. I could possibly just do multiple LUNs for the one or two VM's but... I don't know... Or I can setup one VM per LUN, in the end needing about 12-14 VM's for all our cameras.
I tried to do some online searching but didn't find much for virtualized NVR's... is anyone else doing something similar here? Or am I sort of in very unusual and uncommon territory. Is this a stupid, awful idea? I didn't make the buying decisions here and personally would have preferred just to build new bare metal servers and double the HDD size to 14 TB and be done with it - but I wasn't given that option. I think it will work though and just need to be really detailed about how I set it all up. Thoughts? Suggestions?
Virtualized NVR, what OS?
Re: Virtualized NVR, what OS?
You should have information on the I/O from your current setup to guide the disk requirements.
Rough math will bring you to the CPU usage. Assume each generation is 10% faster per cycle. Remember this will get you a rough estimate of CPU with the same version of ZM. ZM has been improved and uses less CPU now.
VMWare is a tricky thing. A VM only executes when there are enough threads available for the number of vCPUs the VM has been allocated. Getting the right number of vCPUs can increase the performance, too many and too few can hurt.
Storage pools can be used to easily divide the camera data onto different drives. This should allow you to spread to load to different LUNs and balance the I/O to the disks and make the TB limit moot.
Rough math will bring you to the CPU usage. Assume each generation is 10% faster per cycle. Remember this will get you a rough estimate of CPU with the same version of ZM. ZM has been improved and uses less CPU now.
VMWare is a tricky thing. A VM only executes when there are enough threads available for the number of vCPUs the VM has been allocated. Getting the right number of vCPUs can increase the performance, too many and too few can hurt.
Storage pools can be used to easily divide the camera data onto different drives. This should allow you to spread to load to different LUNs and balance the I/O to the disks and make the TB limit moot.
Andy
o||||o
Ubuntu 22.04
ZM 1.36.33
E5-1650-v4 Xeon
16 GB RAM
6 cameras -> 54 FPS modect
o||||o
Ubuntu 22.04
ZM 1.36.33
E5-1650-v4 Xeon
16 GB RAM
6 cameras -> 54 FPS modect
Re: Virtualized NVR, what OS?
Hi Andyrh and thanks for responding.
Not sure what you mean by information, other than what the product documentation states; I haven't done any testing. I have four 10Gbit Ethernet connections per VMware host, but only two of four are active I/O, so in theory VM's should get up to a max of 20 Gbit bandwidth to their storage. The storage on the SAN array is spread across 69 disks in Dell EMC's proprietary ADAPT RAID. I honestly don't think we will know much until we start stressing it.
VMware is indeed a tricky thing especially when it comes to utilizing CPU's. I think we will have to experiment with this as well.
As for storage pools to divide camera data to different drives, i'll look into this.
I'm really hoping some others are already doing something similar and may see this post and can respond. It is always nice to get some guidance from those who have done it. I will keep working on it with consideration to your suggestions though, thank you.
Not sure what you mean by information, other than what the product documentation states; I haven't done any testing. I have four 10Gbit Ethernet connections per VMware host, but only two of four are active I/O, so in theory VM's should get up to a max of 20 Gbit bandwidth to their storage. The storage on the SAN array is spread across 69 disks in Dell EMC's proprietary ADAPT RAID. I honestly don't think we will know much until we start stressing it.
VMware is indeed a tricky thing especially when it comes to utilizing CPU's. I think we will have to experiment with this as well.
As for storage pools to divide camera data to different drives, i'll look into this.
I'm really hoping some others are already doing something similar and may see this post and can respond. It is always nice to get some guidance from those who have done it. I will keep working on it with consideration to your suggestions though, thank you.
Re: Virtualized NVR, what OS?
Your current install should show the data rates for the cameras and you should be able to get the I/O rates from the OS.
For ZM I am a small home user. In my job I am a system admin (OS and storage) and part of the job is to estimate I/O rates and CPU usage for large systems.
PowerVaults (I assume this is the storage) are able to absorb plenty of writes because the cache is large enough to do full stripe writes which minizines the overhead of parity. The problem comes in if you outrun the disk for long enough to fill the cache. I seriously doubt ZM will generate anywhere near enough I/Os to stress that system. It will come down to what else is going on.
In your proposed build I would worry about CPU the RAM.
For what it is worth I have had the best success with ZM on Ubuntu server.
For ZM I am a small home user. In my job I am a system admin (OS and storage) and part of the job is to estimate I/O rates and CPU usage for large systems.
PowerVaults (I assume this is the storage) are able to absorb plenty of writes because the cache is large enough to do full stripe writes which minizines the overhead of parity. The problem comes in if you outrun the disk for long enough to fill the cache. I seriously doubt ZM will generate anywhere near enough I/Os to stress that system. It will come down to what else is going on.
In your proposed build I would worry about CPU the RAM.
For what it is worth I have had the best success with ZM on Ubuntu server.
Andy
o||||o
Ubuntu 22.04
ZM 1.36.33
E5-1650-v4 Xeon
16 GB RAM
6 cameras -> 54 FPS modect
o||||o
Ubuntu 22.04
ZM 1.36.33
E5-1650-v4 Xeon
16 GB RAM
6 cameras -> 54 FPS modect
Re: Virtualized NVR, what OS?
Honestly I think you are overthinking it. I support several vmware based servers, it's not that big a deal.
I have vastly improved ZoneMinder since 1.28. You may find that you need a LOT less hardware than in the past.
Saving h264/h265 instead of billions of jpegs is going to help you a lot with the IO.
You can go with FreeBSD, but I wouldn't unless you REALLY want to. I did make it compile and work, but it is the least tested platform.
If I were you, I would engage my services planning this out and providing support. But I would start with 1 server and see if it handles it. Then add another if it is needed. Put the mysql server on it's own, tuned for that purpose.
1.28.. my god.. and I was shocked the other day when I encountered a 1.30 install..
I have vastly improved ZoneMinder since 1.28. You may find that you need a LOT less hardware than in the past.
Saving h264/h265 instead of billions of jpegs is going to help you a lot with the IO.
You can go with FreeBSD, but I wouldn't unless you REALLY want to. I did make it compile and work, but it is the least tested platform.
If I were you, I would engage my services planning this out and providing support. But I would start with 1 server and see if it handles it. Then add another if it is needed. Put the mysql server on it's own, tuned for that purpose.
1.28.. my god.. and I was shocked the other day when I encountered a 1.30 install..
Re: Virtualized NVR, what OS?
Thanks for the thoughts... you're probably right about overthinking things.
I'll report back how things turn out in a few weeks once i've had time to actually get a vNVR up and running.
As for being on 1.28... oh yeah, I developed an imaging process with a template to make it easy for us and it was so reliable we just kept using that across all our systems - which were pretty much all the same hardware. It's been solid for years. A testament to the progress of ZM I would think.
I'll report back how things turn out in a few weeks once i've had time to actually get a vNVR up and running.
As for being on 1.28... oh yeah, I developed an imaging process with a template to make it easy for us and it was so reliable we just kept using that across all our systems - which were pretty much all the same hardware. It's been solid for years. A testament to the progress of ZM I would think.