ZM on ESXi: config and disk questions

Post here to ask any questions about hardware suitability, configuration in ZoneMinder, or experiences. If you just want to know if something works with ZoneMinder or not, please check the Hardware Compatibility sections in the forum, and the Wiki first. Also search this topic as well.
Post Reply
winstontj
Posts: 28
Joined: Tue Aug 06, 2019 7:56 pm

ZM on ESXi: config and disk questions

Post by winstontj »

First post so go easy on me. I've read the rules, FAQ's, Wiki, etc. I read the part about how to do the math for memory calculator too. (but there's not much out there on CPU/cuda, hdd, etc.) I've done a bunch of work (professionally) with streaming udp and cuda in low latency environments but have never worked with tcp or cameras. I think I have a basic understanding of how ZM works but trying to avoid making ignorant/n00b mistakes. Trying to spec out and config a 6-8 camera system for our house. I'd like to run ZM on a (paid) ESXi VM and push data to vSan disk array.

High level: I want to build a rock solid ZM virtual machine and I don't want to hammer on my disks. I can figure out the networking stuff. Wondering about VM specs/configs, how ZM writes to disk, what do people do for file management, how compressible is the data, how does ZM address logical cores (advertised cores from HV to guest OS), etc. (is it same as every other similar kernel OS)

Going to have 6-8 poe ip cams running over 5e to a catalyst poe switch (no wall-wart injectors). After reading about sv3c "et phone home" I'd like to keep my camera network as locked down as possible (zero outbound NAT, locked down inter-vlan permissions, etc.).

Wondering about switching. I have pfSense router on stick with Cisco L3 core and everything else currently L2. PoE switch is Catalyst 24P 10/100 and 2x GbE SFP ports. Would appreciate feedback on keeping storage within vSan vlan or using smaller nas box on same L2 network as ZM/controller.

ESXi: I have paid ent. plus. There are a couple different ways to approach a ZM VM. Does anyone have success/failure stories?

Cuda/gpu: Does it make a difference? I have enough identical cards to keep my HA cluster but I've never done GPU/cuda within esxi. Last time I looked into it was back 5.x or 6.x and we ended up doing our cuda stuff on bare metal. I don't think esxi lets you do cuda stuff in ha clusters. I'm sure it's possible. Maybe if you have dedicated gpu to entire cluster? I can figure out "how to GPU", really asking "is it worth it" or "does it make a difference".

With udp data I mirror ports at switch then put one copy into ramdrive and second copy gets written to ssd. At 12:01am the file is written to disk, checked back to data vendor and then if good, compressed and written to a big (slow) disk array. Good news about that is data vendor also keeps copy of historical files. Bad news is our cameras don't keep backups. How do I want to handle tcp coming inbound? Do I need to do as described above (mirror ports, ramdrive for zm, second zm to write to ssd) then write, compress and archive daily files during an overnight process?

I'm trying to minimize random writes to a bigger hdd array. How do you guys/girls handle packaging the data stream into bigger chunks so they can be converted into sequential writes vs. random?


Happy to share hardware/software specs. Didn't want to write a TL;DR novel. Happy to answer any questions or provide more detail. Thanks.
winstontj
Posts: 28
Joined: Tue Aug 06, 2019 7:56 pm

Re: ZM on ESXi: config and disk questions

Post by winstontj »

operat0r wrote: Tue Aug 13, 2019 5:45 amYah sounds like your going hard core here I would be interested in chatting and screensharing we may can help each other out...DM me
you may look into something like https://github.com/aol/moloch ( https://www.youtube.com/watch?v=beljO4PJwdw ) not sure what the hell you are trying to do or what kinda show you running over there ...
Hey thanks for the reply and sorry it took me so long to respond to this. I've been doing a good job of blowing up ZM virtual machines so a little busy :D My '/' (home) partitions have been too small so it just blows up the VM eventually. I put it on Ubuntu 18 LTS and let the OS auto-partition. I've been playing with CentOS and setting manual partition sizes as well as looking at your reply/post and trying to direct mappings and paths to where I want things to go. I guess I could do manual partitions during setup on ubuntu too... No idea why I chose CentOS, I guess maybe because I feel like it's more stable and I know the OS a little better.

My esxi stuff is old. It's 5.5u3 Ent. Plus (paid) on xeon 5600 cpus. I have a bunch of cuda capable cards that were decent a few years back but they are pretty old now too. I've done some cuda stuff with them but only on bare metal machines and esxi with pci direct pass-through. If I remember correctly, I don't think esxi 5.5u3 lets you do device pass-through and HA clusters/virtual machine migrations. Even if you have identical cards on every host machine in the cluster I still think it won't let you do it. I think maybe later versions esxi v6.5+ and 7 let you do more gpu stuff but I believe that the hypervisor controls the gpu and there's no direct device pass-through. (I could be totally wrong about that) I thought I had read somewhere that with the combination of nesting (running virtualization inside a virtual machine) and with graphics/gpu integrated into the processor, that hypervisors (at least esxi, citrix, msft) can now handle gpu stuff with high availability, clustering and machine migrations, etc.

My disk setup is pretty straightforward: each host has a small hardware raid0+1 ssd array for guest VMs and vSAN handles all the platter drives via infiniband cards. Each host has a couple ssd's in a stripe/raid0 for on-demand storage if needed, just as an attempt to keep the reads/writes on the main guest vm storage down to a minimum. I think I'm still mentally scarred from back when Seagate had all those 2tb and 3tb drive failures. But also, rebuilding an array with 2-3tb drives took forever. I don't want to think about rebuilding storage arrays with 8tb drives.

I'll try to repond to each of your points.

Thanks for the links and tips about file paths. That's helped a lot both to send data where I want it as well as for partition sizes when I setup a new machine. I've blown up 5 or 6 ubuntu vms because partitions ran out of space and I don't know ZM well enough to know what files I can delete without breaking it.

flat image vs. h264: I haven't looked yet. I can't keep a machine running for longer than 12-14 hours before it blows up so that's lower on the list of priorities. I don't mind the zm event playback. I'm running zm and shinobi in parallel and liking zm much better. Not exactly sure about the gpu yet. I'd like to avoid it if possible (for simplicity, fewer points of failure) but if it offloads a ton of memory and/or CPU use then maybe I'll try it. That's also a little further down the list.

Not sure about the filter to delete stuff. When I asked about file management I was asking more about physical disks, locations, etc. Right now I have operating system plus ZM software running on a (very poorly partitioned) 64gb virtual hard drive. My intent was for that to be ONLY for the Ubuntu server OS and ZM software to live on. I gave the ubuntu virtual machine three extra local (virtual) drives, one is 10gb (log files and event logs), one is 64gb vhd that's currently unused and the third is a 500gb virtual drive for the camera files. Issue I keep running into is that home directory ('/') keeps blowing up. (no space left on device) I believe that's probably what I want to send over to the 64gb vhd but I still don't have a handle on what files are going where. Short-term I'll boot into a live CD, make the partition bigger then start digging into the files a little more.

Bigger picture file management was about locally attached storage or over the network. My home network is pretty solid. I used to wire new office builds and setup all the cabling/switching/telco closets so running copper cable is easy. I have as much as I can over copper, legit firewall, core Cisco L3 switch with everything else Cisco L2 (or esxi vds), etc. We have several Unifi AP's throughout the house and I try to keep as little as possible on wifi. I'd never mount a permanent (mount on boot) network share over wifi... I assume what I have at home is probably very overkill compared to most houses and people using ZM in a residential setting but was wondering what other people are doing.

One of my concerns is power outages. The NAS I had intended to use for "camera storage" is an old d'link dns-321. It's not WOL capable and there's no bios to set the power state (on power failure stay off, resume to last, power on, etc.). My bigger NAS'es are either D'link DNS-1250/1550 (running Linux with actual RAM and Intel Atom processors) or FreeNAS machines. While I have plenty of space on those machines, my worry is that they are wd red nas (not pro) drives and ZM will create a state of constant write to the drives/arrays and therefore they will never have the opportunity to spin down. Part of me says I should dig up an old 2U from the graveyard, throw some drives in it and call it a day. I could figure out a way to HA-sync one ZM on bare metal and one virtual ZM, I probably have about 4 hours of battery and I'm working on an auto-tx switch for the house generator.

For file management my questions were more about physical setup (local, network, backups, HA, etc.) as well as about power failure events and any other crazy things I haven't thought of yet.

I plan on keeping everything for at least 180 days (six months) then purge after that on a daily basis.

Regarding compression: As I just said, I want to keep 6 months of almost constant recording. Maybe I should approach this differently. Rather than thinking I can physically compress the files and archive them, perhaps I should look at not keeping so much data. Maybe something like keeping 30 minutes prior to and after each event. This is a result of an issue with a neighbor and our goal with these cameras is to protect ourselves. Part of what I'm interested in capturing are things like time frames when people come and go. We have been accused of essentially vandalism to our neighbor's yard/property. There have been ongoing accusations to the point that it feels targeted. Being able to go back and see what cars were in our driveway, who left, when and what were we wearing, etc. and then comparing those times to our credit card statements, text messages, phone logs, etc. has been extremely helpful. (basically neighbor can accuse all they want, tell us when it happened and we'll tell you where we were)

As a side note: We have three Unifi AP's in the home. I installed the access points equally distant from each other with the center AP in the back of the home and the two other AP's in the front sides of the home. The guest SSID makes you click a button to accept terms but has no password. It is extremely locked down with content filtering and isolated/sandboxed at the firewall from the rest of my home network. Seeing a mac address hit one AP and move across the other two, combined with radio signal strength lets me know device direction of travel and a fairly accurate proximity/distance from the AP.

So given what I just said above, I want to keep as much as possible for as long as possible without needlessly filling up disks. It would be nice to archive and highly compress some of the older files if possible however if they are not very compressible I'll probably rethink the camera capture/record windows and see if I can cut down overall size of data that way.

would be nice to have just a few videos for each day /thumbnails etc.


Yes. Basically what I'm after to save/record on a daily basis would ideally be n-minutes prior and n-minutes after an event. I realize that's more complicated than it sounds (have to constantly record during the "n-minutes prior" period and then first-in-first-out as it purges... then do you duplicate data or combine events if they happen within an n-minute interval... But yes, I would love to work on something like that whether it be a crude script/cron job or something sexy like an add-on or package for ZM with gui.


logical cores: It's Ubuntu running ZM software/package. When I asked the question I didn't really understand ZM. (I still don't understand ZM :D I wasn't sure (and still don't know) how ZM does motion sense processing... I'll have to look at the ZM code. To be honest, I'll ask a friend to look because it'll probably be over my head. All I know is that different things approach stuff differently. I'm not exactly sure how ESXi displays hyperthreading to a guest machine. Each of my nodes has 12 cores and 12 HT. I know what vSphere shows on the GUI but I'm not sure if a guest VM sees "a core is a core is a core" or if a guest knows the difference between a HT and a true core. Combine that with dual sockets and memory allocation and things get fun. I'll ask someone (and happy to report back if anyone cares).

What I mean by dual sockets is: does ZM care between two sockets, 2 cores each (4 total) or 1 socket with 4 cores? I run a few things that tend to perform better with fewer cores on 2 sockets (but really I think that's about addressing memory)... Was just curious if anyone has gotten down to that level with ZM.

Network bandwidth (so far):
We have two S3VC cameras (one bullet, one dome) installed/running so far and I'm running cable for several more. I feel like your bandwidth is high. I'm seeing closer to ~5.5Mb for two cameras going to both a ZM and Shinobi VMs (running both at same time so I can compare). We were blowing up RBP3B+ with both ZM and Shinobi. The VMs I made are 4 core, 4 gigs ram, 64gb vhdd (ssd raid 0+1). Have flash cache and a bunch of other stuff enabled, it's not quite bare metal but it is close. Bare metal is in the lab environment so CPUs are whatever the 2.66ghz xeons are --maybe x5650 or x5660.

I don't have much experience with stuff that many home/retail people use. I don't know anything about Docker and have never used a Plex server. Those numbers don't look terrible but I don't know much about ZM, have guys who max every bit of CPU/memory/disk I give them and have no clue about your settings. (I don't have a clue about ZM settings on my end so couldn't comment on yours)


GPU:
I think I'm going to hold off for a bit with gpu. I'd really like this to be in a vm within a HA cluster and I'm willing to take a CPU/ram hit by not going GPU. Maybe once I get everything tweaked and running I'll compare two bare metal boxes with/without GPU and see what happens. GPU would have to really reduce cpu and increase performance for me to consider two mirrored standalone machines running zm with gpu.

Does anyone have two ZM machines running in failover/HA mode? (I realize that's more of an Ubuntu thing than a ZM thing) I've been thinking about running one ZM on bare metal and one as VM. If power goes out the bare metal ZM would turn on and sync with the ZM virtual machine then if batteries need to turn off my ESXi hosts the bare metal ZM box (and poe switch) can stay on separate from my ESXi stuff.

not sure what the hell you are trying to do or what kinda show you running over there ...


Trying not to get arrested and honestly at this point it's almost a game.
Neighbor makes accusation: You threw paint thinner over my fence
Me: You just had your house painted
Neighbor: paint thinner is now "chemicals"
Me: OK when did this happen exactly?
My other half: we either weren't home or he was with me that whole time
Neighbor: he must have had/paid a friend to do it while you weren't home
Police incident report: accusation of neighbor putting gasoline into neighbor's fish pond (note "chemicals" turning into gasoline)

To my neighbor: I run 100LL (100 octane low-lead) airplane fuel in my mower, chainsaw, string trimmer, generator, etc. It has zero ethanol and therefore (if kept properly) has a shelf life of over 6 months. It also costs about $6/gallon. Neighbor: You aren't even worth the $2 it would cost for me to waste that much gas... Also to neighbor: If I wanted to throw gas over the fence I would have chosen something with a high percent of ethanol in it because alcohol absorbs (chemically it attaches to) water and therefore fuel with ethanol in it would have done damage to the pumps and filters in your illegal water features. Look at our credit card statements and video feed at gas pumps over last ~5 years. Did I ever put pump gas into a can during that time? Look at video feed at corporate entrance at airport every other month for the last ~5 years. Have I been consistently buying AVGAS? Neighbor: You aren't worth the value of the "chemicals" you found on your property. Bring it.

In a nutshell that's what is going on...

All I want is to record (AND CONTROL) my own data so I can protect myself while at the same time she and I can maintain privacy within our home and on our property by not hosting in the cloud.
Post Reply