Beginner performance questions

[archaic0]

I just found ZoneMinder when looking to replace my old generic NVR device and I'm loving it for the most part. I feel like I have to be missing something though because I'm having performance issues while running on a pretty good PC while others are running this on a Pi fine.

First off, some background. I have two brands of IP cameras (Dahua and Reolink). The Dahua ones have MJPEG AND RTSP support, the Reolinks only have RTSP. Overall, it seems RTSP gives me the best quality, but that's also where I see a performance hit, so I'm wondering how others are running. My cameras are all 3MP and capable of sending 30fps but because I have not been able to get even workable performance at that resolution, I've limited all cameras to 1080P/20fps for testing until I get a handle on this. My target is to get 10 cameras up and running well.

The test PC I'm using is a small form factor PC with a 1TB SSD, Core i7 dual core 3.2Ghz CPU, 8G RAM, running Ubuntu 18.04 LTS. I have alternate hardware slated for the final build that is a Core i7 4770 quad-core 3.4/3.9Ghz CPU, 16G RAM, three 2TB SSDs. Compared to a Pi, both of these should be cruising but I'm maxing out CPU pretty quickly.

One question I have is about the storage options. Using an RTSP source from a camera that is already encoding with H264, I understand I can use the H264 passthrough option (and I have been), but I'm unclear if I also need to save JPEGs. When I was trying an MJPEG setup, it seemed like the 'scrub' performance and quality was MILES better using the JPEGs versus the H264 video, but I don't fully understand the relationship between what is being stored and when those things are used in the UI when reviewing video/events. For a time, I was saving JPEGs and passthrough video for one camera and only JPEGs for the other. I could not tell what the effect of that was. Both cameras have live video and playback with stream, frames, and stills available regardless of these settings.

Currently, I have only 2 cameras recording. I am doing full-time recording, no motion detection. So 2 cameras at 1080p/20fps full time. This is resulting in a 30% CPU load on the current PC and maxes out by the time I get to 5 cameras. That 30% is a bit deceptive though, because it's 30% of 4 threads on a 2 core CPU, so it's probably closer to 80% of total CPU performance. Montage view right now seems to be working well, but CPU load goes to 60% when I do that, which basically feels like maxed. Montage Review with two cameras showing is jumpy at 1x speed, 2-5 fps typically, sometimes 10fps, with enough NO DATA sections to worry me.

I was able to get 6 cameras recording with 3 of them using MJPEG and 3 using RTSP with the CPU load coming in at 60%, but there were noticeable performance issues (with 60% of 4 threads on 2 core CPU being basically max again). For example, using the Montage or Montage Review feature caused max CPU, spinning browser load icons, and I had to close the tab and start over to get back to the Console view. Plus, when I was able to review video, there was a ton of missing video, very poor capture/playback rate (can't tell which) of 1-2fps, and lots of video streaking/tearing.

I feel like the performance bottlenecks at the PC level are causing frames to be dropped both for recording and playback and that gives me an overall impression of not working well. Hopefully someone can help highlight the errors of my ways and point me to the performance sweet spot.

(The annoying thing is that my old NVR is a generic no-brand thing and I had 16 IP cameras going at 1080 with no issues other than hating the software and no mobile viewing options. How is it that such a generic box with what I'm sure is an ATOM single or dual core 1Ghz CPU is able to pull this off while I'm struggling on an i7??)

[archaic0]

I turned off saving JPEGs in my storage settings and dropped my CPU load to 21%, then dropped my fps to 10fps on both cameras and got CPU down to 16%. That doesn't sound terrible, but for only 2 cameras that's an estimated 80% to do all 10 cameras and that's while being limited to 1080P and 10fps.

I'm not sure what my real expectations should be for this, but if people are pulling off even a few cameras on a Pi, then it seems like my performance numbers are WAY off from what is possible.

[archaic0]

I don't know much about the Pi world, but I found this blog about running ZoneMinder on a Pi 3 that seems to suggest that defining 'success' is a bit different than I was thinking. limiting to a single 1080 camera at 5fps is abysmal for a home NVR.

Hopefully someone can shed some light on all of this for me. I really want to like ZoneMinder as my home NVR solution.

I've done some light reading on GPU acceleration to see if that could help, but I'm finding mixed results on that topic. Most things I'm reading say that while they were able to see better performance with GPU acceleration enabled, it wasn't much to write home about. The gains were not double or triple performance for example, but 10-15% maybe.

To get a 10 channel 1080 NVR capturing at 30fps, is it normal/expected to throw a $5,000 server class dual socket 16 thread CPU set of hardware at it? Are others running ZoneMinder on more modest hardware just accepting 1-5fps on a handful of cameras?

If so... how is it that a little no-name generic black box NVR for a couple hundred dollars can do so well compared to this?

[iconnor]

cpu % is typically per core. So if you were maxing out 2 cores, it would be 200%.

There are lots of inefficiencies that will get addressed in subsequent versions, but generally h264 instead of jpeg storage takes a lot less storage, but requires more cpu. And ram.

There is always a tradeoff.

Don't knock the pi too much. 4 real cores, now 8Gb ram. built in h264 decoder means that it is really a powerhouse.

[archaic0]

I’m using an SNMP monitor for the CPU numbers. Locally Ubuntu in the TOP command shows 2.5, 3.5, etc referencing the cores like you mention, with the details on each process showing 150% on a couple processes at the top.

Not knocking on the Pi, great device. I’m shocked that people seem to be happy with an NVR limited to a couple cameras at 5fps. That’s not really much more useful than just using the built in record capabilities of IP cameras to an SD card.

[iconnor]

nvrs tend to be cheap crap with limited capabilities, backed by hardware based decoding. They tend to actually be very nice hardware crippled by dirty hacks on a linux base.

I would love to be able to spend more time hacking and then providing alternate firmware for dvrs...but it is not to be. at the moment.

I realise that I have been annoyed at this since at least 1995. The hardware has always been capable of more than we have been able to access. drivers have always been the issue. THe proprietary drivers worked fine in windows... but there was always more to be done there. The linux drivers were non-existant. Nowadays the hardware is incredibly powerful... but we still dont' have access to the full power of the hardware and everything is so fragmented that we still can't use what is sitting there waiting to be used.

most modern cameras could run ZM on camera. I have spent the last 10 years putting the framework in place for a distributed on-camera zm. but we are still a long ways off, and no one is willing to fund any of it.

[archaic0]

I hear ya. I got the impression that ZoneMinder was pretty mature and capable but maybe it’s just still a bit underpowered (from underfunding). Great for what it is so far, but maybe a ways to go?

Can you help me set some expectations? Is capturing 10 cameras at 1080 attainable, but maybe only at a few fps?

[archaic0]

I think I just need to move on to other options. A dozen other forum deep dives, YouTube videos, and a few others who went dow my same path convinced me I need to move on from ZM.

A couple retail options are probably my best bet, blue Iris or Xeoma, both of which have users running 10-20 1080 to 4K cameras at 20-30fps on hardware similar to mine. Every option has its glitches, nothing is perfect, but that’s where I’m headed next.

Maybe ZM will be viable for me one day but it’s just not there yet. Thanks for the input!

[incompetent-nerd]

archaic0 (and anyone else who is interested in other peoples CPU performance),
I have two zoneminder servers running; one for home and one for work.
Both are running CentOS 7.x with Zoneminder 1.34.12 (H.264 passthrough)

My home server has 9 HikVision 8MP cameras running at 2560x1440 25FPS.
Server is a used 64-core server I bought off eBay a couple of years ago (Dell PowerEdge R815 with Opteron 6386SE CPUs running at 2.8GHz)
Server runs VMWARE and the Zoneminder VM has 16 cores assigned to it.
Inside the VM, O/S reported load average is: 3.86, 3.76, 3.81
Cacti graphs show an average of 31% and a high of 46% CPU usage.

Work server has 17 HiSharp 8MP cameras running at 2560x1440 30FPS.
Server is one of our old retired database servers (HP ProLiant DL380p Gen8 with 16 Hyperthreaded Xeon E5-2665 CPUs running at 2.40GHz).
We wiped it clean and installed CentOS 7.x on bare metal.
O/S reported load average is: 14.91, 14.57, 14.68 (O/S sees 32 CPUs).
No Cacti info for this.

Mildly interesting...
During the winter, the cameras use the same amount of network bandwidth day and night.
During the summer, the nighttime bandwidth is double the daytime bandwidth. I presume because all the bugs attracted to the IR, causing big white streaks all night long.

[SkippyDo]

Insects and IR... I disabled IR in my cameras because insects were attracted to the lights. I can do this because my cameras are all near motion sensor lights, which are tripped when anything (other than insects) are wiggling out there.

[MrMagoo]

I dont think arcaic0 gave this release enough evaluation time.

Having eventually given in to the nag-box reminder to upgrade and gone from 1.32 to 1.34.16 I can see a much better product, once you find your way around.

One thing that tripped me up was the new (to me) playback rate pull down, the default seems to be 1 frame every now and then as the box is blank.
Having selected 1x playback rate its snappier than I remember, thumbs up!
The 4x playback needs to launch the "Benny Hill tune" (possibly a UK only reference )

I do like the timeline feature as well.


Maybe its time to put up the last two cameras.

This is currently running on a Dell SC1435 platform in 16Gb with two quad core Opterons which is about 12 years old.
So old it has a CD drive not a DVD drive to boot from

smpboot: Total of 8 processors activated (46289.46 BogoMIPS)

Upgrading to to 32Gb is more expensive than just buying a faster system with 64Gb so its running until on the SC1435 it breaks or cannot cope.

I have my eye on another Dell workstation rather than a server, individual cores are slower but having three times as many has benefits.
The most recent purchase is a refurbished desktop with "Hexacore processors" it's a few years old and I use it to run KVM test systems.

The SC1435's were used for ESXi testing but are not compatible beyond ESX 6.3

smpboot: Total of 24 processors activated (124529.12 BogoMIPS)

In short dont dismiss Zoneminder from a discussion you read elsewhere, you may have missed something when you set it up.

[iconnor]

RAM requirements in 1.36 are expected to be a lot less than current.

[archaic0]

incompetent-nerd

Thank you for sharing your specs. Those servers are certainly in another world compared to a desktop i7 CPU, so it makes a ton of sense why your experience is different than mine.

I found ZM by way of PI users so I came into this with the expectation that it was able to run on modest hardware. What I've learned however, is that while you are able to run 1 or 2 cameras on a PI at low framerates, you cannot run a whole-house NVR on a PI or seemingly not on desktop hardware either. To run a whole-house NVR, you need server class hardware like what you have. Doesn't have to be brand new, but it does need to be much more than a single socket quad-core.

In the bubble of ZM, this isn't unreasonable. Video is a lot of work, so a respectable beefy server requirement makes sense. However, outside of the bubble of ZM, you find that the power required is by no means consistent with this idea. In the embedded NVR world, you can get 16 channel generic units with minimal (specialized) power, that handle 16 cameras at 1080 or 4k even for a few hundred dollars. Those are highly specialized for the task, so it's not a 'fair' comparison, but it is what I'm trying to replace.

In my case, I installed Windows on my same hardware, installed Blue Iris, started adding my cameras at full 5MP and 8MP resolutions, and in a few minutes had 9 cameras running with continuous recording and only see about 20% CPU load. Compared to having to limit ZM to 1080 and 5 FPS to get anywhere close to 4 cameras being 'acceptable' performance.

I hope this doesn't come off as me 'trashing' ZM, because that's not what I'm trying to do. But rather, I'm just trying to set realistic expectations. If you have desktop-class hardware, then it appears that ZM is not the way to go unless you only have a few cameras and can live with low resolutions and low framerates. This can certainly improve as the product matures, and I'm sure it will. Today, it just doesn't appear capable of meeting my needs on the hardware that I want to use.

And my choice of hardware is specifically not server-class because I don't want the size, heat, power draw, or noise of server-class hardware. If there was no solution available for desktop-class, small, light, quiet hardware, then I'd have to change my plan, but it turns out there is a good option that works very well.

MrMagoo

While I can appreciate that spending a lot more time with ZM may produce some improvements, I can't imagine that any amount of time tweaking could take me from a limit of 2-4 cameras running at reduced resolutions and framerates to 11 or 12 cameras running at full resolution and framerates. At one point I limited myself to a single camera at 1080, 10 FPS, pass-through 264, no snapshots, continuous record - and while this was fine by itself, it was already causing over 10% CPU load. Each camera I added caused a fairly linear increase in load, so there was no way I was going to make it to 9-12 cameras even if I accepted a limit of 1080 and 10 FPS.

You also have server-class hardware. I'm not sure exactly what the performance gap is between that older Opteron CPU is compared to a last-gen i7, but regardless, it seems clear that my issues with ZM are mostly the hardware that I'm using.

I didn't dismiss ZM strictly from reading. I left because it wasn't performing for me when I used my hardware and my cameras no matter what settings I turned off or how many compromises on resolution or FPS that I made. I then questioned why I couldn't get it to perform and that lead to the conclusion that I was using the wrong tool for the job on my hardware after seeing that I was not alone and after seeing others who were having success, but on vastly different hardware.

I'm sure I'll check in on ZM from time to time. I expect that if the developers are passionate about the software, it will continue to evolve. But for now, something else is fitting the bill for me.

[incompetent-nerd]

archaic0,

Not sure if you saw this or not.
viewtopic.php?f=9&t=27537

Wish you the best in your search.

[archaic0]

I did not. Definitely an interesting workaround.

He doesn't mention his hardware, but in his top command I see 2.5 to 3.5 usage with just those 6 cameras, so I think that would be basically maxing out my quad-core well before I got to 12 cameras. In my case though, I only really care about continuous record, so there may be some room for making this idea work on my hardware.

Might be a fun Saturday experiment. Thanks for sharing!