Zoneminder 1.36 not functional

Discussions related to the 1.36.x series of ZoneMinder
dhmsjs
Posts: 23
Joined: Fri May 28, 2021 1:53 pm

Re: Zoneminder 1.36 not functional

Post by dhmsjs »

Just posting this follow-up: I did swap the SD card back into the Pi4 8GB and on that hardware the system runs rock-solid with multiple cameras. Not fragile at all.

So for whatever reason on the RPi 3b, ZM 1.36.5 on Ubuntu 20.04 is fragile (runs, but seems to crash whenever it starts doing meaningful work), while ZM 1.36.0 on Debian 10 Lite seems to handle the exact same work load quite reliably.

Don't know why I thought the RPI3b had 2Gb of memory, but it actually only has 1Gb. So in light of that I'm thinking it's fairly impressive that ZM 1.36.0 runs as well as it does given the low memory resources of the RPi3b.

I'm still curious as to why 1.36.5 on Ubuntu would be so fragile compared to ZM 1.36.0 on Debian in this small-memory situation, but don't really have the time or resources at the moment to look for an answer. Perhaps the change to "waiting on buffer full" vs just dropping frames is the main reason?
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Zoneminder 1.36 not functional

Post by iconnor »

In 1.34 we use an in memory table for monitor status, which seemed like a good idea. Turns out that the memory table type only supports table locks, so every update locks the entire table which did not scale well. So in 1.36 I switched to a regular on disk table, which scale much better. However some people have complained about the extra load on their mysql server and disks. Maybe the answer is to give each monitor it's own status table which seems like a hack, but hey if it works... or perhaps when we have websockets we won't need a monitor status table.

So you might try some db tuning, I suggest starting with my mysqltuner says.
dhmsjs
Posts: 23
Joined: Fri May 28, 2021 1:53 pm

Re: Zoneminder 1.36 not functional

Post by dhmsjs »

Thanks for the details. Will explore db tuning as you suggest.

Gotta say though the RPi4 8GB & Ubuntu 20.04 does seem to be doing quite well with ZM 1.36.5. I am now pushing it to avg cpu loads of 100% (and over -- says 4.16 at the moment) and it doesn't blink. No NaN bug on Ubuntu either.

Just for reference 4 cams MODECT at 720p 10fps with Camera Passthrough _and_ saving JPEG Frames + Analysis Images (I know not typical) plus recording stats puts cpu load in this range on this RPi4 8GB.

Definitely needs fan cooling as noted in other threads. Approx $100 US price is pretty reasonable!
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Zoneminder 1.36 not functional

Post by iconnor »

Yeah that's pretty cool. I was doing 3 720p cams passthrough on a 3b. It has since died though.

I'm a little excited about the future of zm. depressed about the current state of things though. 1080p seems to be the limit of what we can do. Anything else takes too long to decode/translate/motion detect even on modern processors.

So once we implement: h264/5 live view, motion detection on YUV, motion detect on substream, fixing linked monitors UI. Then ZM will be amazing on any platform.

It's a lot of work though, but not a lot of funding ATM.
mikb
Posts: 586
Joined: Mon Mar 25, 2013 12:34 pm

Re: Zoneminder 1.36 not functional

Post by mikb »

bbunge wrote: Sun Aug 22, 2021 4:48 pm
ocascante wrote: Sat Aug 14, 2021 9:19 pm Hello,
I have been a Linux consultant for 36 years ...
I can not match the years of consulting for Linux ...
Nor could Linus Torvalds ... 36 years puts the true creation of Linux at 1985 or before. This is big news!! :)
dhmsjs
Posts: 23
Joined: Fri May 28, 2021 1:53 pm

Re: Zoneminder 1.36 not functional

Post by dhmsjs »

iconnor wrote: Thu Sep 02, 2021 2:55 am ...I'm a little excited about the future of zm. depressed about the current state of things though. 1080p seems to be the limit of what we can do. Anything else takes too long to decode/translate/motion detect even on modern processors.
Nothing to be depressed about. Limits like that just mean we need to overcome them in other ways.
iconnor wrote: Thu Sep 02, 2021 2:55 am ...So once we implement: h264/5 live view, motion detection on YUV, motion detect on substream, fixing linked monitors UI.
^^ Good examples of "other ways."
iconnor wrote: Thu Sep 02, 2021 2:55 am ...Then ZM will be amazing on any platform.
IMHO ZM already is.
iconnor wrote: Thu Sep 02, 2021 2:55 am It's a lot of work though, but not a lot of funding ATM.
I think it is easy for those of us not deep in the details to not recognize how much work it really is. But (from the perspective of a new user anyway) what that effort has created here and now is still quite impressive. ZM is more than just another security system. It is a very powerful, flexible set of tools for managing live video streams. I for one appreciate that difference a whole bunch.

FWIW I have contributed some already and will continue to as I can. To anyone reading this: you should too! It's easy and no amount is too small. Larger amounts are better of course. How much do you appreciate ZM? How much effort would it take for you to re-implement something similar? How much do you spend on coffee every week? Be honest now :-)

Even just $10 from each of the (what is it?) 50,000 current installations could add up to some real money. If you are not actually doing the coding/debugging, what better way is there for you to stimulate further improvements?
sigma_shig
Posts: 3
Joined: Thu Oct 28, 2021 8:01 pm

Re: Zoneminder 1.36 not functional

Post by sigma_shig »

I would like to add my 50 cents to this problem. My configuration:

Code: Select all

ZM 1.36.10.
HP Proliant Server is dedicated to ZM + MySQL
2*Xeon L5639 Hexcore
40GB RAM
Ubuntu Server 20.04 (cloud-init is uninstalled)
3 Cameras
C1: 1280*720
C2=C3: 2592*1944
For All Cameras Video Writer=Camera Passthrough, Mode=MoDect, Analysis FPS=5
And it it really eats up resources Image.
Sure, it doesn't happens for Mode=Monitor, but Modect is required for me. Is it possible to reduce the memory consumption?
sigma_shig
Posts: 3
Joined: Thu Oct 28, 2021 8:01 pm

Re: Zoneminder 1.36 not functional

Post by sigma_shig »

I have removed v 1.36.10 and installed from scratch v1.34.26 and after that manually add cameras. The memory and CPU consumption are dramatically decreased. Image
User avatar
lyallp
Posts: 137
Joined: Fri Nov 26, 2010 2:36 am
Location: Adelaide, Australia

Re: Zoneminder 1.36 not functional

Post by lyallp »

I am a bit of a noob at ZoneMinder.
I am unfamiliar with the throughput specs of your server, but here are my thoughts...
I have been in enterprise linux/unix for longer than I care to remember...

Firstly, 51% cpu on a machine dedicated to ZoneMinder, who cares? It's not doing anything else.
Secondly, 8 cameras (assume colour), * 640*480 Pixels * 3 bytes per pixel (colour) * 15 (frames per second) works out at about 100MB/s
You are using up 1Gb/s network, minimum.
You are using 100MB/s disk I/O to write all these frames.
You are examining 100MB/s for motion, keeping at, lets be generous, 20 frames per camera, in memory.

Get down to the nuts and bolts and think about what the server is being asked to do, do some calculations, is your server capable?

Do you have the sheer bandwidth available?

Are you surprised that things 'hang' if the server is so busy writing to the disk it doesn't have time to look at you?

As suggested by someone else, try halving your frame rate, see if that helps.

I am unsure if you can monitor for motion at a low framerate and only record events at a higher frame rate.

I think you are being a little harsh in saying that 1.36 is non-functional, in this case.
...Lyall
sigma_shig
Posts: 3
Joined: Thu Oct 28, 2021 8:01 pm

Re: Zoneminder 1.36 not functional

Post by sigma_shig »

I know, that nobody cares about 51% of CPU load on a dedicated server, but when I can't even type a command in ssh it is not a good deal. And now I have just 3 cameras configured, but it is just the start of my project. The full capacity is 15+ cameras and I can't connect it based on this resource consumption. And the main question: I posted htop screenshot from v1.34 for the same configuration. This version has the same functionality (from my point of view) but consumes several times fewer resources. This means that such insane consumption was added later.
User avatar
lyallp
Posts: 137
Joined: Fri Nov 26, 2010 2:36 am
Location: Adelaide, Australia

Re: Zoneminder 1.36 not functional

Post by lyallp »

not being able to type in ssh, it does sound like the server is I/O bound, given it has 49% CPU available.

As you are recording all 8 cameras, full time, your I/O will be high, as stated in my earlier post, in the order of 100MB (bytes not bits) per second.

Being new to this, I assume all the cameras are I/P cameras, unlike mine, which is a single USB.

How many network cards does the 'server' have, or is all the network traffic coming down one gigabit(?) network link? In which case, the cameras are going to saturate the network card.

Do you have a 'control' network card and a 'camera' network card, so that if the camera network is saturated you can still access the server via the control network?

What sort of disk storage do you have? Is the primary spinning iron or SSD? 4TB, I am guessing spinning iron.
What throughput does this storage have? Is it RAID, if so, which one? RAID 1 may give speed improvements on spinning iron, not so on SSD.

Maybe setup 'Options -> Storage' as having 2 storage sets, one 'Fast' (SSD) and one 'slow' (Spinning Iron).
Events initially store on SSD, but SSD is expensive, so we off-load them to slow storage with a filter.

Memory, not interested. Focus on the above two first. Once you have them out of the way, then you look at memory and CPU.
If you where using swap, then you don't have enough physical memory, but you are only using 250MB odd of swap, which is in significant in the grand scheme of things. It's a dedicated server, that is what the memory is for. Of course, memory will be sized by the number of cameras, size of frame, colour or black and white, bytes per pixel, frames per second, how many frames you have configured before an event fires, how many frames are kept in memory during processing.

From what I guess of your pc configuration, it is under spec in disk and network, for the use case you are trying to establish.
...Lyall
jonthewise
Posts: 4
Joined: Sat Oct 30, 2021 9:17 pm

Re: Zoneminder 1.36 not functional

Post by jonthewise »

Just wanted to post my two cents on this.

I have 3 IP cameras configured. They all run POE into an Aruba switch, and I had them set to mocord (always recording, plus alarm on motion) at 1080/30fps.

I'm running CentOS7, and using RPMFusion to install zoneminder.

On my old setup, I had a couple of IBM M3 servers, one with FreeNAS on it, and the other with ESXi 6.5. My FreeNAS setup had 8 gig fiber uplinks to the ESXi server for the datastores, and a pair or 1G ethernet links teamed for NFS(for linux mounts) and CIFS(samba/windows sharing). The ESXi system merely had a pair of 1G ethernet connections running to the switch.

The Zoneminder system was confugured to use 4 cores and 32GB of RAM - I think, I have since upgraded, and I'll get to that in a minute - and I had zero issues with recordings or being able to leave a console open to montage and watch all three cameras all day long. The SQL DB was local on the VM, and then I had an NFS mount with 2TB reserved for recordings.

Then, I got a pair of HP DL380 Gen9 servers with far more horsepower and much newer CPUs. This setup has changed a little, now I have a 10G fiber uplink from each system directly to the Aruba (no teaming for the FreeNAS, but I do have 2 uplinks for the ESXi system) and no direct connection from ESX to the NAS, so all the traffic is running over the 10G fiber links. After migrating everything and upgrading to ESXi7, my zoneminder system was chugging along beautifully for a month or two.

Then I went and run a yum update on the zoneminder server, and it updated to the latest version. Now it's unusable. I frequently see CPU usage warnings in vCenter, even after adding 2 more cores (now 6). I have one camera that stays streaming most of the time, but lose the other two signals constantly (I have verified that I can connect directly to the cameras, they are not the problem) and I've even removed all of my zones and switched from mocord to just straight recording, thinking that motion detection was the problem. It's still unusable. So I went into the camera feeds and switched from 30fps down to 10, just to see if that would help. No dice.

I'm going to actually go ahead and spin up a new zoneminder machine and see about installing 1.34 back on it just so that I can use it again, but I wanted to put my .02 in here - there's something definitely wrong with the way the new release is doing things.
mconfortino
Posts: 26
Joined: Sun Oct 24, 2021 8:21 pm
Location: Rawson, Patagonia

Re: Zoneminder 1.36 not functional

Post by mconfortino »

Hi everybody. Just one more example. We have Zoneminder v1.36.10 running on Ubuntu Server 20.04.3. The server is a Dell PowerEdge with two Xeon CPU E5-2670 v3 @ 2.30GHz, for a total of 48 cores. It has 16 GB of RAM. It is streaming more than a hundred cameras on Record mode. We have Axis, Pelco, Bosch, Vivotek, Sony and Hikvision. We started with v1.36.7, witt ZM slowly ocupping all the memory and finally stopping the server. We thought of going to v1.34, but we couldnt because of this:

Image

(When v1.34 starts, it deletes the Latitude and Longitude fields, so no map)
So we upgraded to v1.36.10 and tried to reduce all buffering to a minimum. We standarized on 800x600, 20fps, 24 bit color..
We set Image Buffer Size (frames) to 3, all others to zero. Cameras in passthrough and not saving jpegs.
Now ZM uses just less than 9 GB, with a load of 8.14 and 154 database connections out of 500 available. It appears to be stable.

The same cameras on a HP Proliant, same two proccessors, 96 GB RAM, Ubuntu Server 20.04.3 and ZM v1.36.7 fills the memory in just six hours and kills the server.
jonthewise
Posts: 4
Joined: Sat Oct 30, 2021 9:17 pm

Re: Zoneminder 1.36 not functional

Post by jonthewise »

Having incredible issues finding RPMs to rollback and couldn't build the 1.34 RPMs myself without more work than I had time to put in. Just got updated to 1.36.10 this morning and things seem to be much more solid again. Will report back if I see any issues, but all my cameras are staying up on the montage screen, and I can review the recordings again.
junior466
Posts: 8
Joined: Sun Mar 03, 2019 7:33 pm

Re: Zoneminder 1.36 not functional

Post by junior466 »

Absolutely agree with 1.36 not being functional. I have been spinning up VMs for the past three days trying every single settings out there to see if the memory errors stop but nothing works. I finally found this thread and what a relief. I thought this whole time the issue was with my install...

About to try 1.34.
Post Reply