ZMC not analyzing random monitors

Previous development branch now released as 1.36
Locked
kd4pbs
Posts: 11
Joined: Wed Nov 11, 2020 9:54 pm

ZMC not analyzing random monitors

Post by kd4pbs »

Good day, Zoneminder community.

Long story (TL/DR version):
Right now I am on the latest build as of this morning. Apr. 7, 2021; 1.35.23, and still experiencing the same problem. We have been maintaining this system with updates fairly regularly, at least once every week or two, and I can only say that this issue was first noticed about three to four weeks ago. It's an Ubuntu 20.04.2 system. This system was set up new about five months ago, and up until this issue started, we have been having no problems with it since the first small issues I discovered before it went live.
The problem is that some of my monitors won't detect motion and record. I have a nine camera system, and only four or five of the nine monitors will decide to detect motion. If I let ZM run for a day or so, those same four or five monitors will continue to analyze, and the same four or five monitors which weren't being analyzed will continue to not be analyzed, so it doesn't appear that they spontaneously start working and fail working; when I start Zoneminder, it so far appears that it will be those same monitors with problems. They play fine in the console window, and look fine on ZMNinja, but the monitors which have problems simply refuse to detect motion.
If I restart Zoneminder, chances are very good that the monitors which don't get analyzed will change. If I look at the process list, all nine instances of ZMC will be running and consuming around the normal amount of CPU that they always have.
The cameras are IP based h.264 devices. They were originally set up as camera passthrough for recording, so I was only storing the mpeg files for events - this was successful for several tens of thousands of events over the past handful of months that the system was in use.
To troubleshoot, I changed to encoding and saving jpeg files, changed image buffer sizes to big and small amounts, dropped the Analysis FPS to 1, then 5, then back to default (empty), deleted and recreated all the monitors, toggled the "Analysis Enabled" option for each source, and basically diddled with everything I could think of, including cussing and shaking my finger at the server to no avail. The system still exhibits this problem.
Thinking that it was something I did wrong, I even totally wiped everything back to just the bare metal Ubuntu system - I removed Zoneminder, and dropped the zm database to no avail. Thinking that it was some really really strange thing, I then removed MySQL and Zoneminder in total and then installed a clean ZM install. All for naught. The system still exhibits this issue from a fresh, clean install.
Here's a typical snippet of the events in the log showing that ZM is failing to analyze some of the monitors:

Code: Select all

2021-04-07 10:28:14	Breakroom: 45800 - Analysing at 0.00 fps from 0 - 0=0 / 1617805694.681905 - 1617802641.165685 = 3053.516220
2021-04-07 10:28:14	Front Hallway: 45800 - Analysing at 15.02 fps from 45801 - 45701=100 / 1617805694.661421 - 1617805688.003812 = 6.657609
2021-04-07 10:28:14	Breakroom: 45800 - Capturing at 15.01 fps, capturing bandwidth 492309bytes/sec
2021-04-07 10:28:14	Front Hallway: 45800 - Capturing at 14.99 fps, capturing bandwidth 471484bytes/sec
2021-04-07 10:28:13	Back Door: 46700 - Analysing at 0.00 fps from 0 - 0=0 / 1617805693.792868 - 1617802574.126049 = 3119.666819
2021-04-07 10:28:13	Back Door: 46700 - Capturing at 14.95 fps, capturing bandwidth 497836bytes/sec
2021-04-07 10:28:12	Breezeway: 45600 - Analysing at 15.03 fps from 45601 - 45501=100 / 1617805692.182630 - 1617805685.528825 = 6.653805
2021-04-07 10:28:12	Breezeway: 45600 - Capturing at 14.95 fps, capturing bandwidth 376377bytes/sec
2021-04-07 10:28:11	Master Control: 45600 - Analysing at 15.09 fps from 45601 - 45501=100 / 1617805691.988577 - 1617805685.362517 = 6.626060
2021-04-07 10:28:11	Master Control: 45600 - Capturing at 14.95 fps, capturing bandwidth 501139bytes/sec
2021-04-07 10:28:11	Lobby: 46500 - Analysing at 0.00 fps from 0 - 0=0 / 1617805691.701587 - 1617802591.443599 = 3100.257988
2021-04-07 10:28:11	Lobby: 46500 - Capturing at 14.99 fps, capturing bandwidth 510878bytes/sec
2021-04-07 10:28:11	Loading Dock: 46600 - Analysing at 15.04 fps from 46601 - 46501=100 / 1617805691.427238 - 1617805684.776688 = 6.650550
2021-04-07 10:28:11	Loading Dock: 46600 - Capturing at 15.05 fps, capturing bandwidth 515214bytes/sec
2021-04-07 10:28:08	Telecom Entrance: 46600 - Analysing at 14.96 fps from 46601 - 46501=100 / 1617805688.482486 - 1617805681.797649 = 6.684837
2021-04-07 10:28:08	Telecom Entrance: 46600 - Capturing at 15.01 fps, capturing bandwidth 501294bytes/sec
2021-04-07 10:28:08	Breakroom: 45700 - Analysing at 0.00 fps from 0 - 0=0 / 1617805688.042568 - 1617802641.165685 = 3046.876883
2021-04-07 10:28:08	Front Hallway: 45700 - Analysing at 14.99 fps from 45701 - 45601=100 / 1617805688.003812 - 1617805681.330805 = 6.673007
2021-04-07 10:28:07	Breakroom: 45700 - Capturing at 15.07 fps, capturing bandwidth 508973bytes/sec
2021-04-07 10:28:07	Front Hallway: 45700 - Capturing at 15.01 fps, capturing bandwidth 513055bytes/sec
2021-04-07 10:28:07	Back Door: 46600 - Analysing at 0.00 fps from 0 - 0=0 / 1617805687.102549 - 1617802574.126049 = 3112.976500
2021-04-07 10:28:07	Back Door: 46600 - Capturing at 15.02 fps, capturing bandwidth 506599bytes/sec
2021-04-07 10:28:05	Breezeway: 45500 - Analysing at 15.01 fps from 45501 - 45401=100 / 1617805685.528825 - 1617805678.866396 = 6.662429
2021-04-07 10:28:05	Breezeway: 45500 - Capturing at 15.01 fps, capturing bandwidth 376711bytes/sec
2021-04-07 10:28:05	Master Control: 45500 - Analysing at 14.94 fps from 45501 - 45401=100 / 1617805685.362517 - 1617805678.669643 = 6.692874
2021-04-07 10:28:05	Master Control: 45500 - Capturing at 15.08 fps, capturing bandwidth 511394bytes/sec
2021-04-07 10:28:05	Lobby: 46400 - Analysing at 0.00 fps from 0 - 0=0 / 1617805685.035258 - 1617802591.443599 = 3093.591659
2021-04-07 10:28:04	Lobby: 46400 - Capturing at 15.01 fps, capturing bandwidth 498057bytes/sec
2021-04-07 10:28:04	Loading Dock: 46500 - Analysing at 14.91 fps from 46501 - 46401=100 / 1617805684.776688 - 1617805678.069582 = 6.707106
2021-04-07 10:28:04	Loading Dock: 46500 - Capturing at 14.96 fps, capturing bandwidth 493753bytes/sec
2021-04-07 10:28:01	Telecom Entrance: 46500 - Analysing at 15.02 fps from 46501 - 46401=100 / 1617805681.797649 - 1617805675.140058 = 6.657591
2021-04-07 10:28:01	Telecom Entrance: 46500 - Capturing at 15.01 fps, capturing bandwidth 501678bytes/sec
If I could, I wouldn't mind rolling it back to a version from a few months prior and leave it alone, but it seems there's no real easy way to do that.

Short story:
Help!

Regards,
Matt
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: ZMC not analyzing random monitors

Post by Magic919 »

1.35.16 seems to be a safe haven. You can install that from the .deb file I linked to on another thread.
-
kd4pbs
Posts: 11
Joined: Wed Nov 11, 2020 9:54 pm

Re: ZMC not analyzing random monitors

Post by kd4pbs »

Thanks, Magic. I'll give it a try.
kd4pbs
Posts: 11
Joined: Wed Nov 11, 2020 9:54 pm

Re: ZMC not analyzing random monitors

Post by kd4pbs »

Oops - I don't think I want to do that. I am sure you're a good guy and all, but the last thing I want to do is install a package from an unknown source on a computer at my place of employment. I've been here for 25 years now and I really wouldn't take too kindly to being fired and end up making my family homeless because I installed something on an intranet computer that I knew better than to install. If it isn't on Connor's PPA, I really don't want to touch it.
Anyone else have any ideas?
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: ZMC not analyzing random monitors

Post by knight-of-ni »

If that is the case, then are you sure you want to be running our bleeding edge master branch?
It sounds like our latest 1.24.x release would be more appropriate for your environment.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
kd4pbs
Posts: 11
Joined: Wed Nov 11, 2020 9:54 pm

Re: ZMC not analyzing random monitors

Post by kd4pbs »

Does that mean that the bleeding edge releases could contain malware? This is the reason why I would only run releases that originate from Isaac's PPA and not just some guy's personal archive. I am not concerned with issues related to failing software, and I doubt that some bug would be introduced in Zoneminder which would spontaneously take down our whole intranet. That being said, we were having great service out of the system with a few updates between December and February. The mistake is that I had forgot to tell one of my engineers to not update the zoneminder package every time the system was updated, and it is my fault for not simply disabling the zoneminder apt repository once I had a working system.
If anyone wishes to obtain more detailed info, logs, or tests to attempt to squash this bug, let me know and I'll be happy to oblige.
Regards,
Matt
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: ZMC not analyzing random monitors

Post by Magic919 »

The suggestion of running the 1.34 release, rather than the development version (1.35.x) is hard to argue with.

There's a lot of development work going on and you can drop in on the Slack channel.

I can agree that the 1.35 has been quite useable for a while, but that's not the current state whilst it's being extensively worked on.

I see your point regarding the .deb file I made available. You shouldn't take my word for anything. You can see the details of the file here https://launchpad.net/~iconnor/+archive ... d/20920227 Have a look at the changes file zoneminder_1.35.16~20210122121212-focal_amd64.changes. There are sha1 and sha256 checksums there.
-
kd4pbs
Posts: 11
Joined: Wed Nov 11, 2020 9:54 pm

Re: ZMC not analyzing random monitors

Post by kd4pbs »

Ahh - there we go. That's the info I was looking for. I appreciate that, Magic.
I'm know just enough to be dangerous about most of this kind of stuff, but have enough common sense to know where to draw the line.
I'll have to look back to figure it out, but there were several 1.35.x versions which worked well for us. While we do use the system all the time, it isn't what we'd consider "critical infrastructure", so I can diddle and experiment at will. Also, now that I've seen that it will work quite well for what we need, I can justify sending some dough to the project. I appreciate ALL of you who contribute your time to the project.
Thanks for the lead on the Slack channel.
Regards,
Matt
kd4pbs
Posts: 11
Joined: Wed Nov 11, 2020 9:54 pm

Re: ZMC not analyzing random monitors

Post by kd4pbs »

Incidentally, for those who might wonder, the sha256 hash matches.

Code: Select all

# sha256sum zoneminder_1.35.16_20210122121212-focal_amd64.deb
e4335c9457e7fa18d0bd5774b2736aa17af47d63d981ea01cab614483d94e436  zoneminder_1.35.16_20210122121212-focal_amd64.deb
I had looked for this late yesterday, but couldn't find the web page which Magic linked for the checksums. Thanks again, Magic!
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: ZMC not analyzing random monitors

Post by Magic919 »

You’re welcome.

I’d suggest putting the ZM package on ‘hold’ once done. That way you can only do a fully conscious update. Use apt-mark hold zoneminder.
-
Locked