New User testing, lots of crashes

[The Eye]

Hi there.

Long time Linux user, open source enthusiast and Perl guy trying to get a grip on a surveillance system for my home.
I don't really expect any help/support, just wanted to report what happens if a ZM newbie (but otherwise hopefully not
totally incompetent) makes first encounters. Maybe some dev finds it enlightening.

At least I think ZM could benefit from such "fresh insight".

Scene:

ZoneMinder 1.34.23 running as a TurnkeyLinux appliance (https://www.turnkeylinux.org/zoneminder) on Proxmox
on a nifty https://www.gigabyte.com/Enterprise/Ser ... ST0-rev-11
The VM got 8 cores 4GB RAM and 1TB disk space
Bought two Reolink RLC-410 PoE cams and they do work ... sort of.

Well, first off: Would it be possible - by any means - to pause the log??? I am getting a twitching eye when there are like 20 messages per second (timestaps prove I do not exaggerate) and I do try to copy and paste one of them.

When I connected one cam, after much ado and reading (viewtopic.php?t=29143) it did work in sufficient quality.
When I connected the second cam, only one of them works. Sometimes "Cam1", sometimes "cam2", while wathing this non-deterministic spectacle, the log errors and warnings of zm -s 3 crashing error code 255 fly by. Sorry - no copy & paste, no better memo because I get 1 second to watch it and then the log message is gone with the wind.

Not sure who thought the Log window could be useful at all this way.

Image buffer too small... again. So if I add one cam, I have to increase the buffers in ALL cam (monitor?) configurations? srsly?
Ok, I'll do:

zmc -m 3 crashed, signal 6 (that is level INF - by the way? yes?)

Great now Cam1 works (before that Cam2 did work).

An error, Timezone not being configured, appears - among the myriad of log messages flying by. Of course it is configured (Europe/Prague). *shrug*

Yeah - I know I can filter to see only warnings and up, still I have only one second to try to copy & paste, because there is always said timezone error. goddammit. Ok, dpkg-reconfigure tzdata seems to have stuffed that mouth.

Hm. I see there might be a problem with hardware resources. 4GB RAM not being enough for two 5MP cams... sigh, will give it more and then some day look at the memory consumption atrocities that evidently do happen somewhere in the implementation. Promise.

edit: yeah. one 5MP cam takes 57% of SHM if you have 4GB RAM, so 8GB RAM is good for 3 cams - I planned 10 cams that would be 24GB straight. Do you start multiple gimp instances per captured image to perform some artistic filter on it?

[knight-of-ni]

I don't have a Reolink camera so I can't speak specifically to that model.

However, I can speak to some of the other questions:

Many of features of ZoneMinder were developed a decade ago. The log view is one of them... introduced in zm 1.25 if I recall.

Our next release is coming "soon", and it has the ability to pause the event list, along with a bunch of other, more modern features.

You can expect memory usage to be lower with the new version, but keep in mind that memory consumption is tied to how many key frames your camera is sending. So for example, if that Reloink camera sends 1 keyframe every 30 frames, it is going to require a lot more memory than if it were sending 1 keyframe every 5 frames. Many cameras have the ability to change this.

I typically set my own cameras to ~6 fps with a keyframe every 6 frames... so a new keyframe is received every second.

This memory usage is allocated in /dev/shm, which is why it is shown at the top of the web console. Keep an eye on it. When you run out, the symptoms you see are what you described... one camera works while another (identical) camera does not. Sounds like you might have figured this out.

[jperkins]

TurnkeyLinux appliance (https://www.turnkeylinux.org/zoneminder) on Proxmox

Well, first off: Would it be possible - by any means - to pause the log???

Yea the log thing got worse on a recent update. I posted about it here. In my case it looked like new logs were coming in when in fact they were switching order. viewtopic.php?f=40&t=30435 . I was beginning to wonder if I was the only one. I usually hit clear a few times so that I can see if fresh logs are coming in. yes it is broken. when I get serious I

Code: Select all

 'tail -f /var/log/zm/da_proper_logfile'

Maybe it would be a good idea to do an install on some bare metal without the proxmox / turnkey to get a reference. I installed via the https://wiki.zoneminder.com/Ubuntu_Serv ... der_1.34.x YMMV. Of course my style is kvm on bare metal and I like to be able to edit raw config files so things like proxmox dont make me feel warm and fuzzy. pfsense is my exception though. of course this rambling probably has nothing to do with your issues.

taking the time to learn about zoneminders logging has helped me. not only settting the debug level but also enabling debug for the particular daemons of zoneminder I am working on.

Yes I can see the age and how it may appear that parts of zoneminder are patched together. I tried to use some other open source projects but have decided zoneminder is the best fit for me. I'd used it years ago with a single wifi cam and a cellphone as a cam. Now have a full fledged system and it is working pretty well. good luck oh and I really like zmninja . very polished

[jperkins]

FWIW. I know nothing about camera models and did what I did before purchasing my cameras.
mixed set of mostly negative reviews mentioning zoneminder on amazon for your camera model

https://www.amazon.com/Reolink-Surveill ... B07C5JWK4K

Camera doesn’t meet my needs as a part of a Zoneminder system. Smearing and poor picture quality!
Flash based user interface is poorly written and lacking
I use mine with Zoneminder and record at 5MP. They are very bright at night. Much more detail than others I own.
Overall a very good camera.
. However, many people report that the cameras will disconnect from time to time
Stop! Don't waste your money! They even smear and pixelate on still images.,There are many Zoneminder forum postings regarding smearing and poor picture quality.
Overall, I'm very pleased with these cameras. They work well with every VMS I've tried so far, minus ZoneMinder and iSpy.

hey it is amazon so go figure :=)

[The Eye]

Thanks for your feedback.

@jperkins:

I bought the RLC-410 5MP camera, because it got a fairly decent review here: https://www.youtube.com/watch?v=LZJX8CKR5KI
I also am using ZoneMinder despite it being only "2nd"(?) in this review: https://www.youtube.com/watch?v=6Q5bZxlUhPs - mostly because I am a fan of crufty but otherwise mighty applications which are open source and do not verndor-lock-in you.

Yeah, I experienced the smearing for the RLC-410 in ZoneMinder too, until I applied the sauce mentioned in viewtopic.php?t=29143 (as mentioned). Now the quality is great. No smearing, perfect picture under any light conditions - including IR mode. Just as the YT review said. Reolink support had like 8 hours reaction time to my support request of FW update problem - and now I am running Firmware 136.
Since Firmware 111 no Flash interface anymore.

Hell, I may even install a SSH on it and play tetris: https://github.com/hn/reolink-camera

In short: very happy with that camera. And by that I mean the picture quality especially in ZM. I see no point in running it on bare hardware, as I am the master of the whole virtualization and hyperconvergence stack there is (Hardware -> Proxmox -> ZM VM). Most of the hardware I gave the VM is paravirtualized anyway.

My only beef with ZM now is the resource requirements, but I guess that is an optimization problem "only". Of course in order to turn the right knobs, I will have to descend more into the inner workings - but I'm used to that.

[jperkins]

My only beef with ZM now is the resource requirements, but I guess that is an optimization problem "only". Of course in order to turn the right knobs, I will have to descend more into the inner workings - but I'm used to that.

Meybe start by reducing your resolution, framerate, and analysis rate greatly. reduce the size of your zones. I am new here and that has helped me a lot.

feeding zoneminder 2 streams from each camera has helped me out. analyze a smaller resolutions sub stream and save it all (MoCord).
then when events come trigger the high resolution feed and save only the motion ( I was hoping for no analysis but it still uses some for some reason) total of 8 cameras h264 passthrough :
I have 8 monitors running MoCord on sub channels. they are doing all the analysis. (4CIF 20/10 FPS) ( monitors 1-8 )
have 8 more monitors running NoDect with '0' zones setup as linked to the first 8 channels. (4MP 20/10 fps) ( monitors 10-17)
viewtopic.php?f=40&t=30470&p=120223#p120019


yea and when I looked up your 'enterprise' system with an 'atom' processor I had to chuckle. forgive my ignorance cause my stuff is all over 10 years old and prolly an atom aint an atom anymore. looks like that thing is geared to move data, was wondering about its ability for 'analysis'

[The Eye]

Meybe start by reducing your resolution, framerate, and analysis rate greatly. reduce the size of your zones. I am new here and that has helped me a lot.

That is not optimization, that is making concessions. And yes, the time for that may come, but not yet.
Indeed, I can tweak the amount of SHM the system allocates via

Code: Select all

tmpfs     /dev/shm       tmpfs       defaults,size=2560M      0 0

in /etc/fstab.

Then it doesn't take the default of 50% of total memory but whatever is requested. So I am back to 4GB and giving the two cams their 2.5GB of shared memory. The rest 1.5GB seems plenty for ZoneMinder. /dev/shm usage is at 89%, I might squeeze it more tight, but not for now.
Doing this, it looks like I might remain somewhere at 14GB-16GB memory requirements for 10 (such or similar) cams, with shared memory around 13GB. That's a huge improvement compared to the formerly projected 24GB and moves it into acceptable space.

CPU usage (for the 8 cores I gave it) is at 20% so theoretically 10 cams would eat these 8 cores, but I assume I will do more analysis than "just Mocord" so maybe a tweak or two here first. I don't think a timestamp is necessary, because the cams do it already, so watermarking it there - not sure how much power that takes - one more time seems unnecessary.

yea and when I looked up your 'enterprise' system with an 'atom' processor I had to chuckle. forgive my ignorance cause my stuff is all over 10 years old and prolly an atom aint an atom anymore. looks like that thing is geared to move data, was wondering about its ability for 'analysis'

Atom is merely some marketing name. Much like Celeron, Pentium and Xeon.
The CPU in question here is a C3958 Denverton - https://ark.intel.com/content/www/us/en ... 0-ghz.html
it is 2 years old already, but having 16 cores with a TDP of 31W is nice.

I also configured the CPU type of the VM as "host", which means I am passing through the various CPU features and let ZM and/or ffmpeg use them (whatever they deem can be used).

[Andyrh]

Building a ZM server can be a challenge (fun). There are many variables.
The way I did it was to figure out what I wanted and what I needed. This gave me the requirements and the nice to haves.

Requirements:
All modcord
All at a framerate that is reasonable
All full resolution
Not to break the bank on storage
Enough cameras to cover the whole yard
Cheap (old) computer

My 1st iteration was 2 cameras at 10FPS. Built it on a gen 1 i3. Worked great until my requirements were changed to 3 cameras, the i3 was having issues keeping up, a few dropped frames when it was busy.

My 2nd (current) iteration is a gen 4 i7 with 16GB RAM. 5 cameras, 3-8mp mostly at 8FPS, some at 10FPS all modcord. No dropped frames and I predict I can add 2-4 more 8mp cameras before I see an issue again. Currently running under 50% typical load, 8GB shared memory @ 35% and 6GB free RAM.

Some things are not concessions, like FPS. As has been said in this forum, people are in jail based on 5FPS. ZM is not making movies, it is capturing events for later review. If you want high FPS, buy lots of cycles. If you want long pre-event captures buy lots of RAM. If you want high resolutions, buy both. All of these choices simply fall into the requirements that tell you what CPU you need and how much RAM is needed.

I hope the helps alleviate some of the stress of building a ZM system.