Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Discussions related to the 1.36.x series of ZoneMinder
yannlieb
Posts: 14
Joined: Fri Mar 19, 2021 9:40 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by yannlieb »

I have tweaked a bit the setup and removed one camera at the same time, so I can't be completely sure what is the driver of the improvement on my system.
I haven't seen the memory consumption per zmc process decrease, however, I have lot less crashes on events, which make me think that the analysis is definitely more efficient in term of memory and cpu consumption.
User avatar
swolsen
Posts: 31
Joined: Sat Mar 20, 2021 9:39 pm
Location: Schenectady, NY, USA

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by swolsen »

I have four cameras, three are 5MP and one 6MP set at 1080P. One is 24x7 Modect and the others are only set to Modect when I'm away for the day. Of the four zmc processes running only one is much higher than the other three and slowly growing, in terms of memory usage.

Image
User avatar
swolsen
Posts: 31
Joined: Sat Mar 20, 2021 9:39 pm
Location: Schenectady, NY, USA

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by swolsen »

Not sure if this helps but I was able to lower the increasing memory of the zmc process by setting the camera from Modect to Monitor then back to Modect. This way I don't have to reboot the system. I can set up a cron job to change the monitor function every couple of days.
Image
ergamus
Posts: 27
Joined: Sat May 29, 2021 10:09 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ergamus »

swolsen wrote: Mon Jul 05, 2021 2:36 am Not sure if this helps but I was able to lower the increasing memory of the zmc process by setting the camera from Modect to Monitor then back to Modect. This way I don't have to reboot the system. I can set up a cron job to change the monitor function every couple of days.
Image
You can do the same with zmpkg.pl restart (like choosing the Restart run-state in the WebGUI), albeit that does restart all the camera processes. I have the ability to restart/shutdown the server disabled in options. Your lucky though, one of my cameras usually maxes out memory every 4 hours or so.
tommisgr
Posts: 23
Joined: Mon Aug 31, 2020 6:28 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by tommisgr »

After facing same issues (ram usage rising - error on zmc process - monitor stop working restarting)

I disabled in the 3 monitor with ram usage (over 1GB up to 4GB)
storage - Save JPEGs -> disabled

was on Analysis images only.
Now running smoothly with 6GB of free RAM (used to be 1Gb and lower and was running a script to restart ZM when mem too low)
iconnor wrote: Mon May 31, 2021 6:45 pm What is your keyframe interval?

Also, you have analysis fps set?
zarkov1994
Posts: 12
Joined: Mon Jul 14, 2014 9:14 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by zarkov1994 »

I've just tried changing this setting (storage - Save JPEGs -> disabled) on my x4 5MP cams and it does seem to solve the memory overrun issue. I've set buffers back to 0 now too and everything's looking good from a memory perspective (sitting around 6gb used. each ZMC under 1GB each at the moment).

tommisgr wrote: Thu Jul 08, 2021 6:46 am After facing same issues (ram usage rising - error on zmc process - monitor stop working restarting)

I disabled in the 3 monitor with ram usage (over 1GB up to 4GB)
storage - Save JPEGs -> disabled

was on Analysis images only.
Now running smoothly with 6GB of free RAM (used to be 1Gb and lower and was running a script to restart ZM when mem too low)

[
Zoneminder 1.36.15 running under:
Fedora 35, Ryzen 5 3600x, 24Gb RAM, 512Gb WD Blue SN570 System SDD, 4TB Seagate Skyhawk Event HDD
Numerous Reolink RLC-410 (Wired IP)
Banksy
Posts: 1
Joined: Tue Mar 29, 2022 11:23 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by Banksy »

I just experienced this issue under v1.36.12 today. Figured, I would post that disabling saving jpegs seems to be the solution.

There were a lot of motion events and the CPU could not keep up, load was ~21 on an 8 core CPU. Once everyone left CPU load went back to ~2, but memory on 2 processes stayed around 4GB+ each many hours later:

before.png
before.png (148.29 KiB) Viewed 178846 times



As suggested above, I tried changing monitor 8 from Nodect -> Mocord -> Nodect, but it did not reduce memory usage. Then I disabled saving jpegs and memory dropped from 5447M to 445M! (also, you can see that there is 1 less process running):

after.png
after.png (145.74 KiB) Viewed 178846 times



Any idea, why saving analysis & jpegs causes excessive memory usage when CPU falls behind?
ZMNoobja
Posts: 14
Joined: Fri Feb 18, 2022 12:41 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ZMNoobja »

Would disabling the JPEG setting end up disabling the event recording entirely? Forgive me as I'm jumping in here.. I'm experiencing almost this exact issue.

1.36.12, 56 cameras, 3fps each, VGA resolution. System is a Dell R710, 120GB RAM, 24TB space in the RAID config.

We restart the machine and everything runs great. We can even see zmaudit.pl running happily and doing its thing. After about 30min to a few hours, we get into a completely unresponsive state and have to hard shutdown.

My engineer and I have been racking our brains trying to find what's causing the issue to no avail. Happy to provide any additional info that might be helpful to the original issue.

JL
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by Magic919 »

Turning off JPEGs won’t stop events recording.
-
ZMNoobja
Posts: 14
Joined: Fri Feb 18, 2022 12:41 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ZMNoobja »

Magic919 wrote: Tue Apr 19, 2022 6:29 pm Turning off JPEGs won’t stop events recording.
Forgive me - How are the events stored at that point, then? I thought they were stored as JPEGs and then compiled to make the 'video' upon exporting the event. Sorry to sidetrack :)
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by Magic919 »

Using 'Camera passthrough' it stores the video as MP4.
-
ZMNoobja
Posts: 14
Joined: Fri Feb 18, 2022 12:41 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ZMNoobja »

Excellent and greatly appreciated clarification!
ZMNoobja
Posts: 14
Joined: Fri Feb 18, 2022 12:41 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ZMNoobja »

Back again with another question... This suggestion to disable decoding seems to disable the live viewing of the cameras. Is that correct?

Wondering if there is a middle-ground between features and performance. At this point, it seems that we cannot run the system as intended without significantly lowering overall camera performance. Even then, it doesn't seem to actually be stable.

Our other system is running on 1.30.4 with over 80 cameras and 12GB RAM. It runs like a top! We initially matched the settings in this new instance with 120GB RAM and roughly 55 cams so that we had a good place to start, but after making a lot of the mentioned workarounds and improvements, we're still seeing the issues that are mentioned throughout this thread. Mainly the RAM issue, but now I'm seeing a lot of logs regarding, "Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up" and "You have set the max video packets in the queue to 100. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting."

Let me take a quick moment to say THANK YOU to the developers and the community for their continued work on this project!!

ZMN
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by iconnor »

In 1.37 dev we have more options like decode only keyframes, and ondemand (only when watching) etc. Helps a lot. That along with using janus for h264 live view are drastically reducing the cpu requirements.

In 1.36 land, 1.36.15 (soon) should help a lot with the issues seen here.

If you want the same performance as 1.30, then turn off saving to mp4. You've be saving a billion jpegs, using a lot momre storage and bandwidth but cpu/ram performance should be the same (or better) as 1.30.
ZMNoobja
Posts: 14
Joined: Fri Feb 18, 2022 12:41 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ZMNoobja »

Thanks for the replies, iconnor!

In wanting to try to disable the suggested mp4 saving, I want to make sure I'm looking at the right setting. I know there is one under Images called OPT_FFMPEG. I'm thinking that can't be the setting you're referring to as that seems to be for on demand video generation.

Also, if I'm understanding correctly, are you saying that the system is autogenerating mp4 videos by default and storing them somewhere? I don't believe we've seen any mp4s being generated, so that would be huge info to know.

Also - to add my system specs:
Dell R710 Server
24x Intel Xeon X5670 @ 2.9GHz (2 Sockets)
120GB RAM
(x6) Dell 6TB 3.5in 7200RPM SAS HDD
56 Cams, running at VGA Low quality substreams, 2fps ea

Thanks again!
ZMN
Post Reply