Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Discussions related to the 1.36.x series of ZoneMinder
fvalley2
Posts: 5
Joined: Sat Jun 25, 2022 12:44 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by fvalley2 »

Hi,

Thanks to iconor for a great, great tool that keeps getting better.

I am having the same, exact same, issues over here. 16 GB ram spilling into and maxing out 4 GB swap. The gains on CPU cycles are great. Went from about 1.3 down to .8 and sometimes .6, awesome. Can't keep rebooting everyday though.

I've read the thread and tried all the suggestions. System was fine prior to upgrade to 1.36.19 on Ubuntu Server 20.04.

Assuming there's no fix yet, can someone point me towards info about backing up database, hopefully camera configurations too, planning to go back to working system backup with 1.34 installed.

Thanks.
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by Magic919 »

Just use standard tools to dump the DB. Mysqldump is usual. Camera configs live in there.
-
fvalley2
Posts: 5
Joined: Sat Jun 25, 2022 12:44 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by fvalley2 »

Thanks. Too much data to go that route, decided to try RAM upgrade from 16G to 32G. I don't know if this will be helpful to anyone and as far as I know simply adding more ram is not the solution to this type of issue but, who doesn't want to max out ram anyway, right?


Results -

After reboot, memory use in htop went up to ~15G, just as before upgrade, held there for an hour or so before I fell asleep. When I woke up today it had climbed to 17G. Seems to be ticking up .1G evry hour or so maybe. Process memory use was 43% in 3 highest processes initially, now 47%.

Searched cron in wiki, this worked.

30 14 * * * /usr/bin/zmsystemctl.pl restart
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by Magic919 »

You might just manage to outrun it using upgrades. Can't blame you for trying :-)

I run Modect on a couple of my cameras and those are the ones that eat the RAM. RAM use goes up during the detection and doesn't always settle to the previous level.

Code: Select all

 total          1478716K
 total          2372480K
 total          2372480K
 total          1494676K
-
fvalley2
Posts: 5
Joined: Sat Jun 25, 2022 12:44 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by fvalley2 »

Thanks for that info. Man I love this peice of software and I'm about to make another donation to its humans.

Ended up reinstalling from scratch on Ubuntu 22.04 Jammy. Repository install 1.36.12. Turned out one of the last things I did was switch from disabled to passthrough on the video writer. RAM doubled from 2 to 4 and the next day it was over 20. Back to disabled and back down to 2 gigs for now. Went to passthrough months ago believing it saved CPU cycles some but load is down around .8 now with an i7 capturing 1.5MB/s. Maybe that helps someone else.

Thanks/best,

~••~
Ralphred
Posts: 32
Joined: Fri Jul 08, 2022 12:23 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by Ralphred »

Was having memory leak issues after a 3.6.19 upgrade, so read this thread (thx for the input :) ).
By setting buffer sizes for the monitor (all @50) and dropping my iframe interval from 25 to 12 (@25fps) the memory usage of zmc is now stable, I can even see it releasing memory back to the system, as opposed to just "hoarding" it as it did before.
I only have a small system and keep relatively small amounts of data, so I appreciate now everyone is not in a position to emulate these tactics, but it helps to document these things, especially if we expect more memory usage from zmc when going forward to plan new system builds.
ameterasu
Posts: 14
Joined: Thu Jun 16, 2022 9:53 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ameterasu »

I'm sharing similar results from 1.36.21 ubuntu 20.04.

Ram consumption is increasing gradually.

Image Buffer Size: 3
Max. Image Buffer Size: 0

20 cameras, all Record no analysis only decode. Almost all cameras Hikvision H264+.

I've tried setting Max. Image Buffer Size 500 on one camera. It calculated 3.5Gigs of max ram usage. But the actual result is 6gigs and increasing.
msedge_rQlCNQc6an.png
msedge_rQlCNQc6an.png (67.05 KiB) Viewed 162285 times
VM has 96gigs of ram, starts at about ~60gigs, ends up filling all inc. swap.
msedge_1xQaEa15KJ.png
msedge_1xQaEa15KJ.png (23.9 KiB) Viewed 162285 times
There seems like there is a memory leak I can not identify.
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by Magic919 »

I'm not suggesting there's not a problem, but unlimited max buffers and H.264+ will likely pour fuel on the flames.
-
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by iconnor »

Honeslty h264+ or any stream with huge keyframe interval is a big problem. The only thing I can think of to do about it is to separate the events and recording concepts so that we are ALWAYS recording and just noting when an event starts and stops. This increases disk IO though.
lightguy48
Posts: 101
Joined: Sun Nov 15, 2015 7:19 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by lightguy48 »

I just upgraded to 1.36.24 this weekend and now all of my monitors/cameras that had no issues on .21 are reporting that they're not seeing the 2nd keyframe. My framerate is 6fps and the keyframe is 12 and my buffer is 51.

Maybe this is flawed logic but I have my pre and post buffers at 25 each and when I had my max buffers at 50 or below it would give errors on not enough buffers but 51 frames for buffers should be plenty for the keyframe I don't know what has happened.

I'm running h264 with passthrough and mocord mode.
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by iconnor »

It's flawed code on my part. It may be that your keyframe interval is too frequent! I have a patch already in master to fix this. You might play aroudn with keyframe interval on your camera to actually increase it, or if running on ubuntu, follow the proposed ppa to get the fix before its actually released. (I will likely update it tomorrow).
ameterasu
Posts: 14
Joined: Thu Jun 16, 2022 9:53 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ameterasu »

I dont have a lot of experience in these kinds of stuff. I tried to take htop ss's 3 days apart. Maybe someone can identify why the cameras have increasing ram requirements.

Edit: Ubuntu 20.04 - Zoneminder 1.36.21, All cameras H264+ Hikvision, Min Buffer 3, Max. Buffer 300, Calculated Max. Ram per camera is 3,5Gb.
Configuration is almost default.
msedge_SkVWXQzRY2.png
msedge_SkVWXQzRY2.png (60.81 KiB) Viewed 162176 times
This is couple hours of runtime.
msedge_P885nagbEi.png
msedge_P885nagbEi.png (65.54 KiB) Viewed 162176 times
This is 4 days+ runtime.
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by iconnor »

Does it actually keep increasing or does it just max out at a large value?
ameterasu
Posts: 14
Joined: Thu Jun 16, 2022 9:53 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ameterasu »

I dont know yet, added more ram and will report back in couple days.

But i had times when i could not even use webui because there was not enough ram i guess. Not with exact same camera config.
ameterasu
Posts: 14
Joined: Thu Jun 16, 2022 9:53 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by ameterasu »

Hello once more;

I see that the memory has been stabilized at about 80gb. There seems to be no memory leak.

For some reason it went up to 110gb, then dropped down to 80gb over 3 days. After that its been stable at around 80gb.

In the past I had 96gb instead of 128gb. Maybe because I did not have enough ram for this period before stabilizing my system was seizing.

Thank you for your helps.
Post Reply