Page 5 of 6

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Sat Jun 25, 2022 12:57 am
by fvalley2
Hi,

Thanks to iconor for a great, great tool that keeps getting better.

I am having the same, exact same, issues over here. 16 GB ram spilling into and maxing out 4 GB swap. The gains on CPU cycles are great. Went from about 1.3 down to .8 and sometimes .6, awesome. Can't keep rebooting everyday though.

I've read the thread and tried all the suggestions. System was fine prior to upgrade to 1.36.19 on Ubuntu Server 20.04.

Assuming there's no fix yet, can someone point me towards info about backing up database, hopefully camera configurations too, planning to go back to working system backup with 1.34 installed.

Thanks.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Sat Jun 25, 2022 6:40 am
by Magic919
Just use standard tools to dump the DB. Mysqldump is usual. Camera configs live in there.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Sun Jun 26, 2022 2:53 pm
by fvalley2
Thanks. Too much data to go that route, decided to try RAM upgrade from 16G to 32G. I don't know if this will be helpful to anyone and as far as I know simply adding more ram is not the solution to this type of issue but, who doesn't want to max out ram anyway, right?


Results -

After reboot, memory use in htop went up to ~15G, just as before upgrade, held there for an hour or so before I fell asleep. When I woke up today it had climbed to 17G. Seems to be ticking up .1G evry hour or so maybe. Process memory use was 43% in 3 highest processes initially, now 47%.

Searched cron in wiki, this worked.

30 14 * * * /usr/bin/zmsystemctl.pl restart

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Mon Jun 27, 2022 12:21 pm
by Magic919
You might just manage to outrun it using upgrades. Can't blame you for trying :-)

I run Modect on a couple of my cameras and those are the ones that eat the RAM. RAM use goes up during the detection and doesn't always settle to the previous level.

Code: Select all

 total          1478716K
 total          2372480K
 total          2372480K
 total          1494676K

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Mon Jul 04, 2022 5:55 pm
by fvalley2
Thanks for that info. Man I love this peice of software and I'm about to make another donation to its humans.

Ended up reinstalling from scratch on Ubuntu 22.04 Jammy. Repository install 1.36.12. Turned out one of the last things I did was switch from disabled to passthrough on the video writer. RAM doubled from 2 to 4 and the next day it was over 20. Back to disabled and back down to 2 gigs for now. Went to passthrough months ago believing it saved CPU cycles some but load is down around .8 now with an i7 capturing 1.5MB/s. Maybe that helps someone else.

Thanks/best,

~••~

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Fri Jul 08, 2022 2:04 pm
by Ralphred
Was having memory leak issues after a 3.6.19 upgrade, so read this thread (thx for the input :) ).
By setting buffer sizes for the monitor (all @50) and dropping my iframe interval from 25 to 12 (@25fps) the memory usage of zmc is now stable, I can even see it releasing memory back to the system, as opposed to just "hoarding" it as it did before.
I only have a small system and keep relatively small amounts of data, so I appreciate now everyone is not in a position to emulate these tactics, but it helps to document these things, especially if we expect more memory usage from zmc when going forward to plan new system builds.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Fri Aug 05, 2022 11:09 am
by ameterasu
I'm sharing similar results from 1.36.21 ubuntu 20.04.

Ram consumption is increasing gradually.

Image Buffer Size: 3
Max. Image Buffer Size: 0

20 cameras, all Record no analysis only decode. Almost all cameras Hikvision H264+.

I've tried setting Max. Image Buffer Size 500 on one camera. It calculated 3.5Gigs of max ram usage. But the actual result is 6gigs and increasing.
msedge_rQlCNQc6an.png
msedge_rQlCNQc6an.png (67.05 KiB) Viewed 256613 times
VM has 96gigs of ram, starts at about ~60gigs, ends up filling all inc. swap.
msedge_1xQaEa15KJ.png
msedge_1xQaEa15KJ.png (23.9 KiB) Viewed 256613 times
There seems like there is a memory leak I can not identify.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Fri Aug 05, 2022 12:48 pm
by Magic919
I'm not suggesting there's not a problem, but unlimited max buffers and H.264+ will likely pour fuel on the flames.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Sun Aug 07, 2022 2:19 pm
by iconnor
Honeslty h264+ or any stream with huge keyframe interval is a big problem. The only thing I can think of to do about it is to separate the events and recording concepts so that we are ALWAYS recording and just noting when an event starts and stops. This increases disk IO though.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Mon Aug 08, 2022 2:27 pm
by lightguy48
I just upgraded to 1.36.24 this weekend and now all of my monitors/cameras that had no issues on .21 are reporting that they're not seeing the 2nd keyframe. My framerate is 6fps and the keyframe is 12 and my buffer is 51.

Maybe this is flawed logic but I have my pre and post buffers at 25 each and when I had my max buffers at 50 or below it would give errors on not enough buffers but 51 frames for buffers should be plenty for the keyframe I don't know what has happened.

I'm running h264 with passthrough and mocord mode.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Mon Aug 08, 2022 11:41 pm
by iconnor
It's flawed code on my part. It may be that your keyframe interval is too frequent! I have a patch already in master to fix this. You might play aroudn with keyframe interval on your camera to actually increase it, or if running on ubuntu, follow the proposed ppa to get the fix before its actually released. (I will likely update it tomorrow).

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Tue Aug 09, 2022 7:38 am
by ameterasu
I dont have a lot of experience in these kinds of stuff. I tried to take htop ss's 3 days apart. Maybe someone can identify why the cameras have increasing ram requirements.

Edit: Ubuntu 20.04 - Zoneminder 1.36.21, All cameras H264+ Hikvision, Min Buffer 3, Max. Buffer 300, Calculated Max. Ram per camera is 3,5Gb.
Configuration is almost default.
msedge_SkVWXQzRY2.png
msedge_SkVWXQzRY2.png (60.81 KiB) Viewed 256504 times
This is couple hours of runtime.
msedge_P885nagbEi.png
msedge_P885nagbEi.png (65.54 KiB) Viewed 256504 times
This is 4 days+ runtime.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Tue Aug 09, 2022 4:28 pm
by iconnor
Does it actually keep increasing or does it just max out at a large value?

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Tue Aug 09, 2022 10:05 pm
by ameterasu
I dont know yet, added more ram and will report back in couple days.

But i had times when i could not even use webui because there was not enough ram i guess. Not with exact same camera config.

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Posted: Mon Aug 22, 2022 11:31 am
by ameterasu
Hello once more;

I see that the memory has been stabilized at about 80gb. There seems to be no memory leak.

For some reason it went up to 110gb, then dropped down to 80gb over 3 days. After that its been stable at around 80gb.

In the past I had 96gb instead of 128gb. Maybe because I did not have enough ram for this period before stabilizing my system was seizing.

Thank you for your helps.