High Ram usage

Previous development branch now released as 1.36
phgerin
Posts: 11
Joined: Tue Feb 09, 2021 10:28 am

Re: High Ram usage

Post by phgerin »

Does anyone have the last 3.15.16 focal amd64 version.
I can't find it in the archive.
Thanks !
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: High Ram usage

Post by Magic919 »

-
phgerin
Posts: 11
Joined: Tue Feb 09, 2021 10:28 am

Re: High Ram usage

Post by phgerin »

hmm ? I can't see any built file there.
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: High Ram usage

Post by Magic919 »

Oops. I presume they do some cleaning up. Was there when I grabbed it a while back.

Try this www.magic919.co.uk/zoneminder_1.35.16~2 ... _amd64.deb
-
phgerin
Posts: 11
Joined: Tue Feb 09, 2021 10:28 am

Re: High Ram usage

Post by phgerin »

Many thanks !
dashie
Posts: 6
Joined: Thu Jan 05, 2017 11:39 am

Re: High Ram usage

Post by dashie »

I'm also having memory issues, which seems caused mostly by recording, memory usage grow quickly when recording, and memory is never released (unless kill of zmc processes).

already bumped the memory of the server to 55G which even at that size ends up exhausted fast. I'm at ~6G RES per zmc process right now.

I've already changed the "Image Buffer Size (frames)" to 2 on all monitors but that doesn't seemed to help.

for informations my monitors are pretty much all:
source type: ffmpeg in modect and analysis enabled
analysis/max/alarm FPS empty
source rtsp/TCP/32bit/1920x1080
save frames+analysis if available, writer H264 passthrough, outputcodec and encoder auto and container mp4 (or it saves mkv and the browser doesn't know what to do with that in auto)
buffers: 2, 25, 25, 25, 500, 1
and misc is default
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: High Ram usage

Post by Magic919 »

Last suggestion I saw was 5 for the buffers.

Reducing the interval between keyframes (on the cameras) will help.
-
dashie
Posts: 6
Joined: Thu Jan 05, 2017 11:39 am

Re: High Ram usage

Post by dashie »

I think the buffers was a suggestion of around 2 for the image buffer size one.

I've left it at two, and changed warmup/pre/post/stream to 5 and alarm frame count to 1.

as for the keyframe, cannot be changed on any of my cameras, so no workaround there.
topminttwo
Posts: 1
Joined: Mon Mar 22, 2021 7:30 am

Re: High Ram usage

Post by topminttwo »

I answered before I had done much testing. I see it now. Some zmc processes are using enormous amounts of memory, even in Monitor mode.
User avatar
iconnor
Posts: 2879
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor »

I havn't been able to reproduce the high ram usage in monitor mode here. Or conceive of how it would happen.

If doing passthrough we have to keep everything from the last keyframe. We only have to keep the encoded data though. So this shouldn't be too much. My 1080p h265 cam is using 74Mb of ram.

If doing motion detection, we decode to raw buffers and these take up a lot of ram. However if motion detection is keeping up, it frees that ram as it goes, so we still don't use much ram.

The only issue is that if the analysis thread gets backed up for some reason (slow db or slow disk or bug in code) the capture process will keep capturing and stuffing data into the queue.

So you guys who are seeing this ram use, I need debug level 4 logs for zmc.
User avatar
iconnor
Posts: 2879
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor »

I just pushed a change that makes ZM only buffer since the last keyframe when using passthrough. Otherwise we don't need to do that, so people using Encode or saving jpegs might see a ram reduction.
phgerin
Posts: 11
Joined: Tue Feb 09, 2021 10:28 am

Re: High Ram usage

Post by phgerin »

I have an i7 Ubuntu Focal with 16GB of ram and 16 cams (all the same, same config).

If I start Zoneminder with all the cams set to "none" and then I turn them to "modect" one by one, I have 4,4BG ram usage and zmc process are max 114mb of ram.

If I reboot the computer with all the cam enabled, the total ram usage is 15 Gb with 2Gb of swap.
zmc process are between 80Mb and 1,6Gb of ram.

The problem seems to occur when CPU usage is very high.
dmidthun
Posts: 6
Joined: Mon Mar 22, 2021 7:50 pm

Re: High Ram usage

Post by dmidthun »

I'm noticing that the memory issue is only happening when the cameras are set to record of any kind, and then drops back off when the event(s) are complete and the process starts over again (~10 mins) . I also noticed that there are 4 zmc processes per camera enabled. Could this also be attributing to the resource hogging? I have my install running with 24G of ram, 12G of swap and can literally watch the usage increase even though there is little to no change in the image being captured (an empty room in my house that gets no foot traffic -- ever). 10 rtsp cameras set to monitor == 4.5~5.5G RAM; change 1 to Mocord, system will again start hoarding memory.

Isaac, is there possibly a memory leak in the zmc process when in record mode?


Thanks,
Dan

Setup:
Ubuntu 18.04
ZM 1.35/master (latest)
8 Merkury 720p 'smartcams; hacked'
Ovirt 4.4.4 KVM Env - 2CPU, 24G RAM, 72G vdrive; 60sda 12sdb(swap) - Nimble backend storage @ 10G FC
dmidthun
Posts: 6
Joined: Mon Mar 22, 2021 7:50 pm

Re: High Ram usage

Post by dmidthun »

iconnor wrote: Mon Mar 22, 2021 3:12 pm So you guys who are seeing this ram use, I need debug level 4 logs for zmc.
Forgive my ignorance - where/how can i get this information to give you?


Dan
User avatar
iconnor
Posts: 2879
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor »

Possible? Yes.

You didn't say whether you are doing passthrough or encode. Encoding takes a ton of ram too (default settings will buffer 20 frames in the encoder).
Passthrough can buffer some stuff but it is encoded data and should just be ref counted so shouldn't amount to much.

dmidthun wrote: Wed Mar 24, 2021 6:20 pm I'm noticing that the memory issue is only happening when the cameras are set to record of any kind, and then drops back off when the event(s) are complete and the process starts over again (~10 mins) . I also noticed that there are 4 zmc processes per camera enabled. Could this also be attributing to the resource hogging? I have my install running with 24G of ram, 12G of swap and can literally watch the usage increase even though there is little to no change in the image being captured (an empty room in my house that gets no foot traffic -- ever). 10 rtsp cameras set to monitor == 4.5~5.5G RAM; change 1 to Mocord, system will again start hoarding memory.

Isaac, is there possibly a memory leak in the zmc process when in record mode?


Thanks,
Dan

Setup:
Ubuntu 18.04
ZM 1.35/master (latest)
8 Merkury 720p 'smartcams; hacked'
Ovirt 4.4.4 KVM Env - 2CPU, 24G RAM, 72G vdrive; 60sda 12sdb(swap) - Nimble backend storage @ 10G FC
Locked