High Ram usage

Forum for questions and support relating to 1.35.x development only.
User avatar
iconnor
Posts: 1530
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor » Thu Mar 25, 2021 5:15 pm

Options -> Logging
Check DEBUG
LOG_DEBUG_TARGET => _zmc
LOG_DEBUG_LEVLE => 4

Restart zoneminder.

Now /var/log/zm/zmc_m[monitor id].log will have the logs.

Cramac1
Posts: 76
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 » Sat Mar 27, 2021 5:07 am

At 1.35.16, it took me 500MB for 1 camera, after the 1.5GB update. How to fix? Buffer reduced to 2
cameras just stand to record.

debug log:
https://disk.yandex.ru/d/NpQVhj0I5GjPMQ
https://disk.yandex.ru/d/59TnGhjHmWSm4A

Cramac1
Posts: 76
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 » Sat Mar 27, 2021 2:58 pm

Magic919 wrote:
Mon Mar 01, 2021 1:46 pm
Oops. I presume they do some cleaning up. Was there when I grabbed it a while back.

Try this www.magic919.co.uk/zoneminder_1.35.16~2 ... _amd64.deb
can you update the link?

Magic919
Posts: 489
Joined: Wed Sep 18, 2013 6:56 am

Re: High Ram usage

Post by Magic919 » Sat Mar 27, 2021 3:40 pm

I've reinstated it. Please try now.

Cramac1
Posts: 76
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 » Sat Mar 27, 2021 3:59 pm

Magic919 wrote:
Sat Mar 27, 2021 3:40 pm
I've reinstated it. Please try now.
txh

dmidthun
Posts: 5
Joined: Mon Mar 22, 2021 7:50 pm

Re: High Ram usage

Post by dmidthun » Sun Mar 28, 2021 1:53 am

iconnor wrote:
Thu Mar 25, 2021 5:13 pm
Possible? Yes.

You didn't say whether you are doing passthrough or encode. Encoding takes a ton of ram too (default settings will buffer 20 frames in the encoder).
Passthrough can buffer some stuff but it is encoded data and should just be ref counted so shouldn't amount to much.


--I'm using passthough. I tried to keep it as simple as possible... though sometimes it gets a little fickle. I left most of the settings close to default minus some tweaks from the upgrade (buffer from 20->2, logging, api settings etc)

dmidthun wrote:
Wed Mar 24, 2021 6:20 pm
I'm noticing that the memory issue is only happening when the cameras are set to record of any kind, and then drops back off when the event(s) are complete and the process starts over again (~10 mins) . I also noticed that there are 4 zmc processes per camera enabled. Could this also be attributing to the resource hogging? I have my install running with 24G of ram, 12G of swap and can literally watch the usage increase even though there is little to no change in the image being captured (an empty room in my house that gets no foot traffic -- ever). 10 rtsp cameras set to monitor == 4.5~5.5G RAM; change 1 to Mocord, system will again start hoarding memory.

Isaac, is there possibly a memory leak in the zmc process when in record mode?


Thanks,
Dan

Setup:
Ubuntu 18.04
ZM 1.35/master (latest)
8 Merkury 720p 'smartcams; hacked'
Ovirt 4.4.4 KVM Env - 2CPU, 24G RAM, 72G vdrive; 60sda 12sdb(swap) - Nimble backend storage @ 10G FC

Cramac1
Posts: 76
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 » Wed Apr 07, 2021 9:47 am

who tried it, fixed it?

P_Dmitrij
Posts: 1
Joined: Tue May 04, 2021 5:36 pm

Re: High Ram usage

Post by P_Dmitrij » Tue May 04, 2021 11:59 pm

Hi all, I've faced the same problem and would like to share my findings. But first of all, many thanks to the ZM developers and contributors for such a great development! Although it cost me couple of weeks to get more familiar with the product and to build a more or less useful setup, I think the ZM is actually the best CCTV solution on the free software market. Especially the version 1.35 with the option to disable video decoding! It just a killer feature for me, which allows to implement a hi-res CCTV system with very limited hardware resources.

My last setup, where the problem exists is follow:

Code: Select all

Server:
i3-4330T with 16GB RAM
OS Proxmox
VM with 2 cores and 8GB RAM limit
Ubuntu 20.04.2 LTS (Focal Fossa)
Zoneminder v1.35.27
Storage: WD Purple WD20PURZ thru NFS (served by Proxmox)

ZoneMinder settings:
Camera 1: Hikvision DS-2CD2142FWD
	Stream 1 - "Monitor-1"
		Mode: RECORD, no analysis, no decoding, camera passthrough
		Source: FFMPEG TCP H.264 2688x1520x32@20fps, I-Frames 20, ~800kB/s
		Linked with Monitor-3
		Buffers: 
			Image Buffer Size			21 (>1 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			1
	Stream 2 - "Monitor-3"
		Mode: MODECT, frames+analysis, video writer disabled
		Source: FFMPEG TCP MJPEG 640x360x32@20fps ~300kB/s
		Buffers: 
			Image Buffer Size			100 (5 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			3

Camera 2: HiWatch IPC-B542
	Stream 1 - "Monitor-4"
		Mode: RECORD, no analysis, no decoding, camera passthrough
		Source: FFMPEG TCP H.264 2688x1520x32@25fps, I-Frames 20, ~900kB/s
		Linked with Monitor-7
		Buffers: 
			Image Buffer Size			26 (>1 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		25 (1 sec)
			Post Event Image Count		125 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			1
	Stream 2 - "Monitor-7"
		Mode: MODECT, frames+analysis, video writer disabled
		Source: FFMPEG TCP MJPEG 640x360x32@20fps ~700kB/s
		Buffers: 
			Image Buffer Size			100 (5 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			3
During testing of the setup, I've got a lot of error messages like
You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.
It was little bit confusing, because only MJPEG sources are decoded and analysed, and CPU utilisation is nor really high all the time. Then I've noticed, that sometimes the memory consumption is quite high. For better analysis I've installed a Zabbix agent and collected some metrics. The most interesting are the memory usage, CPU usage and memory consumption by the ZMC processes. This is how last 24h looks like:
Zabbix_ZM_day.png
Zabbix_ZM_day.png (215.67 KiB) Viewed 166 times
It is quite an obvious here, that the ZMC process of the Monitor-3 (light-blue line) and of the Monitor-7 (light-green line), where motion scan is activated, are growing till end of memory and then been restarted. Depending on amount of alarms in a period, the cycle can be shorter or longer. For example, in the night there are a lot of alarms because of spider webs in IR lights, so the cycle is quite a short. In the day there are not much movements in the camera view and the cycle is longer. Btw, the shared memory (/dev/shm) utilization stays all the time around 26%.

Then I've tried to take a look deeply in the ZMC processes. So I've catched a moment, when a ZMC process just jumped from about 1.8Gb to 3.1Gb in memory. This is the period between 13:38 and 13:41, which has 2 alarms (light-blue line):
Zabbix_ZMC.png
Zabbix_ZMC.png (48.57 KiB) Viewed 166 times
Using the command "pmap -XX <id>" I've collected detailed process memory information at the beginning and at the end of the period. The report is saved as an OpenOffice table. The interesting thing here is that the process constantly allocates blocks of memory each around 64kB and amount of such blocks just grows with the time. The sum of sizes of the blocks is almost the same as the process size by itself. So it really looks like as a memory leak. All details can be found in the table.

Unfortunately I have no experience with unix-based development at all and just a little with administration. So it can be easly, that my assumption is wrong. Anyway I'll be glad to assist with the problem as much as I can.

lavid
Posts: 1
Joined: Fri Dec 04, 2020 6:00 am

Re: High Ram usage

Post by lavid » Thu May 06, 2021 9:12 pm

Just wanted to add that I'm also getting endless RAM usage when I set Maximum Image Buffer Size (frames) to 0 and when I cap it at... any number, I get endless messages of

Code: Select all

You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	
. ZM was working pretty well for me with the previous version, but I saw the "release coming soon" message elsewhere and wanted to dip my toes in. Happy to offer whatever debugging logs are needed to help get this resolved.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests