High Ram usage

Previous development branch now released as 1.36
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor »

Options -> Logging
Check DEBUG
LOG_DEBUG_TARGET => _zmc
LOG_DEBUG_LEVLE => 4

Restart zoneminder.

Now /var/log/zm/zmc_m[monitor id].log will have the logs.
Cramac1
Posts: 111
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 »

At 1.35.16, it took me 500MB for 1 camera, after the 1.5GB update. How to fix? Buffer reduced to 2
cameras just stand to record.

debug log:
https://disk.yandex.ru/d/NpQVhj0I5GjPMQ
https://disk.yandex.ru/d/59TnGhjHmWSm4A
Cramac1
Posts: 111
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 »

Magic919 wrote: Mon Mar 01, 2021 1:46 pm Oops. I presume they do some cleaning up. Was there when I grabbed it a while back.

Try this www.magic919.co.uk/zoneminder_1.35.16~2 ... _amd64.deb
can you update the link?
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: High Ram usage

Post by Magic919 »

I've reinstated it. Please try now.
-
Cramac1
Posts: 111
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 »

Magic919 wrote: Sat Mar 27, 2021 3:40 pm I've reinstated it. Please try now.
txh
dmidthun
Posts: 6
Joined: Mon Mar 22, 2021 7:50 pm

Re: High Ram usage

Post by dmidthun »

iconnor wrote: Thu Mar 25, 2021 5:13 pm Possible? Yes.

You didn't say whether you are doing passthrough or encode. Encoding takes a ton of ram too (default settings will buffer 20 frames in the encoder).
Passthrough can buffer some stuff but it is encoded data and should just be ref counted so shouldn't amount to much.


--I'm using passthough. I tried to keep it as simple as possible... though sometimes it gets a little fickle. I left most of the settings close to default minus some tweaks from the upgrade (buffer from 20->2, logging, api settings etc)

dmidthun wrote: Wed Mar 24, 2021 6:20 pm I'm noticing that the memory issue is only happening when the cameras are set to record of any kind, and then drops back off when the event(s) are complete and the process starts over again (~10 mins) . I also noticed that there are 4 zmc processes per camera enabled. Could this also be attributing to the resource hogging? I have my install running with 24G of ram, 12G of swap and can literally watch the usage increase even though there is little to no change in the image being captured (an empty room in my house that gets no foot traffic -- ever). 10 rtsp cameras set to monitor == 4.5~5.5G RAM; change 1 to Mocord, system will again start hoarding memory.

Isaac, is there possibly a memory leak in the zmc process when in record mode?


Thanks,
Dan

Setup:
Ubuntu 18.04
ZM 1.35/master (latest)
8 Merkury 720p 'smartcams; hacked'
Ovirt 4.4.4 KVM Env - 2CPU, 24G RAM, 72G vdrive; 60sda 12sdb(swap) - Nimble backend storage @ 10G FC
Cramac1
Posts: 111
Joined: Wed Aug 29, 2018 11:29 am

Re: High Ram usage

Post by Cramac1 »

who tried it, fixed it?
P_Dmitrij
Posts: 5
Joined: Tue May 04, 2021 5:36 pm

Re: High Ram usage

Post by P_Dmitrij »

Hi all, I've faced the same problem and would like to share my findings. But first of all, many thanks to the ZM developers and contributors for such a great development! Although it cost me couple of weeks to get more familiar with the product and to build a more or less useful setup, I think the ZM is actually the best CCTV solution on the free software market. Especially the version 1.35 with the option to disable video decoding! It just a killer feature for me, which allows to implement a hi-res CCTV system with very limited hardware resources.

My last setup, where the problem exists is follow:

Code: Select all

Server:
i3-4330T with 16GB RAM
OS Proxmox
VM with 2 cores and 8GB RAM limit
Ubuntu 20.04.2 LTS (Focal Fossa)
Zoneminder v1.35.27
Storage: WD Purple WD20PURZ thru NFS (served by Proxmox)

ZoneMinder settings:
Camera 1: Hikvision DS-2CD2142FWD
	Stream 1 - "Monitor-1"
		Mode: RECORD, no analysis, no decoding, camera passthrough
		Source: FFMPEG TCP H.264 2688x1520x32@20fps, I-Frames 20, ~800kB/s
		Linked with Monitor-3
		Buffers: 
			Image Buffer Size			21 (>1 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			1
	Stream 2 - "Monitor-3"
		Mode: MODECT, frames+analysis, video writer disabled
		Source: FFMPEG TCP MJPEG 640x360x32@20fps ~300kB/s
		Buffers: 
			Image Buffer Size			100 (5 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			3

Camera 2: HiWatch IPC-B542
	Stream 1 - "Monitor-4"
		Mode: RECORD, no analysis, no decoding, camera passthrough
		Source: FFMPEG TCP H.264 2688x1520x32@25fps, I-Frames 20, ~900kB/s
		Linked with Monitor-7
		Buffers: 
			Image Buffer Size			26 (>1 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		25 (1 sec)
			Post Event Image Count		125 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			1
	Stream 2 - "Monitor-7"
		Mode: MODECT, frames+analysis, video writer disabled
		Source: FFMPEG TCP MJPEG 640x360x32@20fps ~700kB/s
		Buffers: 
			Image Buffer Size			100 (5 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			3
During testing of the setup, I've got a lot of error messages like
You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.
It was little bit confusing, because only MJPEG sources are decoded and analysed, and CPU utilisation is nor really high all the time. Then I've noticed, that sometimes the memory consumption is quite high. For better analysis I've installed a Zabbix agent and collected some metrics. The most interesting are the memory usage, CPU usage and memory consumption by the ZMC processes. This is how last 24h looks like:
Zabbix_ZM_day.png
Zabbix_ZM_day.png (215.67 KiB) Viewed 20163 times
It is quite an obvious here, that the ZMC process of the Monitor-3 (light-blue line) and of the Monitor-7 (light-green line), where motion scan is activated, are growing till end of memory and then been restarted. Depending on amount of alarms in a period, the cycle can be shorter or longer. For example, in the night there are a lot of alarms because of spider webs in IR lights, so the cycle is quite a short. In the day there are not much movements in the camera view and the cycle is longer. Btw, the shared memory (/dev/shm) utilization stays all the time around 26%.

Then I've tried to take a look deeply in the ZMC processes. So I've catched a moment, when a ZMC process just jumped from about 1.8Gb to 3.1Gb in memory. This is the period between 13:38 and 13:41, which has 2 alarms (light-blue line):
Zabbix_ZMC.png
Zabbix_ZMC.png (48.57 KiB) Viewed 20163 times
Using the command "pmap -XX <id>" I've collected detailed process memory information at the beginning and at the end of the period. The report is saved as an OpenOffice table. The interesting thing here is that the process constantly allocates blocks of memory each around 64kB and amount of such blocks just grows with the time. The sum of sizes of the blocks is almost the same as the process size by itself. So it really looks like as a memory leak. All details can be found in the table.

Unfortunately I have no experience with unix-based development at all and just a little with administration. So it can be easly, that my assumption is wrong. Anyway I'll be glad to assist with the problem as much as I can.
lavid
Posts: 4
Joined: Fri Dec 04, 2020 6:00 am

Re: High Ram usage

Post by lavid »

Just wanted to add that I'm also getting endless RAM usage when I set Maximum Image Buffer Size (frames) to 0 and when I cap it at... any number, I get endless messages of

Code: Select all

You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	
. ZM was working pretty well for me with the previous version, but I saw the "release coming soon" message elsewhere and wanted to dip my toes in. Happy to offer whatever debugging logs are needed to help get this resolved.
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor »

I fixed another mem leak over the weekend. Please update and report.
P_Dmitrij
Posts: 5
Joined: Tue May 04, 2021 5:36 pm

Re: High Ram usage

Post by P_Dmitrij »

Hello iconnor, many thanks for the update! Now the memory consumption looks much better:
ZM_Memory_after_Update.png
ZM_Memory_after_Update.png (109.51 KiB) Viewed 19878 times
The graph shows, that before the update the memory consumption was growing constantly and now it is quite stable. I will continue to collect stats and let you know if the problem will appear again.
lavid
Posts: 4
Joined: Fri Dec 04, 2020 6:00 am

Re: High Ram usage

Post by lavid »

Still racking up a lot of these:

Code: Select all

2021-05-12 15:51:57	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m14		320413	WAR	You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:57	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m14		320413	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:57	zmc_m14		320413	WAR	You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:57	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:56	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:56	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:56	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:56	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:56	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
and getting a lot of small events. You can see the normal length (10 minute) events before the update here:
Screen Shot 2021-05-12 at 3.53.09 PM.png
Screen Shot 2021-05-12 at 3.53.09 PM.png (393.87 KiB) Viewed 19873 times
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor »

You may just need to set it higher. Increase gradually until they go away.
Or reduce your keyframe interval.
Alternatively there may be something going on that is blocking the event writing process.

Will need debug level 3 logs.

You aren't by any chance using de-interlacing?
P_Dmitrij
Posts: 5
Joined: Tue May 04, 2021 5:36 pm

Re: High Ram usage

Post by P_Dmitrij »

It looks, that ZMC takes much more buffers, as it needed for Pre Event Image Count and Key Frames. For testing I've configured my HiWatch camera to constant bitrate with 25 fps and 12 key frames:
cam02_main_stream.png
cam02_main_stream.png (76.28 KiB) Viewed 19719 times
Then I've limited the Maximum Image Buffer Size with 30 frames:
zmc_cam02_new.png
zmc_cam02_new.png (67.25 KiB) Viewed 19718 times
As for my understanding, it should be enough to set the limit even to 25 frames, because 25 frames are needed for the Pre Event Image Count and they will contain definitely at least one key frame. The Analysis function is disabled at this monitor. But actually I've got just lot of errors like

Code: Select all

You have set the max video packets in the queue to 30. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.
It seems, that we have other factors, that increasing the needed buffer size.
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: High Ram usage

Post by iconnor »

You havn't been reading along.

Set your imagebuffercount to 3.
Locked