High Ram usage

[iconnor]

Options -> Logging
Check DEBUG
LOG_DEBUG_TARGET => _zmc
LOG_DEBUG_LEVLE => 4

Restart zoneminder.

Now /var/log/zm/zmc_m[monitor id].log will have the logs.

[Cramac1]

At 1.35.16, it took me 500MB for 1 camera, after the 1.5GB update. How to fix? Buffer reduced to 2
cameras just stand to record.

debug log:
https://disk.yandex.ru/d/NpQVhj0I5GjPMQ
https://disk.yandex.ru/d/59TnGhjHmWSm4A

[Cramac1]

Oops. I presume they do some cleaning up. Was there when I grabbed it a while back.

Try this www.magic919.co.uk/zoneminder_1.35.16~2 ... _amd64.deb

can you update the link?

[Magic919]

I've reinstated it. Please try now.

[Cramac1]

I've reinstated it. Please try now.

txh

[dmidthun]

Possible? Yes.

You didn't say whether you are doing passthrough or encode. Encoding takes a ton of ram too (default settings will buffer 20 frames in the encoder).
Passthrough can buffer some stuff but it is encoded data and should just be ref counted so shouldn't amount to much.


--I'm using passthough. I tried to keep it as simple as possible... though sometimes it gets a little fickle. I left most of the settings close to default minus some tweaks from the upgrade (buffer from 20->2, logging, api settings etc)

I'm noticing that the memory issue is only happening when the cameras are set to record of any kind, and then drops back off when the event(s) are complete and the process starts over again (~10 mins) . I also noticed that there are 4 zmc processes per camera enabled. Could this also be attributing to the resource hogging? I have my install running with 24G of ram, 12G of swap and can literally watch the usage increase even though there is little to no change in the image being captured (an empty room in my house that gets no foot traffic -- ever). 10 rtsp cameras set to monitor == 4.5~5.5G RAM; change 1 to Mocord, system will again start hoarding memory.

Isaac, is there possibly a memory leak in the zmc process when in record mode?


Thanks,
Dan

Setup:
Ubuntu 18.04
ZM 1.35/master (latest)
8 Merkury 720p 'smartcams; hacked'
Ovirt 4.4.4 KVM Env - 2CPU, 24G RAM, 72G vdrive; 60sda 12sdb(swap) - Nimble backend storage @ 10G FC

[Cramac1]

who tried it, fixed it?

[P_Dmitrij]

Hi all, I've faced the same problem and would like to share my findings. But first of all, many thanks to the ZM developers and contributors for such a great development! Although it cost me couple of weeks to get more familiar with the product and to build a more or less useful setup, I think the ZM is actually the best CCTV solution on the free software market. Especially the version 1.35 with the option to disable video decoding! It just a killer feature for me, which allows to implement a hi-res CCTV system with very limited hardware resources.

My last setup, where the problem exists is follow:

Code: Select all

Server:
i3-4330T with 16GB RAM
OS Proxmox
VM with 2 cores and 8GB RAM limit
Ubuntu 20.04.2 LTS (Focal Fossa)
Zoneminder v1.35.27
Storage: WD Purple WD20PURZ thru NFS (served by Proxmox)

ZoneMinder settings:
Camera 1: Hikvision DS-2CD2142FWD
	Stream 1 - "Monitor-1"
		Mode: RECORD, no analysis, no decoding, camera passthrough
		Source: FFMPEG TCP H.264 2688x1520x32@20fps, I-Frames 20, ~800kB/s
		Linked with Monitor-3
		Buffers: 
			Image Buffer Size			21 (>1 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			1
	Stream 2 - "Monitor-3"
		Mode: MODECT, frames+analysis, video writer disabled
		Source: FFMPEG TCP MJPEG 640x360x32@20fps ~300kB/s
		Buffers: 
			Image Buffer Size			100 (5 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			3

Camera 2: HiWatch IPC-B542
	Stream 1 - "Monitor-4"
		Mode: RECORD, no analysis, no decoding, camera passthrough
		Source: FFMPEG TCP H.264 2688x1520x32@25fps, I-Frames 20, ~900kB/s
		Linked with Monitor-7
		Buffers: 
			Image Buffer Size			26 (>1 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		25 (1 sec)
			Post Event Image Count		125 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			1
	Stream 2 - "Monitor-7"
		Mode: MODECT, frames+analysis, video writer disabled
		Source: FFMPEG TCP MJPEG 640x360x32@20fps ~700kB/s
		Buffers: 
			Image Buffer Size			100 (5 sec)
			Maximum Image Buffer Size	0
			Warmup Frames				0
			Pre Event Image Count		20 (1 sec)
			Post Event Image Count		100 (5 sec)
			Stream Replay Image Buffer	0
			Alarm Frame Count			3

During testing of the setup, I've got a lot of error messages like

You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.

It was little bit confusing, because only MJPEG sources are decoded and analysed, and CPU utilisation is nor really high all the time. Then I've noticed, that sometimes the memory consumption is quite high. For better analysis I've installed a Zabbix agent and collected some metrics. The most interesting are the memory usage, CPU usage and memory consumption by the ZMC processes. This is how last 24h looks like:

Zabbix_ZM_day.png (215.67 KiB) Viewed 38581 times

It is quite an obvious here, that the ZMC process of the Monitor-3 (light-blue line) and of the Monitor-7 (light-green line), where motion scan is activated, are growing till end of memory and then been restarted. Depending on amount of alarms in a period, the cycle can be shorter or longer. For example, in the night there are a lot of alarms because of spider webs in IR lights, so the cycle is quite a short. In the day there are not much movements in the camera view and the cycle is longer. Btw, the shared memory (/dev/shm) utilization stays all the time around 26%.

Then I've tried to take a look deeply in the ZMC processes. So I've catched a moment, when a ZMC process just jumped from about 1.8Gb to 3.1Gb in memory. This is the period between 13:38 and 13:41, which has 2 alarms (light-blue line):

Zabbix_ZMC.png (48.57 KiB) Viewed 38581 times

Using the command "pmap -XX <id>" I've collected detailed process memory information at the beginning and at the end of the period. The report is saved as an OpenOffice table. The interesting thing here is that the process constantly allocates blocks of memory each around 64kB and amount of such blocks just grows with the time. The sum of sizes of the blocks is almost the same as the process size by itself. So it really looks like as a memory leak. All details can be found in the table.

Unfortunately I have no experience with unix-based development at all and just a little with administration. So it can be easly, that my assumption is wrong. Anyway I'll be glad to assist with the problem as much as I can.

[lavid]

Just wanted to add that I'm also getting endless RAM usage when I set Maximum Image Buffer Size (frames) to 0 and when I cap it at... any number, I get endless messages of

Code: Select all

You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	

. ZM was working pretty well for me with the previous version, but I saw the "release coming soon" message elsewhere and wanted to dip my toes in. Happy to offer whatever debugging logs are needed to help get this resolved.

[iconnor]

I fixed another mem leak over the weekend. Please update and report.

[P_Dmitrij]

Hello iconnor, many thanks for the update! Now the memory consumption looks much better:

ZM_Memory_after_Update.png (109.51 KiB) Viewed 38296 times

The graph shows, that before the update the memory consumption was growing constantly and now it is quite stable. I will continue to collect stats and let you know if the problem will appear again.

[lavid]

Still racking up a lot of these:

Code: Select all

2021-05-12 15:51:57	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m14		320413	WAR	You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:57	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m14		320413	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:57	zmc_m14		320413	WAR	You have set the max video packets in the queue to 50. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:57	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:57	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:56	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:56	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:56	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-12 15:51:56	zmc_m13		320408	WAR	You have set the max video packets in the queue to 20. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
2021-05-12 15:51:56	zmc_m13		320408	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135

and getting a lot of small events. You can see the normal length (10 minute) events before the update here:

Screen Shot 2021-05-12 at 3.53.09 PM.png (393.87 KiB) Viewed 38291 times

[iconnor]

You may just need to set it higher. Increase gradually until they go away.
Or reduce your keyframe interval.
Alternatively there may be something going on that is blocking the event writing process.

Will need debug level 3 logs.

You aren't by any chance using de-interlacing?

[P_Dmitrij]

It looks, that ZMC takes much more buffers, as it needed for Pre Event Image Count and Key Frames. For testing I've configured my HiWatch camera to constant bitrate with 25 fps and 12 key frames:

cam02_main_stream.png (76.28 KiB) Viewed 38137 times

Then I've limited the Maximum Image Buffer Size with 30 frames:

zmc_cam02_new.png (67.25 KiB) Viewed 38136 times

As for my understanding, it should be enough to set the limit even to 25 frames, because 25 frames are needed for the Pre Event Image Count and they will contain definitely at least one key frame. The Analysis function is disabled at this monitor. But actually I've got just lot of errors like

Code: Select all

You have set the max video packets in the queue to 30. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.

It seems, that we have other factors, that increasing the needed buffer size.

[iconnor]

You havn't been reading along.

Set your imagebuffercount to 3.