Strange Memory consuption pattern Decode Ondemand vs KeyFrames

Current Development version likely to have breaking changes
Post Reply
MarsRover
Posts: 21
Joined: Sat Apr 30, 2022 10:34 am

Strange Memory consuption pattern Decode Ondemand vs KeyFrames

Post by MarsRover »

Hi again!
1.37.15_20220507113926-focal
I've noticed strange zmc behavior when Decoding is changed between KeyFrames and Ondemand, and when there's no Decoding demand (Record always, passthrough, no Analysis and no cameras are watched Live - "pure" DVR setup).

KeyFrames -> Ondemand:
CPU goes DOWN for most zmc processes >x2 this is understandable. Stays the same after. But for some znc processes it goes up, this is strange..
Memory for most zmc processes goes UP about x2 - this is strange. Stays the same after. Goes down for some zmc processes, this is also strange, although it's what I was expecting to acheive

The same, but mirrored, happens on the swithcback.

Why this is happening?

Zabbix Graph for zmc processes attached (system with 12 cameras connected. Bulk change set Decoding='Ondemand' in zm.Monitiors table and then systemctl restart zoneminder at ~ 23:03:00. Bold lines are the sum of %CPU or %MEM for all zmc processes (right axis). All other lines reflect CPU and Mem metrics for individual zmc processes (left axis). Recording SectionLength=600.
All metrics are collected once per minute by parsing ps output

Code: Select all

 ~$ ps -C zmc -o %mem -o "|%C|" -o cmd
%MEM|%CPU|CMD
 6.7| 1.6|/usr/bin/zmc -m 1
 7.8| 1.8|/usr/bin/zmc -m 2
 6.8| 1.7|/usr/bin/zmc -m 3
 6.6| 2.1|/usr/bin/zmc -m 4
 7.9| 2.1|/usr/bin/zmc -m 5
 7.8| 2.1|/usr/bin/zmc -m 6
 8.2| 1.9|/usr/bin/zmc -m 8
 7.0| 1.8|/usr/bin/zmc -m 9
11.1| 2.3|/usr/bin/zmc -m 11
11.0| 2.7|/usr/bin/zmc -m 14
 0.1| 1.4|/usr/bin/zmc -m 15
 0.1| 1.3|/usr/bin/zmc -m 16
Attachments
Zabbix Graphs
Zabbix Graphs
KeyFrames_to_Ondemand.jpg (305.2 KiB) Viewed 4719 times
User avatar
iconnor
Posts: 2881
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Strange Memory consuption pattern Decode Ondemand vs KeyFrames

Post by iconnor »

Do you have rtsp server turned on for any of the monitors? It counts as a viewer.

I'm not seeing this here.

Turn on debug at level 3 and send me the log. isaac@zoneminder.com
MarsRover
Posts: 21
Joined: Sat Apr 30, 2022 10:34 am

Re: Strange Memory consuption pattern Decode Ondemand vs KeyFrames

Post by MarsRover »

No rtsp server, no Janus Live Stream for all cameras.

Code: Select all

"JanusEnabled":false
"RTSPServer":false
Even thumbnails are off on the server level.

debug log for _zmc_mX component?

offtopic:
how do I query events with EndDateTime IS NULL via API?
/api/events/index/MonitorId:4/EndDateTime%20IS:NULL.json gives me:
"check the manual that corresponds to your MySQL server version for the right syntax to use near '= 'NULL'"
MarsRover
Posts: 21
Joined: Sat Apr 30, 2022 10:34 am

Re: Strange Memory consuption pattern Decode Ondemand vs KeyFrames

Post by MarsRover »

OK. More pictures. I've created a new Dashboard to illustrate different Decoding settings. No other setting are changed.
I have 6 servers with 8-12 cameras of different type, FPS and resolution on each, and I'm able to reproduce this behavior on any of them.
segment 1: Ondemand
segment 2: KeyFrames + Ondemand
segment 3: KeyFrames Only
more meaningful than thousand words. CPU is Red, Memory is Yellow
Debug Log:
tail | grep 'Last Viewed'

Code: Select all

05/11/22 20:16:40.869666 zmc_m4[437322].DB1-zm_monitor.h/636 [Last viewed 1652282200 seconds ago]
05/11/22 20:16:40.879522 zmc_m4[437322].DB1-zm_monitor.h/636 [Last viewed 1652282200 seconds ago]
05/11/22 20:16:40.930528 zmc_m4[437322].DB1-zm_monitor.h/636 [Last viewed 1652282200 seconds ago]
05/11/22 20:16:40.930567 zmc_m4[437322].DB1-zm_monitor.h/636 [Last viewed 1652282200 seconds ago]
05/11/22 20:16:40.949591 zmc_m4[437322].DB1-zm_monitor.h/636 [Last viewed 1652282200 seconds ago]
tail | grep -i 'no Viewers' - nothing
Attachments
Zabbix Graphs
Zabbix Graphs
Ondeamand+KeyFrames.jpg (270.41 KiB) Viewed 4626 times
MarsRover
Posts: 21
Joined: Sat Apr 30, 2022 10:34 am

Re: Strange Memory consuption pattern Decode Ondemand vs KeyFrames

Post by MarsRover »

Playing around with decoding for some more time, this is what I've found:
Thumbnails Enabled
Ondemand eats more memory than KeyFrames Only.
KeyFrames + Ondemand eats even more, because (my guess) decoded keyframes aren't used for thumbnail generation and zmc does the same as with Decoding='Ondemand' for thumbnails.

Thumbnails Disabled
See no point in KeyFrames + Ondemand if Analysis and thumbnails are off.

Debug=3 log
https://drive.google.com/file/d/18671l2 ... sp=sharing
Post Reply