zmc cpu usage with monitor and h264 passthrough

Forum for questions and support relating to the 1.31.x releases only.
ExplodingLemur
Posts: 20
Joined: Mon Sep 17, 2007 5:48 pm

zmc cpu usage with monitor and h264 passthrough

Post by ExplodingLemur »

I set up a monitor with h264 passthrough for an 8MP camera. CPU usage on an i3-4130T is around 70% of one core:
6937 www-data 20 0 1183804 735668 672544 R 70.4 9.1 0:50.17 zmc

Should it be this high when no users are actually looking at the video stream?

Camera configuration: https://imgur.com/a/NJBxwdf
Nothing pops up in the logs.
I installed 1.31.45~20180731103016-bionic in Ubuntu 18.04 via iconnor's zoneminder-master PPA

Thanks!
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: zmc cpu usage with monitor and h264 passthrough

Post by bbunge »

ExplodingLemur wrote: Mon Aug 06, 2018 4:26 pm I set up a monitor with h264 passthrough for an 8MP camera. CPU usage on an i3-4130T is around 70% of one core:
6937 www-data 20 0 1183804 735668 672544 R 70.4 9.1 0:50.17 zmc

Should it be this high when no users are actually looking at the video stream?

Camera configuration: https://imgur.com/a/NJBxwdf
Nothing pops up in the logs.
I installed 1.31.45~20180731103016-bionic in Ubuntu 18.04 via iconnor's zoneminder-master PPA

Thanks!
Bionic has been reported to use more resources and take a performance hit due to security "fixes." It was reported in the ZM 1.30.x area that disabling the security fixes/patches returns performance but with an added security risk.

Of course, we can not recommend disabling a security feature but you are free to expose your network to whatever you want...
ExplodingLemur
Posts: 20
Joined: Mon Sep 17, 2007 5:48 pm

Re: zmc cpu usage with monitor and h264 passthrough

Post by ExplodingLemur »

I looked at this thread and disabled the Spectre V2 and page table isolation mitigations (confirmed the grub settings took effect at next boot via dmesg). It actually increased my CPU utilization. A single monitor, no recording, set to h264 passthrough is now taking 80-90% CPU.
skidd
Posts: 18
Joined: Tue Oct 06, 2015 1:52 pm

Re: zmc cpu usage with monitor and h264 passthrough

Post by skidd »

just wanted to bump this and re-ask the same question. See if there is any insight.
I too am seeing a high CPU load on "record" passthrough H264 streams. Should this be the case? or should passthrough be cheap on cpu?
I have to be honest, I think on a previous version of Zoneminder I had running, the 264 streams in pass-through were infact using very little CPU. I can't remember what version I had, but it was from the repo and off of the master branch. I have not yet tried going back to that version since I'd have to stand up a new database for the older one.. but.. I might do it just to see if it's infact version specific.

I am running the latest from the repo, 1.31.45. compiled and run on an Arch based server with a 4-core cpu.
I have a few cameras on my system, but I've turned them all off except one in order to try and find the load issue.
My cameras all have dual streams setup. all ffmpeg rtsp streams, one MJPG low-res (5fps@704x576) for the analysis, and one H264 high-res (12fps@2048x1536) for non-stop passthrough recording. With triggers linked of course. Currently, I have all but the one high-res "record" left running. htop shows "zmc -m 3" running at an average of 45% CPU. Everything actually seems to work as expected... jpg images on the low-res feed with good events triggered. and nothing but a bunch of mp4 files on the high-res feed. The low-res MJPG based feed uses about 10% cpu (combined zmc & zma), about what I expected.

If I run ffmpeg at my command line on the same machine, just reading the stream with copy on the video codec straight to an mp4 file, it uses at best 2-3% cpu. (ffmpeg -i "rtsp://<user>:<pwd>@cam03:554/cam/realmonitor?channel=1&subtype=0" -c:v copy f.mp4)

I've tried turning on and off as many different things that might affect this as I can find. Nothing seems to lower the cpu load on this one zmc process. No matter what mode I try setting the camera to, it stays about about 45% cpu.

The one thing I noticed, that seemed the most strange. If I switch the H264 feed from "Record" to "Mocord", it adds the zma process to the list. This process only takes about 20% cpu load. the existing zmc process stays at the same cpu load. That's only about 10% higher than the zma process on the low-res feed. This sure seems to me like the high-res zmc process is still somehow extracting images from it, and feeding it into the system when in just "record" mode.

Thanks for anybody who might be able to shed some light on this.
P.S. I've been using ZoneMinder for about 6 years, and always loved it. Just never liked the added cpu load on the 264 based cameras. This pass-through feature, along with mjpg based analysis I think is the best feature yet.

Shane
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: zmc cpu usage with monitor and h264 passthrough

Post by iconnor »

zmc always decodes the video, no matter what. Maybe some future version won't, but right now, it always decodes. You are decoding an 8MP stream... without hardware acceleration... I am not surprised that it is taking 70% of your cpu.
ExplodingLemur
Posts: 20
Joined: Mon Sep 17, 2007 5:48 pm

Re: zmc cpu usage with monitor and h264 passthrough

Post by ExplodingLemur »

What is the purpose of decoding the video stream regardless of settings? If it's set to pass-through only, this serves no purpose.
jwarfin
Posts: 41
Joined: Mon Jul 23, 2018 4:36 am

Re: zmc cpu usage with monitor and h264 passthrough

Post by jwarfin »

ExplodingLemur wrote: Thu Aug 09, 2018 3:31 am I looked at this thread and disabled the Spectre V2 and page table isolation mitigations (confirmed the grub settings took effect at next boot via dmesg). It actually increased my CPU utilization. A single monitor, no recording, set to h264 passthrough is now taking 80-90% CPU.
The Linux Spectre patches have a big impact on I/O - not just general computation. Turning off the Spectre mitigations improved your I/O, resulting in more camera data processed per second and this elevated your CPU utilization. In essence: your system was somewhat I/O bound with the Spectre mitigations enabled, but with the mitigations off you're now somewhat CPU bound.

i3-4130T is just a dual core Haswell @ 2.9ghz. High CPU utilization would be expected for an 8MP 32bit h264 stream @ 30fps. If your goal is to stick with the CPU but achieve a lower load, then I don't see how you could avoid dropping the frame rate and perhaps the resolution too.
ExplodingLemur
Posts: 20
Joined: Mon Sep 17, 2007 5:48 pm

Re: zmc cpu usage with monitor and h264 passthrough

Post by ExplodingLemur »

jwarfin wrote: Thu Aug 23, 2018 3:58 am i3-4130T is just a dual core Haswell @ 2.9ghz. High CPU utilization would be expected for an 8MP 32bit h264 stream @ 30fps. If your goal is to stick with the CPU but achieve a lower load, then I don't see how you could avoid dropping the frame rate and perhaps the resolution too.
If all it's doing is spooling the h.264 directly to a ring buffer, the CPU shouldn't be this busy. Per iconner however
iconnor wrote: Wed Aug 22, 2018 10:07 pmzmc always decodes the video, no matter what.
so even if it's doing absolutely nothing that requires processing the video, it's still decoding it. For monitor it shouldn't need to do anything, and for record it should only have to write the h.264 stream to disk without decompressing or transcoding it. It seems an odd architecture decision to have everything completely dependent on decoded video streams, if someone could explain that would be handy so at least I knew why ZoneMinder worked this way.
jwarfin
Posts: 41
Joined: Mon Jul 23, 2018 4:36 am

Re: zmc cpu usage with monitor and h264 passthrough

Post by jwarfin »

The stream has to be decoded. Otherwise, how could you view the live video via the ZM GUI? I have little experience with 1.31.x (so far), but I expect the h264 passthrough feature allows recording the stream frames directly to disk without re-encoding. That would present less CPU load.

But, after looking at the images of your camera settings again I noticed the "Optional Encoder Options" box. I'm wondering if you need to specify "-c copy" there to tell ffmpeg to explictly passthrough the h264 stream. Or does 1.31.x internally do that when passthrough is selected?
skidd
Posts: 18
Joined: Tue Oct 06, 2015 1:52 pm

Re: zmc cpu usage with monitor and h264 passthrough

Post by skidd »

I was hoping that the stream would just be passed through and stored as is (-c:v copy -c:a copy). My use case has my dual stream camera feeding a MJPEG stream to ZM for motion detection, event review, montage review and just "Hey, who's at the front door". I've found that the rate ZM can feed to my browser, the JPG files for review is excellent. The 264 stream as expected is a tad slower. Then, I intended to have the high-res 264 stream always in a 24/7 7-week rotation. If I found something in the low-res MJPG stream that needed more detailed look , I'd queue up the 264 high-res stream at that location and zoom in on what I was looking for.

As for live-view in ZM.. it would seem to me that it "should" be possible to real-time passthrough the 264 stream into it's MP4 container, and at the same time, serve back that same passthrough stream to the UI.

Perhaps this scenario is a possible future feature addition?
ExplodingLemur
Posts: 20
Joined: Mon Sep 17, 2007 5:48 pm

Re: zmc cpu usage with monitor and h264 passthrough

Post by ExplodingLemur »

jwarfin wrote: Thu Aug 23, 2018 3:58 am i3-4130T is just a dual core Haswell @ 2.9ghz. High CPU utilization would be expected for an 8MP 32bit h264 stream @ 30fps. If your goal is to stick with the CPU but achieve a lower load, then I don't see how you could avoid dropping the frame rate and perhaps the resolution too.
I replaced it with an i7-7700T. It's still 70-80% CPU per stream.
User avatar
zd59
Posts: 102
Joined: Wed Jan 18, 2017 1:39 pm
Location: EU - Slovenia

Re: zmc cpu usage with monitor and h264 passthrough

Post by zd59 »

H!

post:
skidd » Fri Aug 24, 2018 2:48 pm

This should be the right feature of Zoneminder.
Was also surprised, it's not.
All modern IP cameras stream at least two streams with a reason.
The most logical use of them is to minimize CPU and GPU usage by this scenario, I also use:
1. Low res (VGA) stream - video processing & motion detection and live stream watching (computer monitor, tablet, smartphone)
Video processing on low resolution is not CPU/GPU demanding compared to high resolution stream.
2. high resolution (multimega pixel) stream is starting recording (passthrough) with linked triggering from motion detection on low resolution stream, else nothing. So high resolution stream on idle do not use any CPU/GPU, when recording passthrough, minimal.

Expected this behaviour of Zoneminder when passthrough recording was introduced in 1.3.1.x Else passthrough have no benefit except lower disk usage, which is not so important with growing disk spaces. Its easier to bought another disk, replacing a CPU with faster is a lot $$$ issue.

Please developers to consider the above suggestion.
russell_i_brown
Posts: 42
Joined: Wed Mar 18, 2009 9:46 am
Location: Peterborough, England

Re: zmc cpu usage with monitor and h264 passthrough

Post by russell_i_brown »

I'm in the same(ish) boat. Upgraded to 1.32 in the hope that I could record 24/7 from a number of High-Res cameras with passthrough, only to find that zmc still eats CPU in RECORD mode.

However.... I have a thought...

Could one record N minute chunks from the cameras with ffmpeg and inject 'Continuous' events into the zoneminder database?

This would get us low-load continuous recording and the convenience of having the 'events' available through the ZM interface.

Does this sound bonkers or worth playing with?
russell_i_brown
Posts: 42
Joined: Wed Mar 18, 2009 9:46 am
Location: Peterborough, England

Re: zmc cpu usage with monitor and h264 passthrough

Post by russell_i_brown »

OK... I couldn't resist a quick experiment and it does seem to do the job.

I set the camera function to None and running the following script, the camera collects events and they replay through ZM (with a curious log entry see later).

Here's the, crude, script (note: I'm running this on ZM 1.32.0).

Code: Select all

#!/bin/bash
#
#       Grab a video stream from an IP camera into a temp file.
#       Inject an 'Events' record into the ZM Database.
#       Create the directory in the zm storage area
#       Move the mp4 to the right place
#
#       You might have to install mediainfo which I use to grab
#       the frame count.
#
#       ALL VERY CRUDE and merely proof of concept.
#
#       Version 0.1     -       18/ 9/18        First hack
#
#       TODO - Error checking, parameterise things, read paths etc
#       from database.... and lots more :-)
#
MONITOR_ID=29
WIDTH=2048
HEIGHT=1536
# Recording chunk time in seconds
RECORD_TIME=600

STORE_PATH=/var/cache/zoneminder/events/$MONITOR_ID

tmpfile=$(mktemp $STORE_PATH/Rec-XXXXXX.mp4)

while true
do
        #
        #       Record
        START_TIME=`date --rfc-3339=seconds`
        echo ffmpeg -y -loglevel quiet -t $RECORD_TIME -i "rtsp://view:trendcam@192.168.1.135/Streaming/Channels/1" -c:v copy $tmpfile
        ffmpeg -y -loglevel quiet -t $RECORD_TIME -i "rtsp://view:trendcam@192.168.1.135/Streaming/Channels/1" -c:v copy $tmpfile
        END_TIME=`date --rfc-3339=seconds`
        FILESIZE=$(stat -c%s "$tmpfile")
        FRAMES=`mediainfo --Output="Video;%FrameCount%" $tmpfile`

        #
        #       Inject into ZM database
        THIS_ID=`mysql -NBqr zm <<-EOF
                INSERT INTO Events (MonitorId,StorageId,Name,Cause,StartTime,EndTime,Width,Height,Length,Frames,Videoed,DiskSpace,Scheme,StateId,AlarmFrames,SaveJPEGs,Notes)
                values ($MONITOR_ID,0,"New Event","Continuous","$START_TIME","$END_TIME",$WIDTH,$HEIGHT,$RECORD_TIME,$FRAMES,1,$FILESIZE,"Medium",1,0,0,"Injected from ffmpeg capture");
                SET @last_id = LAST_INSERT_ID();
                UPDATE Events SET Name=CONCAT("Event-",@last_id),DefaultVideo=CONCAT(@last_id,"-video.mp4") where Id=@last_id;
                SELECT @last_id;
        EOF`
        if [ "$THIS_ID" -le 0 ]
        then
                echo Woops. Something went wrong with the Mysql injection
                exit 1
        fi
        #
        #       Move recording to the right place
        THIS_PATH=$STORE_PATH/`date --rfc-3339=date`/$THIS_ID
        mkdir -p $THIS_PATH
        mv -f $tmpfile $THIS_PATH/$THIS_ID-video.mp4
        chown -R www-data.www-data $THIS_PATH
        #
done
This creates an mp4 recording direct from the camera every RECORD_TIME seconds, stuffs an entry into the ZM database and puts the mp4 file in the right place in zm's storage.

The events show up on ZM's console screen and play like any other recording.

The log does produce a weird error on playback:
zm-error1.png
zm-error1.png (30.68 KiB) Viewed 28481 times
which I really don't understand. Something to do with the way I'm calling ffmpeg?

So, there we go. A crude way of recording high-res 24/7 without huge CPU load and having the recordings available through the normal ZM interface.

Comments? Suggestions?
User avatar
zd59
Posts: 102
Joined: Wed Jan 18, 2017 1:39 pm
Location: EU - Slovenia

Re: zmc cpu usage with monitor and h264 passthrough

Post by zd59 »

russell_i_brown you are a genious!
Thank you so much.
The exact one I need, with a small change.
I would like to low resolution stream in "modect" trigger event recording by high stream (h264 passthrough) in "none" mode.
Would you be so nice to change your script to do the above?

And for the developers of a Zoneminder - that script should be included in ZM source code.

Regards
Locked