Record High-res H264 streams 24/7 with low CPU Load

If you've made a patch to quick fix a bug or to add a new feature not yet in the main tree then post it here so others can try it out.
Post Reply
russell_i_brown
Posts: 13
Joined: Wed Mar 18, 2009 9:46 am
Location: Peterborough, England
Contact:

Record High-res H264 streams 24/7 with low CPU Load

Post by russell_i_brown » Wed Sep 19, 2018 11:53 am

Even in RECORD mode with H264 passthrough enabled, Zoneminder extracts frames from the stream (confirmed by iconner in this thread). This obviously creates a system load and a significant one when there's a number of high-res cameras involved.

The attached script uses ffmpeg to record the high-res H264 stream from cameras in N second chunks and inserts event records into the Zoneminder database for every chunk so that they can be accessed and managed via the normal interface.

For my setup, I'm recording high-res streams from six IP cameras 24/7 in ten minute chunks and doing MODECT on the low-res streams. This gives me motion events on the low-res and I can go and look at the 3840x2160 mp4 for the appropriate time if needed. This does mean you end with two cameras (low-res MODECT and high-res NONE) in the zoneminder console but hey-ho, I can live with that.

To set this up, firstly get a Monitor for your camera working in MONITOR mode at high-res (the script extracts the Height, Width, Path and StoragePath for a given Monitor from the DB).

Once working, set the new Monitor to 'NONE' so Zoneminder isn't running zmc on it.

Install the attached script (zmrecord.sh) in /usr/local/bin with execute perms.

Test it by running "zmrecord.sh <Monitor_Id>" and you should see it recording into <your_monitors_storage_path>/Monitor_Id/Rec-XXXXX. As each chunk is finished (set the chunk size in seconds by changing RECORD_TIME at the top of the script) , the script moves the mp4 to the right place in your storage, tells Zoneminder's DB all about it and starts recording the next chunk.

Copy the attached systemd (Ugh!) zmrecord@.service script to /lib/systemd/system (or wherever your systemd scripts live) and enable it for each camera with

Code: Select all

systemctl enable zmrecord@<MonitorId>.service
For multiple cameras (for example 8, 12 and 14) you would do:

Code: Select all

systemctl enable zmrecord@8.service zmrecord@12.service zmrecord@14.service
This will tell your system to fire up the ffmpeg recording once it's booted to multi-user mode. You can, of course, start or stop the zmrecord process for individual cameras through the normal systemctl commands:

Code: Select all

systemctl start zmrecord@12.service
      or
systemctl stop zmrecord@12.service
Here's a 'top' on my system recording high-res streams from six cameras with zmc/zma running on various low-res streams (the highest load are some old Axis cameras pulling mjpgs). The ffmpegs reading the high-res hardly feature whereas before, trying to record these high-res streams through Zoneminder, my system was on it's knees.

top.png
top.png (42.17 KiB) Viewed 2851 times
and here's a region of my Zoneminder console showing the 'None' monitors gathering events.

zm.png
zm.png (58.18 KiB) Viewed 2851 times

Enjoy. Comments & suggestions welcome.


Note: the script will only work with Storage scheme 'Medium' (it's what I use).
Note2: You'll need to install the 'mediainfo' package as the script uses it to extract the frame count from the mp4.
Attachments
zmrecord.zip
(1.84 KiB) Downloaded 83 times

russell_i_brown
Posts: 13
Joined: Wed Mar 18, 2009 9:46 am
Location: Peterborough, England
Contact:

Re: Record High-res H264 streams 24/7 with low CPU Load

Post by russell_i_brown » Fri Sep 21, 2018 7:24 am

Small update to the script - I screwed up setting the StorageId in the event. This will only effect you if you're using multiple storage areas.

Attached is version 0.3

There's also an issue with zmaudit getting enthusiastic about clearing out the events as there are no associated Frame records (Frames don't exist as we're recording native H264).

I need to do a little thinking about the right solution so for now I've just set AUDIT_MIN_AGE to 604800 (a week) while I come up with a sensible solution.
Attachments
zmrecord.zip
(1.86 KiB) Downloaded 120 times

Eiserfeyer
Posts: 1
Joined: Thu Apr 13, 2017 9:17 am

Re: Record High-res H264 streams 24/7 with low CPU Load

Post by Eiserfeyer » Thu Oct 11, 2018 6:30 pm

Thanks for your work. Your thoughts seem to have gone in the same direction as mines... But you seem to have the solution :mrgreen:
I will try to check this script at weekend an will let you know if it also works for me.

tigerA
Posts: 7
Joined: Thu Sep 27, 2018 6:35 am

Re: Record High-res H264 streams 24/7 with low CPU Load

Post by tigerA » Wed Oct 17, 2018 12:55 pm

From my own experience.
I set: Monitor -> Source -> Ffmpeg. Protocol RTSP.
In Storage tab: Save JPEGs -> Disabled, VideoWriter - > H264 Camera Passtrough

Pedulla
Posts: 133
Joined: Thu Nov 27, 2014 11:16 am
Location: Portland, Or

Re: Record High-res H264 streams 24/7 with low CPU Load

Post by Pedulla » Mon Nov 12, 2018 9:29 pm

Are you getting audio with this too?

ebeng
Posts: 4
Joined: Tue May 28, 2019 6:30 pm

Re: Record High-res H264 streams 24/7 with low CPU Load

Post by ebeng » Tue May 28, 2019 10:22 pm

Thanks for the script.

I've had to create a <.mylogin.cnf> file for the MySQL statements, because when cleartext passowrd is entered in the script, you will get warning from MySQL:

Code: Select all

mysql_config_editor set --login-path=local --host=localhost --user=db_user --password
I've replaced all the

Code: Select all

mysql -NBqr zm -e 
to

Code: Select all

mysql --login-path=local -NBqr zm -e 
because of the database for ZoneMinder is with a user/pass.

However, bumped up to following error when testing the zmrecord script:

Code: Select all

root@zoneminder:~# zmrecord.sh 1
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1
/usr/local/bin/zmrecord.sh Monitor Data Error. Got: Width 1920 1080 rtsp://xxx:xxx@10.10.10.3:554/cam/realmonitor?channel=1&subtype=0 0 Height  ReadPath  StorePath /1
root@zoneminder:~#
This is when executing the first 2 commands in the script.

Code: Select all

ubuntu@zoneminder:~$ mysql --login-path=local -NBqr zm -e "select Width,Height,Path,StorageId from Monitors where Id=1"
1920    1080    rtsp://xxx:xxx@10.10.10.3:554/cam/realmonitor?channel=1&subtype=0     0
ubuntu@zoneminder:~$ mysql --login-path=local -NBqr zm -e "select Path from Storage where Id=0"
/var/cache/zoneminder/events
ubuntu@zoneminder:~$
So, I did split up the command to get the variables:

Code: Select all

read -r WIDTH <<< `mysql --login-path=local -NBqr zm -e "select Width from Monitors where Id=$MONITOR_ID"`
read -r HEIGHT <<< `mysql --login-path=local -NBqr zm -e "select Height from Monitors where Id=$MONITOR_ID"`
read -r READPATH <<< `mysql --login-path=local -NBqr zm -e "select Path from Monitors where Id=$MONITOR_ID"`
read -r STORAGEID <<< `mysql --login-path=local -NBqr zm -e "select StorageId from Monitors where Id=$MONITOR_ID"`
read -r STORAGE <<< `mysql --login-path=local -NBqr zm -e "select Path from Storage where Id=$STORAGEID"`
STORE_PATH=$STORAGE/$MONITOR_ID
This started the stream to be captured !!

Code: Select all

root      7929  7921  0 20:40 pts/1    00:00:00 ffmpeg -y -loglevel quiet -t 600 -i rtsp://xxx:xxx@10.10.10.3:554/cam/realmonitor?channel=1&subtype=0 -c:v copy /var/cache/zoneminder/events/1/Rec-m4b5nR.mp4
Now i've copied the zmrecord@.service script to the folder:

Code: Select all

root@zoneminder:/lib/systemd/system# systemctl enable zmrecord@1.service zmrecord@4.service zmrecord@5.service zmrecord@6.service
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@1.service to /lib/systemd/system/zmrecord@.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@4.service to /lib/systemd/system/zmrecord@.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@5.service to /lib/systemd/system/zmrecord@.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@6.service to /lib/systemd/system/zmrecord@.service.
when want to start the stream for the ID=1 camera, it still tries to run mysql with the user 'root':

Code: Select all

May 28 22:09:56 zoneminder systemd[1]: Started Record Stream from Camera 1.
May 28 22:09:56 zoneminder zmrecord.sh[3979]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
May 28 22:09:56 zoneminder zmrecord.sh[3979]: message repeated 4 times: [ ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)]
May 28 22:09:56 zoneminder zmrecord.sh[3979]: /usr/local/bin/zmrecord.sh Monitor Data Error. Got: Width  Height  ReadPath  StorePath /1
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Main process exited, code=exited, status=1/FAILURE
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Unit entered failed state.
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Failed with result 'exit-code'.
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Service hold-off time over, scheduling restart.
May 28 22:09:56 zoneminder systemd[1]: Stopped Record Stream from Camera 1.

It seems to be that the SYSTEMCTL is running with root, I've also defined in the service file + make it start after zoneminder + mysql:

Code: Select all

[Unit]
Description=Record Stream from Camera %I
After=mysql.service ssh.service networking.service zoneminder.service


[Service]
Type=simple
ExecStart=/usr/local/bin/zmrecord.sh %I
Restart=on-failure
User=root

[Install]
WantedBy=multi-user.target


after you did create the <.mylogin.cnf>, just copy this file also to the root. It will work via there.

THX for the script!!!!

With my server, the CPU is like 1-2% now for everything.... before with ZoneMinder for every capture it was like 80%!!

This is how the HIGH-RES is being recorde

Code: Select all

 1489 root      20   0  427940  30608  24020 S   0.3  0.2   0:00.33 ffmpeg
 1488 root      20   0  428004  30516  23900 S   0.3  0.2   0:00.25 ffmpeg
 1492 root      20   0  428000  30484  23840 S   0.3  0.2   0:00.37 ffmpeg
 1487 root      20   0  427948  30312  23712 S   0.3  0.2   0:00.33 ffmpeg

This is for the MODECT... huge difference!!!

Code: Select all

 1954 www-data  20   0 2493396 1.443g 1.177g R  92.7  9.2   1:26.94 zmc
 1887 www-data  20   0 2493352 1.442g 1.177g R  98.3  9.2   2:17.66 zmc
 1783 www-data  20   0 2493220 1.442g 1.177g R  95.0  9.2   4:18.39 zmc
 1922 www-data  20   0 2493488 1.442g 1.177g R 107.0  9.2   1:49.09 zmc
 1964 www-data  20   0 1572148 1.206g 1.173g S  21.6  7.7   0:21.19 zma
 1891 www-data  20   0 1562024 1.196g 1.171g S  21.6  7.6   0:29.87 zma
 1932 www-data  20   0 1562024 1.195g 1.171g S  22.6  7.6   0:21.95 zma
 1803 www-data  20   0 1560004 1.194g 1.172g S  20.6  7.6   0:58.16 zma

tuxmos
Posts: 16
Joined: Fri Oct 30, 2015 11:13 am

Re: Record High-res H264 streams 24/7 with low CPU Load

Post by tuxmos » Sat Jun 29, 2019 10:33 am

Thanks for the work. It works so well. I have made adjustments to myself:
Sometimes a stream stops, or stops recording. The service will not restart automatically. You would have to install a watchdoc for it. First of all, I helped to restart all stream services once an hour. Unfortunately, it always remains an mp4 corpse ($ tmpfile) in the root folder of the camera. I inserted a "rm $ STORE_PATH / *. Mp4" in zmrecord.sh after "STORE_PATH = $ STORAGE / $ MONITOR_ID".

Furthermore, all new events in this recording are now called "Event-NUMBER", regardless of "EventPrefix". That would have to be corrected. Is very helpful in filtering the events.
EDIT:
Insert after read Parameters:
read -r EPREFIX <<< `mysql -NBqr ZM -e "select EventPrefix from Monitors where Id=$MONITOR_ID"`
and change "Event-" in UPDATE Events SET Name=CONCAT(... to "$EPREFIX"

What will probably be even more difficult, is that the whole with rotated cameras does not really work / can work. Or? Can ffmpeg also pass the value "Orientation", so that e.g. a camera that is 90 ° rotated, so that a high-profile video is recorded correctly?

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests