Script to convert all events to daily videos

Add any particular hints or tricks you have found to help with your ZoneMinder experience.
mikeyintn
Posts: 5
Joined: Wed Aug 17, 2011 5:46 pm

Script to convert all events to daily videos

Postby mikeyintn » Mon May 16, 2016 8:24 pm

One of my older ZM servers is starting to fill up and I decided it was time to rebuild it. We're required to retain all footage and I wanted to make it a little more accessible if anyone needed to look up events from back over the past four years. Here's a quick-n-dirty bash script I wrote to encode each day's events to a single video file by monitor using ffmpeg. Later, I modified it to run as a cron job to only dump out the previous week's worth of footage and I put a comment in the file how to do this if desired. I was getting about 18:1 compression ratio when comparing the original images with the encoded footage which still looked pretty sharp. But of course you can always adjust this it to be more or less compressed to your liking.

zm_export.sh.zip
(1.57 KiB) Downloaded 186 times


Hopefully it can help someone else who needs to export all the footage to nearline storage or whatnot.
If you have ways of making it better, please do so.. and re-attach I'd like to see what you did!

alabamatoy
Posts: 64
Joined: Sun Jun 05, 2016 2:53 pm

Re: Script to convert all events to daily videos

Postby alabamatoy » Tue Aug 16, 2016 10:37 pm

Very very cool. Thank you greatly.

alabamatoy
Posts: 64
Joined: Sun Jun 05, 2016 2:53 pm

Re: Script to convert all events to daily videos

Postby alabamatoy » Sat Sep 17, 2016 12:41 pm

New, somewhat improved, albeit possibly slower version. This takes as input a numerical month and day and exports all the found events to 1-hour long AVI files. I have been using it for some time now, and it seems to run reliably. I have another very short script which calls it with the correct month and day, and then once its done it deletes the no-longer-necessary events, and also deletes AVI files which are older than a predetermined age, to prevent the drive from filling up. You may need to make adjustments to the ffmpeg call for your specific scenario and settings. Choosing the "verbose" setting in the script is fun to see all the stuff happening, but it produces HUGE text output from the ffmpeg compilation.

https://drive.google.com/open?id=0BytuBa0wusr1aWlfMU96bjVMR3c

Please LMK if you find issues or problems with it. I hope some find this useful. Thanks to mikeyintn for the starting point from which to work.

alabamatoy
Posts: 64
Joined: Sun Jun 05, 2016 2:53 pm

Re: Script to convert all events to daily videos

Postby alabamatoy » Fri Oct 21, 2016 6:18 pm

Due to a very recent change to ffmpeg, you need to add "-safe 0" to the ffmpeg command line in the script, otherwise it fails with an "unsafe filename" error. Apparently a recent change to ffmpeg added functionality that wont allow wildcard filenames by default, so you have to turn the safe filename functionality off.

Falka
Posts: 3
Joined: Sun Jul 31, 2016 6:59 pm

Re: Script to convert all events to daily videos

Postby Falka » Tue Nov 22, 2016 2:56 pm

mikeyintn wrote:One of my older ZM servers is starting to fill up and I decided it was time to rebuild it. We're required to retain all footage and I wanted to make it a little more accessible if anyone needed to look up events from back over the past four years. Here's a quick-n-dirty bash script I wrote to encode each day's events to a single video file by monitor using ffmpeg. Later, I modified it to run as a cron job to only dump out the previous week's worth of footage and I put a comment in the file how to do this if desired. I was getting about 18:1 compression ratio when comparing the original images with the encoded footage which still looked pretty sharp. But of course you can always adjust this it to be more or less compressed to your liking.
zm_export.sh.zip



Hello!
Thank you for sharing the script!

I got stuck at:

Code: Select all

----- this is the directory and filename to save to ------
/mnt/sda1/ 2016-11-06_cam_12.avi
ffmpeg version N-80901-gfebc862 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
  configuration: --extra-libs=-ldl --prefix=/opt/ffmpeg --mandir=/usr/share/man --enable-avresample --disable-debug --enable-nonfree --enable-gpl --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-decoder=amrnb --disable-decoder=amrwb --enable-libpulse --enable-libfreetype --enable-gnutls --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libvorbis --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libspeex --enable-libass --enable-avisynth --enable-libsoxr --enable-libxvid --enable-libvidstab
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 47.100 /  6. 47.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[concat @ 0xa1ae5c0] Unsafe file name '/var/cache/zoneminder/events/12/16/11/06/00/27/36/%05d-capture.jpg'
/tmp/dirs_to_process: Operation not permitted


Can you help me solve this?

Regards,
Falka

Ps.:

Alabamatoy's script also fails at:

Code: Select all

Starting at 2016. nov. 22., kedd, 16:00:44 CET
looking for month = 11 and day = 21

rm: cannot remove '/tmp/*.list': No such file or directory
Finding...

Sorting...

Looking for specific files to compile...


./zm_export_1day.sh: line 148: /tmp/hours.list: No such file or directory


Even if i manually create the hours.list file:

Code: Select all

Starting at 2016. nov. 22., kedd, 16:04:28 CET
looking for month = 11 and day = 21

Finding...

Sorting...

Looking for specific files to compile...


./zm_export_1day.sh: line 148: /tmp/hours.list: No such file or directory

Vaso
Posts: 6
Joined: Mon Sep 19, 2016 6:31 am

Re: Script to convert all events to daily videos

Postby Vaso » Wed Jan 11, 2017 12:50 pm

Alabamatoy's script is really ugly as mentioned inside )))
It will not work for you unless you have the same directory structure as its author has.
1.
If you have events_dir like this:
/usr/share/zoneminder/www/events versus author's /zmdata/events
you should shift arguments number by 3 in lines 88--94

Code: Select all

    CAM="$(echo $EVENTDIRS |awk -F/ '{print $4}')"
    YEAR="$(echo $EVENTDIRS |awk -F/ '{print $5}')"
    MONTH="$(echo $EVENTDIRS |awk -F/ '{print $6}')"
    DAY="$(echo $EVENTDIRS |awk -F/ '{print $7}')"
    HOUR="$(echo $EVENTDIRS |awk -F/ '{print $8}')"
    MIN="$(echo $EVENTDIRS |awk -F/ '{print $9}')"
    DUNNO="$(echo $EVENTDIRS |awk -F/ '{print $10}')"

    to

    CAM="$(echo $EVENTDIRS |awk -F/ '{print $7}')"
    YEAR="$(echo $EVENTDIRS |awk -F/ '{print $8}')"
    MONTH="$(echo $EVENTDIRS |awk -F/ '{print $9}')"
    DAY="$(echo $EVENTDIRS |awk -F/ '{print $10}')"
    HOUR="$(echo $EVENTDIRS |awk -F/ '{print $11}')"
    MIN="$(echo $EVENTDIRS |awk -F/ '{print $12}')"
    DUNNO="$(echo $EVENTDIRS |awk -F/ '{print $13}')"
   


2.
In line 112 "/zmdata/events" have to be replaced with "$events_dir"

3.
In the same line "00" makes impossible to convert events that have "seconds" != "00".
Thats why "DUNNO" should be reincarnate to "SECONDS" and paste in place of "00".

4.
Search method is nightmare!!
Script searches ALL events. What for? We need only ONE day!!

I have resolve above issues in attached script. Try it.
Hope this helps.

P.S. I edit this script in haste so some correction may be a little half-assed. Feel free to improve.
Attachments
zm_export_1day.sh.gz
(2.03 KiB) Downloaded 13 times

ConcreteRooster
Posts: 10
Joined: Fri Jan 24, 2014 6:22 pm

Re: Script to convert all events to daily videos

Postby ConcreteRooster » Thu Feb 02, 2017 4:51 pm

Here's yet another variation on the same theme, this time in Python.

This script works by taking in a top-level directory to search for JPG files; those files will be concatenated into a video file using ffmpeg. Additionally, if requested, each JPG can be run through the ImageMagick program "convert" to scale the size up or down. (Now that I'm writing this I suspect ffmpeg has a way to do this internally; I've been using "convert" for years so it was natural/intuitive for me to use that.) You can disable this by setting the scale argument to 0 or 100 (default is 50).

Here's the --help for an overview:

Code: Select all

$ ./zm_mkvid.py -h
usage: zm_mkvid.py [-h] [--event-dir EVENTDIR] [--verbosity LEVEL] [--dry-run]
                   [--scale SCALE_PCT] [--prefix PREFIX]

Concatenate Zoneminder event image files into video

optional arguments:
  -h, --help            show this help message and exit
  --event-dir EVENTDIR, -d EVENTDIR
                        Top level zoneminder event directory to walk for image
                        files
  --verbosity LEVEL, -v LEVEL
                        Set verbosity/logging level. Valid options: ['INFO',
                        'DEBUG', 'CRITICAL', 'WARNING', 'ERROR']
  --dry-run, -n         Don't actually do anything, just pretend.
  --scale SCALE_PCT, -s SCALE_PCT
                        Specify scale factor for ImageMagick convert.
  --prefix PREFIX, -p PREFIX
                        Prefix for output video file name


So, example command (all on one line):

Code: Select all

./zm_mkvid.py --event-dir=/var/lib/zoneminder/events/2/17/02/02/ --verbosity=DEBUG --scale 75 --prefix=capture_monitor2


This would create a video file named "capture_monitor2_20170202_HHMMSS-20170202_HHMMSS.mkv", where the two HHMMSS fields are replaced by the times of the first and last events for Feb 2, 2017. Since verbosity is set to DEBUG, it will print a bunch of stuff that most people won't be interested in. The images will also be scaled to 75% of their original size before being fed into ffmpeg.

Note I had to append ".txt" to the attachment filename for the forum software to accept it. Obviously you'll want to remove the .txt and "chmod +x".

This was developed/tested under CentOS 7.3, using ZM 1.30.0 from zmrepo with the ImageMagick package installed (provides the "convert" program). It "should" work on other platforms, but might require some tweaking.

Hope someone finds it useful!

zm_mkvid.py.txt
Remove .txt extension
(9.88 KiB) Downloaded 15 times

ConcreteRooster
Posts: 10
Joined: Fri Jan 24, 2014 6:22 pm

Re: Script to convert all events to daily videos

Postby ConcreteRooster » Sun Feb 19, 2017 4:52 pm

Here's a slightly updated version of the script above. Tweaks:
  • --event-dir argument can be passed multiple times
  • --event-dir argument(s) is(are) interpreted as shell glob pattern(s)
  • --help output has been updated to indicate defaults (e.g. 50% scaling)

Enjoy!
Attachments
zm_mkvid.py.txt
(10.38 KiB) Downloaded 6 times

ConcreteRooster
Posts: 10
Joined: Fri Jan 24, 2014 6:22 pm

Re: Script to convert all events to daily videos

Postby ConcreteRooster » Fri Feb 24, 2017 10:19 pm

Another update:
  • Bugfix: the previous --event-dir enhancements could create a situation where the scaled files (or symlinks if not scaling) had colliding filenames. Fixed.
  • Enhancement: added a --no-cleanup option which, if used, will stop the script from cleaning up temp/intermediate files (likely only useful for debugging)
Attachments
zm_mkvid.py.txt
(11.08 KiB) Not downloaded yet


Return to “ZoneMinder Hints & Tips”

Who is online

Users browsing this forum: No registered users and 1 guest