Integrated automated DB backup

If you've made a patch to quick fix a bug or to add a new feature not yet in the main tree then post it here so others can try it out.
User avatar
rdmelin
Posts: 863
Joined: Wed Oct 29, 2003 2:23 pm
Location: Ellensburg, WA USA

Integrated automated DB backup

Post by rdmelin »

I have put together a group of scripts that provide automatic complete zm db backup and recovery.

The system consist of four scripts:

zm_db_backup This script uses mysqldump to backup the tables that contain your complete configuration including monitors and zones. The backup file is written in /var/lib/zm. This can be run manually anytime configuration changes are made. A provision is made to run it weekly automatically.

zmeventdump This script is to be run by a zm filter of the "Automatically execute" type. It writes a .sql file in each event directory that the restore utilty finds and reloads into the db.
Please note that events can only be acted on by one "Automatically execute" filter. Any events already acted upon by such a filter will not be backed up.

zm_db_restore This utility reinitializes the db, restores the config backup, and searches the events path to reload all the events.
Again please note that if zm is running zmaudit.pl will delete events found in the filesystem but not in the db or vice versa. So ensure that zm is stopped when attempting to use zm_db_restore. If you reinstall zm from RPM you can use zm_db_restore instead of zminit to initialize and restore the db before starting zm.
If you are trying to recover important data, and any of this is unclear, please ask for assistance. Once events are deleted from a linux filesystem they will be extremely difficult to recover.

/etc/logrotate.d/zm This taps the function of logrotate to make automatic weekly backups of the configs.

These scripts are included in the latest Mandriva zm_1.22.2 RPM and thus on the new ZMliveCD. The shell scripts are found in /usr/lib/zm/bin

To finish setup of the automatic system the user will need to create a filter to catch the desired events. Something like "Id equal to or greater than 0" will catch all events for complete coverage. Save with the option "Automatically execute command on all matches: " and use the full path to zmeventdump. This command in a terminal will set up this filter on a new installation.

Code: Select all

echo "INSERT INTO Filters VALUES ('EventDump','a:4:{s:5:\"terms\";a:1:{i:0;a:3:{s:4:\"attr\";s:9:\"MonitorId\";s:2:\"op\";s:2:\">=\";s:3:\"val\";s:1:\"0\";}}s:10:\"sort_field\";s:8:\"DateTime\";s:8:\"sort_asc\";s:1:\"1\";s:5:\"limit\";s:0:\"\";}'/usr/lib/zm/bin/zmeventdump',0);" |mysql -u zm_admin -pzaq1xsw2 zm
Then run /usr/lib/zm/bin/zm_db_backup once manually since lograte will ignore it if the backup file doesn't exist.

Check that /var/lib/zm/zm_backup.sql has been created, and that events have a .sql file. If so you arebacked up against database corruption, failed upgrade attempts :) and other unforseens.

With minor modification these should be adaptable to FC or most any distro.

Here they are:

Code: Select all

#!/bin/bash
#===============================================================================
#
#         FILE:  zm_db_backup
#
#        USAGE:  ./zm_db_backup
#
#  DESCRIPTION:  Uses mysqldump to backup the config info in the zm DB
#      OPTIONS:  ---  None
# REQUIREMENTS:  ---  mysqldump
#         BUGS:  ---
#        NOTES:  ---
#       AUTHOR:   Ross Melin <rdmelin@yahoo.com>
#      COMPANY:
#      VERSION:  2.0
#      CREATED:  05/26/2006 06:21:00 AM PDT
#     REVISION:  ---
#===============================================================================

# Edit these to suit your configuration
ZM_CONFIG=/etc/zm.conf
source $ZM_CONFIG

MYSQLDUMP=/usr/bin/mysqldump
BACKUP_PATH=/var/lib/zm
BACKUP_FILE=zm_backup.sql
DUMPOPTS="--user=$ZM_DB_USER --password=$ZM_DB_PASS --opt"
TABLES="Config Filters Groups Monitors States TriggersX10 Users Zones"
OUTFILE="$BACKUP_PATH/$BACKUP_FILE"


echo "--
--- Created by zm_db_backup for ZoneMinder Version $ZM_VERSION
--" >  $OUTFILE

$MYSQLDUMP $DUMPOPTS zm $TABLES >> $OUTFILE

exit 0

Code: Select all

#!/bin/bash
#===============================================================================
#
#         FILE:  zmeventdump
#
#        USAGE:  ./zmeventdump <MonitorName>/<EventId>
#
#  DESCRIPTION:  Uses mysqldump to create a .sql file for individual zm
#                events to make Event table recovery possible by doing a
#                'find' search in ZoneMinder the events directory
#
#      OPTIONS:  ---  None
# REQUIREMENTS:  ---  mysqldump
#         BUGS:  ---
#        NOTES:  ---
#       AUTHOR:   Ross Melin <rdmelin@yahoo.com>
#      COMPANY:
#      VERSION:  2.0
#      CREATED:  05/26/2006 06:21:00 AM PDT
#     REVISION:  ---
#===============================================================================

# Edit these to suit your configuration
ZM_CONFIG=/etc/zm.conf
EVENTS_DIR=/var/www/html/zm/events
MYSQLDUMP=/usr/bin/mysqldump
# The rest should not need editing

# Get the mysql user and password
source $ZM_CONFIG

EVENT_PATH=$1
EVENT_ID=$(echo $1 |cut -f 2 -d / )
MYDUMPOPTS="--user=$ZM_DB_USER --password=$ZM_DB_PASS --skip-opt --compact --quick --no-create-info"


# Dump the sql statements needed to reload the Events, Frames and Stats tables

echo "--- ZM_DB_VERSION=$ZM_VERSION
        " >  $EVENTS_DIR/$EVENT_PATH/.sql

$MYSQLDUMP $MYDUMPOPTS --where="Id=$EVENT_ID" zm Events >> $EVENTS_DIR/$EVENT_PATH/.sql
$MYSQLDUMP $MYDUMPOPTS --where="Eventid=$EVENT_ID" zm Frames >> $EVENTS_DIR/$EVENT_PATH/.sql
$MYSQLDUMP $MYDUMPOPTS --where="Eventid=$EVENT_ID" zm Stats >> $EVENTS_DIR/$EVENT_PATH/.sql


exit 0

Code: Select all

#!/bin/bash
#===============================================================================
#
#         FILE:  zm_db_restore
#
#        USAGE:  ./zm_db_restore
#
#  DESCRIPTION:  Restore a ZoneMinder DB from a backup created by zm_db_backup
#
#      OPTIONS:  ---
# REQUIREMENTS:  ---
#         BUGS:  ---
#        NOTES:  ---
#       AUTHOR:   (),
#      COMPANY:
#      VERSION:  1.0
#      CREATED:  05/29/2006 04:45:06 PM PDT
#     REVISION:  ---
#===============================================================================

ZM_CONFIG=/etc/zm.conf
ZM_BACKUP=/var/lib/zm/zm_backup.sql
EVENTS_DIR=/var/www/html/zm/events

loadcfg() {
        if [ -f $ZM_CONFIG ]; then
                . $ZM_CONFIG
        else
                echo "ERROR: $ZM_CONFIG not found."
                exit 1
        fi
}

chkcfg(){
for n in ZM_DB_HOST ZM_DB_NAME ZM_DB_USER ZM_DB_PASS; do
        eval "val=\$$n"
        if [ "$val" = "" ]; then
                echo "ERROR($ZM_CONFIG): $n should not be empty."
                echo "Enter a $n for ZM to use the Database."
                        if [ "$n" = "ZM_DB_PASS" ]; then
                                echo -n "Will not echo on screen $n : "
                                stty -echo    # Turns off screen echo.
                                read newval
                                stty echo     # Restores screen echo.
                                echo ""
                                ### The following can be used to generate a random password
                                # randstr newval 16
                        else
                                echo -n "$n : "
                                read newval
                        fi
                cp $ZM_CONFIG /tmp/$$ &&
                sed 's/^'$n='.*$/'$n=$newval'/g' /tmp/$$ >$ZM_CONFIG

        fi
done

        if [ "$ZM_DB_HOST" = "localhost" ]
        then
                ClientHost=localhost
        else
                ClientHost=`hostname`
        fi
}

reloadcfg(){
        loadcfg
}

chk_backup_ver(){
        if
                [ -e $ZM_BACKUP  ]
        then
                BACKUP_VER=$(cat $ZM_BACKUP | head -n 2 |tail -n 1 |cut -f 8 -d " ")
        else
                echo "$ZM_BACKUP doesn't exist"
                exit 1
        fi


        if
                [ $BACKUP_VER != $ZM_VERSION ]
        then
                echo "$ZM_BACKUP is from version $BACKUP_VER"
                echo "ZoneMinder version is $ZM_VERSION"
                exit 1
        fi
}

getmylogin(){
        echo  "Enter MySQL Administrator username"
        echo  "(Default: root and password is blank)"
        echo -n "MySQL Admin: "
        read MYADMIN
        echo -n "Password:  "
        read MYPASS
        if [ "X$MYPASS" != "X" ]; then MYPASS="-p$MYPASS"; fi
        echo "\q" |mysql -u $MYADMIN $MYPASS || exit 0

}

checkfordb(){

        if
                echo "show databases" |mysql -u $MYADMIN "$MYPASS" |grep zm
        then
                echo "A $ZM_DB_NAME database exists."
                while [ true ]
                        do
                                echo "Choose one of the following options:"
                                echo "[D]rop the old database and reinitialize"
                                echo "[E]xit and do nothing"
                                read OPTION
                                case $OPTION in
                                        "D"|"d")
                                        echo "drop database zm;"|mysql -u $MYADMIN $MYPASS
                                        return
                                        ;;
                                        "E"|"e")
                                        exit 0
                                        ;;
                                esac
                        done

        fi
}

initdb(){
        sql=/tmp/zm.crdb.sql
        echo "" >$sql
        chmod 600 $sql

        echo "CREATE DATABASE $ZM_DB_NAME;" >>$sql
        echo "USE $ZM_DB_NAME;" >>$sql

        echo "GRANT all on $ZM_DB_NAME.* TO '$ZM_DB_USER'@'$ClientHost' IDENTIFIED BY '$ZM_DB_PASS';" >>$sql

        cat $sql | mysql -B -h $ZM_DB_HOST -u $MYADMIN $MYPASS
        rm -f $sql

        cat $ZM_PATH_UPDATE/zm_create.sql | mysql -h $ZM_DB_HOST -u $ZM_DB_USER -p$ZM_DB_PASS $ZM_DB_NAME

}

restoredb(){
        if
                [ -e $ZM_BACKUP  ]
        then
                cat $ZM_BACKUP | mysql -h $ZM_DB_HOST -u $ZM_DB_USER -p$ZM_DB_PASS $ZM_DB_NAME
        else
                echo "$ZM_BACKUP doesn't exist"
                exit 1
        fi
}

restore_events(){
        for SQL in  $(find $EVENTS_DIR -name .sql)
        do
                cat $SQL | mysql -h $ZM_DB_HOST -u $ZM_DB_USER -p$ZM_DB_PASS $ZM_DB_NAME
        done
}

loadcfg
chkcfg
reloadcfg
chk_backup_ver
getmylogin
checkfordb
initdb
restoredb
restore_events
exit 0
And /etc/logrotate.d/zm

Code: Select all

# First the log files

/var/log/zm/*log {
    weekly
    rotate 3
    notifempty
    missingok
}

Now the weekly db backup

/var/lib/zm/zm_backup.sql {
weekly
rotate 3
missingok
compress
postrotate
/usr/lib/zm/bin/zm_db_backup
endscript
}
Last edited by rdmelin on Mon Feb 11, 2008 3:15 pm, edited 10 times in total.
jameswilson
Posts: 5111
Joined: Wed Jun 08, 2005 8:07 pm
Location: Midlands UK

Post by jameswilson »

excellent Thanks Ross
James Wilson

Disclaimer: The above is pure theory and may work on a good day with the wind behind it. etc etc.
http://www.securitywarehouse.co.uk
User avatar
zoneminder
Site Admin
Posts: 5215
Joined: Wed Jul 09, 2003 2:07 pm
Location: Bristol, UK
Contact:

Post by zoneminder »

Yes, thanks for posting these Ross. Can you stick a reference to them in the Wiki as well if not there already.
Phil
User avatar
MJN
Posts: 251
Joined: Wed Jan 17, 2007 10:29 am
Location: Wiltshire, UK

Post by MJN »

I've just ran these scripts on my Ubuntu installation however hit a slight snag which I circumvented, but would appreciate some clarification as to a) what a particular line is doing, and b) whether my 'fix' was correct (it seemed to do the job!).

The line in question is the following from zm_db_restore:

Code: Select all

cat $ZM_PATH_UPDATE/zmschema.sql | mysql -h $ZM_DB_HOST -u $ZM_DB_USER -p$ZM_DB_PASS $ZM_DB_NAME
Firstly, $ZM_PATH_UPDATE wasn't specified (I'm guessing this should come from zm.conf? Mine didn't have it in... however it does have a $ZM_PATH_BUILD which led me on to my fix....

zmchema.sql didn't exist either, anywhere. However I found zm_create.sql in $ZM_PATH_BUILD/db/ and guessed that this is a 'fresh' database? (please, no sniggering at my lack of SQL knowledge) and lo-and-behold making the necessary changes appeared to work (i.e. I deleted my entire /var/lib/mysql/zm directory/database, ran the restore and everything's back up-and-running.

Could someone have a quick look over what I did and say whether it was correct? Was zm_create.sql the correct file to use (I'm using v1.22.3 for what that's worth). And howcome the default script didn't work out of the box? Should I have zmschema.sql somewhere?

If it's not obvious by now I know *nothing* about SQL hence I was flying blind on this one.... and whilst I landed it I'd like to know exactly how!

Cheers,

Mathew

P.S. Real ignorance time - does it not suffice to simply backup (and restore) the /var/lib/mysql/zm and <webroot>/events directories at the filesystem level...? Does this work? Would it cover everything?
User avatar
rdmelin
Posts: 863
Joined: Wed Oct 29, 2003 2:23 pm
Location: Ellensburg, WA USA

Post by rdmelin »

Looks like you done good. :)
And howcome the default script didn't work out of the box?
I developed the script on and for the livecd installation which uses zm RPMs. Some adjustments are needed to create packages (RPMs and I assume the same will be true for DEBs). For example there will be no valid $ZM_PATH_BUILD since zm isn't built on the system. The zmschema vs zm_create thing is a carry over from some versions ago when IIRC the source package changed but I kept the old name in the RPM package. If I change it now it will mean maintaining multiple versions of these scripts, and that ain't likely.
User avatar
MJN
Posts: 251
Joined: Wed Jan 17, 2007 10:29 am
Location: Wiltshire, UK

Post by MJN »

That's great - many thanks for the speedy follow-up.

I hear what you're saying re the lack of ZM_PATH_BUILD - I didn't realise at the time but I guess it makes sense that on a packaged build it would have all the necessary files to hand that I had (temporarily) in /usr/local/src. Given they've now gone I've kept a copy of zm_create.sql local to the scripts for future use.

Do you think therefore that my use of zm_create.sql was the right thing to do? It certainly seems to be working fine - and looking at what .sql files I had available it seemed the most likely candidate - all the others were for upgrading from previous versions.

Oh, and forgive my manners for not saying so earlier, but thanks for the scripts - they work great! (now... ;))

Mathew
User avatar
rdmelin
Posts: 863
Joined: Wed Oct 29, 2003 2:23 pm
Location: Ellensburg, WA USA

Post by rdmelin »

Do you think therefore that my use of zm_create.sql was the right thing to do?
Exactly. As I say, zmschema.sql was the old name which I have retained in the RPM packages, zm_create.sql is the current name for the correct file.
thanks for the scripts - they work great! (now... Wink)
You're welcome. I'm glad someone is using them. And good job getting them going on Ubuntu.
Last edited by rdmelin on Sun Jun 17, 2007 2:17 pm, edited 1 time in total.
User avatar
robi
Posts: 477
Joined: Sat Mar 17, 2007 10:48 am

Post by robi »

Hm. For some reason I don't get the sql file between the events, but the big one is there in \var\lib\zm. Filters set up okay.

Any ideas?
v1.25.0 + Ubuntu Linux 12.04 Server
User avatar
rdmelin
Posts: 863
Joined: Wed Oct 29, 2003 2:23 pm
Location: Ellensburg, WA USA

Post by rdmelin »

Well first double check that it's not there. It's .sql so is hidden. Try ls -a in an event directory, or if using a file manager like konqueror set view to show hidden files.
User avatar
robi
Posts: 477
Joined: Sat Mar 17, 2007 10:48 am

Post by robi »

rdmelin wrote:Well first double check that it's not there. It's .sql so is hidden. Try ls -a in an event directory, or if using a file manager like konqueror set view to show hidden files.
Nope, they're not there. Only jpegs.
v1.25.0 + Ubuntu Linux 12.04 Server
User avatar
rdmelin
Posts: 863
Joined: Wed Oct 29, 2003 2:23 pm
Location: Ellensburg, WA USA

Post by rdmelin »

OK so the .sql files are created by a script "zmeventdump" run by filter "EventDump". The other backups are are created by a script run by logrotate. So the issue is with the filter.

Posting this running from ZMliveCD 1.22.3 I switched the preconfigured monitor to modect and forced an alarm with the link in the watch window. This snippet is from /var/log/zm/zmfilter.log
06/17/07 15:19:08.899562 zmfilter[4939].DBG [Sleeping for 60 seconds]
06/17/07 15:20:08.905726 zmfilter[4939].DBG [Checking filter 'EventDump', execute]
06/17/07 15:20:08.906772 zmfilter[4939].DBG [Checking event 1]
06/17/07 15:20:08.906995 zmfilter[4939].INF [Executing '/usr/lib/zm/bin/zmeventdump 1/1']
06/17/07 15:20:13.311537 zmfilter[4939].DBG [Output: ]
06/17/07 15:20:13.312669 zmfilter[4939].DBG [Sleeping for 60 seconds]
So "out of the box" the filter is running the script.

The output of "ls -a' in the event directory is:
[guest guest]$ ls -a /var/www/html/zm/events/1/1/
./ 062-capture.jpg 125-capture.jpg 188-capture.jpg
../ 063-capture.jpg 126-capture.jpg 189-capture.jpg
001-capture.jpg 064-capture.jpg 127-capture.jpg 190-capture.jpg
002-capture.jpg 065-capture.jpg 128-capture.jpg 191-capture.jpg
...
058-capture.jpg 121-capture.jpg 184-capture.jpg 247-capture.jpg
059-capture.jpg 122-capture.jpg 185-capture.jpg 248-capture.jpg
060-capture.jpg 123-capture.jpg 186-capture.jpg .sql
061-capture.jpg 124-capture.jpg 187-capture.jpg
So again "out of the box" the script is working correctly.

Look for clues (or lack of them)in /var/log/zm/zmfilter.log to see if the filter is running. If you made changes to the filter and saved them you can use the liveCD as a reference to correct it.
User avatar
robi
Posts: 477
Joined: Sat Mar 17, 2007 10:48 am

Post by robi »

Well I have in zmfilter.log
06/18/07 21:13:58.872734 zmfilter[4668].DBG [Sleeping for 60 seconds]
06/18/07 21:14:58.876332 zmfilter[4668].DBG [Checking filter 'EventDump', execute]
06/18/07 21:14:58.884996 zmfilter[4668].DBG [Checking event 16457]
06/18/07 21:14:58.885296 zmfilter[4668].INF [Executing '/usr/lib/zm/bin/zmeventdump 8/16457']
06/18/07 21:14:58.899448 zmfilter[4668].DBG [Output: ]
06/18/07 21:14:58.900749 zmfilter[4668].DBG [Checking filter 'PurgeWhenFull', delete]
06/18/07 21:14:58.904682 zmfilter[4668].DBG [Sleeping for 60 seconds]
and my ls -a still lists only jpegs. :?
v1.25.0 + Ubuntu Linux 12.04 Server
User avatar
rdmelin
Posts: 863
Joined: Wed Oct 29, 2003 2:23 pm
Location: Ellensburg, WA USA

Post by rdmelin »

Try running the script from a konsole and see if any errors.

"sudo -u apache /usr/lib/zm/bin/zmeventdump 8/16457" or any convenient event in place of 8/16457
User avatar
robi
Posts: 477
Joined: Sat Mar 17, 2007 10:48 am

Post by robi »

It says "No such file or directory".
I have my things in var\www\zm\events\... the /html is not there in the path, as I moved the location. This must be it... thanks.
Where should I fix?
v1.25.0 + Ubuntu Linux 12.04 Server
User avatar
rdmelin
Posts: 863
Joined: Wed Oct 29, 2003 2:23 pm
Location: Ellensburg, WA USA

Post by rdmelin »

Edit /usr/lib/zm/bin/zmeventdump

Change the line that reads
EVENTS_DIR=/var/www/html/zm/events
Post Reply