Some easy tips on improving ZM 1.28.x performance

Forum for questions and support relating to the 1.28.x releases only.
Locked
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Some easy tips on improving ZM 1.28.x performance

Post by asker »

Hello, I thought it may be useful for me to share some tips on performance improvement for an "Average Joe" environment

After the simple optimizations I did listed below, my average load balance went down from a consistent 6.x to between 0.8 to 1.3 when monitoring and between 2.xx - 4.xx during record on motion (4.xx when actual recoding is in effect due to motion along with audit kicked in and me logged in)

I think these are excellent improvements, without me having to reduce quality of capture. I unfortunately did not keep a tab of what changes resulted in what improvements.

Disclaimer: I discovered ZM last week and have only spent 2-3 days setting it up, so obviously I am no expert. Some of my assumptions may be incorrect.

Definition of "Average Joe Environment"
  • You are not running ZM on a high end machine
  • You don't have a dedicated computer for ZM - you also use it for other purposes
  • You don't really want to reduce quality of images captured - after all, this is the era of HD videos. Why would you want to buy good quality cameras only to record very low quality images?
  • You probably want ZM to dump images and events to an external NAS that you already have instead of clogging up local HDD space and causing panics - you want to store at least a week's worth of recordings

Where I started and where I am

I installed Zoneminder on Ubuntu 14.04 Desktop. After initial install, and configuring 3 cameras (two cameras of 1280x960 and one with 640x480) to record on motion detect, my load average was consistently above 6. Access was slow, various buffer overruns in logs etc.

My computer specs
  • 2GB RAM (from summing up output from dmidecode)
  • AMD Athlon(tm) II Neo K325 Dual-Core Processor, 1.3GHz (from /proc/cpuinfo)
  • Cache: 1MB
  • HDD: 100G (It's a dual partition computer, one running Win7 when needed)
  • External NAS: I have an 8TB external NAS mapped to this computer via NFS (this is where I store all the events/images of ZM - I don't use local HDD. The local HDD only stores the MySQL tables which are the structures, but not the image data)
What else do I have running on the machine besides ZM
As I said, this is not a dedicated machine. I suspect this is the case with many of you.
  • I have Ubuntu Desktop installed (not server)
  • I have OpenVPN server running - I use this machine as a VPN gateway when I occasionally need to connect remotely to other LAN machines
  • I have TeamViewer running (expensive) -- I sometimes need to access the full UI of the ubuntu desktop (I know tightVNC etc are more efficient, but I have TV running on every other machine and it was just convenient)
My ZoneMinder set up
  • ZM 1.28.1 and associated software is installed on the machine above
  • All images are dumped by ZM to my NAS via a mapped NFS folder (not using local HDD)
  • Currently controlling 2 Foscam FI9831W cameras (1280x960) and 1 Foscam FI8198W camera (640x480). Will soon be installing two more FI9831W
  • Recording mode: Set to "detect on motion" for the 2 hi-res cameras and just motion on the low-res one. I change 'run states' based on time of day as well (later)
List of Optimizations I did

Ubuntu changes, not related to ZM

As I said earlier, I need to use Ubuntu desktop as it is used for other stuff. Changing to Ubuntu server was not a convenient option (if it came to it, I would).
Instead, I replaced the fancy ubuntu desktop with lubuntu's LXDE desktop. I completely removed the current unity desktop and replaced it with LXDE. Ideally, if I had known all of this before, I would have started off with the full lubuntu distribution, but I did not want to go through the re-install process. I'd highly recommend this.


Changes related to ZM

lib-jpeg turbo and your hopes of super performance improvement
The first thing you come across in ZM performance optimization searches on the Internet is the possibility of replacing your jpeg libraries with lib-jpeg turbo. To do that, you need to install lib-jpeg turbo first and re-compile ZM from source so it picks up the libraries. Just switching LD_LIBRARY_PATH to the new libs does not work, even if you name them the same (potentially dangerous, but...). BUT WAIT. Before you waste your time on it, you may already be using it. If you are running Ubuntu, its very likely your system already has and is already using this optimized library. I wasted a good amount of time making ZM work with a freshly downloaded turbo library, only to be told later by user knnniggett that I already had it (see here). So before you go about wasting half a day, do yourself a favor and check that first.

Then, I read through ZM's own Wiki for load management here. There are various good tips, but given this day and age of HD cameras, NAS storages I did not want to reduce capture quality. Nor did I want to reduce FPS to 1 per second. Plus a lot of the tips are probably a little outdated - recent versions seem to have already taken care of various tips there

List of my tips that helped in significant load reduction (average)

All of these changes can be made after you log in as admin and click on the "Options" link at the top right.
  • Changes Made to Options-->Display:
  • Not performance related. But oh man! When you first install ZM it comes up with a UI that looks straight out of 1970. I groaned. Well, ZM comes with a "flat" Skin too.
    **Please** use it. Nice looking and responsive. ZM_SKIN is the parameter to change. The only problem is this UI has a bug. If you create run states, you will find you can't save a new run state. As a work around, just type in the name, hit Tab and the Save button works)

    Changes Made to Options-->System:
  • FILTER_RELOAD_DELAY set to 3600 (1 hr). This is the duration after which ZM checks if you have either created or made changes to filters. While these changes are save to the DB immediately, the running ZM process only checks after this duration. I think the default was 60s. Unless you are constantly changing your filters, I see no reason why 1 hr would not be sufficient.
  • FILTER_EXECUTE_INTERVAL set to 3600 (1 hr). Default was again something like 60. This is the duration in seconds after which ZM will run through all filters created and execute those that match. I have one filter set, running in the background -- one is to purge all events and images after 5 days. Obviously I don't need ZM to check as often as 60 seconds. The only event I find where 1 hr may be too late if you have configured your ZM to receive emails for motion detect. To be honest, after playing with the ZM motion detect options, I've come to the conclusion that you either live with fine tuning to a level where some obvious motions will be missed, or, broad tune it to a point where you will get *many* false motion detects. I'd rather opt for the latter. If I set it up to email me every time a false detect happened, I'd be deluged. So anyway, for me this interval was sufficient. YMMV
  • AUDIT_CHECK_INTERVAL set to 3600 (1 hr). Again, I think default was around a minute (forgot). This is the interval after which ZMA does an audit of the DB and Filesystem to make sure there are no corruptions etc. Again, I think a 1 hr checkpoint is very reasonable.
  • OPT_FRAME_SERVER currently unchecked, but I will enable it to see if it makes a difference. Basically, if I understood it right, the ZM analysis daemon (ZMA) both analyzes image frames (needed for motion detect) and writes the images to disk. If you have a slow disk, the writing I/O may take more time and while ZMA does that too, it may not be able to analyze new frames if the rate is high relative to the CPU/memory it has. If you check this, a new processes is spawned that takes care of writing the images to disk after analysis (ZMF). Seems like a good idea, but unless I see logs stating problems, I may do this later.
  • BTW, I strongly recommend enabling OPT_USE_AUTH, using AUTH_TYPE as remote and configuring users, so you need to authenticate before you log into ZM's interface. Its default install allows
    anyone with the URL to directly access everything

    System-->Config
    make sure these are enabled:
  • CPU_EXTENSIONS, FAST_IMAGE_BLENDS, OPT_ADAPTIVE_SKIP
  • I disabled CREATE_ANALYSIS_IMAGES. Basically, if you use motion detect, you will see red outlines around areas that detected motion. Given the amount of false positives in ZM, I find this pretty useless, unless I am fine tuning motion detection algorithms. If there is real motion I care about (an intruder, something falling etc), it will be pretty evident looking at the image captured

    System-->Web
    Okay, not performance related but I found this useful:
  • WEB_EVENT_SORT_ORDER - change to desc if you want to see latest events on the top (assuming you are sorting on DateTime)
    I also increased events per page to 50 (WEB_EVENTS_PER_PAGE), checked WEB_LIST_THUMBS and set WEB_LIST_THUMB_WIDTH to 64. These are all, strictly speaking 'ahi-performance' but the impact was not noticeable and I believe only generated when you actually load the page via your browser dynamically (guessing). However, they do make the UI easier to use

    System-->Logging
  • LOG_LEVEL_SYSLOG is currently set to INFO. It is useful to see at least INFO level logs. I suspect I will see more benefit if I make it none, but I'm satisfied at my current load level. If you look at the Log window, even INFO level logs are very copious. Logs involve Disk I/O. Unless you are absolutely sure of your system, I'd recommend keeping LOG at atleast INFO.

    Configuring Zones for better detection (not applicable if you are in continuous record mode without motion detection, or monitor mode)

    Warning: If you have created custom zones BE CAREFUL OF FIDDLING, PANNING, MOVING YOUR CAMERA. If you've moved the camera around ,chances are your zone layer is not properly aligned to what you first created. So its best NOT to move the camera after defining zones. If your camera is not fixed, go back and check your zones once in a while. (This warning is theoretical - I haven't moved my cameras, but it seems obvious to me zones will get messed up if its view moves. I am pretty sure ZM won't be using a complex algorithm to auto adjust its zone points...)

    ZM's Zone documentation explains zoning in detail and I think people should have an idea of this. It helps. You can read up about all sorts of Zone types and algorithms, but I'm just putting in information that I consider 'basic' and everyone should consider:
  • When you first add a camera, an "All" zone is created that basically monitors the entire canvas for activity.
  • Look at your camera image and see if you can cut out certain areas - example, ceiling, side walls etc. Focus on defining a zone where there may be movement
  • You can approach this in two ways, you either reduce the active zone to areas where you have movement, or, overlay inactive zones on top of the full canvas active zone. When I first discovered zoning, I quickly landed up creating 4 zones per camera - adding layers of inactivity on top of my default full canvas zone. It then occurred to me this will likely be more expensive because for each frame ZM has to apply the inactive layers to the active canvas, get the difference and then run modect. Furthermore, I also realized that unless you have a PhD in motion detection, it is very hard to define modect parameters that don't result in many false positives while making sure actual motion is not missed.
  • So, I took the following approach - I reduced my active zone monitoring area and also made sure I did not create a very complex polygon frame with many points. I really don't see the need of exactly matching inactive contours -- it adds to algorithmic complexity and given the amount of false positives, I thought it was better to keep it simple. YMMV of course.
  • Bottom line, I have a max of 2 zones in my cameras, both active and simple polygon shapes. They work pretty accurately and don't miss motions (and of course generates gobs of false positives, which I am fine with)
  • Obviously, how many zones you need are completely upto your situation - just keep in mind, don't keep layering zone on top of zone - think of simplifying existing ones. I've seen tutorials on the net where people have described how to create accurate zones carefully crafting the zone around flower vases, chairs etc. Folks, this is not a photoshop mask :) By adding gobs of polygon points, you are adding to the complexity of ZM's motion detection range.

    Using a secondary stream of your camera

    While I am not in favor of drastic reductions to capture size or FPS, it may make a lot of sense to tune down from say 1080p to 720p. 720p is also excellent quality.
    Unfortunately, I have not yet found a mechanism where I can monitor at 1080p while recording at 720p at the same time thru ZM (you can always use 720 with ZM and access the camera directly for 1080p). You can use run states, but you won't be able to simultaneously record and view at two different rates through ZM (please correct me if I am wrong). Different cameras have different ways in which you can create a 'main stream' and a 'sub stream'. You can specify your main stream to be full resolution and sub stream to be, say, 720p. For Foscam FI9831W, the RTSP urls then become:

    Code: Select all

    rtsp://user:password@ip:port/videoMain
    rtsp://user:password@ip:port/videoSub
    
    You can choose to point to one or the other in your "Source" tab of ZM.

    Using Run States

    This part may or may not apply to you. And this is possibly the least intuitive UI element in ZM :-) When you log in, you will see a status on stop saying "Running" or "Stopped" indicating that ZM is, well, running or stopped. As it turns out, if you click on it, you can create other "States". The ZM document does describe this, but who reads everything? So anyway, a "state" is nothing but a snapshot of your current configuration. You can create different states (which camera to set on what mode, other parameters in your system via the UI etc). Once the states are created and stored, you can simply change states by running /usr/bin/zmpkg.pl yournewstate

    And to make scheduled state changes, you use the linux cron facility to change states based on day of the week and time. This helps in sustained performance management. If you really don't need continuous recording or motion recording all the time, use this to give breaks to your system

    Here is my setup: I want my basement motion detector to be on all the time, 7 days a week. But I want my family room camera to only record motion when my family and I are not sitting around. So here is my crontab configuration for that:

    Code: Select all

    # motionmonitor - all in monitor
    # motionrecord - all in modect
    #familyroommonitor - family room in monitor, basement in record
    
    # cron format - m h  dom mon dow   command
    
    ## MON-FRI RULES
    # switch to motion recording at 8:45 am M-F
    45 8 * * 1-5 /usr/bin/zmpkg.pl motionrecord
    
    # switch to motion monitor at 6:30pm M-F
    30 18 * * 1-5 /usr/bin/zmpkg.pl familyroommonitor
    
    # switch to motion recording 8:45PM M-F
    45 20 * * 1-5 /usr/bin/zmpkg.pl motionrecord
    
    # switch to motion monitor 7AM M-F
    0 7 * *  1-5 /usr/bin/zmpkg.pl familyroommonitor
    
    ## SAT-SUN RULES
    #switch to motion recording - my weekend plans vary so just record
    0 0 * *  6-7 /usr/bin/zmpkg.pl motionrecord
    #switch to motion monitor at 7AM Sat-Sun
    #0 7 * *  6-7 /usr/bin/zmpkg.pl familyroommonitor
    
Last edited by asker on Thu Mar 05, 2015 11:53 am, edited 3 times in total.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Some easy tips on improving ZM 1.28.x performance

Post by bbunge »

Good start! Will have to chew on some of your stuff..

Have been running and learning ZM for several years and just got into Run States today and got it to work to set mocord back to modect at night on 13 of 16 cameras and back to mocord in the morning. Am interested in your weekend cron!

bb
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Some easy tips on improving ZM 1.28.x performance

Post by asker »

Thanks for reading bbunge. My weekend cron task is just to keep modect on all the time, its weekdays where I switch.
bbunge wrote:Good start! Will have to chew on some of your stuff..

Have been running and learning ZM for several years and just got into Run States today and got it to work to set mocord back to modect at night on 13 of 16 cameras and back to mocord in the morning. Am interested in your weekend cron!

bb
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
jkemp
Posts: 13
Joined: Thu Mar 05, 2015 10:20 pm

Re: Some easy tips on improving ZM 1.28.x performance

Post by jkemp »

NAS setup?
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Some easy tips on improving ZM 1.28.x performance

Post by bbunge »

Use another drive:

http://www.zoneminder.com/wiki/index.ph ... Hard_Drive

Talks about using NAS.

bb
jkemp
Posts: 13
Joined: Thu Mar 05, 2015 10:20 pm

Re: Some easy tips on improving ZM 1.28.x performance

Post by jkemp »

Thanks, bbunge.
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: Some easy tips on improving ZM 1.28.x performance

Post by knight-of-ni »

Speaking of performance, everyone should spend some time tweaking the performance of their mysql server. It is guaranteed to noticeably decrease the response time of the web ui. The two most important things you can do is to enable the query cache (it is off by default) and set up an innodb buffer pool that is a little larger than the size of your database. There are tips floating around the forum and the wiki for tuning your mysql (primarly using the tool mysqltuner). If I recall correctly, bbunge just wrote up a guide so maybe he can provide some links on the subject.

Because your system only has 2GB of ram, you are likely not going to be able to increase you innodb buffer pool sufficiently.
Zoneminder already uses /dev/shm for mapped memroy (sits in ram) and you are not going have a enough left over to put the database into ram. Keep in mind that, until your disk filles up the 8TB, your database is going to keep growing.

I'm not sure I agree with setting the audit daemon to run every hour, rather than every ~15 minutes. Increasing the time it runs, also increases the amount of work it has to do when it does run. Running it more often evens out the load, while running it every hour will cause a spike in resource consumption every hour. This can cause disk i/o related problems. You likely won't notice this until your 8TB disk fill ups completely. Until that happens, the system does not have any events to delete.

On a slightly different note, there is a risk associated with using a remote mounted filesystem. You have to be very careful that zoneminder never starts before the remote filesystem is mounted. It only takes one time, and zoneminder will delete all your events from the database when it can no longer find them on the disk. Once you restore the NAS, the events will be there on the drive but the metadata will be gone from the dB..... so zoneminder will proceed to delete the events from the disk. Yes, you've set zmaudit to run every hour, but we all know that Murhpy's law tells us your system will experience this issue at the 59th minute of the hour.

One way to mitigate this potential problem is to regularly backup your database. That way if you do lose your events from the dB, you've got a backup and can recover all but the most recent.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Some easy tips on improving ZM 1.28.x performance

Post by asker »

knnniggett wrote:Speaking of performance, everyone should spend some time tweaking the performance of their mysql server. It is guaranteed to noticeably decrease the response time of the web ui. The two most important things you can do is to enable the query cache (it is off by default) and set up an innodb buffer pool that is a little larger than the size of your database. There are tips floating around the forum and the wiki for tuning your mysql (primarly using the tool mysqltuner). If I recall correctly, bbunge just wrote up a guide so maybe he can provide some links on the subject.
Great - thanks - I will look into this next

Keep in mind that, until your disk filles up the 8TB, your database is going to keep growing.
Actually, I have a filter running in ZM that purges records after 5 days.

I'm not sure I agree with setting the audit daemon to run every hour, rather than every ~15 minutes. Increasing the time it runs, also increases the amount of work it has to do when it does run. Running it more often evens out the load, while running it every hour will cause a spike in resource consumption every hour. This can cause disk i/o related problems. You likely won't notice this until your 8TB disk fill ups completely. Until that happens, the system does not have any events to delete.
Hmm, good point on load averaging. Frankly, its hard to know unless I benchmark 1hr vs 15 mins. ZMA seems to be a heavy process, so I can theoretically imagine both will have pro/cons.
On a slightly different note, there is a risk associated with using a remote mounted filesystem. You have to be very careful that zoneminder never starts before the remote filesystem is mounted. It only takes one time, and zoneminder will delete all your events from the database when it can no longer find them on the disk. Once you restore the NAS, the events will be there on the drive but the metadata will be gone from the dB..... so zoneminder will proceed to delete the events from the disk. Yes, you've set zmaudit to run every hour, but we all know that Murhpy's law tells us your system will experience this issue at the 59th minute of the hour.

One way to mitigate this potential problem is to regularly backup your database. That way if you do lose your events from the dB, you've got a backup and can recover all but the most recent.
Yes, true. I've already taken care of that situation -> I've modified ZM init script to wait till my NFS directory mounts.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: Some easy tips on improving ZM 1.28.x performance

Post by knight-of-ni »

Very good. I bet others could benefit from the modifications you made to the Init script. Can you post it?
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Some easy tips on improving ZM 1.28.x performance

Post by asker »

Sure. I add this to start(). mymedia is where I mount the primary directory. It can be improved - for example, if it never loads, or is down, ZM won't start etc. It also doesn't take into account what happens if my NAS fails while ZM is active. I haven't gotten around to covering all scenarios. Frankly, I'm fine with ZM removing my events if my NAS fails for whatever reason. I've never had it happen so far, so I'm ok with it when it does.

Code: Select all

nas_mounted=`mountpoint -q /mymedia && echo "YES" || echo "NO"`
while [ "$nas_mounted" = "NO" ]
do
        echo "Hrmph. Waiting for NAS.... (hopefully not forever)"
        sleep 3;
        nas_mounted=`mountpoint -q /mymedia && echo "YES" || echo "NO"`
done
knnniggett wrote:Very good. I bet others could benefit from the modifications you made to the Init script. Can you post it?
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
dazed
Posts: 66
Joined: Thu Feb 13, 2014 5:32 pm

Re: Some easy tips on improving ZM 1.28.x performance

Post by dazed »

BBunge posted this recently... I only needed to change my PATH_SWAP
Now I just need to figure out how to do the MySQL settings thing.


"If you have the memory you can put some of those functions on RAM disk such as:

PATH_MAP = /dev/shm

PATH_SWAP= /dev/shm"
dazed
Posts: 66
Joined: Thu Feb 13, 2014 5:32 pm

Re: Some easy tips on improving ZM 1.28.x performance

Post by dazed »

This was the only write-up by bbunge that I could find related to mysql tuning
http://www.zoneminder.com/wiki/index.ph ... Zoneminder

I have it installed now but havent done the above link.

I also Increased my Apache2 RAM to 256mb from 128MB per a recommendation post to another member.
Edit /etc/php5/apache2/php.ini

bbunge wrote
find memory_limit = 128M and start by going to 256M. Restart Apache Mine is set to 512M as I have 16 cameras doing mocord and up to six people watching the montage at one time.
Locked