A guide to managing system load with ZoneMinder

[Flash_]

A guide to managing system load with ZoneMinder
(Written with IP Cameras in mind)


Zoneminder is a superb application in every way, but it does a job that needs a lot of horsepower especially when using multiple IP cameras. IP Cams require an extra level of processing to analogue cards as the jpg or mjpeg images need to be decoded before analysing. This needs grunt. If you have lots of cameras, you need lots of grunt.


Why do ZM need so much grunt?
Think what Zoneminder is actually doing. In modect mode ZM is:
1. Fetching a jpeg from the camera. (Either in single part or multipart stream)
2. Decoding the jpeg image.
3. Comparing the zoned selections to the previous image or images and applying rules.
4. If in alarm state, writing that image to the disk and updating the mysql database.

If you're capturing at five frames per second, the above is repeated five times every second, multiplied by the number of cameras. Decoding the images is what takes the real power from the processor and this is the main reason why analogue cameras which present an image ready-decoded in memory take less work.


How do I know if my computer is overloaded?
If your CPU is running at 100% all the time, it's probably overloaded (or running at exact optimisation). If the load is consistently high (over 10.0 for a single processor) then Bad Things happen - like lost frames, unrecorded events etc. Occasional peaks are fine, normal and nothing to worry about.

Zoneminder runs on Linux, Linux measures system load using "load", which is complicated but gives a rough guide on what the computer is doing at any given time. Zoneminder shows Load on the main page (top right) as well as disk space. Typing "uptime" on the command line will give a similar guide, but with three figures to give a fuller measure of what's happening over a period of time but for the best guide to see what's happening, install "htop" - which gives easy to read graphs for load, memory and cpu usage.

A load of 1.0 means the processor has "just enough to do right now". Also worth noting that a load of 4.0 means exactly the same for a quad processor machine - each number equals a single processor's workload. A very high load can be fine on a computer that has a stacked workload - such as a machine sending out bulk emails, or working its way through a knotty problem; it'll just keep churning away until it's done. However - Zoneminder needs to process information in real time so it can't afford to stack its jobs, it needs to deal with them right away.

For a better and full explanation of Load: http://en.wikipedia.org/wiki/Load_%28computing%29


My load is too high, how can I reduce it?
Zoneminder is /very/ tweakable and it's possible to tune it to compromise. The following are good things to try, in no particular order;

Change the jpeg libraries. In most distributions Linux uses standard jpeg libraries which although fine for most things, don't use the MMX functions in nearly all modern processors. Check whether your cpu supports mmx by running "cpuid |grep MMX" which should give you a line or two along the lines of "MMX instructions". If so, give the libs a try. Most people report their load halves simply by using these libs. http://www.zoneminder.com/forums/viewtopic.php?t=6419 gives more info. Nobody's posted there to say it broke their system... Yet.

If your camera allows you to change image size, think whether you can get away with smaller images. Smaller pics = less load. 320x240 is usually ok for close-up corridor shots.

Go Black and White. Colour pictures use twice to three times the CPU, memory and diskspace but give little benefit to identification.

Reduce frames per second. Halve the fps, halve the workload. If your camera supports fps throttling (Axis do), try that - saves ZM having to drop frames from a stream. 2-5 fps seems to be widely used.

Experiment with using jpeg instead of mjpeg. Some users have reported it gives better performance, but YMMV.

Tweak the zones. Keep them as small and as few as possible. Stick to one zone unless you really need more.

Schedule. If you are running a linux system at near capacity, you'll need to think carefully about things like backups and scheduled tasks. updatedb - the process which maintains a file database so that 'locate' works quickly, is normally scheduled to run once a day and if on a busy system can create a heavy increase on the load. The same is true for scheduled backups, especially those which compress the files. Re-schedule these tasks to a time when the cpu is less likely to be busy, if possible - and also use the "nice" command to reduce their priority. (crontab and /etc/cron.daily/ are good places to start)


More expensive options:

Increase RAM. If your system is having to use disk swap it will HUGELY impact performance in all areas. Again, htop is a good monitor - but first you need to understand that because Linux is using all the memory, it doesn't mean it needs it all - linux handles ram very differently to Windows/DOS and caches stuff. htop will show cached ram as a different colour in the memory graph. Also check that you're actually using a high memory capable kernel - many kernels don't enable high memory by default.

Faster CPU. Simple but effective. Zoneminder also works very well with multiple processor systems out of the box (if SMP is enabled in your kernel). The load of different cameras is spread across the processors.


What about disks and bandwidth?

In most modern pc-based servers, disk I/O is more than adequate for the speeds involved in capturing from multiple cameras in most scenarios.

A typical 100mbit LAN will cope with most setups easily. If you're feeding from cameras over smaller or internet links, obviously fps will be much lower.

Disk and Bandwidth calculators are referenced on the Zoneminder wiki here: http://www.zoneminder.com/wiki/index.ph ... _for_ZM.3F

[cordel]

Thanks for this Flash_
And Thank you for taking the time to post it.

Cheers,
Corey

[jameswilson]

yes very useful. perhaps you would like it on the wiki!

Great advice there

[Flash_]

Thanks and thanks. And it's wiki'd already - my first wiki!

[SlovakJoe]

In most modern pc-based servers, disk I/O is more than adequate for the speeds involved in capturing from multiple cameras in most scenarios.

Disk IO has been my main bottleneck. In the test server I have running, anytime I go to delete a number of events, the deletion causes the system load to increase tremendously. During normal recording conditions the RAM is underutilized, CPU is underutilized, and the recording works smoothly. So far so good. But when anywhere from 5 to 50 GB worth of events are deleted, the system load jumps from say .5 up to 4. At that point Apache drops requests and I can no longer view the main ZM console window. The Apache problem is likely due to this being a test system and using old hardware.

The disk IO performance when deleting and recording at the same time is causing a significant spike in system load. This isn't a surprise, but I'm wondering if other people have problems like this as well? It's to be expected when you have all your events on a single IDE drive. When I move this to production, the production server will most likely use RAID 0 (or 5) or JBOD which will be of great help.

[coke]

I'm pretty new at the ZoneMinder game, but in my non-scientific experience so far, RAID 0 is worlds better than one drive. My initial tests set to record all on one drive took ages to delete old events, and pegged the CPU. Reformatted the system (2 320gb drives) into 1 RAID 0 640GB, and it's running like a champ, deleting piles of dirs (more than rm -r can usually handle in one swipe) fairly instantly.

May or may not have something to do with it now being Reiserfs instead of ext3.

[SlovakJoe]

Thanks for the update Coke. Like I said, the performance problem is to be expected given my hardware setup. It has nothing to do with ZoneMinder. And as for deleting all those files, after a certain point, you can't even use "rm -rf" anymore. You have to use "find" and pipe the output to "rm". Fortunately, ZM takes care of this all.

Either way, I'd never use a single drive in a production system unless it only had a couple of cameras.

In addition to JBOD for disk spanning, I'm considering switching from Ubuntu to Fedora for the built in LVM support. With LVM you can take multiple disks of different size and group them together into 1 logical drive. This would make life a lot easier for those of us with handfuls of spare 120 and 160 gig drives.

[zoneminder]

If you use the original flat filesystem then deleting files from directories with tens of thousands of events in will be slow. The deep filesystem available in 1.23.x is much quicker as no directory has more than a handful of other directories in.

[SlovakJoe]

If you use the original flat filesystem then deleting files from directories with tens of thousands of events in will be slow. The deep filesystem available in 1.23.x is much quicker as no directory has more than a handful of other directories in.

When I first read about that feature I was very excited. However, the documentation states that this feature is not ready for production. My test server is running ZM 1.23.1 and in the configuration window, the help for that feature states "Be warned: deep storage should be considered in beta for now, if you value your data do not select it.". Maybe 1.23.2 has fixed this?

If it is in fact a stable feature, then I'll switch to that and compare the performance of a single IDE drive using deep directories VS RAID 0 drives using the flat file structure.

Either way, RAID 0 should eliminate any serious disk IO issues.

[Lee Sharp]

Consider the filesystem as well. The Live CDs for ZM and most other distributions use ext3 by default. While robust, ext3 is NOT a performance filesystem. I have been thinking a while about benchmarking a few filesystems with ZM to see the difference. Most of them have better delete performance than ext3.

[sajithvk]

I have been using zoneminder for a few months on test basis, and it works perfectly for my requirements, including motion detection, recording etc. Currently, we have a requirement to implement large scale surveillance, covering around 70 IP cameras, each at 640x480 resolution (some PTZ, some fixed). Is it advisable to use zm for this situation? I feel everything ok, but the shared memory required may be a big issue.......

Has anyone tested zm on 50+ ip cameras? Or can any one give me tips on how to go about it?