Rebuilding for processor specific opt..

Forum for questions and support relating to the 1.28.x releases only.
Locked
Pedulla
Posts: 167
Joined: Thu Nov 27, 2014 11:16 am
Location: Portland, Or

Rebuilding for processor specific opt..

Post by Pedulla »

Hi ZM Community,
I'm working on my 4th ZM install and this one is a doozy.
Between 20-24 cams of the Hikvision 2CDxx32F variety (3Mpxl)
Ubuntu 14.04 Server-64bit-32G-1TB-SSD-Xeon E3-1241 (quad-core)

I want to make sure I'm getting them most from my hardware.
Several questions could use some clarification:
1- Can a working 1.28.1 install be rebuilt/recompiled for further optimization?
2- Do I have to rebuild to utilize libjpeg-turbo? (FAQ suggests I do, but Ubuntu may already use it)
3- Can I rebuild a working (not production) system so GCC utilizes processor specific optimizations or do I need to do a from source complete install?
4- I've read several posts where the dependencies are so diverse for ZM it's almost not worth the effort to build from source; really?

Disclosure: 4 yr Linux user, 3 raspberry pi's, full-time mint 17.3 desktop, I'm a package baby - the last program I compiled was Turbo-Pascal on some timeshare host system in the '80's so I've got a little catching up to do...

Thank you in advance for you answers, suggestions and advise.
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: Rebuilding for processor specific opt..

Post by knight-of-ni »

The short answer is you are talking about doing a lot of work, with very little perceptible difference in performance.
Keep in mind that, even if you are successful going down this path, you will have to repeat the process each time a new version of ZoneMinder is released.
We steer normal users away from this path specifically for this reason.
1- Can a working 1.28.1 install be rebuilt/recompiled for further optimization?
Sure. You could pass parameters like O3 and mtune to the compiler, but I doubt you would be able to see a difference in performance.
2- Do I have to rebuild to utilize libjpeg-turbo? (FAQ suggests I do, but Ubuntu may already use it)
Both Ubuntu and RedHat based distros like CentOS ship with libjpeg-turbo functions already installed. ZoneMinder already takes advantage of this. It might not be called "libjpeg-turbo" on the distro in question, but Googling will reveal Ubuntu added this optimization since around 12.04 if I recall.
3- Can I rebuild a working (not production) system so GCC utilizes processor specific optimizations or do I need to do a from source complete install?
See my answer to #1. Additionally, you want to follow the instructions for rebuilding ZoneMinder into a package rather than from source. Instructions are on our github site.
4- I've read several posts where the dependencies are so diverse for ZM it's almost not worth the effort to build from source; really?
All the major distros have moved away from installing system software from source. For example, Perl doesn't even support "make uninstall" anymore ....so once you install ZoneMinder from-source, there is no longer any automatic way to remove it completely from the target system. That's one of many things you just have to know when building from source. The ffmpeg API changes frequently and it is quite common to run into build issues with that. You also have to manually install the ZoneMinder startup scripts, the scripts for the new tmpfiles.d service, and log rotation scripts. I could go on.

The good news is there are recent ZoneMinder packages for most major distros, which take care of all that for you.
This is win-win for everyone. It creates a better experience for the end-user and allows the few of us on the development team to focus on improving the source code, add new features, etc. rather than answering "how do I build" questions.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
Pedulla
Posts: 167
Joined: Thu Nov 27, 2014 11:16 am
Location: Portland, Or

Re: Rebuilding for processor specific opt..

Post by Pedulla »

Happy New Year everyone!

@knnniggett, Thank you for the thorough response. 'nuf said, I'll stick with what I've got and see if I can spend some time tuning in other areas.

I've put up 8 cameras at this point (it's a non-profit gig so I work in my spare time) with dual streams. Low-res 4fps 352x240 (modetect) and Hi-res 1920x1080 15fps (nodetect-link'd). My shared mem is @50%

This puts my processor usage at ~4 out of 8. At this rate I won't make 20 cameras.

First thing I see is that I'll lower the fps for the hi-res to 10 or so.
I'll start looking into the MySQL and Apache processes to see if there is anything jumping out at me.

Any other contemporary tuning suggestions are welcome at this point!
Pedulla
Posts: 167
Joined: Thu Nov 27, 2014 11:16 am
Location: Portland, Or

Re: Rebuilding for processor specific opt..

Post by Pedulla »

Quick update.
Setting innodb_file_per_table as suggested in the link below cut CPU avg utilization back to below 3. I should be able to hit 20 cams now.

http://zoneminder.readthedocs.org/en/la ... ecommended
Linwood
Posts: 75
Joined: Thu Jun 25, 2015 6:40 pm

Re: Rebuilding for processor specific opt..

Post by Linwood »

With regard to optimization and a cmake build, one thing I found is that I couldn't make the optoin actually change optimization. When I would add

-DCMAKE_CXX_FLAGS like -O0, internally it did things like this:

DHAVE_SENDFILE4_SUPPORT=1 -O0 -march=native -Wall -D__STDC_CONSTANT_MACROS -O2

I.e. it added the -O0 in at the beginning but still left the hard coded -O2 in at the end, defeating the option I added. If you try doing the CXX optimization flag by using the DCMAKE_CXX_FLAGS, be sure to check the log to see if it had any effect. You might find it necessary to patch the scripts.

I should note I tried this way back in August and a lot has happened to the code since then, so it may not be an issue.

I should also add I am/was a real novice with CMAKE and I may have simply screwed this up, but it's at least conceptually pretty foolproof in terms of how you set it. I think the scripts had a bug (at least back then).
User avatar
iconnor
Posts: 2879
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Rebuilding for processor specific opt..

Post by iconnor »

I might be interesting for someone to try compiling with -O3, but I doubt there would be much benefit. We already use hand-rolled assembly for a lot of the heavy lifting in motion detection. The rest of the heavy computation (h264 decoding, etc) comes from ffmpeg, so you would have to recompile that too.

Things to monitor are:

#1 cpu utilisation
#2 IO wait times.

You can easily saturate disk IO. Especially if you do something like raid5 (you didn't say that you were, but lots of people do that). You need to understand that ZM is almost all disk writing, so you need to optimize for that case.

In my setups, I use multiple disks, with no raid, and assign cameras to them until the drives are always busy, then add another drive.
Linwood
Posts: 75
Joined: Thu Jun 25, 2015 6:40 pm

Re: Rebuilding for processor specific opt..

Post by Linwood »

iconnor wrote: You can easily saturate disk IO. Especially if you do something like raid5 (you didn't say that you were, but lots of people do that). You need to understand that ZM is almost all disk writing, so you need to optimize for that case.
That's a fair point. i have those same cameras, and am running on a light duty PC with 7 cameras BUT I only do 4 fps, and I use a stripe set (aka Raid-0).

Your biggest win by far is a reduced frame rate or resolution, and of those giving up frame rate is probably the easiest to tolerate, though you may have some areas where lower resolution is acceptable as well, e.g. only close-in views.

By the way, if not obvious, reduce the frame rate in the camera NOT in the monitor definition -- the monitor definition will not work for IP cameras to reduce frame rate.
Pedulla
Posts: 167
Joined: Thu Nov 27, 2014 11:16 am
Location: Portland, Or

Re: Rebuilding for processor specific opt..

Post by Pedulla »

iconnor wrote: In my setups, I use multiple disks, with no raid, and assign cameras to them until the drives are always busy, then add another drive.
Hi IConnor, (BTW, Great work on the packaging)
Interesting, I know how to check I/O wait times with top/htop, I'll check that out.
I'm currently running 2-SSD drives striped.

Leaving the topic somewhat, I'm curious, how do you go about assigning camera's to different drives?
I'll test that on my next build.

Thanks again!
User avatar
iconnor
Posts: 2879
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Rebuilding for processor specific opt..

Post by iconnor »

The assign cameras to specific drives is part of a branch of mine called storageareas. There is a ppa at ppa:iconnor/zoneminder-storageareas. It will likely go into 1.30.
Locked