Performance tuning time

Forum for questions and support relating to the 1.30.x releases only.
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Performance tuning time

Post by Baylink »

As noted in a comment elsewhere, I've got a fresh clean shiny-new install on a Dell SC1430; dual-quad 2GHz Xeon; 6GB RAM; Adaptec 2410 SATA hardware RAID1 of 1TB WDs; Ubuntu 16.04LTS all updated, and 1.30.4 installed from iconnor's PPA (thanks, dude). 12 TrendNet cameras are wired over cat 5e to one of three Gigabit POE switch, two trunked to the third one in the main rack.

Set up my first TrendNet IP-314PI camera using the OnVIF prober, which worked fine (though it didn't always find all the cameras for the pulldown list, at random), and then cloned the rest of the monitors, changing name and IP.

The camera settings are 1280x720 H.264, 10fps, I-frame every 60, "higher" video quality, and a 6144kbps encoding limit. 720p is as low as the main encoder will go; they're 4mp cameras.

The monitor settings are 25/25 blend, and 40/20/20/20 frames. My SHMs are about 120MB per camera, for a total usage of 43% or so. All three monitor frame rates are set to 15fps.

With all 12 cameras in monitor mode, everything is stable, all monitors work, and I can montage all of them on one page. With no viewers running, the machine runs at an LA between 7 and 9, which isn't bad for an 8-core machine, though I'm told I can potentially halve it, by people running similar systems.

Switching all but one camera to modect...

38.22. 15 minute average.

Clearly, I have some tuning to do.

I know I remember seeing some pages on .. I think, the wiki, that suggest how to determine where the pinch points are in a running install, but I can't find them. Anyone got a favorite write up on how to figure out *what* is causing the load to go that high? Once I know where, I can likely figure out how to fix it. One user suggestion was that mysql wasn't keeping up.

For the record, at 38, this 8-core CPU has very little keyboard lag, and actually manages to run passably; it's losing connections and dropping frames, but it still never just collapsed.

My previous record was a 4-core Dell 2600 that went catatonic at 48, and died at 63. :-)
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Performance tuning time

Post by bbunge »

Lower your blend to 6.25
All the FPS settings in ZM leave blank
Lower the frame rate at the cameras to 5 FPS. Zoneminder is not an HD movie creator...
You may need to lower the iframe to 10 if you get smearing
With RTSP use TCP
Check the WIKI install instructions : https://wiki.zoneminder.com/Ubuntu_Serv ... e_easy_way
Target colorspace 32 bit
Make sure you are using ffmpeg
You may need to allocate space for MySQL innodb_buffer_pool_size. Use mysqltuner to see what is going on with MySQL. The above WIKI instructions show you how to switch to mariadb which is what I use on my old Dell i5 Quad.

bb
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

Wasn't aiming for HD, but I have some passage tracks that take up a small enough part of the frame that I was worried about not getting a clear shot at 5fps. I'll give it a try.

Adjust both blend %ages to 6.25, I assume?

Haven't been seeing smearing, so far.

Pretty sure I'm on TCP already.

FFmpeg/32 already set, and I did follow the Easy Way -- though there's at least one bug in it; I'll go back and find it.

I'll be back there tomorrow, and I was already gonna run tuner. Happy to switch to maria if it's believed a solid changeout by the devteam, sure.

Thanks.
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Performance tuning time

Post by bbunge »

Baylink wrote: Wed Aug 23, 2017 12:34 am Wasn't aiming for HD, but I have some passage tracks that take up a small enough part of the frame that I was worried about not getting a clear shot at 5fps. I'll give it a try.

I'm running one 20 year old Axis @ 1 FPS 352x288 and it catches all movement! Most of my cameras are @ 3 FPS with a couple @ 5 FPS 320x240 and we still ID thieves!

Adjust both blend %ages to 6.25, I assume?

Yes

Haven't been seeing smearing, so far.

Pretty sure I'm on TCP already.

FFmpeg/32 already set, and I did follow the Easy Way -- though there's at least one bug in it; I'll go back and find it.

I'll be back there tomorrow, and I was already gonna run tuner. Happy to switch to maria if it's believed a solid changeout by the devteam, sure.

Red Hat/Centos uses Mariadb by default. Just get the changes in my.cnf as detailed in the WIKI instructions

Thanks.
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

bbunge wrote: Wed Aug 23, 2017 2:04 am I'm running one 20 year old Axis @ 1 FPS 352x288 and it catches all movement! Most of my cameras are @ 3 FPS with a couple @ 5 FPS 320x240 and we still ID thieves!
This is a college campus, and I'm covering more space than I really ought to with each camera (and I'm still at 12 cameras).
Adjust both blend %ages to 6.25, I assume?
Yes
Cool.
Red Hat/Centos uses Mariadb by default. Just get the changes in my.cnf as detailed in the WIKI instructions
Will do. If that's the "no change engine" thing, I already did.
User avatar
iconnor
Posts: 2900
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Performance tuning time

Post by iconnor »

I would say set your max analysis framerate to 2 or 3(Keep recording framerate blank or higher than what the camera will send). I find these work just fine. I may introduce an option to only do motion analysis only on I-frames or something. Long and short is, you don't need to do motion analysis on every frame. You can also get trickier and have the motion analysis on a second lower fps stream, which triggers the high fps stream.

Minimise your zones, and crank up the sensitivity ( the zones default toi percent, switch to pixels, and make the min blob area, etc 5000 or less). Your machine should do that number of cameras no problem. May have to do more analysis to figure out where the bottleneck is. I recommend installing atop. raid1 on spinning rust is slow. You might think about adding more disks. I don't really think that slow disks are the problem though. Not with just 12 cameras.

You can also give my ppa at ppa:iconnor/zoneminder-storageareas a try. It has a few performance enhancements. Shouldn't REALLY make difference, but might. It is somewhat stable at the moment.
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

TCP confirmed, switched blends to 6/6. Still above 30.

Seeing lots of this in the mysql.log:

2017-08-23T16:43:33.034346Z 12569 [Note] Aborted connection 12569 to db: 'zm' us
er: 'zmuser' host: 'localhost' (Got an error reading communication packets)

Like, 65 of them. Thought it was more at 10KB, but not.

Odd thing: when I got to the machine today after leaving it in Modect-All Monday afternoon... it was stopped. Linux still running, but ZM stopped.

Is there a file where mode changes are audited?

[ Edit: I see in other older threads it will shut itself down if the load gets out of hand, a policy with which I'm not sure I agree, but that's for another time. ]

mysqltuner also complains that my mysql is only 32 bit, or that the OS is (it is; I don't know why). At even 6GB of main RAM I don't think that's critical.

Can I do an inplace upgrade from my to maria? Or do I have to dump?
Last edited by Baylink on Wed Aug 23, 2017 7:03 pm, edited 1 time in total.
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

Update: I changed the buffers from 40/20/20/20 to 60/20/20/20 with no appreciable difference.

Second update: I just set monitors 5-12 to None. Load drops supralinearly, to 3ish

Going up to 6, I'm at 7, peaking to 9. Pretty linear so far.
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Performance tuning time

Post by bbunge »

Baylink wrote: Wed Aug 23, 2017 5:37 pm TCP confirmed, switched blends to 6/6. Still above 30.

Seeing lots of this in the mysql.log:

2017-08-23T16:43:33.034346Z 12569 [Note] Aborted connection 12569 to db: 'zm' us
er: 'zmuser' host: 'localhost' (Got an error reading communication packets)

Like, 65 of them. Thought it was more at 10KB, but not.

Odd thing: when I got to the machine today after leaving it in Modect-All Monday afternoon... it was stopped. Linux still running, but ZM stopped.

Is there a file where mode changes are audited?

[ Edit: I see in other older threads it will shut itself down if the load gets out of hand, a policy with which I'm not sure I agree, but that's for another time. ]

mysqltuner also complains that my mysql is only 32 bit, or that the OS is (it is; I don't know why). At even 6GB of main RAM I don't think that's critical.

Can I do an inplace upgrade from my to maria? Or do I have to dump?
You had better check the Ubuntu install you used. I have never seen mysqltuner complain that MySQL is 32 bit as I use 64 bit on just about everything I can.

You need to dump the zm database before upgrading to Mariadb. https://wiki.zoneminder.com/Ubuntu_Serv ... _installed

Are you running a desktop version of Ubuntu?
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

16.04LTS. (LTS does mean 'server', does it not?) uname says i686, so yeah, it's the 32 bit version.

With the exception of the amount of RAM mysql can allocate, is that really gonna kill me?

And what am I supposed to do about the character set? phpMYadmin tells me what it is (and it wasn't what I expected), but I don't think it's what's in that wiki article either... Force the new one to what the current one was?
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Performance tuning time

Post by bbunge »

LTS=Long Term Support

I do not do a full server install as there are things included in the server package I do not use. I use what I call a basic install from the netinst package:

Ubuntu basic install can be done from the net install CD (https://www.ubuntu.com/download/alternative-downloads) This is creates a CD from the file mini.iso. When prompted (tasksel) install standard system utilities, OpenSSH Server, and LAMP Server. This is the minimum that is required to set up and run Zoneminder and does not load other apps that are not needed.

I also install Webmin to administer the server.

Yes, using the 32 bit version may be a problem. A 32 bit system uses only 3.5 GIG of RAM but may use more if it is configured properly. But why limit yourself with a 32 bit OS on a beast of a machine?

You may want to start over with a fresh 64 bit install. Not sure if you could reuse your database on a 64 bit system but you could try it.
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

Turns out the backup server just hit my bench, so I'll install that with server64, maria and the PPA, and then dump the database and reload it.
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

Oh, and just for clarification, the question I was asking was "it is not correct that only the server install is available for LTS?" That's not the case, I guess.

I had done the full server install, as I had other plans.

Reinstall on the other machine this afternoon.
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Performance tuning time

Post by bbunge »

Baylink wrote: Sat Aug 26, 2017 5:17 pm Oh, and just for clarification, the question I was asking was "it is not correct that only the server install is available for LTS?" That's not the case, I guess.

I had done the full server install, as I had other plans.

Reinstall on the other machine this afternoon.
https://wiki.ubuntu.com/LTS

Desktop and server can be LTS. Even number versions (12, 14, 16) are LTS for Ubuntu and get 5 year support. Nothing wrong with the full server version just my choice to not load up what I don't need.
Baylink
Posts: 338
Joined: Sun Jun 19, 2005 3:19 am

Re: Performance tuning time

Post by Baylink »

Well, dropping from 10fps to 6 brought me from 32 to about 25 at idle.

skip-name-resolution, on the other hand, brought my 12 camera idle to...

8ish.

I can live with 8 on an 8 core machine; it'll carry me till I find the next step. A single montage monitoring brings it back up to 14 to 16, so that'll be next. I had to download mysqltuner, cause the version in the repos is incompatible. I'll probably do the mariadb replacement this weekend.
Locked