Suddenly high system load due to ZMC process

Forum for questions and support relating to the 1.28.x releases only.
Locked
Markymark
Posts: 50
Joined: Fri Jan 03, 2014 7:43 am

Suddenly high system load due to ZMC process

Post by Markymark »

Hi,

ZM was running like a charme for a few weeks until suddenly system load tripeled due to the ZMC processes. I can not recall any changes to the system. System load recovers to 'normal' after a few hours or immediately after reboot. I will be happy to dig into the details ... but I have not the slightest idea were to start.

Is this a known issue? Any ideas for further investigation?

Any help apprechiated.

Rgrds,

Mark
aaronl
Posts: 33
Joined: Wed Aug 19, 2015 9:11 pm

Re: Suddenly high system load due to ZMC process

Post by aaronl »

Do you have filters set up to remove old data?
if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar charts.
options: k,l,m,c,d,t.

Notice however, that the memory may show only 1% free. This is sort of a lie, use 'free -m' to get a better number on this. See http://www.linuxatemyram.com/ if you are not familiar with that.

Basically you need to find which computer subsystem is the one that is the bottleneck.

Aaron
Markymark
Posts: 50
Joined: Fri Jan 03, 2014 7:43 am

Re: Suddenly high system load due to ZMC process

Post by Markymark »

Aaron,

>Do you have filters set up to remove old data?
No, not yet. There is still plenty of free SSD space available. There is not that much old data anyway.

>if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar

I am currently using a modified version of 'rpimonitor' (I am on a Cubietruck) for performance measurement. So I have the historical data for CPU and Memory. Memory is fine all the time during high load. CPU frequency is constantly on the top line at the same time. I just installed 'nmon' for actual disk usage inspection at high load. At the moment everything is normal.

>Notice however, that the memory may show only 1% free.
Yep, know this.

Any further hints highly apprechiated.

Greetings,

Mark
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Suddenly high system load due to ZMC process

Post by asker »

You should start by posting logs around the time the load spikes.
Option->Logs

Markymark wrote:Aaron,

>Do you have filters set up to remove old data?
No, not yet. There is still plenty of free SSD space available. There is not that much old data anyway.

>if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar

I am currently using a modified version of 'rpimonitor' (I am on a Cubietruck) for performance measurement. So I have the historical data for CPU and Memory. Memory is fine all the time during high load. CPU frequency is constantly on the top line at the same time. I just installed 'nmon' for actual disk usage inspection at high load. At the moment everything is normal.

>Notice however, that the memory may show only 1% free.
Yep, know this.

Any further hints highly apprechiated.

Greetings,

Mark
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Markymark
Posts: 50
Joined: Fri Jan 03, 2014 7:43 am

Re: Suddenly high system load due to ZMC process

Post by Markymark »

@asker

I just re-checked the log files but could not find anything unusual. I will refine the log level and take a look again as soon as "high load" happens again. Since yesterday evening everything runs fine again.

Mark
Markymark
Posts: 50
Joined: Fri Jan 03, 2014 7:43 am

Re: Suddenly high system load due to ZMC process

Post by Markymark »

Hi,

sorry for my late reply but it took some time until the issue happend again.

Here are the facts:

- System load tripels suddenly.
- No IO bottleneck, 'top' shows 0.0 wa, 'vmstat' shows no 'bi/bo' above baseline.
- 'top' shows two 'zmc' processes above 80 %CPU, a third 'zmc' is at baseline as expected (3 cams at all).
- Free/available memory start to decrease at the time high load starts.
- After reboot system load is fine again.
- Cameras are pointed at stable environment. No moving objects in front at time of high load.
- Log files are normal during the time of high load. Log settings are still reserved.

I will bulk the logging now. Maybe anyone can analyse from the facts available so far.

Thanks in advance.

Rgrds,

Mark
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Suddenly high system load due to ZMC process

Post by bbunge »

It is possible that MySQL is causing the high load especially if you have a lot of events and you have never "adjusted" the settings. Most of teh time you can run Zoneminder with default settings in MySQL. I have had to make some adjustments as the number of events recorded has increased. I recommend you install and run mysqltuner for recommendations.

Here are some settings I've changed:

innodb_file_per_table = 1
innodb_buffer_pool_size = 512m

tmp_table_size = 32M
max_heap_table_size = 32M

Note that adding innodb_file_per_table to a running system will not change anything. You can convert your databases and I can look up the procedure if you need it.

bb
Markymark
Posts: 50
Joined: Fri Jan 03, 2014 7:43 am

Re: Suddenly high system load due to ZMC process

Post by Markymark »

Hi bb,

we are talking about maybe 10 events a day, not more. At the moment arround 400 events are present at all.

Honestly spoken I have a very limited understanding of MySQL, yet. I will take a close look at 'mysqltuner' and get a better understanding of your recommendation. Just one qustion in advance: would you expect a higher CPU load for the 'zmc' processes if at last MySQL has the performance issue?

Thanks again,

Mark
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Suddenly high system load due to ZMC process

Post by bbunge »

Yes. Well it may be zmaudit that runs up MySQL. Default settings for zmaudit are to run every 900 seconds. This checks ZM database and file system and puts them in sync.

Did not see which Linux flavor and version you are running..

bb
Markymark
Posts: 50
Joined: Fri Jan 03, 2014 7:43 am

Re: Suddenly high system load due to ZMC process

Post by Markymark »

Hi bb,

OK, understood.

This is the 'mysqltuner' output so far:

Code: Select all

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.44-0+deb7u1
[OK] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in InnoDB tables: 2M (Tables: 18)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] Total fragmented tables: 18

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 5h 4m 39s (6K q [0.329 qps], 454 conn, TX: 63M, RX: 1M)
[--] Reads / Writes: 37% / 63%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[OK] Maximum possible memory usage: 597.8M (29% of installed RAM)
[OK] Slow queries: 0% (0/6K)
[OK] Highest usage of available connections: 7% (12/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/103.0K
[OK] Query cache efficiency: 83.8% (3K cached / 4K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 32 sorts)
[OK] Temporary tables created on disk: 24% (79 on disk / 317 total)
[OK] Thread cache hit rate: 97% (12 created / 454 connections)
[OK] Table cache hit rate: 89% (59 open / 66 opened)
[OK] Open file limit used: 4% (48/1K)
[OK] Table locks acquired immediately: 100% (1K immediate / 1K locks)
[OK] InnoDB data size / buffer pool: 2.0M/128.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Enable the slow query log to troubleshoot bad queries
It seems to be a good idea to give it some more time for proper analysis (next to last recommendation).

>Did not see which Linux flavor and version you are running..

Sorry, I forgot to answer. Debain 7.8 "Wheezy".
Markymark
Posts: 50
Joined: Fri Jan 03, 2014 7:43 am

Re: Suddenly high system load due to ZMC process

Post by Markymark »

>Yes. Well it may be zmaudit that runs up MySQL.

Just to make this clear ... even if this reflects my minor comprehension: It is the Capture Deamon (zmc process) that suddenly has a tripeled CPU load.
Locked