Suddenly high system load due to ZMC process
Suddenly high system load due to ZMC process
Hi,
ZM was running like a charme for a few weeks until suddenly system load tripeled due to the ZMC processes. I can not recall any changes to the system. System load recovers to 'normal' after a few hours or immediately after reboot. I will be happy to dig into the details ... but I have not the slightest idea were to start.
Is this a known issue? Any ideas for further investigation?
Any help apprechiated.
Rgrds,
Mark
ZM was running like a charme for a few weeks until suddenly system load tripeled due to the ZMC processes. I can not recall any changes to the system. System load recovers to 'normal' after a few hours or immediately after reboot. I will be happy to dig into the details ... but I have not the slightest idea were to start.
Is this a known issue? Any ideas for further investigation?
Any help apprechiated.
Rgrds,
Mark
Re: Suddenly high system load due to ZMC process
Do you have filters set up to remove old data?
if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar charts.
options: k,l,m,c,d,t.
Notice however, that the memory may show only 1% free. This is sort of a lie, use 'free -m' to get a better number on this. See http://www.linuxatemyram.com/ if you are not familiar with that.
Basically you need to find which computer subsystem is the one that is the bottleneck.
Aaron
if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar charts.
options: k,l,m,c,d,t.
Notice however, that the memory may show only 1% free. This is sort of a lie, use 'free -m' to get a better number on this. See http://www.linuxatemyram.com/ if you are not familiar with that.
Basically you need to find which computer subsystem is the one that is the bottleneck.
Aaron
Re: Suddenly high system load due to ZMC process
Aaron,
>Do you have filters set up to remove old data?
No, not yet. There is still plenty of free SSD space available. There is not that much old data anyway.
>if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar
I am currently using a modified version of 'rpimonitor' (I am on a Cubietruck) for performance measurement. So I have the historical data for CPU and Memory. Memory is fine all the time during high load. CPU frequency is constantly on the top line at the same time. I just installed 'nmon' for actual disk usage inspection at high load. At the moment everything is normal.
>Notice however, that the memory may show only 1% free.
Yep, know this.
Any further hints highly apprechiated.
Greetings,
Mark
>Do you have filters set up to remove old data?
No, not yet. There is still plenty of free SSD space available. There is not that much old data anyway.
>if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar
I am currently using a modified version of 'rpimonitor' (I am on a Cubietruck) for performance measurement. So I have the historical data for CPU and Memory. Memory is fine all the time during high load. CPU frequency is constantly on the top line at the same time. I just installed 'nmon' for actual disk usage inspection at high load. At the moment everything is normal.
>Notice however, that the memory may show only 1% free.
Yep, know this.
Any further hints highly apprechiated.
Greetings,
Mark
Re: Suddenly high system load due to ZMC process
You should start by posting logs around the time the load spikes.
Option->Logs
Option->Logs
Markymark wrote:Aaron,
>Do you have filters set up to remove old data?
No, not yet. There is still plenty of free SSD space available. There is not that much old data anyway.
>if you are on linux, get familiar with 'nmon', it can show you the cpu, memory and disk all in real time with colored bar
I am currently using a modified version of 'rpimonitor' (I am on a Cubietruck) for performance measurement. So I have the historical data for CPU and Memory. Memory is fine all the time during high load. CPU frequency is constantly on the top line at the same time. I just installed 'nmon' for actual disk usage inspection at high load. At the moment everything is normal.
>Notice however, that the memory may show only 1% free.
Yep, know this.
Any further hints highly apprechiated.
Greetings,
Mark
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.
Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Re: Suddenly high system load due to ZMC process
@asker
I just re-checked the log files but could not find anything unusual. I will refine the log level and take a look again as soon as "high load" happens again. Since yesterday evening everything runs fine again.
Mark
I just re-checked the log files but could not find anything unusual. I will refine the log level and take a look again as soon as "high load" happens again. Since yesterday evening everything runs fine again.
Mark
Re: Suddenly high system load due to ZMC process
Hi,
sorry for my late reply but it took some time until the issue happend again.
Here are the facts:
- System load tripels suddenly.
- No IO bottleneck, 'top' shows 0.0 wa, 'vmstat' shows no 'bi/bo' above baseline.
- 'top' shows two 'zmc' processes above 80 %CPU, a third 'zmc' is at baseline as expected (3 cams at all).
- Free/available memory start to decrease at the time high load starts.
- After reboot system load is fine again.
- Cameras are pointed at stable environment. No moving objects in front at time of high load.
- Log files are normal during the time of high load. Log settings are still reserved.
I will bulk the logging now. Maybe anyone can analyse from the facts available so far.
Thanks in advance.
Rgrds,
Mark
sorry for my late reply but it took some time until the issue happend again.
Here are the facts:
- System load tripels suddenly.
- No IO bottleneck, 'top' shows 0.0 wa, 'vmstat' shows no 'bi/bo' above baseline.
- 'top' shows two 'zmc' processes above 80 %CPU, a third 'zmc' is at baseline as expected (3 cams at all).
- Free/available memory start to decrease at the time high load starts.
- After reboot system load is fine again.
- Cameras are pointed at stable environment. No moving objects in front at time of high load.
- Log files are normal during the time of high load. Log settings are still reserved.
I will bulk the logging now. Maybe anyone can analyse from the facts available so far.
Thanks in advance.
Rgrds,
Mark
Re: Suddenly high system load due to ZMC process
It is possible that MySQL is causing the high load especially if you have a lot of events and you have never "adjusted" the settings. Most of teh time you can run Zoneminder with default settings in MySQL. I have had to make some adjustments as the number of events recorded has increased. I recommend you install and run mysqltuner for recommendations.
Here are some settings I've changed:
innodb_file_per_table = 1
innodb_buffer_pool_size = 512m
tmp_table_size = 32M
max_heap_table_size = 32M
Note that adding innodb_file_per_table to a running system will not change anything. You can convert your databases and I can look up the procedure if you need it.
bb
Here are some settings I've changed:
innodb_file_per_table = 1
innodb_buffer_pool_size = 512m
tmp_table_size = 32M
max_heap_table_size = 32M
Note that adding innodb_file_per_table to a running system will not change anything. You can convert your databases and I can look up the procedure if you need it.
bb
Re: Suddenly high system load due to ZMC process
Hi bb,
we are talking about maybe 10 events a day, not more. At the moment arround 400 events are present at all.
Honestly spoken I have a very limited understanding of MySQL, yet. I will take a close look at 'mysqltuner' and get a better understanding of your recommendation. Just one qustion in advance: would you expect a higher CPU load for the 'zmc' processes if at last MySQL has the performance issue?
Thanks again,
Mark
we are talking about maybe 10 events a day, not more. At the moment arround 400 events are present at all.
Honestly spoken I have a very limited understanding of MySQL, yet. I will take a close look at 'mysqltuner' and get a better understanding of your recommendation. Just one qustion in advance: would you expect a higher CPU load for the 'zmc' processes if at last MySQL has the performance issue?
Thanks again,
Mark
Re: Suddenly high system load due to ZMC process
Yes. Well it may be zmaudit that runs up MySQL. Default settings for zmaudit are to run every 900 seconds. This checks ZM database and file system and puts them in sync.
Did not see which Linux flavor and version you are running..
bb
Did not see which Linux flavor and version you are running..
bb
Re: Suddenly high system load due to ZMC process
Hi bb,
OK, understood.
This is the 'mysqltuner' output so far:
It seems to be a good idea to give it some more time for proper analysis (next to last recommendation).
>Did not see which Linux flavor and version you are running..
Sorry, I forgot to answer. Debain 7.8 "Wheezy".
OK, understood.
This is the 'mysqltuner' output so far:
Code: Select all
-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.44-0+deb7u1
[OK] Operating on 32-bit architecture with less than 2GB RAM
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in InnoDB tables: 2M (Tables: 18)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] Total fragmented tables: 18
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 5h 4m 39s (6K q [0.329 qps], 454 conn, TX: 63M, RX: 1M)
[--] Reads / Writes: 37% / 63%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[OK] Maximum possible memory usage: 597.8M (29% of installed RAM)
[OK] Slow queries: 0% (0/6K)
[OK] Highest usage of available connections: 7% (12/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/103.0K
[OK] Query cache efficiency: 83.8% (3K cached / 4K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 32 sorts)
[OK] Temporary tables created on disk: 24% (79 on disk / 317 total)
[OK] Thread cache hit rate: 97% (12 created / 454 connections)
[OK] Table cache hit rate: 89% (59 open / 66 opened)
[OK] Open file limit used: 4% (48/1K)
[OK] Table locks acquired immediately: 100% (1K immediate / 1K locks)
[OK] InnoDB data size / buffer pool: 2.0M/128.0M
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Enable the slow query log to troubleshoot bad queries
>Did not see which Linux flavor and version you are running..
Sorry, I forgot to answer. Debain 7.8 "Wheezy".
Re: Suddenly high system load due to ZMC process
>Yes. Well it may be zmaudit that runs up MySQL.
Just to make this clear ... even if this reflects my minor comprehension: It is the Capture Deamon (zmc process) that suddenly has a tripeled CPU load.
Just to make this clear ... even if this reflects my minor comprehension: It is the Capture Deamon (zmc process) that suddenly has a tripeled CPU load.