Zoneminder keeps crashing

Forum for questions and support relating to the 1.30.x releases only.
Locked
ebenezer191
Posts: 6
Joined: Thu Mar 21, 2019 2:26 pm

Zoneminder keeps crashing

Post by ebenezer191 »

Hello,
I've zoneminder 1.30.4+dfsg-2~bpo9+1 on Debian 9, installed from backports. It works with php-fpm7.0 on Nginx.
I've installed zoneminder about 6 weeks ago, configured, installed the webcam, everything was fine for what I had tested.
Since maybe 10 days ago zoneminder keeps crashing.
Here something from the log during the crash:

Code: Select all

Mar 21 15:56:51 mycomputer zmaudit[16269]: FAT [Can't execute: Lost connection to MySQL server during query]
Mar 21 15:56:51 mycomputer zmaudit[16269]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Mar 21 15:56:51 mycomputer zmdc[16218]: ERR ['zmaudit.pl -c' exited abnormally, exit status 255]
Mar 21 15:56:51 mycomputer zmdc[16218]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Mar 21 15:56:53 mycomputer zmwatch[16274]: FAT [Can't execute: MySQL server has gone away]
Mar 21 15:56:53 mycomputer zmwatch[16274]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Mar 21 15:57:27 mycomputer zmc_dvideo1[16262]: INF [gspca_sn9c20x: 21000 - Capturing at 19.61 fps]
Mar 21 15:57:47 mycomputer zmfilter[16263]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
Mar 21 15:58:18 mycomputer zmc_dvideo1[16262]: INF [gspca_sn9c20x: 22000 - Capturing at 19.61 fps]
Mar 21 15:58:47 mycomputer zmfilter[16263]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
Mar 21 15:59:09 mycomputer zmc_dvideo1[16262]: INF [gspca_sn9c20x: 23000 - Capturing at 19.61 fps]
Mar 21 15:59:47 mycomputer zmfilter[16263]: FAT [Can't execute 'SELECT * FROM Filters WHERE Background = 1 and(  AutoArchive = 1#012                or AutoVideo = 1#012                or AutoUpload = 1#012                or AutoEmail = 1#012                or AutoMessage = 1#012                or AutoExecute = 1#012                or AutoDelete = 1#012             ) ORDER BY Name': MySQL server has gone away]

I don't have a clue about what's exactly going on, but I can add that I suspect the crash first occurred after a recent upgrade of the whole php suite, included php-fpm7.0...
Losing the connection to the db is weird, maybe there's a timeout to increase somewhere, I don't know

Thank you for the help
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Zoneminder keeps crashing

Post by iconnor »

perhaps you are running out of mysql connections. It defaults to 100 I think... each monitor can use 3 or 4 connections, plus more for every viewer.

I improved this is 1.32, including showing how many are in use in the ui. Check your mysql logs.
ebenezer191
Posts: 6
Joined: Thu Mar 21, 2019 2:26 pm

Re: Zoneminder keeps crashing

Post by ebenezer191 »

Thank you for your answer.
In my my.cnf I didn't set the parameter maxconnection. At the moment I only have one camera in function, to make it easier to investigate the problem. The mysql server log, which I've just enabled, shows:

Code: Select all

3 Query     insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( '1553246596.80221', 'zmfilter', '8658', '-2', 'ERR', 'Can\'t execute filter \'SELECT E.*,\n       unix_timestamp(E.StartTime) as Time,\n       M.Name as MonitorName,\n       M.DefaultRate,\n       M.DefaultScale\n         FROM Events as E\n         INNER JOIN Monitors as M on M.Id = E.MonitorId\n          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100\', ignoring: MySQL server has gone away', 'zmfilter.pl', NULL )
On MariaDB there's only the ZM DB it seems weird to me that it's not able to bear one db. And moreover, as I've before pointed out, when I installed it, it was working... :roll:

Thanks
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Zoneminder keeps crashing

Post by bbunge »

Run mysqltuner to see recommendations. Note that the value may not be in the my.cnf or one of the other config files for the database server but the server is using a default value (151 for Mariadb). It is possible the my.cnf is pretty bare. You can add a value to the file:

[mysqld]
max_connections=[desired new maximum number]

See: https://www.ionos.com/community/hosting ... ons-error/
ebenezer191
Posts: 6
Joined: Thu Mar 21, 2019 2:26 pm

Re: Zoneminder keeps crashing

Post by ebenezer191 »

Thank you for the tip about mysqltuner, I've changed some values according to the suggestions, and also set max_connections=200 but the server keeps crashing

Code: Select all

zmaudit[15538]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
zmdc[15495]: ERR ['zmaudit.pl -c' exited abnormally, exit status 255]
 zmdc[15495]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
 zmfilter[15530]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
 zmwatch[15545]: FAT [Can't execute: MySQL server has gone away]
 zmfilter[15530]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
 zmfilter[15530]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
 zmfilter[15530]: FAT [Can't execute 'SELECT * FROM Filters WHERE Background = 1 and(  AutoArchive = 1#012                or AutoVideo = 1#012                or AutoUpload = 1#012                or AutoEmail = 1#012                or AutoMessage = 1#012                or AutoExecute = 1#012                or AutoDelete = 1#012             ) ORDER BY Name': MySQL server has gone away]
 zmc_dvideo1[15528]: INF [gspca_sn9c20x: 5000 - Capturing at 2.72 fps]
 zmc_dvideo1[15528]: INF [gspca_sn9c20x: 6000 - Capturing at 2.60 fps]
mah... :roll:
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Zoneminder keeps crashing

Post by bbunge »

200 may not be enough. I have a server with a lot of folks viewing montage so I set connections to 1000.

Check your memory especially tmpfs. Linux uses half of the RAM for tmpfs and Zoneminder can fill that space, /dev/shm, if you use high resolution camera settings.
ebenezer191
Posts: 6
Joined: Thu Mar 21, 2019 2:26 pm

Re: Zoneminder keeps crashing

Post by ebenezer191 »

Thank you for your answer. Unfortunately it doesn't matter how high I set the value of max_connections, it keeps crashing. I also checked the temporary fs and and it's still almost empty. Please note that I'm currently running only a small usb cam at low resolution (640px), I would wonder if such environment could cause all those troubles to my system (but I'm not a zm expert).
What I don't understand (well, among the other issues) is why before the same system was working without problems. The only change I'm aware of, is the update of php, which let me think this could be the cause, although I don't understand how.
Thanks.
ebenezer191
Posts: 6
Joined: Thu Mar 21, 2019 2:26 pm

Re: Zoneminder keeps crashing

Post by ebenezer191 »

I've stopped logging to the web gui and the db; now it works without crashing, and when it does, it restarts without doing anything
jwarfin
Posts: 41
Joined: Mon Jul 23, 2018 4:36 am

Re: Zoneminder keeps crashing

Post by jwarfin »

ebenezer191 wrote: Fri Mar 29, 2019 4:02 pm I've stopped logging to the web gui and the db; now it works without crashing, and when it does, it restarts without doing anything
Your comment reminded me that I had to do the same sort of thing. My initial ZM 1.30.4 (on Ubuntu 18.04) setup ran fairly well at first with just one cam with logging to DB enabled. But, after adding several more cams ZM became increasingly unstable with odd monitor errors and race conditions. I eventually suspected some sort of edge condition related to DB contention/conflict. Tried additional DB tunings (via mysqltuner) multiple times over several weeks but to no avail. My system has a 6 core 12 thread CPU, 16 gigs of RAM and ample fast disk storage - the ZM stability issues were not related to lack of resources or bandwidth. Anyway...

To debug further, I planned to do the typical exercise: systematically disable/enable some things/settings and observe results. Turning off all logging to DB was the first thing I tried and THAT immediately restored stability.

Also (after the above), based on a bug report I read someplace (can't find a reference now) I also I set both LOG_LEVEL_SYSLOG and LOG_LEVEL_FILE to Info. This is actually redundant, but doing this (supposedly) avoids tripping an odd bug that can result in a weird ZM exception error. Doing this also meant I had to make some changes related to how log files are managed...

I added some rsyslog rules to assure all rsyslog related ZM log entries actually go to /var/log/zm.log. To do this I added a /etc/rsyslog.d/10-zm_log.conf file, with contents like this:
:programname, startswith, "zm" /var/log/zm.log
& stop
:programname, isequal, "check" /var/log/zm.log
& stop
:programname, isequal, "check2" /var/log/zm.log
& stop
:programname, startswith, "web_" /var/log/zm.log
& stop
This ZM logging setup does result in redundant logging to multiple logs in /var/log/zm AND a "unified" log /var/log/zm.log. Not exactly elegant, but my ZM setup remains 100% stable with this.

For log rotation, my /etc/logrotate.d/zonenminder config file contents:
#
# Zoneminder logs directly to files in /var/log/zm/
# (not via syslog/rsyslog). Thus rotate must be done
# via Zoneminder native log rotation facility.
#
/var/log/zm/*.log {
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
/usr/bin/zmpkg.pl logrot >>/dev/null 2>&1 || :
endscript
}
#
# Zoneminder uses syslog/rsyslog to log to /var/log/zm.log.
# Thus rotate is handled the standard way for syslog/rsyslog logs.
#
/var/log/zm.log {
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
My ZM 1.30.4 setup has been rock stable (over a year now) ever since making these changes.

Note: the examples above are for an Ubuntu 18.04 system.
Last edited by jwarfin on Sat Mar 30, 2019 5:32 pm, edited 1 time in total.
ebenezer191
Posts: 6
Joined: Thu Mar 21, 2019 2:26 pm

Re: Zoneminder keeps crashing

Post by ebenezer191 »

Thank you
Locked