[H264] For time to time I find ZM partially stopped

Forum for questions and support relating to the 1.30.x releases only.
Locked
abi
Posts: 61
Joined: Fri Oct 23, 2015 11:25 am

[H264] For time to time I find ZM partially stopped

Post by abi »

I made this topic before, however I decided to delete it as I thought I can figure out what is happening by myself. However,now I totally ran out of ideas.
So, the problem is that sometime (say, 2 times per week) I found ZM in rather strange state.
It runs, however for zmdc.pl, no sock file or pid exists. It leaves most of stuff running, but for webconsole the status is 'stopped' and I have no ability to view live video. From the other side, api works and instance can easily be accessed from ZMNinja.

I found no server side problems or logs, like OOM or similar activity from the server itself to stop ZM. Also, with debug mode for zmdc.pl I found no registered event to stop ZM as well. From the code, it looks like that deletion of pid file logs.

However, logs show that when ZM enters this state most all processes are being restarted. (by watchdog?). And, actually all of them are restarted. Disappearance of pid worries me. I disabled rc.d script, to exclude possible clash b/w ZM itself and rc.d, so it looks like supernatural event to me.

Code: Select all

root@zm:~ # ps -aux
USER    PID %CPU %MEM     VSZ    RSS TT  STAT STARTED     TIME COMMAND
www   72316  4.5  1.3  522036 330148  -  SJ   04:45   29:20.88 /usr/local/bin/zma -m 1
www   72308  4.4  1.3  513844 324008  -  SJ   04:45   30:00.99 /usr/local/bin/zma -m 3
www   72294  4.2  1.6  652224 407356  -  SJ   04:45   42:54.21 /usr/local/bin/zmc -m 6
www   98461  4.0  1.1  483132 287888  -  SJ   15:29    0:08.63 /usr/local/www/zoneminder-h264/cgi-bin/nph-zms (zms)
www   72292  3.5  1.6  666560 412048  -  SJ   04:45   42:03.24 /usr/local/bin/zmc -m 7
www   72287  2.8  1.6  654272 409820  -  SJ   04:45   33:25.38 /usr/local/bin/zmc -m 3
www   72295  2.7  1.7  664512 417852  -  SJ   04:45   28:35.27 /usr/local/bin/zmc -m 1
www   72306  2.0  0.6  338340 148484  -  SJ   04:45   14:23.74 /usr/local/bin/zma -m 12
www   72309  2.0  0.6  334244 145656  -  SJ   04:45   13:39.69 /usr/local/bin/zma -m 8
www   72286  1.8  0.6  330148 143040  -  SJ   04:45   13:09.51 /usr/local/bin/zma -m 2
www   72313  1.8  0.6  330148 143048  -  SJ   04:45   13:14.38 /usr/local/bin/zma -m 4
www   72310  1.6  0.6  326052 140436  -  SJ   04:45   12:56.05 /usr/local/bin/zma -m 5
www   72326  1.4  1.7  664512 420848  -  SJ   04:45   17:56.85 /usr/local/bin/zmc -m 10
www   72314  1.1  1.3  505652 317864  -  SJ   04:45   12:38.19 /usr/local/bin/zma -m 9
www   97774  0.8  0.1   64916  13224  -  SJ   15:16    0:07.89 php-fpm: pool www (php-fpm)
www   72297  0.7  0.7  394800 182784  -  SJ   04:45   12:47.91 /usr/local/bin/zmc -m 4
www   72285  0.6  0.7  402992 183672  -  SJ   04:45   23:14.62 /usr/local/bin/zmc -m 12
www   72299  0.6  0.7  394800 182824  -  SJ   04:45   16:19.95 /usr/local/bin/zmc -m 2
www   72284  0.5  0.7  398896 178720  -  SJ   04:45   20:44.79 /usr/local/bin/zmc -m 5
www   72298  0.5  1.7  658368 420232  -  SJ   04:45   24:22.86 /usr/local/bin/zmc -m 11
www   72293  0.4  0.7  398896 186688  -  SJ   04:45   15:02.46 /usr/local/bin/zmc -m 8
www   72296  0.3  1.7  678848 434904  -  SJ   04:45   19:22.49 /usr/local/bin/zmc -m 9
root   8057  0.0  0.0   10464   1508  -  SsJ  Wed22    0:03.16 /usr/sbin/syslogd -c -ss
mysql  8113  0.0  0.0   13144      0  -  IWsJ -        0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --pid-file=/var/db/mysql/zm.dacha.abinet.ru.
mysql  9425  0.0  1.7 2042436 425136  -  SJ   Wed22   10:41.60 /usr/local/libexec/mysqld --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --plugin-dir=/usr/local/lib/mysql/plugin --skip-perfo
root   9435  0.0  0.0   58772   2764  -  SsJ  Wed22    0:03.50 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
root   9440  0.0  0.0   20260      0  -  IWsJ -        0:00.00 nginx: master process /usr/local/sbin/nginx
www    9441  0.0  0.0   20260   2124  -  SJ   Wed22    0:18.50 nginx: worker process (nginx)
www    9449  0.0  0.0   10428      0  -  IWsJ -        0:00.00 daemon: /usr/local/sbin/fcgiwrap[9450] (daemon)
www    9450  0.0  0.0   10408      0  -  IWJ  -        0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9453  0.0  0.0   10408    812  -  IJ   Wed22    0:01.13 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9454  0.0  0.0   10408    812  -  IJ   Wed22    0:00.68 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9455  0.0  0.0   10408    812  -  IJ   Wed22    0:01.19 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9456  0.0  0.0   10408    812  -  IJ   Wed22    0:01.02 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9457  0.0  0.0   10408    812  -  IJ   Wed22    0:00.91 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9458  0.0  0.0   10408    812  -  IJ   Wed22    0:00.65 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9459  0.0  0.0   10408    812  -  IJ   Wed22    0:00.67 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9460  0.0  0.0   10408    812  -  IJ   Wed22    0:01.63 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9461  0.0  0.0   10408    812  -  IJ   Wed22    0:00.89 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9462  0.0  0.0   10408    836  -  IJ   Wed22    0:00.67 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9463  0.0  0.0   10408    836  -  IJ   Wed22    0:00.44 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9464  0.0  0.0   10408    836  -  IJ   Wed22    0:01.61 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9465  0.0  0.0   10408    824  -  IJ   Wed22    0:00.83 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9466  0.0  0.0   10408    812  -  SJ   Wed22    0:00.95 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9467  0.0  0.0   10408      0  -  IWJ  -        0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9468  0.0  0.0   10408      0  -  IWJ  -        0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9469  0.0  0.0   10408      0  -  IWJ  -        0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www    9470  0.0  0.0   10408      0  -  IWJ  -        0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
root   9492  0.0  0.0   55676   3408  -  IsJ  Wed22    0:00.00 /usr/sbin/sshd
root   9496  0.0  0.0   12564    532  -  SsJ  Wed22    0:00.27 /usr/sbin/cron -J 15 -s
www   72161  0.0  0.1  111652  37612  -  IJ   04:44    0:01.14 /usr/local/bin/perl -wT /usr/local/bin/zmfilter.pl
www   72196  0.0  0.1   99324  29272  -  IJ   04:44    0:22.63 /usr/local/bin/perl -wT /usr/local/bin/zmaudit.pl -c
www   72307  0.0  0.1  479028  28580  -  SJ   04:45    0:10.29 /usr/local/bin/zma -m 11
www   72311  0.0  0.9  479028 222824  -  SJ   04:45    0:10.94 /usr/local/bin/zma -m 7
www   72312  0.0  0.9  479028 218236  -  SJ   04:45    0:10.99 /usr/local/bin/zma -m 6
www   72315  0.0  0.2  479028  53924  -  SJ   04:45    0:10.26 /usr/local/bin/zma -m 10
www   78341  0.0  0.1   84976  16784  -  SJ   07:17    0:06.11 /usr/local/bin/perl -wT /usr/local/bin/zmwatch.pl
root  97554  0.0  0.0   43732   2300  2  IJ   15:13    0:00.01 login [pam] (login)
root  97555  0.0  0.0   19600   3104  2  SJ   15:13    0:00.01 -csh (csh)
root  98658  0.0  0.0   21164   2120  2  R+J  15:33    0:00.00 ps -aux
www   24965  0.0  0.1   84976  17104  0- SJ   Wed22    1:47.17 /usr/local/bin/perl -wT /usr/local/bin/zmdc.pl startup
ps output shows normal running instance. You can expect the same when ZM runs normally.

Code: Select all

root@zm:~ # ls -la /var/run/zm
total 44
drwxr-xr-x  2 www   www    42 Nov 26 15:29 .
drwxr-xr-x  7 root  wheel  20 Nov 26 04:15 ..
-rw-------  1 www   www     0 Nov 24 15:58 zms-034453.lock
srwxr-xr-x  1 www   www     0 Nov 24 15:58 zms-034453s.sock
-rw-------  1 www   www     0 Nov 24 17:50 zms-147744.lock
srwxr-xr-x  1 www   www     0 Nov 24 17:50 zms-147744s.sock
-rw-------  1 www   www     0 Nov 24 11:13 zms-186381.lock
srwxr-xr-x  1 www   www     0 Nov 24 11:13 zms-186381s.sock
-rw-------  1 www   www     0 Nov 24 15:59 zms-263888.lock
srwxr-xr-x  1 www   www     0 Nov 24 15:59 zms-263888s.sock
-rw-------  1 www   www     0 Nov 24 13:46 zms-320506.lock
srwxr-xr-x  1 www   www     0 Nov 24 13:46 zms-320506s.sock
-rw-------  1 www   www     0 Nov 24 15:57 zms-344291.lock
srwxr-xr-x  1 www   www     0 Nov 24 15:57 zms-344291s.sock
-rw-------  1 www   www     0 Nov 24 15:57 zms-381352.lock
srwxr-xr-x  1 www   www     0 Nov 24 15:57 zms-381352s.sock
-rw-------  1 www   www     0 Nov 24 11:15 zms-661443.lock
srwxr-xr-x  1 www   www     0 Nov 24 11:15 zms-661443s.sock
-rw-------  1 www   www     0 Nov 25 20:11 zms-728404.lock
srwxr-xr-x  1 www   www     0 Nov 25 20:11 zms-728404s.sock
-rw-------  1 www   www     0 Nov 24 21:16 zms-741737.lock
srwxr-xr-x  1 www   www     0 Nov 24 21:16 zms-741737s.sock
-rw-------  1 www   www     0 Nov 24 21:17 zms-770522.lock
srwxr-xr-x  1 www   www     0 Nov 24 21:17 zms-770522s.sock
-rw-------  1 www   www     0 Nov 24 13:46 zms-832133.lock
srwxr-xr-x  1 www   www     0 Nov 24 13:46 zms-832133s.sock
-rw-------  1 www   www     0 Nov 24 17:45 zms-891026.lock
srwxr-xr-x  1 www   www     0 Nov 24 17:45 zms-891026s.sock
-rw-------  1 www   www     0 Nov 24 21:18 zms-891784.lock
srwxr-xr-x  1 www   www     0 Nov 24 21:18 zms-891784s.sock
-rw-------  1 www   www     0 Nov 24 17:50 zms-925406.lock
srwxr-xr-x  1 www   www     0 Nov 24 17:50 zms-925406s.sock
-rw-------  1 www   www     0 Nov 26 15:29 zms-937205.lock
srwxr-xr-x  1 www   www     0 Nov 26 15:29 zms-937205s.sock
-rw-------  1 www   www     0 Nov 24 15:58 zms-969847.lock
srwxr-xr-x  1 www   www     0 Nov 24 15:58 zms-969847s.sock
-rw-------  1 www   www     0 Nov 24 11:14 zms-976765.lock
srwxr-xr-x  1 www   www     0 Nov 24 11:14 zms-976765s.sock
-rw-------  1 www   www     0 Nov 24 11:14 zms-981259.lock
srwxr-xr-x  1 www   www     0 Nov 24 11:14 zms-981259s.sock
-rw-------  1 www   www     0 Nov 24 15:57 zms-996497.lock
srwxr-xr-x  1 www   www     0 Nov 24 15:57 zms-996497s.sock
Some stalle files (I suspect they are not connected to the issue) and absence of zmdc sock file. But stalled files older than event date mean the folder was not emptied by someone.

Code: Select all

11/26/16 04:23:42.071851 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:23:42.073617 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:23:42, pid = 71491]
11/26/16 04:23:42.073689 zmdc[71491].INF ['zmc -m 10' started at 16/11/26 04:23:42]
11/26/16 04:34:27.768526 zmdc[24965].INF ['zmc -m 10' sending stop to pid 71491 at 16/11/26 04:34:27]
11/26/16 04:34:33.806223 zmdc[24965].WAR ['zmc -m 10' has not stopped at 16/11/26 04:34:33. Sending KILL to pid 71491]
11/26/16 04:34:33.818137 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:34:33.919177 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:34:33.921147 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:34:33, pid = 71834]
11/26/16 04:34:33.921149 zmdc[71834].INF ['zmc -m 10' started at 16/11/26 04:34:33]
11/26/16 04:39:14.151747 zmdc[24965].INF ['zmc -m 10' sending stop to pid 71834 at 16/11/26 04:39:14]
11/26/16 04:39:17.842886 zmdc[24965].INF ['zmc -m 10' exited normally]
11/26/16 04:39:22.019604 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:39:22.021440 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:39:22, pid = 71969]
11/26/16 04:39:22.021451 zmdc[71969].INF ['zmc -m 10' started at 16/11/26 04:39:22]
11/26/16 04:44:18.168526 zmdc[24965].INF ['zmc -m 10' sending stop to pid 71969 at 16/11/26 04:44:18]
11/26/16 04:44:24.184824 zmdc[24965].WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:24. Sending KILL to pid 71969]
11/26/16 04:44:24.196526 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:44:34.061568 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:44:34.063398 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:44:34, pid = 72141]
11/26/16 04:44:34.063415 zmdc[72141].INF ['zmc -m 10' started at 16/11/26 04:44:34]
11/26/16 04:44:34.323494 zmdc[24965].INF ['zmc -m 10' sending stop to pid 72141 at 16/11/26 04:44:34]
11/26/16 04:44:34.425456 zmdc[24965].INF ['zmc -m 5' exited normally]
11/26/16 04:44:34.452650 zmdc[24965].INF ['zma -m 11' exited normally]
11/26/16 04:44:34.503089 zmdc[24965].INF ['zma -m 3' exited normally]
11/26/16 04:44:34.525917 zmdc[24965].INF ['zma -m 8' exited normally]
11/26/16 04:44:34.591736 zmdc[24965].INF ['zmc -m 3' exited normally]
11/26/16 04:44:34.591910 zmdc[24965].INF ['zma -m 2' exited normally]
11/26/16 04:44:34.606082 zmdc[24965].INF ['zmaudit.pl -c' exited, signal 14]
11/26/16 04:44:34.606200 zmdc[24965].WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:34. Sending KILL to pid 72141]
11/26/16 04:44:34.607035 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:44:34.639984 zmdc[24965].INF ['zmc -m 8' exited normally]
11/26/16 04:44:34.640155 zmdc[24965].INF [Starting pending process, zma -m 2]
11/26/16 04:44:34.641750 zmdc[24965].INF ['zma -m 2' starting at 16/11/26 04:44:34, pid = 72145]
11/26/16 04:44:34.641754 zmdc[72145].INF ['zma -m 2' started at 16/11/26 04:44:34]
11/26/16 04:44:34.641917 zmdc[24965].INF [Starting pending process, zmaudit.pl -c]
11/26/16 04:44:34.643239 zmdc[24965].INF ['zmaudit.pl -c' starting at 16/11/26 04:44:34, pid = 72146]
11/26/16 04:44:34.643407 zmdc[24965].INF [Starting pending process, zmc -m 5]
11/26/16 04:44:34.643394 zmdc[72146].INF ['zmaudit.pl -c' started at 16/11/26 04:44:34]
11/26/16 04:44:34.645506 zmdc[72147].INF ['zmc -m 5' started at 16/11/26 04:44:34]
11/26/16 04:44:34.648921 zmdc[24965].INF ['zmc -m 5' starting at 16/11/26 04:44:34, pid = 72147]
11/26/16 04:44:34.649255 zmdc[24965].INF [Starting pending process, zma -m 8]
11/26/16 04:44:34.651519 zmdc[24965].INF ['zma -m 8' starting at 16/11/26 04:44:34, pid = 72148]
11/26/16 04:44:34.651795 zmdc[24965].INF [Starting pending process, zma -m 3]
11/26/16 04:44:34.653824 zmdc[72148].INF ['zma -m 8' started at 16/11/26 04:44:34]
11/26/16 04:44:34.654123 zmdc[24965].INF ['zma -m 3' starting at 16/11/26 04:44:34, pid = 72149]
11/26/16 04:44:34.654390 zmdc[24965].INF [Starting pending process, zma -m 11]
11/26/16 04:44:34.655363 zmdc[72149].INF ['zma -m 3' started at 16/11/26 04:44:34]
11/26/16 04:44:34.670264 zmdc[24965].INF ['zma -m 11' starting at 16/11/26 04:44:34, pid = 72150]
11/26/16 04:44:34.670566 zmdc[24965].INF [Starting pending process, zmc -m 3]
11/26/16 04:44:34.672604 zmdc[72150].INF ['zma -m 11' started at 16/11/26 04:44:34]
11/26/16 04:44:34.673627 zmdc[72151].INF ['zmc -m 3' started at 16/11/26 04:44:34]
11/26/16 04:44:34.701424 zmdc[24965].INF ['zmc -m 3' starting at 16/11/26 04:44:34, pid = 72151]
11/26/16 04:44:34.701728 zmdc[24965].INF [Starting pending process, zmc -m 8]
11/26/16 04:44:34.721405 zmdc[72152].INF ['zmc -m 8' started at 16/11/26 04:44:34]
11/26/16 04:44:34.744183 zmdc[24965].INF ['zmc -m 8' starting at 16/11/26 04:44:34, pid = 72152]
11/26/16 04:44:34.795404 zmdc[24965].INF ['zmc -m 1' exited normally]
11/26/16 04:44:34.795688 zmdc[24965].INF [Starting pending process, zmc -m 1]
11/26/16 04:44:34.798172 zmdc[72154].INF ['zmc -m 1' started at 16/11/26 04:44:34]
11/26/16 04:44:34.799116 zmdc[24965].INF ['zmc -m 1' starting at 16/11/26 04:44:34, pid = 72154]
11/26/16 04:44:34.802345 zmdc[24965].INF ['zma -m 11' sending stop to pid 72150 at 16/11/26 04:44:34]
11/26/16 04:44:34.915106 zmdc[24965].INF ['zmwatch.pl' exited, signal 14]
11/26/16 04:44:34.967582 zmdc[24965].INF ['zmc -m 9' exited normally]
11/26/16 04:44:35.061434 zmdc[24965].INF ['zma -m 6' exited normally]
11/26/16 04:44:35.169509 zmdc[24965].INF ['zmc -m 7' exited normally]
11/26/16 04:44:35.186856 zmdc[24965].INF ['zma -m 5' exited normally]
11/26/16 04:44:35.188310 zmdc[24965].INF ['zma -m 12' exited normally]
11/26/16 04:44:35.188568 zmdc[24965].WAR ['zma -m 11' has not stopped at 16/11/26 04:44:35. Sending KILL to pid 72150]
11/26/16 04:44:35.189506 zmdc[24965].INF ['zma -m 11' crashed, signal 8]
11/26/16 04:44:35.191605 zmdc[24965].INF ['zma -m 1' exited normally]
11/26/16 04:44:35.218225 zmdc[24965].INF ['zma -m 7' exited normally]
11/26/16 04:44:35.218453 zmdc[24965].INF [Starting pending process, zma -m 12]
11/26/16 04:44:35.221114 zmdc[24965].INF ['zma -m 12' starting at 16/11/26 04:44:35, pid = 72155]
11/26/16 04:44:35.221351 zmdc[24965].INF [Starting pending process, zmwatch.pl]
11/26/16 04:44:35.221808 zmdc[72155].INF ['zma -m 12' started at 16/11/26 04:44:35]
11/26/16 04:44:35.223589 zmdc[24965].INF ['zmwatch.pl' starting at 16/11/26 04:44:35, pid = 72156]
11/26/16 04:44:35.223925 zmdc[72156].INF ['zmwatch.pl' started at 16/11/26 04:44:35]
11/26/16 04:44:35.224160 zmdc[24965].INF [Starting pending process, zmc -m 7]
11/26/16 04:44:35.232935 zmdc[72157].INF ['zmc -m 7' started at 16/11/26 04:44:35]
11/26/16 04:44:35.237318 zmdc[24965].INF ['zmc -m 7' starting at 16/11/26 04:44:35, pid = 72157]
11/26/16 04:44:35.247146 zmdc[24965].INF [Starting pending process, zma -m 7]
11/26/16 04:44:35.263310 zmdc[24965].INF ['zma -m 7' starting at 16/11/26 04:44:35, pid = 72158]
11/26/16 04:44:35.269104 zmdc[24965].INF [Starting pending process, zma -m 6]
11/26/16 04:44:35.287334 zmdc[72159].INF ['zma -m 6' started at 16/11/26 04:44:35]
11/26/16 04:44:35.298115 zmdc[72158].INF ['zma -m 7' started at 16/11/26 04:44:35]
11/26/16 04:44:35.340504 zmdc[24965].INF ['zma -m 6' starting at 16/11/26 04:44:35, pid = 72159]
11/26/16 04:44:35.375050 zmdc[24965].INF ['zmc -m 6' exited normally]
11/26/16 04:44:35.413586 zmdc[24965].INF ['zma -m 10' exited normally]
11/26/16 04:44:35.439142 zmdc[24965].INF ['zmc -m 2' exited normally]
11/26/16 04:44:35.572021 zmdc[24965].ERR ['zma -m 8' exited abnormally, exit status 255]
11/26/16 04:44:35.572343 zmdc[24965].INF ['zma -m 4' exited normally]
11/26/16 04:44:35.572476 zmdc[24965].INF ['zmfilter.pl' exited, signal 14]
11/26/16 04:44:35.572609 zmdc[24965].INF ['zma -m 9' exited normally]
11/26/16 04:44:35.572773 zmdc[24965].INF [Starting pending process, zmfilter.pl]
11/26/16 04:44:35.576444 zmdc[24965].INF ['zmfilter.pl' starting at 16/11/26 04:44:35, pid = 72161]
11/26/16 04:44:35.576773 zmdc[24965].INF [Starting pending process, zmc -m 9]
11/26/16 04:44:35.601208 zmdc[24965].INF ['zmc -m 9' starting at 16/11/26 04:44:35, pid = 72162]
11/26/16 04:44:35.601560 zmdc[24965].INF [Starting pending process, zma -m 4]
11/26/16 04:44:35.605847 zmdc[24965].INF ['zma -m 4' starting at 16/11/26 04:44:35, pid = 72163]
11/26/16 04:44:35.606188 zmdc[24965].INF [Starting pending process, zma -m 9]
11/26/16 04:44:35.608953 zmdc[24965].INF ['zma -m 9' starting at 16/11/26 04:44:35, pid = 72164]
11/26/16 04:44:35.609267 zmdc[24965].INF [Starting pending process, zma -m 10]
11/26/16 04:44:35.610789 zmdc[72161].INF ['zmfilter.pl' started at 16/11/26 04:44:35]
11/26/16 04:44:35.618376 zmdc[72162].INF ['zmc -m 9' started at 16/11/26 04:44:35]
11/26/16 04:44:35.642563 zmdc[72165].INF ['zma -m 10' started at 16/11/26 04:44:35]
11/26/16 04:44:35.642480 zmdc[24965].INF ['zma -m 10' starting at 16/11/26 04:44:35, pid = 72165]
11/26/16 04:44:35.643165 zmdc[24965].INF [Starting pending process, zma -m 1]
11/26/16 04:44:35.606945 zmdc[72163].INF ['zma -m 4' started at 16/11/26 04:44:35]
11/26/16 04:44:35.658254 zmdc[72164].INF ['zma -m 9' started at 16/11/26 04:44:35]
11/26/16 04:44:35.693056 zmdc[72166].INF ['zma -m 1' started at 16/11/26 04:44:35]
11/26/16 04:44:35.705370 zmdc[24965].INF ['zma -m 1' starting at 16/11/26 04:44:35, pid = 72166]
11/26/16 04:44:35.706075 zmdc[24965].INF ['zmc -m 4' exited normally]
11/26/16 04:44:35.706361 zmdc[24965].INF ['zmc -m 11' exited normally]
11/26/16 04:44:35.706520 zmdc[24965].INF [Starting pending process, zmc -m 11]
11/26/16 04:44:35.718976 zmdc[72167].INF ['zmc -m 11' started at 16/11/26 04:44:35]
11/26/16 04:44:35.761429 zmdc[24965].INF ['zmc -m 11' starting at 16/11/26 04:44:35, pid = 72167]
11/26/16 04:44:35.761706 zmdc[24965].INF ['zmc -m 12' exited normally]
11/26/16 04:44:35.761860 zmdc[24965].INF [Starting pending process, zmc -m 2]
11/26/16 04:44:35.764254 zmdc[24965].INF ['zmc -m 2' starting at 16/11/26 04:44:35, pid = 72168]
11/26/16 04:44:35.764501 zmdc[24965].INF [Starting pending process, zma -m 5]
11/26/16 04:44:35.766803 zmdc[24965].INF ['zma -m 5' starting at 16/11/26 04:44:35, pid = 72169]
11/26/16 04:44:35.777354 zmdc[72169].INF ['zma -m 5' started at 16/11/26 04:44:35]
11/26/16 04:44:35.817024 zmdc[72168].INF ['zmc -m 2' started at 16/11/26 04:44:35]
11/26/16 04:44:35.869099 zmdc[24965].INF [Starting pending process, zmc -m 6]
11/26/16 04:44:36.067858 zmdc[24965].INF ['zmc -m 6' starting at 16/11/26 04:44:36, pid = 72170]
11/26/16 04:44:36.069954 zmdc[24965].INF [Starting pending process, zmc -m 4]
11/26/16 04:44:36.059442 zmdc[72170].INF ['zmc -m 6' started at 16/11/26 04:44:36]
11/26/16 04:44:36.111350 zmdc[24965].INF ['zmc -m 4' starting at 16/11/26 04:44:36, pid = 72171]
11/26/16 04:44:36.130271 zmdc[24965].INF [Starting pending process, zmc -m 12]
11/26/16 04:44:36.134701 zmdc[72173].INF ['zmc -m 12' started at 16/11/26 04:44:36]
11/26/16 04:44:36.250270 zmdc[24965].INF ['zmc -m 12' starting at 16/11/26 04:44:36, pid = 72173]
11/26/16 04:44:36.295638 zmdc[72171].INF ['zmc -m 4' started at 16/11/26 04:44:36]
11/26/16 04:44:36.489515 zmdc[24965].ERR ['zma -m 6' exited abnormally, exit status 255]
11/26/16 04:44:36.507134 zmdc[24965].ERR ['zma -m 12' exited abnormally, exit status 255]
11/26/16 04:44:36.523456 zmdc[24965].ERR ['zma -m 4' exited abnormally, exit status 255]
11/26/16 04:44:36.759421 zmdc[24965].ERR ['zma -m 9' exited abnormally, exit status 255]
11/26/16 04:44:36.765435 zmdc[24965].INF ['zmaudit.pl -c' exited, signal 14]
11/26/16 04:44:40.064168 zmdc[24965].INF [Starting pending process, zma -m 11]
11/26/16 04:44:40.065965 zmdc[24965].INF ['zma -m 11' starting at 16/11/26 04:44:40, pid = 72179]
11/26/16 04:44:40.065993 zmdc[72179].INF ['zma -m 11' started at 16/11/26 04:44:40]
11/26/16 04:44:40.066135 zmdc[24965].INF [Starting pending process, zma -m 8]
11/26/16 04:44:40.067516 zmdc[24965].INF ['zma -m 8' starting at 16/11/26 04:44:40, pid = 72180]
11/26/16 04:44:40.067707 zmdc[72180].INF ['zma -m 8' started at 16/11/26 04:44:40]
11/26/16 04:44:40.897470 zmdc[24965].INF ['zma -m 5' exited, signal 14]
11/26/16 04:44:40.897665 zmdc[24965].INF ['zma -m 8' exited, signal 14]
11/26/16 04:44:40.897748 zmdc[24965].INF ['zma -m 1' exited, signal 14]
11/26/16 04:44:40.897826 zmdc[24965].INF ['zma -m 11' exited, signal 14]
11/26/16 04:44:40.897904 zmdc[24965].INF ['zma -m 7' exited, signal 14]
11/26/16 04:44:40.898849 zmdc[24965].INF ['zma -m 10' exited normally]
11/26/16 04:44:41.005378 zmdc[24965].INF [Starting pending process, zma -m 12]
11/26/16 04:44:41.007252 zmdc[24965].INF ['zma -m 12' starting at 16/11/26 04:44:41, pid = 72192]
11/26/16 04:44:41.007256 zmdc[72192].INF ['zma -m 12' started at 16/11/26 04:44:41]
11/26/16 04:44:41.007415 zmdc[24965].INF [Starting pending process, zma -m 6]
11/26/16 04:44:41.008802 zmdc[24965].INF ['zma -m 6' starting at 16/11/26 04:44:41, pid = 72193]
11/26/16 04:44:41.008967 zmdc[24965].INF [Starting pending process, zma -m 4]
11/26/16 04:44:41.008945 zmdc[72193].INF ['zma -m 6' started at 16/11/26 04:44:41]
11/26/16 04:44:41.010665 zmdc[24965].INF ['zma -m 4' starting at 16/11/26 04:44:41, pid = 72194]
11/26/16 04:44:41.010827 zmdc[24965].INF [Starting pending process, zma -m 9]
11/26/16 04:44:41.010757 zmdc[72194].INF ['zma -m 4' started at 16/11/26 04:44:41]
11/26/16 04:44:41.012547 zmdc[24965].INF ['zma -m 9' starting at 16/11/26 04:44:41, pid = 72195]
11/26/16 04:44:41.012718 zmdc[24965].INF [Starting pending process, zmaudit.pl -c]
11/26/16 04:44:41.012685 zmdc[72195].INF ['zma -m 9' started at 16/11/26 04:44:41]
11/26/16 04:44:41.014147 zmdc[24965].INF ['zmaudit.pl -c' starting at 16/11/26 04:44:41, pid = 72196]
11/26/16 04:44:41.014609 zmdc[72196].INF ['zmaudit.pl -c' started at 16/11/26 04:44:41]
11/26/16 04:44:42.097973 zmdc[24965].INF ['zmc -m 5' exited normally]
11/26/16 04:44:42.406296 zmdc[24965].INF ['zmc -m 12' exited normally]
11/26/16 04:44:43.984170 zmdc[24965].INF ['zmc -m 3' exited normally]
11/26/16 04:44:44.711074 zmdc[24965].INF ['zmc -m 8' exited normally]
11/26/16 04:44:44.955188 zmdc[24965].INF ['zmc -m 7' exited normally]
11/26/16 04:44:45.061656 zmdc[24965].INF [Starting pending process, zma -m 7]
11/26/16 04:44:45.063442 zmdc[24965].INF ['zma -m 7' starting at 16/11/26 04:44:45, pid = 72199]
11/26/16 04:44:45.063431 zmdc[72199].INF ['zma -m 7' started at 16/11/26 04:44:45]
11/26/16 04:44:45.063614 zmdc[24965].INF [Starting pending process, zma -m 10]
11/26/16 04:44:45.064986 zmdc[24965].INF ['zma -m 10' starting at 16/11/26 04:44:45, pid = 72200]
11/26/16 04:44:45.065145 zmdc[24965].INF [Starting pending process, zma -m 1]
11/26/16 04:44:45.065121 zmdc[72200].INF ['zma -m 10' started at 16/11/26 04:44:45]
11/26/16 04:44:45.066796 zmdc[24965].INF ['zma -m 1' starting at 16/11/26 04:44:45, pid = 72201]
11/26/16 04:44:45.066966 zmdc[24965].INF [Starting pending process, zma -m 5]
11/26/16 04:44:45.066974 zmdc[72201].INF ['zma -m 1' started at 16/11/26 04:44:45]
11/26/16 04:44:45.068647 zmdc[24965].INF ['zma -m 5' starting at 16/11/26 04:44:45, pid = 72202]
11/26/16 04:44:45.068735 zmdc[72202].INF ['zma -m 5' started at 16/11/26 04:44:45]
11/26/16 04:44:45.213579 zmdc[24965].INF ['zmc -m 1' exited normally]
11/26/16 04:44:45.452572 zmdc[24965].ERR ['zma -m 5' exited abnormally, exit status 255]
11/26/16 04:44:45.661341 zmdc[24965].INF ['zmc -m 6' exited normally]
11/26/16 04:44:45.695243 zmdc[24965].ERR ['zma -m 7' exited abnormally, exit status 255]
11/26/16 04:44:45.698459 zmdc[24965].ERR ['zma -m 1' exited abnormally, exit status 255]
11/26/16 04:44:45.916841 zmdc[24965].INF ['zmc -m 4' crashed, signal 8]
11/26/16 04:44:45.917054 zmdc[24965].INF ['zma -m 4' crashed, signal 8]
11/26/16 04:44:45.917173 zmdc[24965].INF ['zmc -m 2' crashed, signal 8]
11/26/16 04:44:45.917824 zmdc[24965].INF ['zma -m 9' crashed, signal 8]
11/26/16 04:44:45.917935 zmdc[24965].INF ['zma -m 10' crashed, signal 8]
11/26/16 04:44:45.918145 zmdc[24965].INF ['zma -m 6' crashed, signal 8]
11/26/16 04:44:45.918388 zmdc[24965].INF ['zmc -m 9' crashed, signal 8]
11/26/16 04:44:45.918466 zmdc[24965].INF ['zma -m 12' crashed, signal 8]
11/26/16 04:44:45.920350 zmdc[24965].INF ['zmc -m 11' crashed, signal 8]
11/26/16 04:44:47.095878 zmdc[24965].INF [Starting pending process, zmc -m 5]
11/26/16 04:44:47.097689 zmdc[24965].INF ['zmc -m 5' starting at 16/11/26 04:44:47, pid = 72216]
11/26/16 04:44:47.097703 zmdc[72216].INF ['zmc -m 5' started at 16/11/26 04:44:47]
11/26/16 04:44:47.097849 zmdc[24965].INF [Starting pending process, zmc -m 12]
11/26/16 04:44:47.099224 zmdc[24965].INF ['zmc -m 12' starting at 16/11/26 04:44:47, pid = 72217]
11/26/16 04:44:47.099375 zmdc[72217].INF ['zmc -m 12' started at 16/11/26 04:44:47]
11/26/16 04:44:48.056484 zmdc[24965].INF [Starting pending process, zmc -m 3]
11/26/16 04:44:48.058304 zmdc[24965].INF ['zmc -m 3' starting at 16/11/26 04:44:48, pid = 72218]
11/26/16 04:44:48.058304 zmdc[72218].INF ['zmc -m 3' started at 16/11/26 04:44:48]
11/26/16 04:44:49.003622 zmdc[24965].INF [Starting pending process, zmc -m 7]
11/26/16 04:44:49.005479 zmdc[24965].INF ['zmc -m 7' starting at 16/11/26 04:44:49, pid = 72219]
11/26/16 04:44:49.005506 zmdc[72219].INF ['zmc -m 7' started at 16/11/26 04:44:49]
11/26/16 04:44:49.005645 zmdc[24965].INF [Starting pending process, zmc -m 8]
11/26/16 04:44:49.007102 zmdc[24965].INF ['zmc -m 8' starting at 16/11/26 04:44:49, pid = 72220]
11/26/16 04:44:49.007196 zmdc[72220].INF ['zmc -m 8' started at 16/11/26 04:44:49]
11/26/16 04:44:49.023697 zmdc[24965].INF ['zma -m 2' crashed, signal 8]
11/26/16 04:44:50.064426 zmdc[24965].INF [Starting pending process, zmc -m 6]
11/26/16 04:44:50.066325 zmdc[24965].INF ['zmc -m 6' starting at 16/11/26 04:44:50, pid = 72221]
11/26/16 04:44:50.066348 zmdc[72221].INF ['zmc -m 6' started at 16/11/26 04:44:50]
11/26/16 04:44:50.066502 zmdc[24965].INF [Starting pending process, zmc -m 1]
11/26/16 04:44:50.067901 zmdc[24965].INF ['zmc -m 1' starting at 16/11/26 04:44:50, pid = 72222]
11/26/16 04:44:50.068067 zmdc[24965].INF [Starting pending process, zma -m 11]
11/26/16 04:44:50.068067 zmdc[72222].INF ['zmc -m 1' started at 16/11/26 04:44:50]
11/26/16 04:44:50.069714 zmdc[24965].INF ['zma -m 11' starting at 16/11/26 04:44:50, pid = 72223]
11/26/16 04:44:50.069931 zmdc[24965].INF [Starting pending process, zmc -m 9]
11/26/16 04:44:50.069883 zmdc[72223].INF ['zma -m 11' started at 16/11/26 04:44:50]
11/26/16 04:44:50.071614 zmdc[72224].INF ['zmc -m 9' started at 16/11/26 04:44:50]
11/26/16 04:44:50.071772 zmdc[24965].INF ['zmc -m 9' starting at 16/11/26 04:44:50, pid = 72224]
11/26/16 04:44:50.071982 zmdc[24965].INF [Starting pending process, zma -m 8]
11/26/16 04:44:50.073814 zmdc[24965].INF ['zma -m 8' starting at 16/11/26 04:44:50, pid = 72225]
11/26/16 04:44:50.074030 zmdc[24965].INF [Starting pending process, zmc -m 4]
11/26/16 04:44:50.073985 zmdc[72225].INF ['zma -m 8' started at 16/11/26 04:44:50]
11/26/16 04:44:50.075866 zmdc[24965].INF ['zmc -m 4' starting at 16/11/26 04:44:50, pid = 72226]
11/26/16 04:44:50.076091 zmdc[24965].INF [Starting pending process, zmc -m 11]
11/26/16 04:44:50.076013 zmdc[72226].INF ['zmc -m 4' started at 16/11/26 04:44:50]
11/26/16 04:44:50.077947 zmdc[24965].INF ['zmc -m 11' starting at 16/11/26 04:44:50, pid = 72227]
11/26/16 04:44:50.078181 zmdc[24965].INF [Starting pending process, zmc -m 2]
11/26/16 04:44:50.078086 zmdc[72227].INF ['zmc -m 11' started at 16/11/26 04:44:50]
11/26/16 04:44:50.081160 zmdc[24965].INF ['zmc -m 2' starting at 16/11/26 04:44:50, pid = 72228]
11/26/16 04:44:50.081498 zmdc[72228].INF ['zmc -m 2' started at 16/11/26 04:44:50]
11/26/16 04:44:54.011403 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:44:54.013287 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:44:54, pid = 72231]
11/26/16 04:44:54.013336 zmdc[72231].INF ['zmc -m 10' started at 16/11/26 04:44:54]
11/26/16 04:44:54.013455 zmdc[24965].INF [Starting pending process, zma -m 2]
11/26/16 04:44:54.014825 zmdc[24965].INF ['zma -m 2' starting at 16/11/26 04:44:54, pid = 72232]
11/26/16 04:44:54.015068 zmdc[72232].INF ['zma -m 2' started at 16/11/26 04:44:54]
11/26/16 04:44:55.028175 zmdc[24965].INF [Starting pending process, zma -m 12]
11/26/16 04:44:55.029898 zmdc[24965].INF ['zma -m 12' starting at 16/11/26 04:44:55, pid = 72235]
11/26/16 04:44:55.030097 zmdc[24965].INF [Starting pending process, zma -m 7]
11/26/16 04:44:55.030043 zmdc[72235].INF ['zma -m 12' started at 16/11/26 04:44:55]
11/26/16 04:44:55.032038 zmdc[72237].INF ['zma -m 7' started at 16/11/26 04:44:55]
11/26/16 04:44:55.032083 zmdc[24965].INF ['zma -m 7' starting at 16/11/26 04:44:55, pid = 72237]
11/26/16 04:44:55.032353 zmdc[24965].INF ['zma -m 2' exited, signal 14]
11/26/16 04:44:55.032488 zmdc[24965].INF ['zma -m 8' exited, signal 14]
11/26/16 04:44:55.032607 zmdc[24965].INF ['zma -m 11' exited, signal 14]
11/26/16 04:44:55.032754 zmdc[24965].INF [Starting pending process, zma -m 6]
11/26/16 04:44:55.034394 zmdc[24965].INF ['zma -m 6' starting at 16/11/26 04:44:55, pid = 72239]
11/26/16 04:44:55.034579 zmdc[24965].INF [Starting pending process, zma -m 4]
11/26/16 04:44:55.034587 zmdc[72239].INF ['zma -m 6' started at 16/11/26 04:44:55]
11/26/16 04:44:55.036329 zmdc[24965].INF ['zma -m 4' starting at 16/11/26 04:44:55, pid = 72241]
11/26/16 04:44:55.036458 zmdc[72241].INF ['zma -m 4' started at 16/11/26 04:44:55]
11/26/16 04:44:55.036560 zmdc[24965].INF [Starting pending process, zma -m 9]
11/26/16 04:44:55.038510 zmdc[24965].INF ['zma -m 9' starting at 16/11/26 04:44:55, pid = 72243]
11/26/16 04:44:55.038514 zmdc[72243].INF ['zma -m 9' started at 16/11/26 04:44:55]
11/26/16 04:44:55.038752 zmdc[24965].INF [Starting pending process, zma -m 10]
11/26/16 04:44:55.040744 zmdc[24965].INF ['zma -m 10' starting at 16/11/26 04:44:55, pid = 72245]
11/26/16 04:44:55.040850 zmdc[72245].INF ['zma -m 10' started at 16/11/26 04:44:55]
11/26/16 04:44:55.041003 zmdc[24965].INF [Starting pending process, zma -m 1]
11/26/16 04:44:55.044175 zmdc[24965].INF ['zma -m 1' starting at 16/11/26 04:44:55, pid = 72247]
11/26/16 04:44:55.044451 zmdc[24965].INF [Starting pending process, zma -m 5]
11/26/16 04:44:55.047673 zmdc[24965].INF ['zma -m 5' starting at 16/11/26 04:44:55, pid = 72250]
11/26/16 04:44:55.048051 zmdc[72250].INF ['zma -m 5' started at 16/11/26 04:44:55]
11/26/16 04:44:55.050125 zmdc[72247].INF ['zma -m 1' started at 16/11/26 04:44:55]
11/26/16 04:44:55.364314 zmdc[24965].INF ['zmc -m 12' exited normally]
11/26/16 04:44:55.434641 zmdc[24965].INF ['zmc -m 5' exited normally]
11/26/16 04:44:55.661033 zmdc[24965].ERR ['zma -m 12' exited abnormally, exit status 255]
11/26/16 04:44:55.683702 zmdc[24965].ERR ['zma -m 5' exited abnormally, exit status 255]
11/26/16 04:44:56.807401 zmdc[24965].INF ['zmc -m 3' exited normally]
11/26/16 04:44:57.900191 zmdc[24965].INF ['zmc -m 7' exited normally]
11/26/16 04:44:58.629648 zmdc[24965].INF ['zmc -m 8' exited normally]
11/26/16 04:44:59.215459 zmdc[24965].INF ['zmc -m 6' exited normally]
11/26/16 04:44:59.444079 zmdc[24965].INF ['zmc -m 1' exited normally]
11/26/16 04:45:00.064022 zmdc[24965].INF ['zma -m 4' crashed, signal 8]
11/26/16 04:45:00.064425 zmdc[24965].INF ['zma -m 10' crashed, signal 8]
11/26/16 04:45:00.064708 zmdc[24965].INF ['zma -m 7' crashed, signal 8]
11/26/16 04:45:00.064828 zmdc[24965].INF ['zmc -m 2' crashed, signal 8]
11/26/16 04:45:00.064962 zmdc[24965].INF ['zmc -m 4' crashed, signal 8]
11/26/16 04:45:00.065099 zmdc[24965].INF ['zma -m 1' crashed, signal 8]
11/26/16 04:45:00.065210 zmdc[24965].INF ['zmc -m 9' crashed, signal 8]
11/26/16 04:45:00.065390 zmdc[24965].INF ['zma -m 6' crashed, signal 8]
11/26/16 04:45:00.065904 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:45:00.066293 zmdc[24965].INF ['zma -m 9' crashed, signal 8]
11/26/16 04:45:00.067794 zmdc[24965].INF ['zmc -m 11' crashed, signal 8]
11/26/16 04:45:00.108450 zmdc[24965].INF ['zma -m 3' exited normally]
11/26/16 04:45:05.079342 zmdc[24965].INF [Starting pending process, zma -m 3]
11/26/16 04:45:05.081127 zmdc[72283].INF ['zma -m 3' started at 16/11/26 04:45:05]
11/26/16 04:45:05.081185 zmdc[24965].INF ['zma -m 3' starting at 16/11/26 04:45:05, pid = 72283]
11/26/16 04:45:05.081436 zmdc[24965].INF [Starting pending process, zmc -m 5]
11/26/16 04:45:05.082880 zmdc[24965].INF ['zmc -m 5' starting at 16/11/26 04:45:05, pid = 72284]
11/26/16 04:45:05.083053 zmdc[24965].INF [Starting pending process, zmc -m 12]
11/26/16 04:45:05.083010 zmdc[72284].INF ['zmc -m 5' started at 16/11/26 04:45:05]
11/26/16 04:45:05.084759 zmdc[24965].INF ['zmc -m 12' starting at 16/11/26 04:45:05, pid = 72285]
11/26/16 04:45:05.084889 zmdc[72285].INF ['zmc -m 12' started at 16/11/26 04:45:05]
11/26/16 04:45:05.084996 zmdc[24965].INF [Starting pending process, zma -m 2]
11/26/16 04:45:05.086777 zmdc[24965].INF ['zma -m 2' starting at 16/11/26 04:45:05, pid = 72286]
11/26/16 04:45:05.086903 zmdc[72286].INF ['zma -m 2' started at 16/11/26 04:45:05]
11/26/16 04:45:05.704639 zmdc[24965].ERR ['zma -m 3' exited abnormally, exit status 255]
11/26/16 04:45:06.014809 zmdc[24965].INF [Starting pending process, zmc -m 3]
11/26/16 04:45:06.016543 zmdc[24965].INF ['zmc -m 3' starting at 16/11/26 04:45:06, pid = 72287]
11/26/16 04:45:06.016573 zmdc[72287].INF ['zmc -m 3' started at 16/11/26 04:45:06]
11/26/16 04:45:06.970275 zmdc[24965].ERR ['zmwatch.pl' exited abnormally, exit status 255]
11/26/16 04:45:07.071898 zmdc[24965].INF [Starting pending process, zmc -m 7]
I suspect ZM enteres this strange mode here. zmwatch.pl starts to report exit status 255 right after that.

syslog

Code: Select all

Nov 26 04:34:33 zm zmdc[24965]: WAR ['zmc -m 10' has not stopped at 16/11/26 04:34:33. Sending KILL to pid 71491]
Nov 26 04:39:17 zm zma_m10[45484]: WAR [Signal: Lost]
Nov 26 04:41:24 zm zma_m10[45484]: WAR [Signal: Reacquired]
Nov 26 04:44:24 zm zmdc[24965]: WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:24. Sending KILL to pid 71969]
Nov 26 04:44:34 zm zma_m5[25078]: WAR [Event 119408 has zero time, setting to current]
Nov 26 04:44:34 zm zma_m5[25078]: WAR [Signal: Lost]
Nov 26 04:44:34 zm zmdc[24965]: WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:34. Sending KILL to pid 72141]
Nov 26 04:44:34 zm zmwatch[25149]: ERR [Error getting last capture time for  cam11]
Nov 26 04:44:34 zm zma_m1[11038]: WAR [Event 119409 has zero time, setting to current]
Nov 26 04:44:35 zm zmdc[24965]: WAR ['zma -m 11' has not stopped at 16/11/26 04:44:35. Sending KILL to pid 72150]
Nov 26 04:44:35 zm zma_m8[72148]: ERR [Shared data not initialised by capture daemon for monitor cam8]
Nov 26 04:44:35 zm zmdc[24965]: ERR ['zma -m 8' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m12[72155]: ERR [Shared data not initialised by capture daemon for monitor cam12]
Nov 26 04:44:36 zm zma_m6[72159]: ERR [Shared data not initialised by capture daemon for monitor cam6]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 6' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 12' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m4[72163]: ERR [Shared data not initialised by capture daemon for monitor cam4]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 4' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:36 zm zma_m9[72164]: ERR [Shared data not initialised by capture daemon for monitor cam9]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 9' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:36 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:37 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:37 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:37 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:38 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:38 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:38 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:39 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:39 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:39 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m8[72180]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m11[72179]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m12[72192]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:42 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:42 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:42 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:43 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:43 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:43 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:44 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:44 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:44 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:45 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:45 zm zma_m5[72202]: ERR [Shared data not initialised by capture daemon for monitor cam5]
Nov 26 04:44:45 zm zmdc[24965]: ERR ['zma -m 5' exited abnormally, exit status 255]
Nov 26 04:44:45 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:45 zm zma_m7[72199]: ERR [Shared data not initialised by capture daemon for monitor cam7]
Nov 26 04:44:45 zm zmdc[24965]: ERR ['zma -m 7' exited abnormally, exit status 255]
Nov 26 04:44:45 zm zma_m1[72201]: ERR [Shared data not initialised by capture daemon for monitor cam1]
Nov 26 04:44:45 zm zmdc[24965]: ERR ['zma -m 1' exited abnormally, exit status 255]
Nov 26 04:44:49 zm zmdc[72149]: WAR ['zma -m 2' has not stopped at 16/11/26 04:44:49. Sending KILL to pid 72145]
Nov 26 04:44:50 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:50 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:51 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:51 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:52 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:52 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:53 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:53 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:54 zm zma_m2[72232]: WAR [Waiting for capture daemon]
Nov 26 04:44:54 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:54 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m12[72235]: ERR [Shared data not initialised by capture daemon for monitor cam12]
Nov 26 04:44:55 zm zmdc[24965]: ERR ['zma -m 12' exited abnormally, exit status 255]
Nov 26 04:44:55 zm zma_m5[72250]: ERR [Shared data not initialised by capture daemon for monitor cam5]
Nov 26 04:44:55 zm zmdc[24965]: ERR ['zma -m 5' exited abnormally, exit status 255]
Nov 26 04:44:55 zm zma_m7[72237]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m7[72237]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m7[72237]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:59 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:59 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:59 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:45:05 zm zma_m3[72283]: ERR [Shared data not initialised by capture daemon for monitor cam3]
Nov 26 04:45:05 zm zmdc[24965]: ERR ['zma -m 3' exited abnormally, exit status 255]
Nov 26 04:45:05 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:06 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:06 zm zmwatch[72156]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 1", output is "Unable to connect to server"]
Nov 26 04:45:06 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 04:45:07 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:14 zm last message repeated 7 times
Nov 26 04:45:15 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:15 zm zma_m8[72309]: WAR [Waiting for capture daemon]
Nov 26 04:45:15 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:16 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:16 zm zma_m8[72309]: WAR [Waiting for capture daemon]
Nov 26 04:45:16 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:17 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:17 zm zma_m8[72309]: WAR [Waiting for capture daemon]
Nov 26 04:45:17 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:18 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:18 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:19 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:26 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:26 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:26 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:27 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:27 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:27 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:28 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:28 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:28 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:29 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:29 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:29 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:30 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:30 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:30 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:31 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:31 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:32 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:32 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:33 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:33 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:46:04 zm last message repeated 31 times
Nov 26 04:47:37 zm last message repeated 92 times
Nov 26 04:47:51 zm zmwatch[72301]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 04:47:51 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 04:53:21 zm zmwatch[72385]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 04:53:21 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 04:58:41 zm zmwatch[72541]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 04:58:41 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 05:01:31 zm zmwatch[72727]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 05:01:31 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 05:03:41 zm zmwatch[72890]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 05:03:41 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 05:07:31 zm zmwatch[73011]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Monitor #10 is on wifi, so it disconnects from time to time, but I don't think it's triggering the issue.
Any ideas are appreciated. :)
abi
Posts: 61
Joined: Fri Oct 23, 2015 11:25 am

Re: [H264] For time to time I find ZM partially stopped

Post by abi »

BTW, attempt to restart ZM from this state means disaster - when zmdc.pl starts killing old processes, they are immediately restarted. I suspect killing code is unreliable, see ps output

Code: Select all

www   24965  0.0  0.1   84976  17104  0- SJ   Wed22    1:47.17 /usr/local/bin/perl -wT /usr/local/bin/zmdc.pl startup

Code: Select all

my @daemons = (
    'zmc',
    'zma',
    'zmf',
    'zmfilter.pl',
    'zmaudit.pl',
    'zmtrigger.pl',
    'zmx10.pl',
    'zmwatch.pl',
    'zmupdate.pl',
    'zmtrack.pl',
    'zmtelemetry.pl'
);
perl scripts are not killed by sub KillAll

Code: Select all

root@zm:~ # killall zmdc.pl
No matching processes were found
I think, ps should be used to determine pid first.
Locked