[H264] For time to time I find ZM partially stopped
Posted: Sat Nov 26, 2016 12:43 pm
I made this topic before, however I decided to delete it as I thought I can figure out what is happening by myself. However,now I totally ran out of ideas.
So, the problem is that sometime (say, 2 times per week) I found ZM in rather strange state.
It runs, however for zmdc.pl, no sock file or pid exists. It leaves most of stuff running, but for webconsole the status is 'stopped' and I have no ability to view live video. From the other side, api works and instance can easily be accessed from ZMNinja.
I found no server side problems or logs, like OOM or similar activity from the server itself to stop ZM. Also, with debug mode for zmdc.pl I found no registered event to stop ZM as well. From the code, it looks like that deletion of pid file logs.
However, logs show that when ZM enters this state most all processes are being restarted. (by watchdog?). And, actually all of them are restarted. Disappearance of pid worries me. I disabled rc.d script, to exclude possible clash b/w ZM itself and rc.d, so it looks like supernatural event to me.
ps output shows normal running instance. You can expect the same when ZM runs normally.
Some stalle files (I suspect they are not connected to the issue) and absence of zmdc sock file. But stalled files older than event date mean the folder was not emptied by someone.
I suspect ZM enteres this strange mode here. zmwatch.pl starts to report exit status 255 right after that.
syslog
Monitor #10 is on wifi, so it disconnects from time to time, but I don't think it's triggering the issue.
Any ideas are appreciated.
So, the problem is that sometime (say, 2 times per week) I found ZM in rather strange state.
It runs, however for zmdc.pl, no sock file or pid exists. It leaves most of stuff running, but for webconsole the status is 'stopped' and I have no ability to view live video. From the other side, api works and instance can easily be accessed from ZMNinja.
I found no server side problems or logs, like OOM or similar activity from the server itself to stop ZM. Also, with debug mode for zmdc.pl I found no registered event to stop ZM as well. From the code, it looks like that deletion of pid file logs.
However, logs show that when ZM enters this state most all processes are being restarted. (by watchdog?). And, actually all of them are restarted. Disappearance of pid worries me. I disabled rc.d script, to exclude possible clash b/w ZM itself and rc.d, so it looks like supernatural event to me.
Code: Select all
root@zm:~ # ps -aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
www 72316 4.5 1.3 522036 330148 - SJ 04:45 29:20.88 /usr/local/bin/zma -m 1
www 72308 4.4 1.3 513844 324008 - SJ 04:45 30:00.99 /usr/local/bin/zma -m 3
www 72294 4.2 1.6 652224 407356 - SJ 04:45 42:54.21 /usr/local/bin/zmc -m 6
www 98461 4.0 1.1 483132 287888 - SJ 15:29 0:08.63 /usr/local/www/zoneminder-h264/cgi-bin/nph-zms (zms)
www 72292 3.5 1.6 666560 412048 - SJ 04:45 42:03.24 /usr/local/bin/zmc -m 7
www 72287 2.8 1.6 654272 409820 - SJ 04:45 33:25.38 /usr/local/bin/zmc -m 3
www 72295 2.7 1.7 664512 417852 - SJ 04:45 28:35.27 /usr/local/bin/zmc -m 1
www 72306 2.0 0.6 338340 148484 - SJ 04:45 14:23.74 /usr/local/bin/zma -m 12
www 72309 2.0 0.6 334244 145656 - SJ 04:45 13:39.69 /usr/local/bin/zma -m 8
www 72286 1.8 0.6 330148 143040 - SJ 04:45 13:09.51 /usr/local/bin/zma -m 2
www 72313 1.8 0.6 330148 143048 - SJ 04:45 13:14.38 /usr/local/bin/zma -m 4
www 72310 1.6 0.6 326052 140436 - SJ 04:45 12:56.05 /usr/local/bin/zma -m 5
www 72326 1.4 1.7 664512 420848 - SJ 04:45 17:56.85 /usr/local/bin/zmc -m 10
www 72314 1.1 1.3 505652 317864 - SJ 04:45 12:38.19 /usr/local/bin/zma -m 9
www 97774 0.8 0.1 64916 13224 - SJ 15:16 0:07.89 php-fpm: pool www (php-fpm)
www 72297 0.7 0.7 394800 182784 - SJ 04:45 12:47.91 /usr/local/bin/zmc -m 4
www 72285 0.6 0.7 402992 183672 - SJ 04:45 23:14.62 /usr/local/bin/zmc -m 12
www 72299 0.6 0.7 394800 182824 - SJ 04:45 16:19.95 /usr/local/bin/zmc -m 2
www 72284 0.5 0.7 398896 178720 - SJ 04:45 20:44.79 /usr/local/bin/zmc -m 5
www 72298 0.5 1.7 658368 420232 - SJ 04:45 24:22.86 /usr/local/bin/zmc -m 11
www 72293 0.4 0.7 398896 186688 - SJ 04:45 15:02.46 /usr/local/bin/zmc -m 8
www 72296 0.3 1.7 678848 434904 - SJ 04:45 19:22.49 /usr/local/bin/zmc -m 9
root 8057 0.0 0.0 10464 1508 - SsJ Wed22 0:03.16 /usr/sbin/syslogd -c -ss
mysql 8113 0.0 0.0 13144 0 - IWsJ - 0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --pid-file=/var/db/mysql/zm.dacha.abinet.ru.
mysql 9425 0.0 1.7 2042436 425136 - SJ Wed22 10:41.60 /usr/local/libexec/mysqld --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --plugin-dir=/usr/local/lib/mysql/plugin --skip-perfo
root 9435 0.0 0.0 58772 2764 - SsJ Wed22 0:03.50 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
root 9440 0.0 0.0 20260 0 - IWsJ - 0:00.00 nginx: master process /usr/local/sbin/nginx
www 9441 0.0 0.0 20260 2124 - SJ Wed22 0:18.50 nginx: worker process (nginx)
www 9449 0.0 0.0 10428 0 - IWsJ - 0:00.00 daemon: /usr/local/sbin/fcgiwrap[9450] (daemon)
www 9450 0.0 0.0 10408 0 - IWJ - 0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9453 0.0 0.0 10408 812 - IJ Wed22 0:01.13 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9454 0.0 0.0 10408 812 - IJ Wed22 0:00.68 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9455 0.0 0.0 10408 812 - IJ Wed22 0:01.19 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9456 0.0 0.0 10408 812 - IJ Wed22 0:01.02 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9457 0.0 0.0 10408 812 - IJ Wed22 0:00.91 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9458 0.0 0.0 10408 812 - IJ Wed22 0:00.65 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9459 0.0 0.0 10408 812 - IJ Wed22 0:00.67 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9460 0.0 0.0 10408 812 - IJ Wed22 0:01.63 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9461 0.0 0.0 10408 812 - IJ Wed22 0:00.89 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9462 0.0 0.0 10408 836 - IJ Wed22 0:00.67 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9463 0.0 0.0 10408 836 - IJ Wed22 0:00.44 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9464 0.0 0.0 10408 836 - IJ Wed22 0:01.61 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9465 0.0 0.0 10408 824 - IJ Wed22 0:00.83 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9466 0.0 0.0 10408 812 - SJ Wed22 0:00.95 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9467 0.0 0.0 10408 0 - IWJ - 0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9468 0.0 0.0 10408 0 - IWJ - 0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9469 0.0 0.0 10408 0 - IWJ - 0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
www 9470 0.0 0.0 10408 0 - IWJ - 0:00.00 /usr/local/sbin/fcgiwrap -s unix:/var/run/fcgiwrap/fcgiwrap.sock -c 18
root 9492 0.0 0.0 55676 3408 - IsJ Wed22 0:00.00 /usr/sbin/sshd
root 9496 0.0 0.0 12564 532 - SsJ Wed22 0:00.27 /usr/sbin/cron -J 15 -s
www 72161 0.0 0.1 111652 37612 - IJ 04:44 0:01.14 /usr/local/bin/perl -wT /usr/local/bin/zmfilter.pl
www 72196 0.0 0.1 99324 29272 - IJ 04:44 0:22.63 /usr/local/bin/perl -wT /usr/local/bin/zmaudit.pl -c
www 72307 0.0 0.1 479028 28580 - SJ 04:45 0:10.29 /usr/local/bin/zma -m 11
www 72311 0.0 0.9 479028 222824 - SJ 04:45 0:10.94 /usr/local/bin/zma -m 7
www 72312 0.0 0.9 479028 218236 - SJ 04:45 0:10.99 /usr/local/bin/zma -m 6
www 72315 0.0 0.2 479028 53924 - SJ 04:45 0:10.26 /usr/local/bin/zma -m 10
www 78341 0.0 0.1 84976 16784 - SJ 07:17 0:06.11 /usr/local/bin/perl -wT /usr/local/bin/zmwatch.pl
root 97554 0.0 0.0 43732 2300 2 IJ 15:13 0:00.01 login [pam] (login)
root 97555 0.0 0.0 19600 3104 2 SJ 15:13 0:00.01 -csh (csh)
root 98658 0.0 0.0 21164 2120 2 R+J 15:33 0:00.00 ps -aux
www 24965 0.0 0.1 84976 17104 0- SJ Wed22 1:47.17 /usr/local/bin/perl -wT /usr/local/bin/zmdc.pl startup
Code: Select all
root@zm:~ # ls -la /var/run/zm
total 44
drwxr-xr-x 2 www www 42 Nov 26 15:29 .
drwxr-xr-x 7 root wheel 20 Nov 26 04:15 ..
-rw------- 1 www www 0 Nov 24 15:58 zms-034453.lock
srwxr-xr-x 1 www www 0 Nov 24 15:58 zms-034453s.sock
-rw------- 1 www www 0 Nov 24 17:50 zms-147744.lock
srwxr-xr-x 1 www www 0 Nov 24 17:50 zms-147744s.sock
-rw------- 1 www www 0 Nov 24 11:13 zms-186381.lock
srwxr-xr-x 1 www www 0 Nov 24 11:13 zms-186381s.sock
-rw------- 1 www www 0 Nov 24 15:59 zms-263888.lock
srwxr-xr-x 1 www www 0 Nov 24 15:59 zms-263888s.sock
-rw------- 1 www www 0 Nov 24 13:46 zms-320506.lock
srwxr-xr-x 1 www www 0 Nov 24 13:46 zms-320506s.sock
-rw------- 1 www www 0 Nov 24 15:57 zms-344291.lock
srwxr-xr-x 1 www www 0 Nov 24 15:57 zms-344291s.sock
-rw------- 1 www www 0 Nov 24 15:57 zms-381352.lock
srwxr-xr-x 1 www www 0 Nov 24 15:57 zms-381352s.sock
-rw------- 1 www www 0 Nov 24 11:15 zms-661443.lock
srwxr-xr-x 1 www www 0 Nov 24 11:15 zms-661443s.sock
-rw------- 1 www www 0 Nov 25 20:11 zms-728404.lock
srwxr-xr-x 1 www www 0 Nov 25 20:11 zms-728404s.sock
-rw------- 1 www www 0 Nov 24 21:16 zms-741737.lock
srwxr-xr-x 1 www www 0 Nov 24 21:16 zms-741737s.sock
-rw------- 1 www www 0 Nov 24 21:17 zms-770522.lock
srwxr-xr-x 1 www www 0 Nov 24 21:17 zms-770522s.sock
-rw------- 1 www www 0 Nov 24 13:46 zms-832133.lock
srwxr-xr-x 1 www www 0 Nov 24 13:46 zms-832133s.sock
-rw------- 1 www www 0 Nov 24 17:45 zms-891026.lock
srwxr-xr-x 1 www www 0 Nov 24 17:45 zms-891026s.sock
-rw------- 1 www www 0 Nov 24 21:18 zms-891784.lock
srwxr-xr-x 1 www www 0 Nov 24 21:18 zms-891784s.sock
-rw------- 1 www www 0 Nov 24 17:50 zms-925406.lock
srwxr-xr-x 1 www www 0 Nov 24 17:50 zms-925406s.sock
-rw------- 1 www www 0 Nov 26 15:29 zms-937205.lock
srwxr-xr-x 1 www www 0 Nov 26 15:29 zms-937205s.sock
-rw------- 1 www www 0 Nov 24 15:58 zms-969847.lock
srwxr-xr-x 1 www www 0 Nov 24 15:58 zms-969847s.sock
-rw------- 1 www www 0 Nov 24 11:14 zms-976765.lock
srwxr-xr-x 1 www www 0 Nov 24 11:14 zms-976765s.sock
-rw------- 1 www www 0 Nov 24 11:14 zms-981259.lock
srwxr-xr-x 1 www www 0 Nov 24 11:14 zms-981259s.sock
-rw------- 1 www www 0 Nov 24 15:57 zms-996497.lock
srwxr-xr-x 1 www www 0 Nov 24 15:57 zms-996497s.sock
Code: Select all
11/26/16 04:23:42.071851 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:23:42.073617 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:23:42, pid = 71491]
11/26/16 04:23:42.073689 zmdc[71491].INF ['zmc -m 10' started at 16/11/26 04:23:42]
11/26/16 04:34:27.768526 zmdc[24965].INF ['zmc -m 10' sending stop to pid 71491 at 16/11/26 04:34:27]
11/26/16 04:34:33.806223 zmdc[24965].WAR ['zmc -m 10' has not stopped at 16/11/26 04:34:33. Sending KILL to pid 71491]
11/26/16 04:34:33.818137 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:34:33.919177 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:34:33.921147 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:34:33, pid = 71834]
11/26/16 04:34:33.921149 zmdc[71834].INF ['zmc -m 10' started at 16/11/26 04:34:33]
11/26/16 04:39:14.151747 zmdc[24965].INF ['zmc -m 10' sending stop to pid 71834 at 16/11/26 04:39:14]
11/26/16 04:39:17.842886 zmdc[24965].INF ['zmc -m 10' exited normally]
11/26/16 04:39:22.019604 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:39:22.021440 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:39:22, pid = 71969]
11/26/16 04:39:22.021451 zmdc[71969].INF ['zmc -m 10' started at 16/11/26 04:39:22]
11/26/16 04:44:18.168526 zmdc[24965].INF ['zmc -m 10' sending stop to pid 71969 at 16/11/26 04:44:18]
11/26/16 04:44:24.184824 zmdc[24965].WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:24. Sending KILL to pid 71969]
11/26/16 04:44:24.196526 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:44:34.061568 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:44:34.063398 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:44:34, pid = 72141]
11/26/16 04:44:34.063415 zmdc[72141].INF ['zmc -m 10' started at 16/11/26 04:44:34]
11/26/16 04:44:34.323494 zmdc[24965].INF ['zmc -m 10' sending stop to pid 72141 at 16/11/26 04:44:34]
11/26/16 04:44:34.425456 zmdc[24965].INF ['zmc -m 5' exited normally]
11/26/16 04:44:34.452650 zmdc[24965].INF ['zma -m 11' exited normally]
11/26/16 04:44:34.503089 zmdc[24965].INF ['zma -m 3' exited normally]
11/26/16 04:44:34.525917 zmdc[24965].INF ['zma -m 8' exited normally]
11/26/16 04:44:34.591736 zmdc[24965].INF ['zmc -m 3' exited normally]
11/26/16 04:44:34.591910 zmdc[24965].INF ['zma -m 2' exited normally]
11/26/16 04:44:34.606082 zmdc[24965].INF ['zmaudit.pl -c' exited, signal 14]
11/26/16 04:44:34.606200 zmdc[24965].WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:34. Sending KILL to pid 72141]
11/26/16 04:44:34.607035 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:44:34.639984 zmdc[24965].INF ['zmc -m 8' exited normally]
11/26/16 04:44:34.640155 zmdc[24965].INF [Starting pending process, zma -m 2]
11/26/16 04:44:34.641750 zmdc[24965].INF ['zma -m 2' starting at 16/11/26 04:44:34, pid = 72145]
11/26/16 04:44:34.641754 zmdc[72145].INF ['zma -m 2' started at 16/11/26 04:44:34]
11/26/16 04:44:34.641917 zmdc[24965].INF [Starting pending process, zmaudit.pl -c]
11/26/16 04:44:34.643239 zmdc[24965].INF ['zmaudit.pl -c' starting at 16/11/26 04:44:34, pid = 72146]
11/26/16 04:44:34.643407 zmdc[24965].INF [Starting pending process, zmc -m 5]
11/26/16 04:44:34.643394 zmdc[72146].INF ['zmaudit.pl -c' started at 16/11/26 04:44:34]
11/26/16 04:44:34.645506 zmdc[72147].INF ['zmc -m 5' started at 16/11/26 04:44:34]
11/26/16 04:44:34.648921 zmdc[24965].INF ['zmc -m 5' starting at 16/11/26 04:44:34, pid = 72147]
11/26/16 04:44:34.649255 zmdc[24965].INF [Starting pending process, zma -m 8]
11/26/16 04:44:34.651519 zmdc[24965].INF ['zma -m 8' starting at 16/11/26 04:44:34, pid = 72148]
11/26/16 04:44:34.651795 zmdc[24965].INF [Starting pending process, zma -m 3]
11/26/16 04:44:34.653824 zmdc[72148].INF ['zma -m 8' started at 16/11/26 04:44:34]
11/26/16 04:44:34.654123 zmdc[24965].INF ['zma -m 3' starting at 16/11/26 04:44:34, pid = 72149]
11/26/16 04:44:34.654390 zmdc[24965].INF [Starting pending process, zma -m 11]
11/26/16 04:44:34.655363 zmdc[72149].INF ['zma -m 3' started at 16/11/26 04:44:34]
11/26/16 04:44:34.670264 zmdc[24965].INF ['zma -m 11' starting at 16/11/26 04:44:34, pid = 72150]
11/26/16 04:44:34.670566 zmdc[24965].INF [Starting pending process, zmc -m 3]
11/26/16 04:44:34.672604 zmdc[72150].INF ['zma -m 11' started at 16/11/26 04:44:34]
11/26/16 04:44:34.673627 zmdc[72151].INF ['zmc -m 3' started at 16/11/26 04:44:34]
11/26/16 04:44:34.701424 zmdc[24965].INF ['zmc -m 3' starting at 16/11/26 04:44:34, pid = 72151]
11/26/16 04:44:34.701728 zmdc[24965].INF [Starting pending process, zmc -m 8]
11/26/16 04:44:34.721405 zmdc[72152].INF ['zmc -m 8' started at 16/11/26 04:44:34]
11/26/16 04:44:34.744183 zmdc[24965].INF ['zmc -m 8' starting at 16/11/26 04:44:34, pid = 72152]
11/26/16 04:44:34.795404 zmdc[24965].INF ['zmc -m 1' exited normally]
11/26/16 04:44:34.795688 zmdc[24965].INF [Starting pending process, zmc -m 1]
11/26/16 04:44:34.798172 zmdc[72154].INF ['zmc -m 1' started at 16/11/26 04:44:34]
11/26/16 04:44:34.799116 zmdc[24965].INF ['zmc -m 1' starting at 16/11/26 04:44:34, pid = 72154]
11/26/16 04:44:34.802345 zmdc[24965].INF ['zma -m 11' sending stop to pid 72150 at 16/11/26 04:44:34]
11/26/16 04:44:34.915106 zmdc[24965].INF ['zmwatch.pl' exited, signal 14]
11/26/16 04:44:34.967582 zmdc[24965].INF ['zmc -m 9' exited normally]
11/26/16 04:44:35.061434 zmdc[24965].INF ['zma -m 6' exited normally]
11/26/16 04:44:35.169509 zmdc[24965].INF ['zmc -m 7' exited normally]
11/26/16 04:44:35.186856 zmdc[24965].INF ['zma -m 5' exited normally]
11/26/16 04:44:35.188310 zmdc[24965].INF ['zma -m 12' exited normally]
11/26/16 04:44:35.188568 zmdc[24965].WAR ['zma -m 11' has not stopped at 16/11/26 04:44:35. Sending KILL to pid 72150]
11/26/16 04:44:35.189506 zmdc[24965].INF ['zma -m 11' crashed, signal 8]
11/26/16 04:44:35.191605 zmdc[24965].INF ['zma -m 1' exited normally]
11/26/16 04:44:35.218225 zmdc[24965].INF ['zma -m 7' exited normally]
11/26/16 04:44:35.218453 zmdc[24965].INF [Starting pending process, zma -m 12]
11/26/16 04:44:35.221114 zmdc[24965].INF ['zma -m 12' starting at 16/11/26 04:44:35, pid = 72155]
11/26/16 04:44:35.221351 zmdc[24965].INF [Starting pending process, zmwatch.pl]
11/26/16 04:44:35.221808 zmdc[72155].INF ['zma -m 12' started at 16/11/26 04:44:35]
11/26/16 04:44:35.223589 zmdc[24965].INF ['zmwatch.pl' starting at 16/11/26 04:44:35, pid = 72156]
11/26/16 04:44:35.223925 zmdc[72156].INF ['zmwatch.pl' started at 16/11/26 04:44:35]
11/26/16 04:44:35.224160 zmdc[24965].INF [Starting pending process, zmc -m 7]
11/26/16 04:44:35.232935 zmdc[72157].INF ['zmc -m 7' started at 16/11/26 04:44:35]
11/26/16 04:44:35.237318 zmdc[24965].INF ['zmc -m 7' starting at 16/11/26 04:44:35, pid = 72157]
11/26/16 04:44:35.247146 zmdc[24965].INF [Starting pending process, zma -m 7]
11/26/16 04:44:35.263310 zmdc[24965].INF ['zma -m 7' starting at 16/11/26 04:44:35, pid = 72158]
11/26/16 04:44:35.269104 zmdc[24965].INF [Starting pending process, zma -m 6]
11/26/16 04:44:35.287334 zmdc[72159].INF ['zma -m 6' started at 16/11/26 04:44:35]
11/26/16 04:44:35.298115 zmdc[72158].INF ['zma -m 7' started at 16/11/26 04:44:35]
11/26/16 04:44:35.340504 zmdc[24965].INF ['zma -m 6' starting at 16/11/26 04:44:35, pid = 72159]
11/26/16 04:44:35.375050 zmdc[24965].INF ['zmc -m 6' exited normally]
11/26/16 04:44:35.413586 zmdc[24965].INF ['zma -m 10' exited normally]
11/26/16 04:44:35.439142 zmdc[24965].INF ['zmc -m 2' exited normally]
11/26/16 04:44:35.572021 zmdc[24965].ERR ['zma -m 8' exited abnormally, exit status 255]
11/26/16 04:44:35.572343 zmdc[24965].INF ['zma -m 4' exited normally]
11/26/16 04:44:35.572476 zmdc[24965].INF ['zmfilter.pl' exited, signal 14]
11/26/16 04:44:35.572609 zmdc[24965].INF ['zma -m 9' exited normally]
11/26/16 04:44:35.572773 zmdc[24965].INF [Starting pending process, zmfilter.pl]
11/26/16 04:44:35.576444 zmdc[24965].INF ['zmfilter.pl' starting at 16/11/26 04:44:35, pid = 72161]
11/26/16 04:44:35.576773 zmdc[24965].INF [Starting pending process, zmc -m 9]
11/26/16 04:44:35.601208 zmdc[24965].INF ['zmc -m 9' starting at 16/11/26 04:44:35, pid = 72162]
11/26/16 04:44:35.601560 zmdc[24965].INF [Starting pending process, zma -m 4]
11/26/16 04:44:35.605847 zmdc[24965].INF ['zma -m 4' starting at 16/11/26 04:44:35, pid = 72163]
11/26/16 04:44:35.606188 zmdc[24965].INF [Starting pending process, zma -m 9]
11/26/16 04:44:35.608953 zmdc[24965].INF ['zma -m 9' starting at 16/11/26 04:44:35, pid = 72164]
11/26/16 04:44:35.609267 zmdc[24965].INF [Starting pending process, zma -m 10]
11/26/16 04:44:35.610789 zmdc[72161].INF ['zmfilter.pl' started at 16/11/26 04:44:35]
11/26/16 04:44:35.618376 zmdc[72162].INF ['zmc -m 9' started at 16/11/26 04:44:35]
11/26/16 04:44:35.642563 zmdc[72165].INF ['zma -m 10' started at 16/11/26 04:44:35]
11/26/16 04:44:35.642480 zmdc[24965].INF ['zma -m 10' starting at 16/11/26 04:44:35, pid = 72165]
11/26/16 04:44:35.643165 zmdc[24965].INF [Starting pending process, zma -m 1]
11/26/16 04:44:35.606945 zmdc[72163].INF ['zma -m 4' started at 16/11/26 04:44:35]
11/26/16 04:44:35.658254 zmdc[72164].INF ['zma -m 9' started at 16/11/26 04:44:35]
11/26/16 04:44:35.693056 zmdc[72166].INF ['zma -m 1' started at 16/11/26 04:44:35]
11/26/16 04:44:35.705370 zmdc[24965].INF ['zma -m 1' starting at 16/11/26 04:44:35, pid = 72166]
11/26/16 04:44:35.706075 zmdc[24965].INF ['zmc -m 4' exited normally]
11/26/16 04:44:35.706361 zmdc[24965].INF ['zmc -m 11' exited normally]
11/26/16 04:44:35.706520 zmdc[24965].INF [Starting pending process, zmc -m 11]
11/26/16 04:44:35.718976 zmdc[72167].INF ['zmc -m 11' started at 16/11/26 04:44:35]
11/26/16 04:44:35.761429 zmdc[24965].INF ['zmc -m 11' starting at 16/11/26 04:44:35, pid = 72167]
11/26/16 04:44:35.761706 zmdc[24965].INF ['zmc -m 12' exited normally]
11/26/16 04:44:35.761860 zmdc[24965].INF [Starting pending process, zmc -m 2]
11/26/16 04:44:35.764254 zmdc[24965].INF ['zmc -m 2' starting at 16/11/26 04:44:35, pid = 72168]
11/26/16 04:44:35.764501 zmdc[24965].INF [Starting pending process, zma -m 5]
11/26/16 04:44:35.766803 zmdc[24965].INF ['zma -m 5' starting at 16/11/26 04:44:35, pid = 72169]
11/26/16 04:44:35.777354 zmdc[72169].INF ['zma -m 5' started at 16/11/26 04:44:35]
11/26/16 04:44:35.817024 zmdc[72168].INF ['zmc -m 2' started at 16/11/26 04:44:35]
11/26/16 04:44:35.869099 zmdc[24965].INF [Starting pending process, zmc -m 6]
11/26/16 04:44:36.067858 zmdc[24965].INF ['zmc -m 6' starting at 16/11/26 04:44:36, pid = 72170]
11/26/16 04:44:36.069954 zmdc[24965].INF [Starting pending process, zmc -m 4]
11/26/16 04:44:36.059442 zmdc[72170].INF ['zmc -m 6' started at 16/11/26 04:44:36]
11/26/16 04:44:36.111350 zmdc[24965].INF ['zmc -m 4' starting at 16/11/26 04:44:36, pid = 72171]
11/26/16 04:44:36.130271 zmdc[24965].INF [Starting pending process, zmc -m 12]
11/26/16 04:44:36.134701 zmdc[72173].INF ['zmc -m 12' started at 16/11/26 04:44:36]
11/26/16 04:44:36.250270 zmdc[24965].INF ['zmc -m 12' starting at 16/11/26 04:44:36, pid = 72173]
11/26/16 04:44:36.295638 zmdc[72171].INF ['zmc -m 4' started at 16/11/26 04:44:36]
11/26/16 04:44:36.489515 zmdc[24965].ERR ['zma -m 6' exited abnormally, exit status 255]
11/26/16 04:44:36.507134 zmdc[24965].ERR ['zma -m 12' exited abnormally, exit status 255]
11/26/16 04:44:36.523456 zmdc[24965].ERR ['zma -m 4' exited abnormally, exit status 255]
11/26/16 04:44:36.759421 zmdc[24965].ERR ['zma -m 9' exited abnormally, exit status 255]
11/26/16 04:44:36.765435 zmdc[24965].INF ['zmaudit.pl -c' exited, signal 14]
11/26/16 04:44:40.064168 zmdc[24965].INF [Starting pending process, zma -m 11]
11/26/16 04:44:40.065965 zmdc[24965].INF ['zma -m 11' starting at 16/11/26 04:44:40, pid = 72179]
11/26/16 04:44:40.065993 zmdc[72179].INF ['zma -m 11' started at 16/11/26 04:44:40]
11/26/16 04:44:40.066135 zmdc[24965].INF [Starting pending process, zma -m 8]
11/26/16 04:44:40.067516 zmdc[24965].INF ['zma -m 8' starting at 16/11/26 04:44:40, pid = 72180]
11/26/16 04:44:40.067707 zmdc[72180].INF ['zma -m 8' started at 16/11/26 04:44:40]
11/26/16 04:44:40.897470 zmdc[24965].INF ['zma -m 5' exited, signal 14]
11/26/16 04:44:40.897665 zmdc[24965].INF ['zma -m 8' exited, signal 14]
11/26/16 04:44:40.897748 zmdc[24965].INF ['zma -m 1' exited, signal 14]
11/26/16 04:44:40.897826 zmdc[24965].INF ['zma -m 11' exited, signal 14]
11/26/16 04:44:40.897904 zmdc[24965].INF ['zma -m 7' exited, signal 14]
11/26/16 04:44:40.898849 zmdc[24965].INF ['zma -m 10' exited normally]
11/26/16 04:44:41.005378 zmdc[24965].INF [Starting pending process, zma -m 12]
11/26/16 04:44:41.007252 zmdc[24965].INF ['zma -m 12' starting at 16/11/26 04:44:41, pid = 72192]
11/26/16 04:44:41.007256 zmdc[72192].INF ['zma -m 12' started at 16/11/26 04:44:41]
11/26/16 04:44:41.007415 zmdc[24965].INF [Starting pending process, zma -m 6]
11/26/16 04:44:41.008802 zmdc[24965].INF ['zma -m 6' starting at 16/11/26 04:44:41, pid = 72193]
11/26/16 04:44:41.008967 zmdc[24965].INF [Starting pending process, zma -m 4]
11/26/16 04:44:41.008945 zmdc[72193].INF ['zma -m 6' started at 16/11/26 04:44:41]
11/26/16 04:44:41.010665 zmdc[24965].INF ['zma -m 4' starting at 16/11/26 04:44:41, pid = 72194]
11/26/16 04:44:41.010827 zmdc[24965].INF [Starting pending process, zma -m 9]
11/26/16 04:44:41.010757 zmdc[72194].INF ['zma -m 4' started at 16/11/26 04:44:41]
11/26/16 04:44:41.012547 zmdc[24965].INF ['zma -m 9' starting at 16/11/26 04:44:41, pid = 72195]
11/26/16 04:44:41.012718 zmdc[24965].INF [Starting pending process, zmaudit.pl -c]
11/26/16 04:44:41.012685 zmdc[72195].INF ['zma -m 9' started at 16/11/26 04:44:41]
11/26/16 04:44:41.014147 zmdc[24965].INF ['zmaudit.pl -c' starting at 16/11/26 04:44:41, pid = 72196]
11/26/16 04:44:41.014609 zmdc[72196].INF ['zmaudit.pl -c' started at 16/11/26 04:44:41]
11/26/16 04:44:42.097973 zmdc[24965].INF ['zmc -m 5' exited normally]
11/26/16 04:44:42.406296 zmdc[24965].INF ['zmc -m 12' exited normally]
11/26/16 04:44:43.984170 zmdc[24965].INF ['zmc -m 3' exited normally]
11/26/16 04:44:44.711074 zmdc[24965].INF ['zmc -m 8' exited normally]
11/26/16 04:44:44.955188 zmdc[24965].INF ['zmc -m 7' exited normally]
11/26/16 04:44:45.061656 zmdc[24965].INF [Starting pending process, zma -m 7]
11/26/16 04:44:45.063442 zmdc[24965].INF ['zma -m 7' starting at 16/11/26 04:44:45, pid = 72199]
11/26/16 04:44:45.063431 zmdc[72199].INF ['zma -m 7' started at 16/11/26 04:44:45]
11/26/16 04:44:45.063614 zmdc[24965].INF [Starting pending process, zma -m 10]
11/26/16 04:44:45.064986 zmdc[24965].INF ['zma -m 10' starting at 16/11/26 04:44:45, pid = 72200]
11/26/16 04:44:45.065145 zmdc[24965].INF [Starting pending process, zma -m 1]
11/26/16 04:44:45.065121 zmdc[72200].INF ['zma -m 10' started at 16/11/26 04:44:45]
11/26/16 04:44:45.066796 zmdc[24965].INF ['zma -m 1' starting at 16/11/26 04:44:45, pid = 72201]
11/26/16 04:44:45.066966 zmdc[24965].INF [Starting pending process, zma -m 5]
11/26/16 04:44:45.066974 zmdc[72201].INF ['zma -m 1' started at 16/11/26 04:44:45]
11/26/16 04:44:45.068647 zmdc[24965].INF ['zma -m 5' starting at 16/11/26 04:44:45, pid = 72202]
11/26/16 04:44:45.068735 zmdc[72202].INF ['zma -m 5' started at 16/11/26 04:44:45]
11/26/16 04:44:45.213579 zmdc[24965].INF ['zmc -m 1' exited normally]
11/26/16 04:44:45.452572 zmdc[24965].ERR ['zma -m 5' exited abnormally, exit status 255]
11/26/16 04:44:45.661341 zmdc[24965].INF ['zmc -m 6' exited normally]
11/26/16 04:44:45.695243 zmdc[24965].ERR ['zma -m 7' exited abnormally, exit status 255]
11/26/16 04:44:45.698459 zmdc[24965].ERR ['zma -m 1' exited abnormally, exit status 255]
11/26/16 04:44:45.916841 zmdc[24965].INF ['zmc -m 4' crashed, signal 8]
11/26/16 04:44:45.917054 zmdc[24965].INF ['zma -m 4' crashed, signal 8]
11/26/16 04:44:45.917173 zmdc[24965].INF ['zmc -m 2' crashed, signal 8]
11/26/16 04:44:45.917824 zmdc[24965].INF ['zma -m 9' crashed, signal 8]
11/26/16 04:44:45.917935 zmdc[24965].INF ['zma -m 10' crashed, signal 8]
11/26/16 04:44:45.918145 zmdc[24965].INF ['zma -m 6' crashed, signal 8]
11/26/16 04:44:45.918388 zmdc[24965].INF ['zmc -m 9' crashed, signal 8]
11/26/16 04:44:45.918466 zmdc[24965].INF ['zma -m 12' crashed, signal 8]
11/26/16 04:44:45.920350 zmdc[24965].INF ['zmc -m 11' crashed, signal 8]
11/26/16 04:44:47.095878 zmdc[24965].INF [Starting pending process, zmc -m 5]
11/26/16 04:44:47.097689 zmdc[24965].INF ['zmc -m 5' starting at 16/11/26 04:44:47, pid = 72216]
11/26/16 04:44:47.097703 zmdc[72216].INF ['zmc -m 5' started at 16/11/26 04:44:47]
11/26/16 04:44:47.097849 zmdc[24965].INF [Starting pending process, zmc -m 12]
11/26/16 04:44:47.099224 zmdc[24965].INF ['zmc -m 12' starting at 16/11/26 04:44:47, pid = 72217]
11/26/16 04:44:47.099375 zmdc[72217].INF ['zmc -m 12' started at 16/11/26 04:44:47]
11/26/16 04:44:48.056484 zmdc[24965].INF [Starting pending process, zmc -m 3]
11/26/16 04:44:48.058304 zmdc[24965].INF ['zmc -m 3' starting at 16/11/26 04:44:48, pid = 72218]
11/26/16 04:44:48.058304 zmdc[72218].INF ['zmc -m 3' started at 16/11/26 04:44:48]
11/26/16 04:44:49.003622 zmdc[24965].INF [Starting pending process, zmc -m 7]
11/26/16 04:44:49.005479 zmdc[24965].INF ['zmc -m 7' starting at 16/11/26 04:44:49, pid = 72219]
11/26/16 04:44:49.005506 zmdc[72219].INF ['zmc -m 7' started at 16/11/26 04:44:49]
11/26/16 04:44:49.005645 zmdc[24965].INF [Starting pending process, zmc -m 8]
11/26/16 04:44:49.007102 zmdc[24965].INF ['zmc -m 8' starting at 16/11/26 04:44:49, pid = 72220]
11/26/16 04:44:49.007196 zmdc[72220].INF ['zmc -m 8' started at 16/11/26 04:44:49]
11/26/16 04:44:49.023697 zmdc[24965].INF ['zma -m 2' crashed, signal 8]
11/26/16 04:44:50.064426 zmdc[24965].INF [Starting pending process, zmc -m 6]
11/26/16 04:44:50.066325 zmdc[24965].INF ['zmc -m 6' starting at 16/11/26 04:44:50, pid = 72221]
11/26/16 04:44:50.066348 zmdc[72221].INF ['zmc -m 6' started at 16/11/26 04:44:50]
11/26/16 04:44:50.066502 zmdc[24965].INF [Starting pending process, zmc -m 1]
11/26/16 04:44:50.067901 zmdc[24965].INF ['zmc -m 1' starting at 16/11/26 04:44:50, pid = 72222]
11/26/16 04:44:50.068067 zmdc[24965].INF [Starting pending process, zma -m 11]
11/26/16 04:44:50.068067 zmdc[72222].INF ['zmc -m 1' started at 16/11/26 04:44:50]
11/26/16 04:44:50.069714 zmdc[24965].INF ['zma -m 11' starting at 16/11/26 04:44:50, pid = 72223]
11/26/16 04:44:50.069931 zmdc[24965].INF [Starting pending process, zmc -m 9]
11/26/16 04:44:50.069883 zmdc[72223].INF ['zma -m 11' started at 16/11/26 04:44:50]
11/26/16 04:44:50.071614 zmdc[72224].INF ['zmc -m 9' started at 16/11/26 04:44:50]
11/26/16 04:44:50.071772 zmdc[24965].INF ['zmc -m 9' starting at 16/11/26 04:44:50, pid = 72224]
11/26/16 04:44:50.071982 zmdc[24965].INF [Starting pending process, zma -m 8]
11/26/16 04:44:50.073814 zmdc[24965].INF ['zma -m 8' starting at 16/11/26 04:44:50, pid = 72225]
11/26/16 04:44:50.074030 zmdc[24965].INF [Starting pending process, zmc -m 4]
11/26/16 04:44:50.073985 zmdc[72225].INF ['zma -m 8' started at 16/11/26 04:44:50]
11/26/16 04:44:50.075866 zmdc[24965].INF ['zmc -m 4' starting at 16/11/26 04:44:50, pid = 72226]
11/26/16 04:44:50.076091 zmdc[24965].INF [Starting pending process, zmc -m 11]
11/26/16 04:44:50.076013 zmdc[72226].INF ['zmc -m 4' started at 16/11/26 04:44:50]
11/26/16 04:44:50.077947 zmdc[24965].INF ['zmc -m 11' starting at 16/11/26 04:44:50, pid = 72227]
11/26/16 04:44:50.078181 zmdc[24965].INF [Starting pending process, zmc -m 2]
11/26/16 04:44:50.078086 zmdc[72227].INF ['zmc -m 11' started at 16/11/26 04:44:50]
11/26/16 04:44:50.081160 zmdc[24965].INF ['zmc -m 2' starting at 16/11/26 04:44:50, pid = 72228]
11/26/16 04:44:50.081498 zmdc[72228].INF ['zmc -m 2' started at 16/11/26 04:44:50]
11/26/16 04:44:54.011403 zmdc[24965].INF [Starting pending process, zmc -m 10]
11/26/16 04:44:54.013287 zmdc[24965].INF ['zmc -m 10' starting at 16/11/26 04:44:54, pid = 72231]
11/26/16 04:44:54.013336 zmdc[72231].INF ['zmc -m 10' started at 16/11/26 04:44:54]
11/26/16 04:44:54.013455 zmdc[24965].INF [Starting pending process, zma -m 2]
11/26/16 04:44:54.014825 zmdc[24965].INF ['zma -m 2' starting at 16/11/26 04:44:54, pid = 72232]
11/26/16 04:44:54.015068 zmdc[72232].INF ['zma -m 2' started at 16/11/26 04:44:54]
11/26/16 04:44:55.028175 zmdc[24965].INF [Starting pending process, zma -m 12]
11/26/16 04:44:55.029898 zmdc[24965].INF ['zma -m 12' starting at 16/11/26 04:44:55, pid = 72235]
11/26/16 04:44:55.030097 zmdc[24965].INF [Starting pending process, zma -m 7]
11/26/16 04:44:55.030043 zmdc[72235].INF ['zma -m 12' started at 16/11/26 04:44:55]
11/26/16 04:44:55.032038 zmdc[72237].INF ['zma -m 7' started at 16/11/26 04:44:55]
11/26/16 04:44:55.032083 zmdc[24965].INF ['zma -m 7' starting at 16/11/26 04:44:55, pid = 72237]
11/26/16 04:44:55.032353 zmdc[24965].INF ['zma -m 2' exited, signal 14]
11/26/16 04:44:55.032488 zmdc[24965].INF ['zma -m 8' exited, signal 14]
11/26/16 04:44:55.032607 zmdc[24965].INF ['zma -m 11' exited, signal 14]
11/26/16 04:44:55.032754 zmdc[24965].INF [Starting pending process, zma -m 6]
11/26/16 04:44:55.034394 zmdc[24965].INF ['zma -m 6' starting at 16/11/26 04:44:55, pid = 72239]
11/26/16 04:44:55.034579 zmdc[24965].INF [Starting pending process, zma -m 4]
11/26/16 04:44:55.034587 zmdc[72239].INF ['zma -m 6' started at 16/11/26 04:44:55]
11/26/16 04:44:55.036329 zmdc[24965].INF ['zma -m 4' starting at 16/11/26 04:44:55, pid = 72241]
11/26/16 04:44:55.036458 zmdc[72241].INF ['zma -m 4' started at 16/11/26 04:44:55]
11/26/16 04:44:55.036560 zmdc[24965].INF [Starting pending process, zma -m 9]
11/26/16 04:44:55.038510 zmdc[24965].INF ['zma -m 9' starting at 16/11/26 04:44:55, pid = 72243]
11/26/16 04:44:55.038514 zmdc[72243].INF ['zma -m 9' started at 16/11/26 04:44:55]
11/26/16 04:44:55.038752 zmdc[24965].INF [Starting pending process, zma -m 10]
11/26/16 04:44:55.040744 zmdc[24965].INF ['zma -m 10' starting at 16/11/26 04:44:55, pid = 72245]
11/26/16 04:44:55.040850 zmdc[72245].INF ['zma -m 10' started at 16/11/26 04:44:55]
11/26/16 04:44:55.041003 zmdc[24965].INF [Starting pending process, zma -m 1]
11/26/16 04:44:55.044175 zmdc[24965].INF ['zma -m 1' starting at 16/11/26 04:44:55, pid = 72247]
11/26/16 04:44:55.044451 zmdc[24965].INF [Starting pending process, zma -m 5]
11/26/16 04:44:55.047673 zmdc[24965].INF ['zma -m 5' starting at 16/11/26 04:44:55, pid = 72250]
11/26/16 04:44:55.048051 zmdc[72250].INF ['zma -m 5' started at 16/11/26 04:44:55]
11/26/16 04:44:55.050125 zmdc[72247].INF ['zma -m 1' started at 16/11/26 04:44:55]
11/26/16 04:44:55.364314 zmdc[24965].INF ['zmc -m 12' exited normally]
11/26/16 04:44:55.434641 zmdc[24965].INF ['zmc -m 5' exited normally]
11/26/16 04:44:55.661033 zmdc[24965].ERR ['zma -m 12' exited abnormally, exit status 255]
11/26/16 04:44:55.683702 zmdc[24965].ERR ['zma -m 5' exited abnormally, exit status 255]
11/26/16 04:44:56.807401 zmdc[24965].INF ['zmc -m 3' exited normally]
11/26/16 04:44:57.900191 zmdc[24965].INF ['zmc -m 7' exited normally]
11/26/16 04:44:58.629648 zmdc[24965].INF ['zmc -m 8' exited normally]
11/26/16 04:44:59.215459 zmdc[24965].INF ['zmc -m 6' exited normally]
11/26/16 04:44:59.444079 zmdc[24965].INF ['zmc -m 1' exited normally]
11/26/16 04:45:00.064022 zmdc[24965].INF ['zma -m 4' crashed, signal 8]
11/26/16 04:45:00.064425 zmdc[24965].INF ['zma -m 10' crashed, signal 8]
11/26/16 04:45:00.064708 zmdc[24965].INF ['zma -m 7' crashed, signal 8]
11/26/16 04:45:00.064828 zmdc[24965].INF ['zmc -m 2' crashed, signal 8]
11/26/16 04:45:00.064962 zmdc[24965].INF ['zmc -m 4' crashed, signal 8]
11/26/16 04:45:00.065099 zmdc[24965].INF ['zma -m 1' crashed, signal 8]
11/26/16 04:45:00.065210 zmdc[24965].INF ['zmc -m 9' crashed, signal 8]
11/26/16 04:45:00.065390 zmdc[24965].INF ['zma -m 6' crashed, signal 8]
11/26/16 04:45:00.065904 zmdc[24965].INF ['zmc -m 10' crashed, signal 8]
11/26/16 04:45:00.066293 zmdc[24965].INF ['zma -m 9' crashed, signal 8]
11/26/16 04:45:00.067794 zmdc[24965].INF ['zmc -m 11' crashed, signal 8]
11/26/16 04:45:00.108450 zmdc[24965].INF ['zma -m 3' exited normally]
11/26/16 04:45:05.079342 zmdc[24965].INF [Starting pending process, zma -m 3]
11/26/16 04:45:05.081127 zmdc[72283].INF ['zma -m 3' started at 16/11/26 04:45:05]
11/26/16 04:45:05.081185 zmdc[24965].INF ['zma -m 3' starting at 16/11/26 04:45:05, pid = 72283]
11/26/16 04:45:05.081436 zmdc[24965].INF [Starting pending process, zmc -m 5]
11/26/16 04:45:05.082880 zmdc[24965].INF ['zmc -m 5' starting at 16/11/26 04:45:05, pid = 72284]
11/26/16 04:45:05.083053 zmdc[24965].INF [Starting pending process, zmc -m 12]
11/26/16 04:45:05.083010 zmdc[72284].INF ['zmc -m 5' started at 16/11/26 04:45:05]
11/26/16 04:45:05.084759 zmdc[24965].INF ['zmc -m 12' starting at 16/11/26 04:45:05, pid = 72285]
11/26/16 04:45:05.084889 zmdc[72285].INF ['zmc -m 12' started at 16/11/26 04:45:05]
11/26/16 04:45:05.084996 zmdc[24965].INF [Starting pending process, zma -m 2]
11/26/16 04:45:05.086777 zmdc[24965].INF ['zma -m 2' starting at 16/11/26 04:45:05, pid = 72286]
11/26/16 04:45:05.086903 zmdc[72286].INF ['zma -m 2' started at 16/11/26 04:45:05]
11/26/16 04:45:05.704639 zmdc[24965].ERR ['zma -m 3' exited abnormally, exit status 255]
11/26/16 04:45:06.014809 zmdc[24965].INF [Starting pending process, zmc -m 3]
11/26/16 04:45:06.016543 zmdc[24965].INF ['zmc -m 3' starting at 16/11/26 04:45:06, pid = 72287]
11/26/16 04:45:06.016573 zmdc[72287].INF ['zmc -m 3' started at 16/11/26 04:45:06]
11/26/16 04:45:06.970275 zmdc[24965].ERR ['zmwatch.pl' exited abnormally, exit status 255]
11/26/16 04:45:07.071898 zmdc[24965].INF [Starting pending process, zmc -m 7]
syslog
Code: Select all
Nov 26 04:34:33 zm zmdc[24965]: WAR ['zmc -m 10' has not stopped at 16/11/26 04:34:33. Sending KILL to pid 71491]
Nov 26 04:39:17 zm zma_m10[45484]: WAR [Signal: Lost]
Nov 26 04:41:24 zm zma_m10[45484]: WAR [Signal: Reacquired]
Nov 26 04:44:24 zm zmdc[24965]: WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:24. Sending KILL to pid 71969]
Nov 26 04:44:34 zm zma_m5[25078]: WAR [Event 119408 has zero time, setting to current]
Nov 26 04:44:34 zm zma_m5[25078]: WAR [Signal: Lost]
Nov 26 04:44:34 zm zmdc[24965]: WAR ['zmc -m 10' has not stopped at 16/11/26 04:44:34. Sending KILL to pid 72141]
Nov 26 04:44:34 zm zmwatch[25149]: ERR [Error getting last capture time for cam11]
Nov 26 04:44:34 zm zma_m1[11038]: WAR [Event 119409 has zero time, setting to current]
Nov 26 04:44:35 zm zmdc[24965]: WAR ['zma -m 11' has not stopped at 16/11/26 04:44:35. Sending KILL to pid 72150]
Nov 26 04:44:35 zm zma_m8[72148]: ERR [Shared data not initialised by capture daemon for monitor cam8]
Nov 26 04:44:35 zm zmdc[24965]: ERR ['zma -m 8' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m12[72155]: ERR [Shared data not initialised by capture daemon for monitor cam12]
Nov 26 04:44:36 zm zma_m6[72159]: ERR [Shared data not initialised by capture daemon for monitor cam6]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 6' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 12' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m4[72163]: ERR [Shared data not initialised by capture daemon for monitor cam4]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 4' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:36 zm zma_m9[72164]: ERR [Shared data not initialised by capture daemon for monitor cam9]
Nov 26 04:44:36 zm zmdc[24965]: ERR ['zma -m 9' exited abnormally, exit status 255]
Nov 26 04:44:36 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:36 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:37 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:37 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:37 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:38 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:38 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:38 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:39 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:39 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:39 zm zma_m5[72169]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m8[72180]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m11[72179]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m7[72158]: WAR [Waiting for capture daemon]
Nov 26 04:44:40 zm zma_m1[72166]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m12[72192]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:41 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:42 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:42 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:42 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:43 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:43 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:43 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:44 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:44 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:44 zm zma_m6[72193]: WAR [Waiting for capture daemon]
Nov 26 04:44:45 zm zma_m4[72194]: WAR [Waiting for capture daemon]
Nov 26 04:44:45 zm zma_m5[72202]: ERR [Shared data not initialised by capture daemon for monitor cam5]
Nov 26 04:44:45 zm zmdc[24965]: ERR ['zma -m 5' exited abnormally, exit status 255]
Nov 26 04:44:45 zm zma_m9[72195]: WAR [Waiting for capture daemon]
Nov 26 04:44:45 zm zma_m7[72199]: ERR [Shared data not initialised by capture daemon for monitor cam7]
Nov 26 04:44:45 zm zmdc[24965]: ERR ['zma -m 7' exited abnormally, exit status 255]
Nov 26 04:44:45 zm zma_m1[72201]: ERR [Shared data not initialised by capture daemon for monitor cam1]
Nov 26 04:44:45 zm zmdc[24965]: ERR ['zma -m 1' exited abnormally, exit status 255]
Nov 26 04:44:49 zm zmdc[72149]: WAR ['zma -m 2' has not stopped at 16/11/26 04:44:49. Sending KILL to pid 72145]
Nov 26 04:44:50 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:50 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:51 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:51 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:52 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:52 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:53 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:53 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:54 zm zma_m2[72232]: WAR [Waiting for capture daemon]
Nov 26 04:44:54 zm zma_m11[72223]: WAR [Waiting for capture daemon]
Nov 26 04:44:54 zm zma_m8[72225]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m12[72235]: ERR [Shared data not initialised by capture daemon for monitor cam12]
Nov 26 04:44:55 zm zmdc[24965]: ERR ['zma -m 12' exited abnormally, exit status 255]
Nov 26 04:44:55 zm zma_m5[72250]: ERR [Shared data not initialised by capture daemon for monitor cam5]
Nov 26 04:44:55 zm zmdc[24965]: ERR ['zma -m 5' exited abnormally, exit status 255]
Nov 26 04:44:55 zm zma_m7[72237]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:55 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m7[72237]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:56 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m7[72237]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:57 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m1[72247]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m6[72239]: WAR [Waiting for capture daemon]
Nov 26 04:44:58 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:44:59 zm zma_m10[72245]: WAR [Waiting for capture daemon]
Nov 26 04:44:59 zm zma_m4[72241]: WAR [Waiting for capture daemon]
Nov 26 04:44:59 zm zma_m9[72243]: WAR [Waiting for capture daemon]
Nov 26 04:45:05 zm zma_m3[72283]: ERR [Shared data not initialised by capture daemon for monitor cam3]
Nov 26 04:45:05 zm zmdc[24965]: ERR ['zma -m 3' exited abnormally, exit status 255]
Nov 26 04:45:05 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:06 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:06 zm zmwatch[72156]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 1", output is "Unable to connect to server"]
Nov 26 04:45:06 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 04:45:07 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:14 zm last message repeated 7 times
Nov 26 04:45:15 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:15 zm zma_m8[72309]: WAR [Waiting for capture daemon]
Nov 26 04:45:15 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:16 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:16 zm zma_m8[72309]: WAR [Waiting for capture daemon]
Nov 26 04:45:16 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:17 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:17 zm zma_m8[72309]: WAR [Waiting for capture daemon]
Nov 26 04:45:17 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:18 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:18 zm zma_m2[72286]: WAR [Waiting for capture daemon]
Nov 26 04:45:19 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:20 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:21 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:22 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:23 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:24 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m9[72314]: WAR [Waiting for capture daemon]
Nov 26 04:45:25 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:26 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:26 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:26 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:27 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:27 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:27 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:28 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:28 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:28 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:29 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:29 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:29 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:30 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:30 zm zma_m4[72313]: WAR [Waiting for capture daemon]
Nov 26 04:45:30 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:31 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:31 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:32 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:32 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:45:33 zm zma_m11[72307]: WAR [Waiting for capture daemon]
Nov 26 04:45:33 zm zma_m10[72315]: WAR [Waiting for capture daemon]
Nov 26 04:46:04 zm last message repeated 31 times
Nov 26 04:47:37 zm last message repeated 92 times
Nov 26 04:47:51 zm zmwatch[72301]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 04:47:51 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 04:53:21 zm zmwatch[72385]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 04:53:21 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 04:58:41 zm zmwatch[72541]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 04:58:41 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 05:01:31 zm zmwatch[72727]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 05:01:31 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 05:03:41 zm zmwatch[72890]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Nov 26 05:03:41 zm zmdc[24965]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Nov 26 05:07:31 zm zmwatch[73011]: ERR [Unable to run "/usr/local/bin/zmdc.pl restart zmc -m 10", output is "Unable to connect to server"]
Any ideas are appreciated.