segfault on restart, lots of errors in logs, "zmc exited abnormally" and "Unable to read packet from stream"

Forum for questions and support relating to the 1.30.x releases only.
Locked
tolland
Posts: 1
Joined: Wed Jul 25, 2018 10:48 am

segfault on restart, lots of errors in logs, "zmc exited abnormally" and "Unable to read packet from stream"

Post by tolland »

Hi,
I am running zoneminder 1.30.4 from the fusion repos on Centos-7.5.1804 with 8 monitors, also connected to it with zmNinja. I have setup 8 monitors, 2 for each camera, the first is the substream in "modect" mode, and the second is the full quality stream in "monitor" mode.

Currently zoneminder is working, and detects motion correctly (some of the time), and the steams are all viewable in zmNinja and at the console. However the system is unreliable, and the logs full of errors. The behaviour suggests it's load related, but the errors/logging is a complete mess... so it's hard to pick apart what things to change.

The first issue is that upon restart, zoneminder is segfaulting;

Code: Select all

Jul 25 11:05:15 cctv zmdc[2771]: INF [Server shutdown at 18/07/25 11:05:15]
Jul 25 11:05:15 cctv systemd[1]: Starting ZoneMinder CCTV recording and security system...
-- Subject: Unit zoneminder.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit zoneminder.service has begun starting up.
Jul 25 11:05:15 cctv zmpkg[3149]: INF [Sanity checking States table...]
Jul 25 11:05:15 cctv zmpkg[3149]: INF [Command: start]
Jul 25 11:05:15 cctv web_php[744]: ERR [Socket /var/lib/zoneminder/sock/zms-882407s.sock does not exist.  This file is created by zms, and since it does not exist, either zms did not run, or zms exited early.  Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly.  Make sure that ZM is actually recording.  If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.]
Jul 25 11:05:15 cctv zms[3156]: ERR [Got empty memory map file size 0, is the zmc process for this monitor running?]
Jul 25 11:05:15 cctv zms[3156]: ERR [Got signal 11 (Segmentation fault), crashing]
Jul 25 11:05:15 cctv zms[3156]: ERR [Signal address is 0x4, from 0x56167d07882d]
Jul 25 11:05:15 cctv zms[3156]: ERR [Backtrace 0: /usr/libexec/zoneminder/cgi-bin/nph-zms(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x56167d0dcf52]]
Jul 25 11:05:15 cctv zms[3156]: ERR [Backtrace 1: /lib64/libpthread.so.0(+0xf6d0) [0x7fa73e6ed6d0]]
Jul 25 11:05:15 cctv zms[3156]: ERR [Backtrace 2: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZNK7Monitor6GetFPSEv+0xd) [0x56167d07882d]]
Jul 25 11:05:15 cctv zms[3156]: ERR [Backtrace 3: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZN13MonitorStream9runStreamEv+0x48) [0x56167d0855a8]]
Jul 25 11:05:15 cctv zms[3156]: ERR [Backtrace 4: /usr/libexec/zoneminder/cgi-bin/nph-zms(main+0xa61) [0x56167d0434c1]]
Jul 25 11:05:15 cctv zms[3156]: ERR [Backtrace 5: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fa73b055445]]
Jul 25 11:05:15 cctv zms[3156]: ERR [Backtrace 6: /usr/libexec/zoneminder/cgi-bin/nph-zms(+0x20297) [0x56167d044297]]
Jul 25 11:05:15 cctv zms[3156]: INF [Backtrace complete, please execute the following command for more information]
Jul 25 11:05:15 cctv zms[3156]: INF [addr2line -e /usr/libexec/zoneminder/cgi-bin/nph-zms 0x56167d0dcf52 0x7fa73e6ed6d0 0x56167d07882d 0x56167d0855a8 0x56167d0434c1 0x7fa73b055445 0x56167d044297]
Jul 25 11:05:15 cctv zmdc[3158]: INF [Server starting at 18/07/25 11:05:15]
Executing the suggested debugging produces this;

Code: Select all

addr2line -e /usr/libexec/zoneminder/cgi-bin/nph-zms 0x56167d0dcf52 0x7fa73e6ed6d0 0x56167d07882d 0x56167d0855a8 0x56167d0434c1 0x7fa73b055445 0x56167d044297
??:0
??:0
??:0
??:0
??:0
??:0
??:0
The second issue is that the capture daemon frequently crashes;

Code: Select all

Jul 25 11:05:20 cctv zmdc[3158]: INF ['zmc -m 22' starting at 18/07/25 11:05:20, pid = 3246]
Jul 25 11:05:20 cctv zmdc[3246]: INF ['zmc -m 22' started at 18/07/25 11:05:20]
Jul 25 11:05:20 cctv zmdc[3158]: ERR ['zmc -m 22' exited abnormally, exit status 255]
Jul 25 11:05:20 cctv zmdc[3158]: INF [Starting pending process, zmc -m 22]
Jul 25 11:05:20 cctv zmdc[3158]: INF ['zmc -m 22' starting at 18/07/25 11:05:20, pid = 3249]
Jul 25 11:05:20 cctv zmdc[3249]: INF ['zmc -m 22' started at 18/07/25 11:05:20]
Jul 25 11:05:20 cctv zma_m19[3209]: WAR [Waiting for capture daemon]
Jul 25 11:05:20 cctv zmdc[3158]: INF ['zma -m 22' starting at 18/07/25 11:05:20, pid = 3251]
Jul 25 11:05:20 cctv zmdc[3251]: INF ['zma -m 22' started at 18/07/25 11:05:20]
The load on the machine looks like this;

Code: Select all

top - 11:10:20 up 13:43,  3 users,  load average: 1.87, 1.65, 1.73
Tasks: 276 total,   1 running, 275 sleeping,   0 stopped,   0 zombie
%Cpu0  : 18.7 us,  0.7 sy,  0.0 ni, 79.9 id,  0.0 wa,  0.0 hi,  0.7 si,
%Cpu1  : 29.1 us,  0.7 sy,  0.0 ni, 69.3 id,  0.0 wa,  0.0 hi,  1.0 si,
%Cpu2  : 31.0 us,  1.3 sy,  0.0 ni, 67.3 id,  0.0 wa,  0.0 hi,  0.3 si,
%Cpu3  : 28.6 us,  1.0 sy,  0.0 ni, 70.0 id,  0.0 wa,  0.0 hi,  0.3 si,
%Cpu4  : 21.1 us,  1.0 sy,  0.0 ni, 77.9 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu5  : 24.1 us,  0.3 sy,  0.0 ni, 75.6 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu6  : 26.8 us,  0.3 sy,  0.0 ni, 72.6 id,  0.0 wa,  0.0 hi,  0.3 si,
%Cpu7  : 15.2 us,  0.7 sy,  0.0 ni, 83.8 id,  0.0 wa,  0.0 hi,  0.3 si,
KiB Mem : 12137940 total,  6285940 free,  1392416 used,  4459584 buff/cac
KiB Swap:  6160380 total,  6160380 free,        0 used.  9067272 avail Me

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+
 3200 apache    20   0 1203708 189028 107988 S  7.7  1.6   3:05.79
 3187 apache    20   0 1203848 187396 108060 S  6.6  1.5   2:31.25
 3195 apache    20   0 1203772 190820 108068 S  3.2  1.6   1:14.84
 3191 apache    20   0 1720044 701956 616484 S  2.9  5.8   1:09.77
 3206 apache    20   0 1197600 141776 108056 S  0.9  1.2   0:21.07
 3249 apache    20   0 1197760 140704 107984 S  0.7  1.2   0:16.37
 3218 apache    20   0  531188 123856 107752 S  0.7  1.0   0:15.34
 9916 streamt+  20   0  747924  66140   6396 S  0.6  0.5  25:31.15
 3209 apache    20   0  532600 126904 106332 S  0.3  1.0   0:08.24
 3220 apache    20   0  532600 126908 106332 S  0.3  1.0   0:06.72

Performance wise, this machine is significantly more powerful than the machine it replaces, which was a windows 10 box running iSpy, which seemed to handle this setup, and reliably motion detected on these streams. Though that box was stuck at 90% cpu continuously it never crashed in several years of monitoring.


Any suggestions on how to fix these issues?
Locked