zoneminder not very reliable

Forum for questions and support relating to the 1.30.x releases only.
Locked
krbvroc1
Posts: 4
Joined: Mon Nov 13, 2017 3:17 pm

zoneminder not very reliable

Post by krbvroc1 »

I have been using zoneminder 1.30.4 (from the rpmfusion repo) for a few weeks under the very latest Centos 7.4 on a clean VM install. It has not been very reliable - especially recently.

My major issue is that there will be times the system just becomes unusable and a reboot is needed to recover. I am using two ReoLink C2 cameras (right now only using one). The log will show something like:

Code: Select all

2017-11-13 10:28:02.466986 web_php[1712].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [Timed out waiting for msg /var/lib/zoneminder/sock/zms-177743s.sock]
2017-11-13 10:28:02.378241 web_php[1566].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [Socket /var/lib/zoneminder/sock/zms-177743s.sock does not exist.  This file is created by zms, and since it does not exist, either zms did not run, or zms exited early.  Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly.  Make sure that ZM is actually recording.  If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.]
2017-11-13 10:27:59.916397 zms[23723].ERR-zm_monitor.cpp/4213 [Terminating, last frame sent time 1510586879.416237 secs more than maximum of 10.000000]
2017-11-13 10:27:59.367590 web_js[1566].ERR-?view=watch [getStreamCmdResponse stream error: socket_bind( /var/lib/zoneminder/sock/zms-177743w.sock ) failed: Address already in use - checkStreamForErrors()]
2017-11-13 10:27:59.286243 web_php[1566].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [socket_bind( /var/lib/zoneminder/sock/zms-177743w.sock ) failed: Address already in use]
2017-11-13 10:27:57.004406 zms[23715].ERR-zm_monitor.cpp/4213 [Terminating, last frame sent time 1510586876.504189 secs more than maximum of 10.000000]

If during this failure period I bring up the Reolink webpage embedded in the camera I can see the video fine so the camera is not hung or anything.

If I reboot the zoneminder VM it works on a new reboot.

I had just restarted everything and after only a 10 minutes it stopped working. I believe this is the log prior to it stopping working. Not sure if that TDP parse failure corresponds with the exact moment it stops working or not...I am trying to correlate that.

Code: Select all

2017-11-13 10:46:01.146615 zmc_m1[3058].ERR-zm_rtsp.cpp/88 [16: b7 7a d8 f2 da e4 f9 9d be 91 02 7a 8c 1e e3 8a]
2017-11-13 10:46:01.118868 zmc_m1[3058].ERR-zm_rtsp.cpp/86 [Response parse failure, 826 bytes follow]
2017-11-13 10:45:50.951479 web_js[1857].ERR-?view=watch [getStreamCmdResponse stream error: socket_bind( /var/lib/zoneminder/sock/zms-626033w.sock ) failed: Address already in use - checkStreamForErrors()]
2017-11-13 10:45:50.858050 web_php[1857].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [socket_bind( /var/lib/zoneminder/sock/zms-626033w.sock ) failed: Address already in use]
2017-11-13 10:45:50.829368 web_php[1497].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [Socket /var/lib/zoneminder/sock/zms-626033s.sock does not exist.  This file is created by zms, and since it does not exist, either zms did not run, or zms exited early.  Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly.  Make sure that ZM is actually recording.  If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.]
2017-11-13 10:45:48.113437 zms[1898].ERR-zm_monitor.cpp/4213 [Terminating, last frame sent time 10.019719 secs more than maximum of 10.000000]
2017-11-13 10:45:42.568836 zmc_m1[1776].ERR-zm_rtsp.cpp/770 [Unexpected channel selector 192 in RTSP interleaved data]
krbvroc1
Posts: 4
Joined: Mon Nov 13, 2017 3:17 pm

Re: zoneminder not very reliable

Post by krbvroc1 »

Another reboot and about 30 minutes later - segfaults:

Code: Select all

2017-11-13 11:22:07.349123 zms[7715].ERR-zm_signal.cpp/102 [Backtrace 6: /usr/libexec/zoneminder/cgi-bin/nph-zms(+0x20297) [0x5609415a7297]]
2017-11-13 11:22:07.340759 zms[7715].ERR-zm_signal.cpp/102 [Backtrace 5: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fe0d0f7cc05]]
2017-11-13 11:22:07.332320 zms[7715].ERR-zm_signal.cpp/102 [Backtrace 4: /usr/libexec/zoneminder/cgi-bin/nph-zms(main+0xa61) [0x5609415a64c1]]
2017-11-13 11:22:07.324037 zms[7715].ERR-zm_signal.cpp/102 [Backtrace 3: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZN13MonitorStream9runStreamEv+0x48) [0x5609415e85a8]]
2017-11-13 11:22:07.315583 zms[7715].ERR-zm_signal.cpp/102 [Backtrace 2: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZNK7Monitor6GetFPSEv+0xd) [0x5609415db82d]]
2017-11-13 11:22:07.307159 zms[7715].ERR-zm_signal.cpp/102 [Backtrace 1: /lib64/libpthread.so.0(+0xf5e0) [0x7fe0d45325e0]]
2017-11-13 11:22:07.299063 zms[7715].ERR-zm_signal.cpp/102 [Backtrace 0: /usr/libexec/zoneminder/cgi-bin/nph-zms(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x56094163ff52]]
2017-11-13 11:22:07.290503 zms[7715].ERR-zm_signal.cpp/81 [Signal address is 0x4, from 0x5609415db82d]
2017-11-13 11:22:07.282365 zms[7715].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
2017-11-13 11:22:07.272057 zms[7715].ERR-zm_monitor.cpp/533 [Got empty memory map file size 0, is the zmc process for this monitor running?]
2017-11-13 11:22:07.221339 web_js[1498].ERR-?view=watch [getStreamCmdResponse stream error: socket_sendto( /var/lib/zoneminder/sock/zms-674740s.sock ) failed: Connection refused - checkStreamForErrors()]
2017-11-13 11:22:07.100093 web_php[1498].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [socket_sendto( /var/lib/zoneminder/sock/zms-674740s.sock ) failed: Connection refused]
2017-11-13 11:22:06.232463 zms[7696].ERR-zm_signal.cpp/102 [Backtrace 6: /usr/libexec/zoneminder/cgi-bin/nph-zms(+0x20297) [0x562aa4771297]]
2017-11-13 11:22:06.224200 zms[7696].ERR-zm_signal.cpp/102 [Backtrace 5: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd2fc1bcc05]]
2017-11-13 11:22:06.215823 zms[7696].ERR-zm_signal.cpp/102 [Backtrace 4: /usr/libexec/zoneminder/cgi-bin/nph-zms(main+0xa61) [0x562aa47704c1]]
2017-11-13 11:22:06.207425 zms[7696].ERR-zm_signal.cpp/102 [Backtrace 3: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZN13MonitorStream9runStreamEv+0x48) [0x562aa47b25a8]]
2017-11-13 11:22:06.199137 zms[7696].ERR-zm_signal.cpp/102 [Backtrace 2: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZNK7Monitor6GetFPSEv+0xd) [0x562aa47a582d]]
2017-11-13 11:22:06.190709 zms[7696].ERR-zm_signal.cpp/102 [Backtrace 1: /lib64/libpthread.so.0(+0xf5e0) [0x7fd2ff7725e0]]
2017-11-13 11:22:06.182661 zms[7696].ERR-zm_signal.cpp/102 [Backtrace 0: /usr/libexec/zoneminder/cgi-bin/nph-zms(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x562aa4809f52]]
2017-11-13 11:22:06.174155 zms[7696].ERR-zm_signal.cpp/81 [Signal address is 0x4, from 0x562aa47a582d]
2017-11-13 11:22:06.164113 zms[7696].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
2017-11-13 11:22:06.153292 zms[7696].ERR-zm_monitor.cpp/533 [Got empty memory map file size 0, is the zmc process for this monitor running?]
2017-11-13 11:22:06.103586 web_js[1500].ERR-?view=watch [getStreamCmdResponse stream error: socket_sendto( /var/lib/zoneminder/sock/zms-674740s.sock ) failed: Connection refused - checkStreamForErrors()]
2017-11-13 11:22:06.026905 web_php[1500].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [socket_sendto( /var/lib/zoneminder/sock/zms-674740s.sock ) failed: Connection refused]
2017-11-13 11:22:05.015867 zms[7688].ERR-zm_signal.cpp/102 [Backtrace 6: /usr/libexec/zoneminder/cgi-bin/nph-zms(+0x20297) [0x55d2847b0297]]
2017-11-13 11:22:04.999239 zms[7688].ERR-zm_signal.cpp/102 [Backtrace 5: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f968f119c05]]
2017-11-13 11:22:04.992989 web_php[1501].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [Timed out waiting for msg /var/lib/zoneminder/sock/zms-674740s.sock]
2017-11-13 11:22:04.990774 zms[7688].ERR-zm_signal.cpp/102 [Backtrace 4: /usr/libexec/zoneminder/cgi-bin/nph-zms(main+0xa61) [0x55d2847af4c1]]
2017-11-13 11:22:04.982410 zms[7688].ERR-zm_signal.cpp/102 [Backtrace 3: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZN13MonitorStream9runStreamEv+0x48) [0x55d2847f15a8]]
2017-11-13 11:22:04.974052 zms[7688].ERR-zm_signal.cpp/102 [Backtrace 2: /usr/libexec/zoneminder/cgi-bin/nph-zms(_ZNK7Monitor6GetFPSEv+0xd) [0x55d2847e482d]]
2017-11-13 11:22:04.965745 zms[7688].ERR-zm_signal.cpp/102 [Backtrace 1: /lib64/libpthread.so.0(+0xf5e0) [0x7f96926cf5e0]]
2017-11-13 11:22:04.957642 zms[7688].ERR-zm_signal.cpp/102 [Backtrace 0: /usr/libexec/zoneminder/cgi-bin/nph-zms(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x55d284848f52]]
2017-11-13 11:22:04.949183 zms[7688].ERR-zm_signal.cpp/81 [Signal address is 0x4, from 0x55d2847e482d]
2017-11-13 11:22:04.941068 zms[7688].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
2017-11-13 11:22:04.931825 zms[7688].ERR-zm_monitor.cpp/533 [Got empty memory map file size 0, is the zmc process for this monitor running?]
2017-11-13 11:22:04.866779 web_js[1500].ERR-?view=watch [getStreamCmdResponse stream error: Socket /var/lib/zoneminder/sock/zms-674740s.sock does not exist.  This file is created by zms, and since it does not exist, either zms did not run, or zms exited early.  Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly.  Make sure that ZM is actually recording.  If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information. - checkStreamForErrors()]
2017-11-13 11:22:04.775053 web_php[1500].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [Socket /var/lib/zoneminder/sock/zms-674740s.sock does not exist.  This file is created by zms, and since it does not exist, either zms did not run, or zms exited early.  Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly.  Make sure that ZM is actually recording.  If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.]
2017-11-13 11:22:02.434436 zmc_m1[6883].ERR-zm_rtsp.cpp/770 [Unexpected channel selector 249 in RTSP interleaved data]
2017-11-13 11:22:02.117957 zmc_m1[6883].ERR-zm_rtsp.cpp/770 [Unexpected channel selector 62 in RTSP interleaved data]
2017-11-13 11:22:02.011201 zms[7682].ERR-zm_monitor.cpp/3799 [Can't sendto on sd 9: No such file or directory]
2017-11-13 11:22:01.907580 zmc_m1[6883].ERR-zm_rtsp.cpp/770 [Unexpected channel selector 161 in RTSP interleaved data]
2017-11-13 11:22:01.757767 web_js[1500].ERR-?view=watch [getStreamCmdResponse stream error: socket_bind( /var/lib/zoneminder/sock/zms-674740w.sock ) failed: Address already in use - checkStreamForErrors()]
2017-11-13 11:22:01.669209 web_php[1500].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [socket_bind( /var/lib/zoneminder/sock/zms-674740w.sock ) failed: Address already in use]
2017-11-13 11:21:59.743368 zms[6893].ERR-zm_monitor.cpp/4213 [Terminating, last frame sent time 10.004701 secs more than maximum of 10.000000]
krbvroc1
Posts: 4
Joined: Mon Nov 13, 2017 3:17 pm

Re: zoneminder not very reliable

Post by krbvroc1 »

It stopped working again...I did correlate it with the following event in the log - the RTSP 'Response parse failure'

Code: Select all

2017-11-13 13:21:07.430090 zmdc[4964].ERR-zmdc.pl ['zmc -m 1' exited abnormally, exit status 255]
2017-11-13 13:21:07.388508 zmc_m1[20280].FAT-zm_remote_camera_rtsp.cpp/162 [No RTSP sources]
2017-11-13 13:20:59.967071 web_php[20210].ERR-/usr/share/zoneminder/www/includes/functions.php/2033 [Socket /var/lib/zoneminder/sock/zms-492837s.sock does not exist.  This file is created by zms, and since it does not exist, either zms did not run, or zms exited early.  Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly.  Make sure that ZM is actually recording.  If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.]
2017-11-13 13:20:59.346281 zmc_m1[20286].ERR-zm_rtsp.cpp/88 [16: f1 fd e9 47 87 fe 9e 07 a5 09 9f 2c ed 71 6d c7]
2017-11-13 13:20:59.240023 zmc_m1[20286].ERR-zm_rtsp.cpp/86 [Response parse failure, 93 bytes follow]
2017-11-13 13:20:41.567538 zmc_m1[10741].ERR-zm_rtsp.cpp/770 [Unexpected channel selector 2 in RTSP interleaved data]
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: zoneminder not very reliable

Post by bbunge »

First off dump the VM. As for reliable there are a great many of us with systems up with many cameras for months or years with no issues! Look to your install and dump the VM. Also try a Ubuntu server!!!
krbvroc1
Posts: 4
Joined: Mon Nov 13, 2017 3:17 pm

Re: zoneminder not very reliable

Post by krbvroc1 »

I don't doubt that that are many with it working. It was working 'okay' for me for a few weeks. It would occasionally crash which was tolerable. There was a recent firmware update I did on the Reolink - not sure if it is related to that. I might try rolling that back to see. I also notice there were some centos updates to the lame library - not sure if that could be related.

I have been a Centos / RedHat user since the late 90's, it wouldn't make sense to switch my OS. I have run other video related things such as MythTV and video transcoders in virtual machines...I don't believe that is the issue. For a properly sized VM this load is next to nothing. I think ZM is having issues with decoding the RTSP and crashing rather than gracefully recovering. The load is barely above 1, I've allocated 4 CPU's to the VM, 12G of ram, and 1TB of disk.
geraldbrandt
Posts: 129
Joined: Tue Dec 09, 2014 10:20 pm

Re: zoneminder not very reliable

Post by geraldbrandt »

I run mine in a VM just fine. Ubuntu and six cameras 1920x720 or higher res. 3GB RAM, 8 CPUs
User avatar
iconnor
Posts: 2900
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: zoneminder not very reliable

Post by iconnor »

Yeah ZM doesn't like your camera's responses. Maybe try the ffmpeg input type.
Locked