Streaming has stopped working

Forum for questions and support relating to the 1.28.x releases only.
Locked
jrd
Posts: 11
Joined: Wed Jan 04, 2017 7:32 pm

Streaming has stopped working

Post by jrd »

I don't think this is a 1.28.x issue, but that's what I'm running, so I put it here.

I've been running 1.28.1 for a while, on a gentoo server. I recently updated some other OS/utility stuff on that machine, and ZM's ability to stream video has broken. It's still capturing and recording events fine, but attempting to live monitor a camera, or play back a recorded event, just fail with a busted video icon .

Looking at the apache log, when I attempt to stream video, I see stuff like

{code}[Wed Jan 04 10:16:35 2017] [error] [client 192.168.2.12] ERR [socket_sendto( /var/tmp/zm/zms-429673s.sock ) failed: No such file or directory], referer: http://localhost:8090/index.php?view=watch&mid=4
[Wed Jan 04 10:16:45 2017] [error] [client 192.168.2.12] ERR [socket_sendto( /var/tmp/zm/zms-429673s.sock ) failed: No such file or directory], referer: http://localhost:8090/index.php?view=watch&mid=4
[Wed Jan 04 10:16:55 2017] [error] [client 192.168.2.12] ERR [socket_sendto( /var/tmp/zm/zms-429673s.sock ) failed: No such file or directory], referer: http://localhost:8090/index.php?view=watch&mid=4
[Wed Jan 04 12:38:57 2017] [error] [client 192.168.2.12] ERR [socket_sendto( /var/tmp/zm/zms-488222s.sock ) failed: No such file or directory], referer: http://localhost:8090/index.php?view=ev ... c=1&page=1
{/code}

Checking /var/tmp/zm, it's correct, those socket files are not present. Nothing new has been created in that dir since I did my system updates.

I'm looking for hints on how to debug. I've broken something, but it's not clear what. Hints?

Thanks in advance.
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Streaming has stopped working

Post by bbunge »

Did you by any chance upgrade Apache? May be possible that you need to enable cgi again. Another area is the ZM Options Paths - PATH_ZMS.
jrd
Posts: 11
Joined: Wed Jan 04, 2017 7:32 pm

Re: Streaming has stopped working

Post by jrd »

Thanks.

This has turned into more of a rat-hole than I hoped :(

Since I posted, I've tried a number of things, but so far am still stuck.

In the process I upgraded to ZM 1.30.0. I also cut over to Apache 2.4.25. I upgraded all my configs (I think) to match the 2.4 permission scheme, but I'm still losing.

I'm pretty sure the root of the problem is CGI permissions. When I go to hit one of the streaming endpoints (like when I click on one of the cameras to get a live view) the Apache access log records a 403 return, and the error log says "client denied by server configuration" on the attempt to invoke nph-zms. I'm sure that's why my sockets aren't there.

I *think* I've got the cgi config set up correctly, correct path to nph-zms etc. I have a single vhost, with ScriptAlias pointing in the right place. I'm still puzzling over what config settings could be convincing Apache that it shouldn't serve that CGI.

Can anyone point me at a simple standardized config that's expected to work with Apache 2.4?

I'm running gentoo, BTW.

Thanks in advance...
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Streaming has stopped working

Post by bbunge »

jrd
Posts: 11
Joined: Wed Jan 04, 2017 7:32 pm

Re: Streaming has stopped working

Post by jrd »

Yeah, I've been using those.

I've gotten several things sorted out, including apache permissions. (At least part of) ZM appears to be starting. But now I'm not capturing any events.

At present, I'm fairly sure the root of the problem is that zmc instances are not starting. I can review old events with no problem, but I'm not creating any new ones. When I go to the UI and select a camera to monitor, the apache log shows

Code: Select all

[Mon Jan 16 06:57:20.685080 2017] [:error] [pid 29541] [client 192.168.2.6:54724] ERR [getStreamCmdResponse stream error: socket_sendto( /var/tmp/zm/zms-882977s.sock ) failed: Connection refused - checkStreamForErrors()], referer: http://h2.jrd.org/index.php?view=watch&mid=4
[Mon Jan 16 06:57:21.490762 2017] [cgi:error] [pid 29543] [client 192.168.2.6:54736] Premature end of script headers: zms, referer: http://h2.jrd.org/index.php?view=watch&mid=4
The system log shows

Code: Select all

Jan 16 06:56:59 h2 simple[29633]: ERR [Got empty memory map file size 0, is the zmc process for this monitor running?]
Jan 16 06:56:59 h2 simple[29633]: ERR [Got signal 11 (Segmentation fault), crashing]
Jan 16 06:56:59 h2 simple[29633]: ERR [Signal address is 0x4, from 0x44448d]
Jan 16 06:56:59 h2 simple[29633]: ERR [Backtrace 0: /usr/libexec/zoneminder/cgi-bin/zms(_Z14zm_die_handleriP9siginfo_tPv+0x6c) [0x46d38c]]
Jan 16 06:56:59 h2 simple[29633]: ERR [Backtrace 1: /lib64/libpthread.so.0(+0x10ca0) [0x7fbcf1564ca0]]
Jan 16 06:56:59 h2 simple[29633]: ERR [Backtrace 2: /usr/libexec/zoneminder/cgi-bin/zms(_ZNK7Monitor6GetFPSEv+0xd) [0x44448d]]
Jan 16 06:56:59 h2 simple[29633]: ERR [Backtrace 3: /usr/libexec/zoneminder/cgi-bin/zms(_ZN13MonitorStream9runStreamEv+0x48) [0x4509f8]]
Jan 16 06:57:00 h2 simple[29633]: ERR [Backtrace 4: /usr/libexec/zoneminder/cgi-bin/zms(main+0x92c) [0x410a0c]]
Jan 16 06:57:00 h2 simple[29633]: ERR [Backtrace 5: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fbceff91805]]
Jan 16 06:57:00 h2 simple[29633]: ERR [Backtrace 6: /usr/libexec/zoneminder/cgi-bin/zms(_start+0x29) [0x411579]]
Jan 16 06:57:00 h2 simple[29633]: INF [Backtrace complete, please execute the following command for more information]
There are, in fact, no zmc processes running.

Doing a bunch of googling, I see a lot of commentary about this class of problem being caused by insufficient shared memory. I don't think that's what's going on here. I tried disabling all the cameras except one low-rez unit, and there's no change. My shared memory partition shows zero usage.

Happy to hear other ideas. I think my next step is to start instrumenting the zmc code and see what it's doing. Or figure out how to run it under gdb. Hints?

I suppose I should move this thread to the 1.30 area, since that's what I'm running now.
jrd
Posts: 11
Joined: Wed Jan 04, 2017 7:32 pm

Re: Streaming has stopped working

Post by jrd »

Ok, I sort of fixed it.

Running zmc under the debugger revealed that it would start capturing, and streaming a monitor started working. So I dug into why zmc wasn't starting at all.

I found viewtopic.php?f=36&t=25382&p=97009&hili ... ver#p97009 which gave me some clues. Apparently when I upgraded, I ended up with a server-id of 1 for my only server, and that convinced ZM that all my monitors belonged to some other server. It doesn't work to just delete the server entry, but making the one server entry have ID 0, and setting all the monitors to say they're on that server ID, did the trick.

Suggestion for the devs: 1: Logging: on startup, say something about which monitors it's starting up, and which it isn't, and why. 2. On the landing page, maybe in the server column, say something about which server, or <localhost>, or something, to give a clue about how the system thinks this monitor is managed.

Thanks for your help!
Locked