Terminating, last frame sent time 1430120263.528179 secs

Forum for questions and support relating to the 1.28.x releases only.
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

OK, I'm going to give this a go (although I have tried many FPS/key frame combos to no avail), I'll report back.
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by asker »

lowprofile wrote:OK, I'm going to give this a go (although I have tried many FPS/key frame combos to no avail), I'll report back.
Awesome! BTW just today, I had a reason to build ZM from source in a VM and I faced the _exact_ problem you mention - i.e

Code: Select all

ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-465397s.sock ) failed: No such file or directory
I figured out why this was happening - again, I compiled from source and a test branch that is not stable (angular-ui), but this was the reason:

I started tracing the logs of both apache and syslog like so:

Code: Select all

tail -f /var/log/syslog /var/log/apache2/error.log
And I noticed this all-important error:

Code: Select all

[Sun May 03 16:26:44.966888 2015] [cgi:error] [pid 4764] [client 192.168.1.4:49567] script not found or unable to stat: /usr/lib/cgi-bin/nph-zms, referer: http://192.168.1.48/zm/?view=watch&mid=2
Please double check your logs like I did and if this is an error that comes up each time you try and watch live feed, check if you have nph-zms sitting in/usr/lib/cgi-bin or some other directory. In my case it was actually in /local/libexec/zoneminder/cgi-bin/

In my web interface, Options->Path->PATH_ZMS was set to /cgi-bin/ which should be a relative path, so no idea where it was picking /usr/lib from when even in /etc/zm.conf I had ZM_PATH_CGI=/usr/local/libexec/zoneminder/cgi-bin

This may be because I am on an experimental branch, so instead of trying to figure out where it was hardcoded, I just copied zms and nph-zms to /usr/lib/cgi-bin too and boom. I had streaming.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

OK, I set my two monitors up such that the FPS was left blank. One camera I left the settings alone - capturing at 15 FPS with keyframe interval at the default 3

The other I dropped to 3 FPS and set the keyframe interval to 10.

Just before I crashed last night I checked the monitors and bingo - still running. Until I noticed that on the second monitor, the sun was still shining!
I checked the logs - no errors. Just an image on the monitor that was three or four hours old.

So I went back to that camera and put the keyframe interval back to 3 (it was getting late) and restarted zoneminder. This morning, all is well so fingers crossed. If I get the time today I'll set both cameras up properly (I'll have a second monitor for each camera using the low-res stream for monitoring)

I had a good dig through all of the logs, and with the exception of 'Shared data not initialised by capture daemon, some query functions may not be available or produce invalid results' there are no errors or warnings I can find. Fingers crossed

Just FYI I'm installed from package (x64 arch) not compiled from source.
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by asker »

:D Awesome! Thanks for the update.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

Damn :-(

2015-05-04 16:12:46.559886 zms 25803 ERR Terminating, last frame sent time 1430752366.059794 secs more than maximum of 10.000000 zm_monitor.cpp 4155
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by asker »

lowprofile wrote:Damn :-(

2015-05-04 16:12:46.559886 zms 25803 ERR Terminating, last frame sent time 1430752366.059794 secs more than maximum of 10.000000 zm_monitor.cpp 4155

Ouch. I'm going blind here but:

a) Looks like your problem camera is the Escam- can you reconfirm all your settings are similar to this post: https://www.devin.cl/blog/escam-ip-camera/
b) Have you tried with FFMPEG ?
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

I have two similar cameras, one is an Escam QD500, the other is a no-name but it's running the same/similar software. My setup is almost identical to the one shown in the link, with the exception that I have not specified FPS.

ffmpeg does not work well. I get smeared images. Ubuntu 14.04 uses libav (although ffmpeg is touted to return in 15.10 I believe), don't know if that has anything to do with it.

I have returned to messing around with the camera settings. Don't ask me why but it definitely does not like having its FPS set too low. The UI becomes unresponsive and the image stream patchy. I returned it to default settings (25 fps and I frame interval of 3) and it came back to life, then lowered the FPS in stages until I got to 10 where the camera seems happy and my CPU fan isn't turning the server into a hovercraft.

I'm pretty sure that whatever the outcome with this camera, leaving the FPS blank in the monitor config and setting it at the camera is the way to go.
codabiz
Posts: 59
Joined: Sun Jan 04, 2009 10:16 am
Location: London, UK

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by codabiz »

Your findings are similar to mine.

After much experimentation with frame rates and streaming rates, I eventually got my Logitech webcam - Raspberry Pi (MotionPie software) to stream at 960x720 at a stable 4 fps to my ZM 1.28.1/Ubuntu 14.04 LTS server (FPS blank). I am no longer getting:

'Terminating, last frame sent time 1430301155.783266 secs more than maximum of 10.000000'

and general socket errors. :)
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by asker »

Sounds good - looks like you have localized your problem to the optimal FPS setting in the camera and have found one. The only thing is this: if you leave the FPS blank in ZM and that camera goes down, your server CPU will turn into that hovercraft trying to find it -- and there is where the FPS on ZM helps. So for example, if your ideal FPS is 10 in camera, setting it to 15 in ZM should not affect things (unless you have tested that it does) and will save you from the 'camera going down' situation.
lowprofile wrote:I have two similar cameras, one is an Escam QD500, the other is a no-name but it's running the same/similar software. My setup is almost identical to the one shown in the link, with the exception that I have not specified FPS.

ffmpeg does not work well. I get smeared images. Ubuntu 14.04 uses libav (although ffmpeg is touted to return in 15.10 I believe), don't know if that has anything to do with it.

I have returned to messing around with the camera settings. Don't ask me why but it definitely does not like having its FPS set too low. The UI becomes unresponsive and the image stream patchy. I returned it to default settings (25 fps and I frame interval of 3) and it came back to life, then lowered the FPS in stages until I got to 10 where the camera seems happy and my CPU fan isn't turning the server into a hovercraft.

I'm pretty sure that whatever the outcome with this camera, leaving the FPS blank in the monitor config and setting it at the camera is the way to go.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

Still can't get it to stay connected. The issue for me is that I can connect via RTSP with VLC and I don't lose the stream. I can connect with a browser, with Xeoma.... and I don't lose the stream.

last frame sent time 1430120263.528179 secs - to me, that sounds like an arithmetic over/underflow. I could live with it if Zoneminder recovered, but it just dies and that's that.

I'm going to swap that camera out for a dahua I have on the way (from China). It could be something odd with the camera. Other than that, I think I'll just have to ditch Zoneminder which is a pity because (other than staying connected to the camera) it does everything I want.
codabiz
Posts: 59
Joined: Sun Jan 04, 2009 10:16 am
Location: London, UK

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by codabiz »

I wonder, has any anyone experienced this problem ' 'Last frame sent time 1430120263.528179 secs' on ZM versions prior to 1.28.1?
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

I wonder too......

Release 1.28.1 February 14th, 2015 ->Minor release focusing on RTSP improvements.<-
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

Just out of curiosity, where does that 'sent time' come from? is it using the server time, or is it getting it from the camera somehow?
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by asker »

lowprofile wrote:Just out of curiosity, where does that 'sent time' come from? is it using the server time, or is it getting it from the camera somehow?
https://github.com/ZoneMinder/ZoneMinde ... .cpp#L4158

Based on the code, zm_monitor keeps a track of the time the last frame was received by the server (not camera time). If the current time exceeds the last frame sent time by more than max_secs_since_last_sent_frame (which seems to be 10 here), then it bails out.

It seems to me 1430752366.059794 seconds is actually an error - I took a quick look at the code and the last_frame_sent variable is initialized to a valid value only in a few cases. It's possible in your camera, due to an error in receiving frames, the last_frame_sent time is undefined and therefore junk.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
lowprofile
Posts: 19
Joined: Mon Apr 27, 2015 8:01 am

Re: Terminating, last frame sent time 1430120263.528179 secs

Post by lowprofile »

Season 1 episode 9.

Due to the high load on the server caused by libvlc I resolved to get ffmpeg working (it simply didn't). Given that *buntu 14 uses libav I thought I'd look into that.

Checked OPT_FFMPEG and set PATH_FFMPEG to /usr/bin/avconv.
Set remote method to RTP/RTSP because I heard that enforces a TCP connection.
Set WATCH_CHECK_INTERVAL and WATCH_MAX_DELAY to higher values
Created 2 monitors for each camera, a low res one on MODECT linked to a high res one set to NODECT (using the cameras high and low res streams respectively) so the image processing is done at 704 x 480 but the 1280 x 720 images get saved during an alarm as well.

No image distortion, and the cpu load from ZM is under 10%. The wait begins......
Locked