Signal loss causes blue screen that requires camera reset

[empty01]

No. How can I find out? The system in Ubuntu server installation.

[Baylink]

Go to a root shell prompt (#), and type

# locate ffmpeg

if locate is installed, or

# type ffmpeg

if not, and when you find it -- and/or how many of them you have -- do

# ffmpeg -version

Your ZM config pages should have a pathname for ffmpeg that matches one of the ones you get back from those commands (if you have more than one).

You do have the cameras in ffmpeg mode, rather than libvlc or remote, right?

[empty01]

Hi BayLink

ffmpeg -version gives:

ffmpeg version 2.8.11-0ubuntu0.16.04.1 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100

locate ffmpeg gives:
/usr/bin/ffmpeg

Yes, I have set up ZM to use ffmpeg on all cameras.

Anything wrong here?

All best

Mark

[Baylink]

Not necessarily wrong, but I gather that there is a three-point X release of ffmpeg and that sometimes it works better.

[empty01]

So how would I install that?
Thanks
Mark

[SteveGilvarry]

In your first post the logs show all the URL's for camera's having issues are the same, like 5 cameras, that would be unusual for cameras from different manufacturers, all my cameras across brands and sometime models vary the RTSP URL. From quick search are they xvision or all some rebranded Y3K camera, UK company that designs own cameras? And you at one point said you could get the RTSP stream in VLC while blue screen in zm, but more recently said no RTSP stream anywhere until you reboot camera? (Camera web interface is not using RTSP stream)
Based on that I think your cameras RTSP stream might just be crashing, you can test it using ffmpeg to connect to camera using the RTSP url. You can do the same with VLC, but I suggested FFMPEG as it will be same as zm but different. Just connect it as a player let it run and see what happens.
Does seem odd a product would do this as RTSP is pretty much the only way these things connect to everything else days, but it is my best theory based on information at hand. Don't know if you tried things like moving away from h264, chopping resolution right down at the camera, and generally dumbing camera down to see if it has an impact.

[empty01]

Hi Steve
Firstly, thanks for taking the time to look through this rather long list of posts.

Yes, when I posted the logs there were from 5 cameras using the same RTSP URL. Those are the cameras that give me the blue screen most often. Indeed, there are 2 cameras that - with my old computer setup - would give me blue screens the most frequently. But I have experienced the blue screen problem with all cameras at some point over the last 6 months or so.
For 1 of them (RTSP URL = rtsp://******:******@192.168.1.103:554/1/stream1/Profile1), the blue screens problem is very rare.
For 3 of them, which have the same RTSP URL (rtsp://*****:****@192.168.1.108:554/1), the blue screens problem occurs occasionally.
For 8 of them, which have the same RTSP URL (rtsp://192.168.1.193:554/ch01.264), the blue screens problem occurs frequently.
For 2 of the above 8, the blue screens problem occurs predictably ~ once per day.

I've had other cameras from different manufactures and with different RTSP URL, which i've now sold, which also gave the blue screens problem occasionally. Perhaps this is for different reasons?

Things have improved (i.e. the blue screens problem occurs less frequently) since I switched from a dual core 6Gb RAM machine to a 8 core 16 Gb RAM machine.

The cameras are labelled Robox, but I suspect that is some rebranding of the originals from China.

Yes, I realise that my comments about VLC streaming might have seemed contradictory. Currently, when I get the blue screens problem in ZM, I am unable to view its RTSP stream on VLC (whereas I am for non-blue screen cameras). That also suggests to me that the RTSP stream is crashing.
On previous occasions I was able to view the RTSP stream on VLC, even though I had a blue screen in ZM. Perhaps on those occasions the blue screen in ZM was caused by something else.

The best fix I have found is that the 8 cameras that give me the most problems have a function accessed via the camera's web interface to schedule reboots once a day. But given the frequency of the RTSP stream going down, this still meant some cameras would be down for much of the following 24 hours.

I only have the Ubuntu sever install on (i.e. no GUI), so not sure how to view the camera's RTSP stream with FFMPEG - any suggestions?

I'm not keen on chopping down the resolution / dumbing down the cameras, but as an experiment I could try it on the most problematic camera to see if things improve.

Thanks again for looking at these posts.

All best

Mark

[SteveGilvarry]

Wow definitely something not right and a royal pain in the arse.
Things I would play with it move one of the most painful ones back to your old setup, run that by itself just to be sure it is not something performance related. I wonder if a camera going offline is causing zm to start trying to reconnect too fast and impacting the others, so those other two going blue screen due to some issue in them, and then taking others down. But that is just me thinking about stuff. But would be interesting to know if some of the lesser impacted cameras are still doing it on the new server. So in the past you had rtsp crashes in some cameras and also a performance issue affecting the others?

ffmpeg runs on lots of things, but since you must already have vlc then try that instead. Remove one of those 2 *beeping beep beep* cameras that is going blue everyday from zm. Then connect to it direct from VLC and let it run, see if it goes blue, means leaving a desktop running.

And dumbing down settings is just for experimenting, maybe the camera is struggling with 1080 h264 so try 720 and see if it still blue screens, try this in one of the worst 2 and one of the other 8 like that.

All of this of course relies on you wanting to keep playing, but it may just be these cameras are just junked. I run Dahua and Hikvision cameras and have never had any signal loss, one Dahua must have been running for 5 years, and only occasional reboot. Couldn't see anything on Robox, except some DVR mentions here and there.

[Baylink]

What were you 15 minute load averages like on the old box, and what are they like on the new one. Just at idle, with no viewers open?

[empty01]

Loads are way down with the new server. Never measured the old 15 min averages, but it would hover between 3 and 10 - I would guess the average was ~6 (old box is a dual-core Dell Optiplex 760; 6 Gb RAM)

With the Sun Fire x2250, its pretty stable around 2.2 (and that's with a few more cameras). That may be because on the x2250 I installed the naked server version of Ubuntu (no GUI), whereas on my old dual-core Dell Optiplex 760 I had the full-fat Ubuntu install.

As I have pointed out above, with the new and clearly less stressed x2250, my blue screen / RTSP stream loss problems have become much less frequent (only once in the last 5 days), but haven't gone away completely. That suggests to me that a stressed ZM is somehow knocking out the camera's RTSP stream, and once that stream is knocked out, it can't be recovered unless I reboot the camera.

[Baylink]

I do agree with your diagnosis -- and of course, it *really* means that the firmware in the camera isn't rugged enough... hey. Have you checked to see if there's new firmware for the cameras?

[empty01]

Hi
Back to this persistent problem - despite now having an unstressed Sunfire x2250 running with load averages of ~2.5, I am still getting the Signal loss causes blue screen that requires camera reset. What's more, its now happening on cameras that were previously mostly trouble-free. I'm now pretty convinced that some of my cameras are causing the problem (i.e. dropping their RTSP streams) as I can't view the RTSP streams on the blue-screened cameras using VLC, but I can using their web interface.

I've been looking at other solutions and one possible one I have come across is periodically sending the cameras ONVIF systemreboot commands. Does anybody have any experience with sending ONVIF commands to cameras who could give me some guidance on this? I tried installing npm (which seemed to work fine) then the npm package onviv, but that gave the following errors:
/root
└── onvif@0.5.3

npm WARN enoent ENOENT: no such file or directory, open '/root/package.json'
npm WARN root No description
npm WARN root No repository field.
npm WARN root No README data
npm WARN root No license field.

Many thanks in advance
Mark

[empty01]

Hi
So I tried installing Node.js and npm, as it has an onvif implementation. After much more pain, I concluded that it was not possible to get this to work - I couldn't even get basic Node.js and npm working.

Then I came across quatanium/python-onvif. I have a bit of a python background so had a go, and got some onvif commands to work, including the 'SystemReboot' command. So now I have a bunch of python scripts that will reboot my cameras, and want to trigger them when I get signal loss on specific cameras.

Interestingly, when using python-onvif, it turns out I don't need to send the onvif 'SystemReboot' command to get my RTSP stream back - I merely need to create a camera instance and it comes back on. Makes me suspect that zm is not handling the RTSP stream properly.

Anyway, it now turns out that zm filters don't do what they are supposed to do and run user-specified commands.

[ke0eyh]

I have 2 work around/options you might try.

My wife and I use an app called "Cam Viewer Pro" by Robert Chou on Google play $3.99. (There is also a free version) It has a reboot option that I use for 2 of my cameras that have IR cut filter issues that a quick reboot fixes. It only takes me a moment to reboot both cameras using the app. And it works with Zoneminder cameras, they are in the drop down menus. Just add the monitor number. (But I use the IP and port # of the camera not Zoneminder. Otherwise you wouldn't be able to reboot the camera)

I also have a PC in the living room that we leave on 24/7 to view our Zoneminder server running in the garage. It tends to act up and drop a couple cameras after 12/24 hours . Then I get a black box or 2 instead of the cameras and lots of errors in the logs. A refresh of the "Montage" page clears this right up. So I use a Firefox add on "ReloadMatic" by "Pylo". I set it up to auto refresh the page every 15 minutes and that seems to have taken care of the issue.

Zoneminder v1.30.4
Ubuntu Server edition 16.04

The first picture shows the app settings using a USB camera. Second and third pictures show the rebooting of an IP camera.

[empty01]

Back on this topic - hopefully for the last time.
I had my whole Ubuntu server and zoneminder system setup on 2 separate HDDs, one =350Gb and one = 2Tb. All of the blue screen problems described above were with the 2Tb HDD set up. A few weeks ago the 2Tb HDD started kicking up HDD errors so I shutdown and replaced it with the 350Gb one. As far as I am aware, I setup the whole Ubuntu server and zoneminder systems identically on both HDDs. However, I have not experienced any blue screen / signal loss problems at all with the 350Gb system set up.
So my question is, could it have been the 2Tb HDD that was causing these annoying blue screen / signal loss issues? Any thoughts on how that could be happening? Bad sectors? On its last legs?
Many thanks
Mark