SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Forum for questions and support relating to the 1.30.x releases only.
Post Reply
dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by dws » Tue Jan 30, 2018 9:26 pm

I've been beating my head against the keyboard on this one for a bit over a month... I'm missing something, so I'm coming here to ask for advice.

I've got four Ventech IP-PoE cameras that work great, stay running, and give great pictures.... that is, they do all of this up until the moment I start Zoneminder.

I used the ONVIF probe to configure them for ZM, and it spat out the same RTSP URL that I used when attempting to stream from them in VLC and ffmpeg outside of ZoneMinder. VLC and ffmpeg both stream these cameras without issue, and without reboots.

However, when I start Zoneminder, the cameras seem to go nuts. All four of them reboot within one minute of ZM starting, and then they reboot at random intervals as long as ZM is running. When I stop ZM, all four cameras stay up and running.

The system is a Mac Pro with Ubuntu 16.04.3 running on it, 16GB RAM, and two Intel Xeon 5130 CPUs (four processors in total, two per chip).

I have done the following to try to keep the cameras up and running under Zoneminder:
- switched to constant bit rate in the cameras' internal setup
- reduced the frame rate to 5 FPS in the cameras' internal setup
- isolated the cameras to their own PoE switch
- contacted the manufacturer/seller for support (they couldn't/wouldn't help with ZM installations)

The cameras all stream great and without reboots in ONVIFer (an Android app) and the NVSIP app that the manufacturer provides... until I start ZM.

For what it's worth, I also bought two Jooan PoE cameras that seem to work just fine without any issues using the same configurations as the Ventech cams.

These particular Ventech cameras say they support ONVIF 2.4, have PTZ, snapshot capability, and are based on the HI3518EP chip.

So, now, I'm wondering what it is that ZM might be sending to the cameras that's causing them to barf and reboot.

ANY thoughts, advice, etc would be very welcome, since the boss (in his infinite wisdom) threw away the boxes, so I can't return them to Amazon. Yes, they were only $35/ea, but I'd like to be able to use them rather than going through the process of replacing them.

Thanks in advance.
Last edited by dws on Wed Jan 31, 2018 6:32 pm, edited 1 time in total.

dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

Re: Zoneminder causes cameras to reboot randomly

Post by dws » Wed Jan 31, 2018 6:31 pm

I solved this last night and this morning, actually...

In order to get these cameras to work properly, I had to do a few things to the cameras themselves, the OS, and ZoneMinder.

First, Ubuntu currently uses version 2.8.x of ffmpeg (video/audio decoding libraries). For the Ventech cameras to work, version 3.4.x or better was needed. Google how to do that upgrade. This upgrade drastically reduced CPU load and memory use.
Second, in the Ventech cameras themselves, ALL ALARMS MUST BE SHUT OFF. All of the boxes in all of the tabs in the Alarm configurations must be UN-checked.
Third, in the Ventech cameras themselves, the frame and bit rate settings must be changed. The cameras cannot be set to higher than 10 FPS, and to help ffmpeg decode the streams easier, setting CBR instead of VBR is essential.
Fourth is the ZoneMinder settings. When creating a monitor, ZoneMinder is able to probe the network for ONVIF cameras, and will automatically detect the camera, its streams, and set up an ffmpeg stream to monitor and control the cameras. Using this initial probe's settings will cause the cameras to reboot because, I believe, ZoneMinder and your cameras are speaking different dialects of the same language. It's OK to set up the camera this way, but after that monitor has been saved, the user must go back into the ZoneMinder monitor settings and change a few things:
-Under the General tab, change "Source Type" to "Remote"
-Under the Source tab, change "Remote Protocol" to "RTSP" and the Remote Method "RTP/RTSP". "Remote Host" is the IP of the camera ONLY. "Remote Host Port" should be 554, "Remote Host Path" should be set for either "/live0.264" or "/live1.264" depending on the desired resolution (the first is the high res, the second is the low res). 24 bit color, set the resolution to the resolution of the stream, and at the bottom, checkmark the box next to "Use RTSP response Media URL"
The user can set other options as they see fit.

What all of this does is:
-Tells ZoneMinder NOT to send any commands to the camera via the RTSP stream.
-Tells the cameras to JUST be a camera, and not to try to record anything themselves (thus not filling their memory or causing them to crash and reboot).
-Increases the decoding efficiency on the computer so the computer isn't coughing, sputtering, and causing frame losses.

bbunge
Posts: 2364
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by bbunge » Thu Feb 01, 2018 1:44 am

Interesting conclusions. Bur usually ffmpeg RTSP running TCP 32 bit colour works better with modern cameras than the ancient remote. You should turn down the frame rate and key frame interval as well as resolution.

dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by dws » Fri Feb 02, 2018 4:48 am

bbunge wrote:
Thu Feb 01, 2018 1:44 am
Interesting conclusions. Bur usually ffmpeg RTSP running TCP 32 bit colour works better with modern cameras than the ancient remote. You should turn down the frame rate and key frame interval as well as resolution.
In no case will the Ventech cameras stay stable if using the ffmpeg method. I have tried every conceivable combination of settings using that method, and had zero success
Using the remote method, ZM seems to be smart enough to resynchronize when the cameras send "extra" data, usually 84 extra bytes. Instead of bombing out, ZM logs the extra data as a warning and just moves on. I have a feeling that ZM was sending something back to those cameras when it detected the extra data, which was causing the reboots.
I'm not sure what the extra data is, or is for, but this is the only way I was able to keep the cameras from rebooting.
Oddly, the cameras stream beautifully to ONVIFer without reboots for hours, but as soon as I start ZM, they misbehaved.
I don't blame ZM for this; I believe the cameras themselves are the issue, what with the extra data being sent with the streams.

User avatar
knight-of-ni
Posts: 2216
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by knight-of-ni » Fri Feb 02, 2018 4:36 pm

The irony is that the Remote RTSP method makes functions call to the ffmpeg API. Someday the Remote -> RTSP method may go away so a better long term solution would be to investigate why the ffmpeg method isn't working as it should. In the grand scheme of sheer number of cameras, the ffmpeg source type is far better.

Upgrading from ffmpeg 2.8 to 3.x series was a good first step. However, the trouble is that ffmpeg has a rapidly changing API. Upgrading ffmpeg on your system is not enough to get zoneminder to use the new(er) API calls in the new version of ffmpeg. You have to rebuild zoneminder to get that.

Assuming you can afford to play around with your system (i.e. not production), try to build your own zoneminder deb against the version of ffmpeg you have installed. Make sure ffmpeg 2.8 is completely removed before doing this. Instructions on how to use the do_debian_package.sh in our git repo are available in our readthedocs site.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/

dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by dws » Fri Feb 02, 2018 7:20 pm

knnniggett wrote:
Fri Feb 02, 2018 4:36 pm
The irony is that the Remote RTSP method makes functions call to the ffmpeg API. Someday the Remote -> RTSP method may go away so a better long term solution would be to investigate why the ffmpeg method isn't working as it should. In the grand scheme of sheer number of cameras, the ffmpeg source type is far better.

Upgrading from ffmpeg 2.8 to 3.x series was a good first step. However, the trouble is that ffmpeg has a rapidly changing API. Upgrading ffmpeg on your system is not enough to get zoneminder to use the new(er) API calls in the new version of ffmpeg. You have to rebuild zoneminder to get that.

Assuming you can afford to play around with your system (i.e. not production), try to build your own zoneminder deb against the version of ffmpeg you have installed. Make sure ffmpeg 2.8 is completely removed before doing this. Instructions on how to use the do_debian_package.sh in our git repo are available in our readthedocs site.
Yes, I found it ironic, myself... However, these particular cameras are just plain WEIRD... I've got two cheapo Jooan cams on the same server, with the four Ventech cams, and the Jooan cams were basically plug in, scan ONVIF, select stream, and BOOM! up and running. Nary a problem with them at all. The Jooan cams were 2/3 the cost of the Ventech ones... I'll give you two guesses which cams I'm going to buy in the future...

As far as the API calls, as long as ZM is able to talk to FFMPEG, and I can get the video to my system, I don't feel a need to recompile... This is a production system, now... a few weeks ago I would have had time to play with it, but we've just got our other processes running, and the cams are necessary for remote monitoring.

I'm still wondering what the "extra" data that's being sent by the camera is... It doesn't seem to convert to anything ASCII, and I don't recognize the format.

Code: Select all

2018-02-02 13:33:44.070706	zmc_m4		24426	WAR	56: 74 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 38 3a 33 33 3a 34 34 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 39 45 43 45 35 30 39 0d 0a 0d 0a	zm_rtsp.cpp	799
2018-02-02 13:33:44.022522	zmc_m4		24426	WAR	Unexpected format RTSP interleaved data, resyncing by 56 bytes	zm_rtsp.cpp	798
2018-02-02 12:50:48.075320	zmc_m4		24426	WAR	56: 74 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 37 3a 35 30 3a 34 37 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 39 45 43 45 35 30 39 0d 0a 0d 0a	zm_rtsp.cpp	799
2018-02-02 12:50:48.008199	zmc_m4		24426	WAR	Unexpected format RTSP interleaved data, resyncing by 56 bytes	zm_rtsp.cpp	798
2018-02-02 12:17:11.071644	zmc_m1		25797	WAR	55: 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 37 3a 31 37 3a 31 31 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 31 37 37 30 45 32 39 41 0d 0a 0d 0a	zm_rtsp.cpp	799
2018-02-02 12:17:11.017484	zmc_m1		25797	WAR	Unexpected format RTSP interleaved data, resyncing by 55 bytes	zm_rtsp.cpp	798
2018-02-02 12:04:07.118771	zmc_m1		25797	WAR	55: 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 37 3a 30 34 3a 30 37 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 31 37 37 30 45 32 39 41 0d 0a 0d 0a	zm_rtsp.cpp	799
2018-02-02 12:04:07.068171	zmc_m1		25797	WAR	Unexpected format RTSP interleaved data, resyncing by 55 bytes	zm_rtsp.cpp	798
2018-02-02 11:55:50.083711	zmc_m3		24451	WAR	55: 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 36 3a 35 35 3a 34 39 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 35 35 42 32 30 33 31 0d 0a 0d 0a	zm_rtsp.cpp	799
2018-02-02 11:55:50.017633	zmc_m3		24451	WAR	Unexpected format RTSP interleaved data, resyncing by 55 bytes	zm_rtsp.cpp	798
If that "extra" code is what was causing the reboots before, it'd be interesting to find out exactly what it is, what it's intended for, etc...

As I stated, though, the cams are working, now, using the "ancient" remote method..

User avatar
knight-of-ni
Posts: 2216
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by knight-of-ni » Fri Feb 02, 2018 9:14 pm

Some cameras have the ability to do their own motion detection. For those kinds of cameras, the problem becomes how to send that data to the receiving device. A popular way to do that is to send the motion detection meta data as a separate RTP (no S) stream on a different port, much like the old passive ftp transfers back in the day. That of course is very firewall un-friendly. One way to get around the firewall un-friendliness is to interleave the motion detection meta data directly into the stream itself. Of course, the problem then becomes how to un-interleave that data at the other end.

That might be the extra data zoneminder is complaining about.

In any case, and this is very speculative, but if your cameras have any settings regarding motion detection or RTP streams, try turning all that off. You want a basic RTSP stream and nothing else.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/

dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by dws » Fri Feb 02, 2018 9:52 pm

Interesting... because I've got all motion detection turned off in the cameras' firmware. That was part of my testing for a solution before coming upon this fix. At least it's SUPPOSED to be off; one of my complaints about these Ventech cams is that many times the settings don't "take" - they'll show as set, even after logging out and back in, but they don't actually take effect.
You're probably on to something with the interleaved data... and this is probably why the ffmpeg method fails, but the "ancient" remote method works in ZM. This is my guess, anyway.
I do appreciate the explanation, though, because you're helping to fill in some of the blanks in my understanding of how these things work.

mikb
Posts: 395
Joined: Mon Mar 25, 2013 12:34 pm

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by mikb » Sat Feb 03, 2018 8:37 pm

dws wrote:
Fri Feb 02, 2018 7:20 pm
I'm still wondering what the "extra" data that's being sent by the camera is... It doesn't seem to convert to anything ASCII, and I don't recognize the format.
If you mean this line here ...

Code: Select all

2018-02-02 13:33:44.070706	zmc_m4		24426	WAR	56: 74 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 38 3a 33 33 3a 34 34 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 39 45 43 45 35 30 39 0d 0a 0d 0a
That most certainly is ASCII. Frequent 20 (space) and "0d 0a" (CR/LF).

I make that as "te: Fri, Feb 02 2018 18:33:34 GMT(newline)Session: 09ECE509\r\n"

You are getting some headers in there (Date: ... Session: ...) by the look of it. Anyone else got a clue where they are coming from :)

dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by dws » Mon Feb 05, 2018 5:19 pm

You are getting some headers in there (Date: ... Session: ...) by the look of it. Anyone else got a clue where they are coming from :)
These cameras are cheapos that are intended by Ventech to be used on their DVR/NVR... they're not "generic" cams...

The data I'd converted before (it wasn't the set I'd pasted) didn't seem to translate to anything, but this does, so I'm guessing that's housekeeping data for these cams to talk to their NVR/DVR...

Another guess: switching the cams from "ffmpeg" to REMOTE in ZM allows ZM to "ignore" that extra data, and keeps the cams from rebooting... I'm not sure if ZM is requesting resyncs from the cams when it loses its spot in ffmpeg or not, but this was the only way I could get them to work.

Thanks for your insight.

darmach
Posts: 19
Joined: Thu Aug 13, 2015 8:26 am

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by darmach » Wed Apr 17, 2019 2:55 pm

I apparently get the same issue - I have 8 of these, on zoneminder 1.32.3:
https://www.aliexpress.com/item/POE-Wat ... 55494.html

So a different camera - I wouldn't be surprised if it would be the same under the hood, especcially when the streaming address (../live0.264) is the same.

I get exactly the same issue - the cameras reboot often, nothing useful in the logs. Managed to log on via telnet (ipc71a default password for root) and load is ~17 but I guess this might be normal for embedded linux.

And after reading this thread I decided to try out different software. First cams ran for 24 hours with chinese CMS software, then I tried out yahoo which also managed to run the cameras for few hours without any issues.

So the issue might be the same - zoneminder sending some response to some weird data from cams, which they can't handle and crash. I don't see that additional data in the zm logs though. Can some of you guys help me sort this out?

I already tried the fixes from this thread:
- I'm running dockerized ZM, ffmpeg is already 3.4.4.
- All alarm definitions in the cameras are off.
- tried setting zm to remote and settings from this thread, but this fails with:

Code: Select all

Date/Time
Component	Server	PID	Level	Message	File	Line
2019-04-17 16:52:35	zmc_m1		487	INF	Starting Capture version 1.32.3	zmc.cpp	223
2019-04-17 16:52:35	zmc_m1		488	INF	Received new Content-Base in DESCRIBE response header. Updated device Url to: 'rtsp://192.168.1.101:554/live0.264/'	zm_rtsp.cpp	355
2019-04-17 16:52:35	zmc_m1		488	WAR	The device is using a codec that may not be supported. Do not be surprised if things don't work.	zm_rtp_source.cpp	69
2019-04-17 16:52:35	zmc_m1		487	FAT	Unable to locate video stream	zm_remote_camera_rtsp.cpp	188
2019-04-17 16:52:24	zmc_m1		480	WAR	The device is using a codec that may not be supported. Do not be surprised if things don't work.	zm_rtp_source.cpp	69
2019-04-17 16:52:24	zmc_m1		480	INF	Received new Content-Base in DESCRIBE response header. Updated device Url to: 'rtsp://192.168.1.101:554/live0.264/'	zm_rtsp.cpp	355
2019-04-17 16:52:24	zmc_m1		478	INF	Starting Capture version 1.32.3	zmc.cpp	223
2019-04-17 16:52:24	zmc_m1		478	FAT	Unable to locate video stream	zm_remote_camera_rtsp.cpp	188
2019-04-17 16:52:24	zmc_m1		478	INF	Got signal 15 (Terminated), exiting	zm_signal.cpp	40
2019-04-17 16:52:17	zmc_m1		476	FAT	Unable to locate video stream	zm_remote_camera_rtsp.cpp	188
2019-04-17 16:52:17	zmc_m1		477	WAR	The device is using a codec that may not be supported. Do not be surprised if things don't work.	zm_rtp_source.cpp	69
2019-04-17 16:52:17	zmc_m1		477	INF	Received new Content-Base in DESCRIBE response header. Updated device Url to: 'rtsp://192.168.1.101:554/live0.264/'	zm_rtsp.cpp	355
2019-04-17 16:52:17	zmc_m1		476	INF	Starting Capture version 1.32.3	zmc.cpp	223
2019-04-17 16:52:14	zmc_m1		468	FAT	Unable to locate video stream	zm_remote_camera_rtsp.cpp	188
2019-04-17 16:52:14	zmc_m1		470	WAR	The device is using a codec that may not be supported. Do not be surprised if things don't work.	zm_rtp_source.cpp	69

dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by dws » Wed Apr 24, 2019 2:56 pm

Interesting... The initial issue was under 1.30... If you're running 1.32 now, and you have the choice in the camera itself between NTSC and PAL, you could give that a shot. I've found some cameras don't like to respond to ONVIF probes from within ZM unless they're set to PAL.
It may also be that the URL for the video stream on your particular cam is different...
I've since replaced all those Ventech cams with other, higher-resolution devices, and used them for percussive therapy, as they're pretty much crap. The Remote method in 1.32 seems to no longer work for these cams, and their constant rebooting (again, after the upgrade) got so annoying the percussive therapy became a necessity.

dws
Posts: 43
Joined: Mon Jan 15, 2018 2:05 pm

Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly

Post by dws » Wed Apr 24, 2019 2:57 pm

Also, I find myself wishing our cams were identical (and that I still had one for experimentation)... I'd been unable to locate the root password for mine.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 4 guests