I've been beating my head against the keyboard on this one for a bit over a month... I'm missing something, so I'm coming here to ask for advice.
I've got four Ventech IP-PoE cameras that work great, stay running, and give great pictures.... that is, they do all of this up until the moment I start Zoneminder.
I used the ONVIF probe to configure them for ZM, and it spat out the same RTSP URL that I used when attempting to stream from them in VLC and ffmpeg outside of ZoneMinder. VLC and ffmpeg both stream these cameras without issue, and without reboots.
However, when I start Zoneminder, the cameras seem to go nuts. All four of them reboot within one minute of ZM starting, and then they reboot at random intervals as long as ZM is running. When I stop ZM, all four cameras stay up and running.
The system is a Mac Pro with Ubuntu 16.04.3 running on it, 16GB RAM, and two Intel Xeon 5130 CPUs (four processors in total, two per chip).
I have done the following to try to keep the cameras up and running under Zoneminder:
- switched to constant bit rate in the cameras' internal setup
- reduced the frame rate to 5 FPS in the cameras' internal setup
- isolated the cameras to their own PoE switch
- contacted the manufacturer/seller for support (they couldn't/wouldn't help with ZM installations)
The cameras all stream great and without reboots in ONVIFer (an Android app) and the NVSIP app that the manufacturer provides... until I start ZM.
For what it's worth, I also bought two Jooan PoE cameras that seem to work just fine without any issues using the same configurations as the Ventech cams.
These particular Ventech cameras say they support ONVIF 2.4, have PTZ, snapshot capability, and are based on the HI3518EP chip.
So, now, I'm wondering what it is that ZM might be sending to the cameras that's causing them to barf and reboot.
ANY thoughts, advice, etc would be very welcome, since the boss (in his infinite wisdom) threw away the boxes, so I can't return them to Amazon. Yes, they were only $35/ea, but I'd like to be able to use them rather than going through the process of replacing them.
Thanks in advance.
SOLVED: Zoneminder causes Ventech cameras to reboot randomly
SOLVED: Zoneminder causes Ventech cameras to reboot randomly
Last edited by dws on Wed Jan 31, 2018 6:32 pm, edited 1 time in total.
Re: Zoneminder causes cameras to reboot randomly
I solved this last night and this morning, actually...
In order to get these cameras to work properly, I had to do a few things to the cameras themselves, the OS, and ZoneMinder.
First, Ubuntu currently uses version 2.8.x of ffmpeg (video/audio decoding libraries). For the Ventech cameras to work, version 3.4.x or better was needed. Google how to do that upgrade. This upgrade drastically reduced CPU load and memory use.
Second, in the Ventech cameras themselves, ALL ALARMS MUST BE SHUT OFF. All of the boxes in all of the tabs in the Alarm configurations must be UN-checked.
Third, in the Ventech cameras themselves, the frame and bit rate settings must be changed. The cameras cannot be set to higher than 10 FPS, and to help ffmpeg decode the streams easier, setting CBR instead of VBR is essential.
Fourth is the ZoneMinder settings. When creating a monitor, ZoneMinder is able to probe the network for ONVIF cameras, and will automatically detect the camera, its streams, and set up an ffmpeg stream to monitor and control the cameras. Using this initial probe's settings will cause the cameras to reboot because, I believe, ZoneMinder and your cameras are speaking different dialects of the same language. It's OK to set up the camera this way, but after that monitor has been saved, the user must go back into the ZoneMinder monitor settings and change a few things:
-Under the General tab, change "Source Type" to "Remote"
-Under the Source tab, change "Remote Protocol" to "RTSP" and the Remote Method "RTP/RTSP". "Remote Host" is the IP of the camera ONLY. "Remote Host Port" should be 554, "Remote Host Path" should be set for either "/live0.264" or "/live1.264" depending on the desired resolution (the first is the high res, the second is the low res). 24 bit color, set the resolution to the resolution of the stream, and at the bottom, checkmark the box next to "Use RTSP response Media URL"
The user can set other options as they see fit.
What all of this does is:
-Tells ZoneMinder NOT to send any commands to the camera via the RTSP stream.
-Tells the cameras to JUST be a camera, and not to try to record anything themselves (thus not filling their memory or causing them to crash and reboot).
-Increases the decoding efficiency on the computer so the computer isn't coughing, sputtering, and causing frame losses.
In order to get these cameras to work properly, I had to do a few things to the cameras themselves, the OS, and ZoneMinder.
First, Ubuntu currently uses version 2.8.x of ffmpeg (video/audio decoding libraries). For the Ventech cameras to work, version 3.4.x or better was needed. Google how to do that upgrade. This upgrade drastically reduced CPU load and memory use.
Second, in the Ventech cameras themselves, ALL ALARMS MUST BE SHUT OFF. All of the boxes in all of the tabs in the Alarm configurations must be UN-checked.
Third, in the Ventech cameras themselves, the frame and bit rate settings must be changed. The cameras cannot be set to higher than 10 FPS, and to help ffmpeg decode the streams easier, setting CBR instead of VBR is essential.
Fourth is the ZoneMinder settings. When creating a monitor, ZoneMinder is able to probe the network for ONVIF cameras, and will automatically detect the camera, its streams, and set up an ffmpeg stream to monitor and control the cameras. Using this initial probe's settings will cause the cameras to reboot because, I believe, ZoneMinder and your cameras are speaking different dialects of the same language. It's OK to set up the camera this way, but after that monitor has been saved, the user must go back into the ZoneMinder monitor settings and change a few things:
-Under the General tab, change "Source Type" to "Remote"
-Under the Source tab, change "Remote Protocol" to "RTSP" and the Remote Method "RTP/RTSP". "Remote Host" is the IP of the camera ONLY. "Remote Host Port" should be 554, "Remote Host Path" should be set for either "/live0.264" or "/live1.264" depending on the desired resolution (the first is the high res, the second is the low res). 24 bit color, set the resolution to the resolution of the stream, and at the bottom, checkmark the box next to "Use RTSP response Media URL"
The user can set other options as they see fit.
What all of this does is:
-Tells ZoneMinder NOT to send any commands to the camera via the RTSP stream.
-Tells the cameras to JUST be a camera, and not to try to record anything themselves (thus not filling their memory or causing them to crash and reboot).
-Increases the decoding efficiency on the computer so the computer isn't coughing, sputtering, and causing frame losses.
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
Interesting conclusions. Bur usually ffmpeg RTSP running TCP 32 bit colour works better with modern cameras than the ancient remote. You should turn down the frame rate and key frame interval as well as resolution.
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
In no case will the Ventech cameras stay stable if using the ffmpeg method. I have tried every conceivable combination of settings using that method, and had zero success
Using the remote method, ZM seems to be smart enough to resynchronize when the cameras send "extra" data, usually 84 extra bytes. Instead of bombing out, ZM logs the extra data as a warning and just moves on. I have a feeling that ZM was sending something back to those cameras when it detected the extra data, which was causing the reboots.
I'm not sure what the extra data is, or is for, but this is the only way I was able to keep the cameras from rebooting.
Oddly, the cameras stream beautifully to ONVIFer without reboots for hours, but as soon as I start ZM, they misbehaved.
I don't blame ZM for this; I believe the cameras themselves are the issue, what with the extra data being sent with the streams.
-
knight-of-ni
- Posts: 2404
- Joined: Thu Oct 18, 2007 1:55 pm
- Location: Shiloh, IL
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
The irony is that the Remote RTSP method makes functions call to the ffmpeg API. Someday the Remote -> RTSP method may go away so a better long term solution would be to investigate why the ffmpeg method isn't working as it should. In the grand scheme of sheer number of cameras, the ffmpeg source type is far better.
Upgrading from ffmpeg 2.8 to 3.x series was a good first step. However, the trouble is that ffmpeg has a rapidly changing API. Upgrading ffmpeg on your system is not enough to get zoneminder to use the new(er) API calls in the new version of ffmpeg. You have to rebuild zoneminder to get that.
Assuming you can afford to play around with your system (i.e. not production), try to build your own zoneminder deb against the version of ffmpeg you have installed. Make sure ffmpeg 2.8 is completely removed before doing this. Instructions on how to use the do_debian_package.sh in our git repo are available in our readthedocs site.
Upgrading from ffmpeg 2.8 to 3.x series was a good first step. However, the trouble is that ffmpeg has a rapidly changing API. Upgrading ffmpeg on your system is not enough to get zoneminder to use the new(er) API calls in the new version of ffmpeg. You have to rebuild zoneminder to get that.
Assuming you can afford to play around with your system (i.e. not production), try to build your own zoneminder deb against the version of ffmpeg you have installed. Make sure ffmpeg 2.8 is completely removed before doing this. Instructions on how to use the do_debian_package.sh in our git repo are available in our readthedocs site.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
All of these can be found at https://zoneminder.blogspot.com/
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
Yes, I found it ironic, myself... However, these particular cameras are just plain WEIRD... I've got two cheapo Jooan cams on the same server, with the four Ventech cams, and the Jooan cams were basically plug in, scan ONVIF, select stream, and BOOM! up and running. Nary a problem with them at all. The Jooan cams were 2/3 the cost of the Ventech ones... I'll give you two guesses which cams I'm going to buy in the future...knnniggett wrote: ↑Fri Feb 02, 2018 4:36 pm The irony is that the Remote RTSP method makes functions call to the ffmpeg API. Someday the Remote -> RTSP method may go away so a better long term solution would be to investigate why the ffmpeg method isn't working as it should. In the grand scheme of sheer number of cameras, the ffmpeg source type is far better.
Upgrading from ffmpeg 2.8 to 3.x series was a good first step. However, the trouble is that ffmpeg has a rapidly changing API. Upgrading ffmpeg on your system is not enough to get zoneminder to use the new(er) API calls in the new version of ffmpeg. You have to rebuild zoneminder to get that.
Assuming you can afford to play around with your system (i.e. not production), try to build your own zoneminder deb against the version of ffmpeg you have installed. Make sure ffmpeg 2.8 is completely removed before doing this. Instructions on how to use the do_debian_package.sh in our git repo are available in our readthedocs site.
As far as the API calls, as long as ZM is able to talk to FFMPEG, and I can get the video to my system, I don't feel a need to recompile... This is a production system, now... a few weeks ago I would have had time to play with it, but we've just got our other processes running, and the cams are necessary for remote monitoring.
I'm still wondering what the "extra" data that's being sent by the camera is... It doesn't seem to convert to anything ASCII, and I don't recognize the format.
Code: Select all
2018-02-02 13:33:44.070706 zmc_m4 24426 WAR 56: 74 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 38 3a 33 33 3a 34 34 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 39 45 43 45 35 30 39 0d 0a 0d 0a zm_rtsp.cpp 799
2018-02-02 13:33:44.022522 zmc_m4 24426 WAR Unexpected format RTSP interleaved data, resyncing by 56 bytes zm_rtsp.cpp 798
2018-02-02 12:50:48.075320 zmc_m4 24426 WAR 56: 74 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 37 3a 35 30 3a 34 37 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 39 45 43 45 35 30 39 0d 0a 0d 0a zm_rtsp.cpp 799
2018-02-02 12:50:48.008199 zmc_m4 24426 WAR Unexpected format RTSP interleaved data, resyncing by 56 bytes zm_rtsp.cpp 798
2018-02-02 12:17:11.071644 zmc_m1 25797 WAR 55: 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 37 3a 31 37 3a 31 31 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 31 37 37 30 45 32 39 41 0d 0a 0d 0a zm_rtsp.cpp 799
2018-02-02 12:17:11.017484 zmc_m1 25797 WAR Unexpected format RTSP interleaved data, resyncing by 55 bytes zm_rtsp.cpp 798
2018-02-02 12:04:07.118771 zmc_m1 25797 WAR 55: 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 37 3a 30 34 3a 30 37 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 31 37 37 30 45 32 39 41 0d 0a 0d 0a zm_rtsp.cpp 799
2018-02-02 12:04:07.068171 zmc_m1 25797 WAR Unexpected format RTSP interleaved data, resyncing by 55 bytes zm_rtsp.cpp 798
2018-02-02 11:55:50.083711 zmc_m3 24451 WAR 55: 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 36 3a 35 35 3a 34 39 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 35 35 42 32 30 33 31 0d 0a 0d 0a zm_rtsp.cpp 799
2018-02-02 11:55:50.017633 zmc_m3 24451 WAR Unexpected format RTSP interleaved data, resyncing by 55 bytes zm_rtsp.cpp 798
As I stated, though, the cams are working, now, using the "ancient" remote method..
-
knight-of-ni
- Posts: 2404
- Joined: Thu Oct 18, 2007 1:55 pm
- Location: Shiloh, IL
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
Some cameras have the ability to do their own motion detection. For those kinds of cameras, the problem becomes how to send that data to the receiving device. A popular way to do that is to send the motion detection meta data as a separate RTP (no S) stream on a different port, much like the old passive ftp transfers back in the day. That of course is very firewall un-friendly. One way to get around the firewall un-friendliness is to interleave the motion detection meta data directly into the stream itself. Of course, the problem then becomes how to un-interleave that data at the other end.
That might be the extra data zoneminder is complaining about.
In any case, and this is very speculative, but if your cameras have any settings regarding motion detection or RTP streams, try turning all that off. You want a basic RTSP stream and nothing else.
That might be the extra data zoneminder is complaining about.
In any case, and this is very speculative, but if your cameras have any settings regarding motion detection or RTP streams, try turning all that off. You want a basic RTSP stream and nothing else.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
All of these can be found at https://zoneminder.blogspot.com/
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
Interesting... because I've got all motion detection turned off in the cameras' firmware. That was part of my testing for a solution before coming upon this fix. At least it's SUPPOSED to be off; one of my complaints about these Ventech cams is that many times the settings don't "take" - they'll show as set, even after logging out and back in, but they don't actually take effect.
You're probably on to something with the interleaved data... and this is probably why the ffmpeg method fails, but the "ancient" remote method works in ZM. This is my guess, anyway.
I do appreciate the explanation, though, because you're helping to fill in some of the blanks in my understanding of how these things work.
You're probably on to something with the interleaved data... and this is probably why the ffmpeg method fails, but the "ancient" remote method works in ZM. This is my guess, anyway.
I do appreciate the explanation, though, because you're helping to fill in some of the blanks in my understanding of how these things work.
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
If you mean this line here ...
Code: Select all
2018-02-02 13:33:44.070706 zmc_m4 24426 WAR 56: 74 65 3a 20 46 72 69 2c 20 46 65 62 20 30 32 20 32 30 31 38 20 31 38 3a 33 33 3a 34 34 20 47 4d 54 0d 0a 53 65 73 73 69 6f 6e 3a 20 30 39 45 43 45 35 30 39 0d 0a 0d 0a
I make that as "te: Fri, Feb 02 2018 18:33:34 GMT(newline)Session: 09ECE509\r\n"
You are getting some headers in there (Date: ... Session: ...) by the look of it. Anyone else got a clue where they are coming from
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
These cameras are cheapos that are intended by Ventech to be used on their DVR/NVR... they're not "generic" cams...You are getting some headers in there (Date: ... Session: ...) by the look of it. Anyone else got a clue where they are coming from
The data I'd converted before (it wasn't the set I'd pasted) didn't seem to translate to anything, but this does, so I'm guessing that's housekeeping data for these cams to talk to their NVR/DVR...
Another guess: switching the cams from "ffmpeg" to REMOTE in ZM allows ZM to "ignore" that extra data, and keeps the cams from rebooting... I'm not sure if ZM is requesting resyncs from the cams when it loses its spot in ffmpeg or not, but this was the only way I could get them to work.
Thanks for your insight.
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
I apparently get the same issue - I have 8 of these, on zoneminder 1.32.3:
https://www.aliexpress.com/item/POE-Wat ... 55494.html
So a different camera - I wouldn't be surprised if it would be the same under the hood, especcially when the streaming address (../live0.264) is the same.
I get exactly the same issue - the cameras reboot often, nothing useful in the logs. Managed to log on via telnet (ipc71a default password for root) and load is ~17 but I guess this might be normal for embedded linux.
And after reading this thread I decided to try out different software. First cams ran for 24 hours with chinese CMS software, then I tried out Shinobi which also managed to run the cameras for few hours without any issues.
So the issue might be the same - zoneminder sending some response to some weird data from cams, which they can't handle and crash. I don't see that additional data in the zm logs though. Can some of you guys help me sort this out?
I already tried the fixes from this thread:
- I'm running dockerized ZM, ffmpeg is already 3.4.4.
- All alarm definitions in the cameras are off.
- tried setting zm to remote and settings from this thread, but this fails with:
https://www.aliexpress.com/item/POE-Wat ... 55494.html
So a different camera - I wouldn't be surprised if it would be the same under the hood, especcially when the streaming address (../live0.264) is the same.
I get exactly the same issue - the cameras reboot often, nothing useful in the logs. Managed to log on via telnet (ipc71a default password for root) and load is ~17 but I guess this might be normal for embedded linux.
And after reading this thread I decided to try out different software. First cams ran for 24 hours with chinese CMS software, then I tried out Shinobi which also managed to run the cameras for few hours without any issues.
So the issue might be the same - zoneminder sending some response to some weird data from cams, which they can't handle and crash. I don't see that additional data in the zm logs though. Can some of you guys help me sort this out?
I already tried the fixes from this thread:
- I'm running dockerized ZM, ffmpeg is already 3.4.4.
- All alarm definitions in the cameras are off.
- tried setting zm to remote and settings from this thread, but this fails with:
Code: Select all
Date/Time
Component Server PID Level Message File Line
2019-04-17 16:52:35 zmc_m1 487 INF Starting Capture version 1.32.3 zmc.cpp 223
2019-04-17 16:52:35 zmc_m1 488 INF Received new Content-Base in DESCRIBE response header. Updated device Url to: 'rtsp://192.168.1.101:554/live0.264/' zm_rtsp.cpp 355
2019-04-17 16:52:35 zmc_m1 488 WAR The device is using a codec that may not be supported. Do not be surprised if things don't work. zm_rtp_source.cpp 69
2019-04-17 16:52:35 zmc_m1 487 FAT Unable to locate video stream zm_remote_camera_rtsp.cpp 188
2019-04-17 16:52:24 zmc_m1 480 WAR The device is using a codec that may not be supported. Do not be surprised if things don't work. zm_rtp_source.cpp 69
2019-04-17 16:52:24 zmc_m1 480 INF Received new Content-Base in DESCRIBE response header. Updated device Url to: 'rtsp://192.168.1.101:554/live0.264/' zm_rtsp.cpp 355
2019-04-17 16:52:24 zmc_m1 478 INF Starting Capture version 1.32.3 zmc.cpp 223
2019-04-17 16:52:24 zmc_m1 478 FAT Unable to locate video stream zm_remote_camera_rtsp.cpp 188
2019-04-17 16:52:24 zmc_m1 478 INF Got signal 15 (Terminated), exiting zm_signal.cpp 40
2019-04-17 16:52:17 zmc_m1 476 FAT Unable to locate video stream zm_remote_camera_rtsp.cpp 188
2019-04-17 16:52:17 zmc_m1 477 WAR The device is using a codec that may not be supported. Do not be surprised if things don't work. zm_rtp_source.cpp 69
2019-04-17 16:52:17 zmc_m1 477 INF Received new Content-Base in DESCRIBE response header. Updated device Url to: 'rtsp://192.168.1.101:554/live0.264/' zm_rtsp.cpp 355
2019-04-17 16:52:17 zmc_m1 476 INF Starting Capture version 1.32.3 zmc.cpp 223
2019-04-17 16:52:14 zmc_m1 468 FAT Unable to locate video stream zm_remote_camera_rtsp.cpp 188
2019-04-17 16:52:14 zmc_m1 470 WAR The device is using a codec that may not be supported. Do not be surprised if things don't work. zm_rtp_source.cpp 69Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
Interesting... The initial issue was under 1.30... If you're running 1.32 now, and you have the choice in the camera itself between NTSC and PAL, you could give that a shot. I've found some cameras don't like to respond to ONVIF probes from within ZM unless they're set to PAL.
It may also be that the URL for the video stream on your particular cam is different...
I've since replaced all those Ventech cams with other, higher-resolution devices, and used them for percussive therapy, as they're pretty much crap. The Remote method in 1.32 seems to no longer work for these cams, and their constant rebooting (again, after the upgrade) got so annoying the percussive therapy became a necessity.
It may also be that the URL for the video stream on your particular cam is different...
I've since replaced all those Ventech cams with other, higher-resolution devices, and used them for percussive therapy, as they're pretty much crap. The Remote method in 1.32 seems to no longer work for these cams, and their constant rebooting (again, after the upgrade) got so annoying the percussive therapy became a necessity.
Re: SOLVED: Zoneminder causes Ventech cameras to reboot randomly
Also, I find myself wishing our cams were identical (and that I still had one for experimentation)... I'd been unable to locate the root password for mine.