ZM errors in log

[alabamatoy]

Ubuntu 16.04, ZM 1.30.0

Once in a while, something goes awry with one of my monitors. I will see in the logs:

Code: Select all

Jul 17 02:44:01 signal-security zmdc[14770]: WAR ['zmc -m 10' has not stopped at 18/07/17 02:44:01. Sending KILL to pid 14849]
Jul 17 02:44:01 signal-security zmc_m10[31586]: WAR [Unable to parse ffmpeg option 0 '', expecting key=value]
Jul 17 02:44:12 signal-security zmc_m10[31586]: ERR [Open video took more than 10 seconds.]
Jul 17 02:44:12 signal-security zmc_m10[31586]: ERR [Unable to open input rtsp://login:password@10.1.1.168:554/cam/realmonitor?channel=1&subtype=0 due to: Connection timed out]
Jul 17 02:44:12 signal-security zma_m10[14852]: WAR [Signal: Lost]
Jul 17 02:44:12 signal-security zmc_m10[31586]: WAR [Unable to parse ffmpeg option 0 '', expecting key=value]
Jul 17 02:44:12 signal-security zmc_m10[31586]: ERR [Unable to open input rtsp://login:password@10.1.1.168:554/cam/realmonitor?channel=1&subtype=0 due to: Operation now in progress]
Jul 17 02:44:12 signal-security zmc_m10[31586]: WAR [Buffer overrun at index 3, image 53, slow down capture, speed up analysis or increase ring buffer size]

In most cases, this is followed by hundreds if not thousands of lines of "Buffer overrun at..." entries, punctuated by a "...zma may have gone away" message once per second. Usually, after about 10 minutes (and thousands of lines of crap in the log) I will see "Signal: Reacquired" and everything settles down again.

Does anyone have any suggestions for how to fix this?

[knight-of-ni]

Buffer overrun always means the system is being driven too hard. One or more components are not able to keep up.

However, it's unclear what happened first... did the camera go offline and then the cpu blew up trying to reconnect, or are you running your system right on the edge of stability, where an occasional hiccup causes the house of cards to collapse? Providing a full page of logs, including the moment just before things went awry would help tell us that. Also, showing us your system load, /dev/shm, etc is needed too.

If we assume the problem is the former, then the current work-around is to place a value in the max fps that is 1 fps higher than the incoming stream from each camera. This is easy to do so you might as well try it then observe whether or not that fixes the issue.

[alabamatoy]

I dont think the system is anywhere near the edge. The webmin dashboard generally shows less than 50% CPU (even when Im streaming some video and using webmin etc), and a/o couple seconds ago it showed "CPU load averages: 2.37 (1 min) 2.22 (5 mins) 2.29 (15 mins)"

In the log snip that I pasted, looking before the 02:44:01 WAR entry, there was nothing for 30 minutes prior, and about 15 minutes before that was a zmfilter.pl junk alarm cleanup, and before that was about 2 hours of no log entries, all was quiet on the eastern front.

Intuitively, it seems to me that you are correct with "did the camera go offline and then the cpu blew up trying to reconnect". I'm thinking maybe I need a log watcher (swatch?) and when a bunch of these buffer overruns appear, I do a ZM restart? Other suggestions?

[alabamatoy]

This is pretty typical...

[iconnor]

The first few lines tell all. It is taking more than 10 seconds to connect. This is a reconnect attempt, so zma is waiting for there to be frames, but zmc can't get any from the camera.

[alabamatoy]

This is kinda what I figured. So I need to configure something which would recognize this condition based on the log entries (I'm leaning towards swatch) and bounce the cameras (I can cycle power to the cameras via CLI).

I dont know why the cameras will hang like this sometimes. It seems like they go off and smoke a cigarette or something, and come back 15-30 minutes later.