Triggering a script on a ZM error ( ie Camera offline )

Discussions related to the 1.36.x series of ZoneMinder
Post Reply
Burtski
Posts: 28
Joined: Thu May 14, 2020 3:34 pm

Triggering a script on a ZM error ( ie Camera offline )

Post by Burtski »

Hello all,
I have version 1.36.25 running on a dedicated server with a total of 10 cameras feeding 20 monitors.
It is all quite stable.
I am using AMCREST cameras that seem to need to be power cycled from time to time.
Every few days I will have a camera that looses its mind. It will sort of respond but it will not return a valid image.
Simply commanding my POE switch to cycle the power to that camera will fix it for several days.
For the most part it is just annoying. But from time to time an important camera will go offline.

Whenever this happens I will get an entry in the log:
1/4/23, 7:10:28 PM CST zmc_m4 1529980 ERR Failed to capture image from monitor 4 Front_East (1/1) zmc.cpp 305

I am wondering if there is a way to have ZM itself trigger a script when this happens. (The script will cycle the power to the camera)

I have researched a number of ways of doing this by having an external job watch the ZM logs (swatch), but I thought that I would first check if there is some native way to facilitate this within ZM first.

Thanks
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: Triggering a script on a ZM error ( ie Camera offline )

Post by Magic919 »

I'm not aware of anything native.

Seen the approach of filtering the logs or parsing and then firing off alerts.
-
User avatar
iconnor
Posts: 2881
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Triggering a script on a ZM error ( ie Camera offline )

Post by iconnor »

There is in master. We can now trigger a camera reboot (assuming the camera is still alive at all). Looks like the Amcrest control script uses the wrong function for rebooting, but that could be easily fixed.
Burtski
Posts: 28
Joined: Thu May 14, 2020 3:34 pm

Re: Triggering a script on a ZM error ( ie Camera offline )

Post by Burtski »

Does this fire an external script or just try to do a reset via the interface?
In my case I need to run a short script that will cycle the power to that port in my POE switch.

(The problem does not go away if I issue a reset command, it seams to need a power cycle in my case)

Thanks
User avatar
iconnor
Posts: 2881
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Triggering a script on a ZM error ( ie Camera offline )

Post by iconnor »

It is a perl (PTZ Control script) level command. But you can script anything you like...

I think it would be nice if we integrated a ping/online state do action thing....

However you can do this yourself with monit. ping/or monitor actual web page contents if it goes offline run a script the power cycles it from the switch.
Burtski
Posts: 28
Joined: Thu May 14, 2020 3:34 pm

Re: Triggering a script on a ZM error ( ie Camera offline )

Post by Burtski »

I have stared down the path of tail-ing the zmc_mxx.log files.
Firstly, is it safe ( from a future upgrades point ov view ) to use 'zmc.cpp 305' as a match term in my script?
That is, will this remain the same as upgrades happen or should I search for 'ERR Failed to capture image from monitor' ?

I am also noticing that the log file appears to get written only every few minutes with lots of log entries.
Often the last line in the log will be just half a line.
Looks like the file buffer in not getting flushed after each entry.

(I am writing my script such that it will not care, just pointing it out )

{ zm 1.36.25. 10 Cameras ( mostly Amcrest ) feeding 20 monitors }

Thanks
User avatar
iconnor
Posts: 2881
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Triggering a script on a ZM error ( ie Camera offline )

Post by iconnor »

No, line numbers will likely change. I make no promises about text changes although it's fairly unlikely.

Logging is generally buffered unless in debug mode.

What I do is log warnings to syslog and use something like logcheck to email me anything out of the ordinary. Works well enough for me.
Burtski
Posts: 28
Joined: Thu May 14, 2020 3:34 pm

Re: Triggering a script on a ZM error ( ie Camera offline )

Post by Burtski »

Here is my solution that is working fairly well

It just tails syslog looking for events from ZoneMinder.
It parses the event for a line with "Failed to capture image from monitor"
Then figures out what monitor that came from.

I use Plivo to send me a text message when ever a hit occurs.

It turns out that from time to time a single frame will be missing without any ill effect so I just let it go
If there is a second missing frame in a 5 minute window I call a script that cycles the power to that camera's POE port on my switch.

This runs in a detached screen session.

This has been working pretty well.

This definitely could get cleaned up, but this fire is out. I have moved on to the next one. :roll:

BTW, the Amcrest cameras really need a power cycle. Sending the reset via http does not properly reset the cameras
I have 11 cameras and they all do this. Some more often than others.



Code: Select all

import subprocess
import plivo
from datetime import datetime
import time


class Camera:
    def __init__(self,identifier,ip,port):
        self.MuteTime= 5 * 60 # 5 minute mute time.
        self.ResetHoldoffTime = 2 * 60
        self.Muted = False
        self.UnmuteTime = 0
        self.ResetTime = 0
        self.Identifier = identifier
        self.port = port
        self.ip = ip
        self.LastErrorTime = 0

def SendText(cam,reason):
    auth_id = '11111111111111111111111111'
    auth_token = '222222222222222222222'

    client = plivo.RestClient(auth_id,auth_token)
    response = client.messages.create(
        src = '3333333333',
        dst = '4444444444',
        text = f'Camera {cam.Identifier} on port {cam.port} \n {reason}',
    )



def processZMevent(line):
    if 'Failed to capture image from monitor' in line:
        c = line.split()
        CurrentYear = str(datetime.now().year)
        i = CurrentYear + ' '+c[0]+' '+c[1]+' '+c[2]
        SyslogTime = datetime.strptime(i,'%Y %b %d %H:%M:%S')
        SyslogTimestamp = datetime.timestamp(SyslogTime)
        CurrentTimeStamp = time.time()
        for cam in CameraList:
            if cam.Identifier in line:
                print(line)
                if CurrentTimeStamp > cam.UnmuteTime:
                    cam.UnmuteTime = CurrentTimeStamp + cam.MuteTime

                    SendText(cam,'First Missing Frame')
                    print("Sent Text")

                else:
                    print("Got another hit during the mute time")
                    # Got antoher alarm during mute time
                    if CurrentTimeStamp > cam.ResetTime:
                        #It has been a while since we sent a reset
                        cmd = f'./cycle_power {cam.port}'
                        subprocess.run(cmd,shell = True)
                        cam.ResetTime = CurrentTimeStamp + cam.ResetHoldoffTime
                        SendText(cam,'Reset port sent')
                    else:
                        print("Reset in progress, do nothing")

                cam.LastErrorTime = time.time()

if __name__ == '__main__':

    CameraList = []
    CameraList.append(Camera('[zmc_m13]',148,20))
    CameraList.append(Camera('[zmc_m32]',104,5))
    CameraList.append(Camera('[zmc_m14]',110,1))
    CameraList.append(Camera('[zmc_m15]',114,17))
    CameraList.append(Camera('[zmc_m17]',113,19))
    CameraList.append(Camera('[zmc_m18]',127,22))
    CameraList.append(Camera('[zmc_m19]',117,14))
    CameraList.append(Camera('[zmc_m20]',62,00))
    CameraList.append(Camera('[zmc_m25]',107,13))
    CameraList.append(Camera('[zmc_m29]',108,15))
    CameraList.append(Camera('[zmc_m30]',120,1))

    f = subprocess.Popen(['tail','-F','/var/log/syslog'],\
                     stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    while True:
        line = f.stdout.readline().decode("utf-8")
        if 'zm' in line:
            #print ('Zone Minder event')
            # print (line)
            processZMevent(line)


Post Reply