zmeventnotification stopped working

Discussions related to the 1.36.x series of ZoneMinder
Helipil0t
Posts: 10
Joined: Tue May 12, 2020 5:21 pm

zmeventnotification stopped working

Post by Helipil0t »

I'm not sure what's going on here but seemingly overnight my zoneminder setup no longer detects object / faces. What's more, every once in a while the server will crash and all streams will drop. zmc process is killed and drop off the GPU.

I'm running 1.36.32. I tried installing the latest version of zmeventnotifications. Testing an event ID I get the following:

Code: Select all

sudo -u www-data /var/lib/zmeventnotification/bin/zm_detect.py --config /etc/zm/objectconfig.ini  --eventid 26449 --monitorid 4 --debug
11/27/22 09:26:19 zmesdetect_m4[44627] INF ZMLog.py:292 [Setting up signal handler for logs]

11/27/22 09:26:19 zmesdetect_m4[44627] INF ZMLog.py:301 [Switching global logger to ZMLog]

11/27/22 09:26:19 zmesdetect_m4[44627] INF zm_detect.py:284 [---------| app:6.1.28, pyzm:0.3.56, ES:6.1.28
 , OpenCV:4.5.3|------------]

11/27/22 09:26:19 zmesdetect_m4[44627] INF utils.py:404 [Reading config from: /etc/zm/objectconfig.ini]

11/27/22 09:26:19 zmesdetect_m4[44627] INF utils.py:409 [Reading secrets from: /etc/zm/secrets.ini]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_PORTAL]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_USER]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_PASSWORD]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_API_PORTAL]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ML_USER]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ML_PASSWORD]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !PLATEREC_ALPR_KEY]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG1 utils.py:444 [allowing self-signed certs to work...]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:454 [Now checking for monitor overrides]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG3 utils.py:521 [Finally, doing parameter substitution]

11/27/22 09:26:19 zmesdetect_m4[44627] INF zm_detect.py:309 [Importing local classes for Object/Face]

11/27/22 09:26:20 zmesdetect_m4[44627] INF zm_detect.py:334 [Connecting with ZM APIs]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 api.py:72 [API SSL certificate check has been disbled]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 api.py:181 [using username/password for login]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 api.py:210 [Using new token API]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 api.py:217 [Access token expires on:2022-11-27 11:26:20.365089 [7200s]]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 api.py:221 [Refresh token expires on:2022-11-28 09:26:20.365727 [86400s]]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 zm_detect.py:342 [using ml_sequence]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 zm_detect.py:354 [using stream_sequence]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:160 [Resetting models, will be loaded on next run]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 detect_sequence.py:634 [Using automatic locking as we are switching between models]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:51 [Media get SSL certificate check has been disbled]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:99 [Using URL 26449 for stream]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:114 [We will only process frames: ['snapshot', 'alarm']]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:137 [No need to start streams, we are picking images from https://seanstech.ca:64443/zm/index.php?view=image&eid=26449]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 Media.py:271 [Reading https://seanstech.ca:64443/zm/index.php?view=image&eid=26449&fid=snapshot]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 api.py:144 [No need to relogin as access token still has 119.99981745 minutes remaining]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 api.py:272 [make_request called with url=https://seanstech.ca:64443/zm/index.php?view=image&eid=26449&fid=snapshot payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNjY5NTU5MTgwLCJleHAiOjE2Njk1NjYzODAsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ.ZOOUbYGmc9fWPJ7xM9bJ_it9BIuOR0YTxEhAM9W92S4'}]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:654 [perf: Starting for frame:snapshot]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:664 [Sequence of detection types to execute: ['object', 'face', 'alpr']]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:669 [============ Frame: snapshot Running object detection type in sequence ==================]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 detect_sequence.py:174 [Skipping TPU object detection as it is disabled]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 detect_sequence.py:178 [Loading sequence: YoloV4 GPU/CPU]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 detect_sequence.py:179 [Initializing model  type:object with options:{'name': 'YoloV4 GPU/CPU', 'enabled': 'yes', 'object_config': '/var/lib/zmeventnotification/models/yolov4/yolov4.cfg', 'object_weights': '/var/lib/zmeventnotification/models/yolov4/yolov4.weights', 'object_labels': '/var/lib/zmeventnotification/models/yolov4/coco.names', 'object_min_confidence': 0.3, 'object_framework': 'opencv', 'object_processor': 'gpu', 'gpu_max_processes': 1, 'gpu_max_lock_wait': 100, 'cpu_max_processes': 3, 'cpu_max_lock_wait': 100, 'max_detection_size': '90%', 'disable_locks': 'no'}]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:38 [portalock: max:1, name:pyzm_uid33_gpu_lock, timeout:100]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 detect_sequence.py:689 [object has a same_model_sequence strategy of first]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:701 [--------- Frame:snapshot Running variation: #1 -------------]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:136 [detect extracted image dimensions as: 800wx1066h]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:50 [Waiting for pyzm_uid33_gpu_lock portalock...]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:52 [Got pyzm_uid33_gpu_lock portalock]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 yolo.py:85 [|--------- Loading "YoloV4 GPU/CPU" model from disk -------------|]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 yolo.py:100 [perf: processor:gpu Yolo initialization (loading /var/lib/zmeventnotification/models/yolov4/yolov4.weights model from disk) took: 250.32 ms]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:115 [Setting CUDA backend for OpenCV]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 yolo.py:116 [If you did not set your CUDA_ARCH_BIN correctly during OpenCV compilation, you will get errors during detection related to invalid device/make_policy]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 yolo.py:160 [|---------- YOLO (input image: 800w*1066h, model resize dimensions: 416w*416h) ----------|]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 yolo.py:68 [Released pyzm_uid33_gpu_lock portalock]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG1 yolo.py:185 [perf: processor:gpu Yolo detection took: 9649.06 ms]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 yolo.py:220 [perf: processor:gpu Yolo NMS filtering took: 4.14 ms]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:66 [core model detection over, got 4 objects. Now filtering]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:70 [Max object size found to be: 90%]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:78 [Converted 90% to 767520.0]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:101 [Ignoring car [520, 564, 592, 618] as conf. level 0.28317883610725403 is lower than 0.3]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:93 [Ignoring object:aeroplane, as it's area: 871700px exceeds max_object_area of 767520.0px]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:103 [Returning filtered list of 2 objects.]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:711 [This model iteration inside object found: labels: ['person', 'suitcase'],conf:[0.9284842014312744, 0.6206374168395996]]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:424 [Max object size found to be: 90%]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:432 [Converted 90% to 767520.0]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:448 [No polygons, adding full image polygon: {'name': 'full_image', 'value': [(0, 0), (1536, 0), (1536, 2048), (0, 2048)], 'pattern': None}]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG3 detect_sequence.py:228 [resized polygons x=0.5208333333333334/y=0.5205078125: [{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 1066), (0, 1066)], 'pattern': None}]]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:502 [intersection: object:person,POLYGON ((526 426, 796 426, 796 1006, 526 1006, 526 426)) intersects polygon:full_image,POLYGON ((0 0, 800 0, 800 1066, 0 1066, 0 0))]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:518 [Using global match pattern: (person|car|motorbike|bus|truck|boat)]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:526 [full_image intersects object:person[[(526, 426), (796, 426), (796, 1006), (526, 1006)]]]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:502 [intersection: object:suitcase,POLYGON ((365 863, 631 863, 631 1049, 365 1049, 365 863)) intersects polygon:full_image,POLYGON ((0 0, 800 0, 800 1066, 0 1066, 0 0))]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:518 [Using global match pattern: (person|car|motorbike|bus|truck|boat)]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:535 [full_image intersects object:suitcase[[(365, 863), (631, 863), (631, 1049), (365, 1049)]] but does NOT match your detect pattern filter]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:753 [breaking out of same model loop, as matches found and strategy is "first"]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG1 detect_sequence.py:669 [============ Frame: snapshot Running face detection type in sequence ==================]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:190 [Skipping TPU face detection as it is disabled]

Illegal instruction
Clearly it detects the object. But then stops with "Illegal instruction"

When the server crashes I have to restart the zoneminder service in order to get the monitors back online until the next detection runs.. Then it may or may not crash again.
How can I further diagnose this issue?

Thanks!
Helipil0t
Posts: 10
Joined: Tue May 12, 2020 5:21 pm

Re: zmeventnotification stopped working

Post by Helipil0t »

Quick update.. The crash I believe is as a result of running out of ram. dmesg shows:

Code: Select all

Out of memory: Killed process 7665 (zmc) total-vm:8454820kB, anon-rss:488988kB, file-rss:132124kB, shmem-rss:488056kB, UID:33 pgtables:5412kB oom_score_adj:0


I'll up my memory a bit. Should I set "Maximum Image Buffer Size (frames):" ?
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: zmeventnotification stopped working

Post by Magic919 »

I think it’s better to constrain it. I never get oom killed.
-
Helipil0t
Posts: 10
Joined: Tue May 12, 2020 5:21 pm

Re: zmeventnotification stopped working

Post by Helipil0t »

K so that sorts out the crashes. Still can't get zmeventnotification working properly. No idea what's going on with it. The debug doesn't share too many clues.
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: zmeventnotification stopped working

Post by Magic919 »

ZMES is clearly finding objects.

I’d suggest running the instance on the commandline to see the debug. Then manually running the check above and the others that are suggested in the docs. Something must show up.
-
Helipil0t
Posts: 10
Joined: Tue May 12, 2020 5:21 pm

Re: zmeventnotification stopped working

Post by Helipil0t »

Ok so I think there's issues with yolo.py:

Code: Select all

11/28/22 01:35:44 zmesdetect_m3[1310] DBG2 detect_sequence.py:714 [Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/detect_sequence.py", line 710, in detect_stream
    _b,_l,_c,_m = m.detect(image=frame)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/object.py", line 65, in detect
    b,l,c,_model_names = self.model.detect(image)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 168, in detect
    ln = self.get_output_layers()
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 129, in get_output_layers
    output_layers = [
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 130, in <listcomp>
    layer_names[i[0] - 1] for i in self.net.getUnconnectedOutLayers()
IndexError: invalid index to scalar variable.
This is running the latest zmeventnotification from github. So the "hotfix" no longer works. Running OpenCV 4.6.0-dev, Cuda 11.8
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: zmeventnotification stopped working

Post by Magic919 »

Ok.
-
Helipil0t
Posts: 10
Joined: Tue May 12, 2020 5:21 pm

Re: zmeventnotification stopped working

Post by Helipil0t »

So I thought this code was fixed a while ago with the following:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        if self.is_get_unconnected_api_list:
            output_layers = [
                layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        else:
            output_layers = [
                layer_names[i[0] - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        return output_layers
I removed the if else statement and just used:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        output_layers = [
            layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
        ]
        return output_layers
Run again and get the following error:

Code: Select all

11/28/22 10:41:10 zmesdetect_m3[4068] DBG2 detect_sequence.py:714 [Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/detect_sequence.py", line 710, in detect_stream
    _b,_l,_c,_m = m.detect(image=frame)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/object.py", line 65, in detect
    b,l,c,_model_names = self.model.detect(image)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 225, in detect
    i = i[0]
IndexError: invalid index to scalar variable.
I have Python 3.10.6.

Maybe I should use an older version of Python?
Last edited by Helipil0t on Mon Nov 28, 2022 4:19 pm, edited 1 time in total.
Helipil0t
Posts: 10
Joined: Tue May 12, 2020 5:21 pm

Re: zmeventnotification stopped working

Post by Helipil0t »

Update:

I commented out the if statement in the following:

Code: Select all

        # now filter out with configured yolo confidence, so we can see rejections in log
        for i in indices:
           # if not self.is_get_unconnected_api_list:
           #     i = i[0]
            box = boxes[i]
            x = box[0]
            y = box[1]
            w = box[2]
            h = box[3]
And everything seems to be working again.
Wish I knew more about what I was doing. :P Is it the python interpreter not handing indices the same way?
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: zmeventnotification stopped working

Post by Magic919 »

I've no idea.

There is some mention of the topic by Asker. He understands it :-)

viewtopic.php?t=31129
-
lorenjz
Posts: 40
Joined: Fri Feb 28, 2020 12:25 am

Re: zmeventnotification stopped working

Post by lorenjz »

I'm having the same issue however, commenting out the if statement mentioned here
Helipil0t wrote: Mon Nov 28, 2022 4:14 pm Update:

I commented out the if statement in the following:

Code: Select all

        # now filter out with configured yolo confidence, so we can see rejections in log
        for i in indices:
           # if not self.is_get_unconnected_api_list:
           #     i = i[0]
            box = boxes[i]
            x = box[0]
            y = box[1]
            w = box[2]
            h = box[3]
And everything seems to be working again.
Wish I knew more about what I was doing. :P Is it the python interpreter not handing indices the same way?
or the else if mentioned here:
Helipil0t wrote: Mon Nov 28, 2022 3:46 pm So I thought this code was fixed a while ago with the following:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        if self.is_get_unconnected_api_list:
            output_layers = [
                layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        else:
            output_layers = [
                layer_names[i[0] - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        return output_layers
I removed the if else statement and just used:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        output_layers = [
            layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
        ]
        return output_layers
Run again and get the following error:

Code: Select all

11/28/22 10:41:10 zmesdetect_m3[4068] DBG2 detect_sequence.py:714 [Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/detect_sequence.py", line 710, in detect_stream
    _b,_l,_c,_m = m.detect(image=frame)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/object.py", line 65, in detect
    b,l,c,_model_names = self.model.detect(image)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 225, in detect
    i = i[0]
IndexError: invalid index to scalar variable.
I have Python 3.10.6.

Maybe I should use an older version of Python?
change the error but no matter the combination of commented or not for either or both don't allow a successful analysis. Following along for any further developments....
Magic919
Posts: 1381
Joined: Wed Sep 18, 2013 6:56 am

Re: zmeventnotification stopped working

Post by Magic919 »

You might get help in the Slack channel.
-
volfan6415
Posts: 8
Joined: Tue Apr 03, 2018 1:10 pm

Re: zmeventnotification stopped working

Post by volfan6415 »

Is there a fix for this? I am having the same issue.
eresonance
Posts: 1
Joined: Sun Feb 19, 2023 6:25 pm

Re: zmeventnotification stopped working

Post by eresonance »

I have python3.11 and OpenCV 4.7.0 and I see the same issue. I think there's a problem detecting the opencv version.

Open up the yolo.py script:

Code: Select all

sudo vim /usr/local/lib/python3.11/site-packages/pyzm/ml/yolo.py
Go to line 95:

Code: Select all

        (maj, minor, patch) = cv2.__version__.split('.')
        min_ver = int(maj + minor)
        patch = int(patch) if patch.isdigit() else 0
        if min_ver >= 45 and patch >=4:
            # see https://github.com/opencv/opencv/issues/20923
            # we need to modify Yolo code not to expect a nested structure
            g.logger.Debug(1, 'You are using OpenCV >= 4.5.4, making sure we fix getUnconnectedOutLayers() API')
            self.is_get_unconnected_api_list = True
We want that to look like:

Code: Select all

        (maj, minor, patch) = cv2.__version__.split('.')
        min_ver = int(maj + minor + patch)
        if min_ver >= 454:
            # see https://github.com/opencv/opencv/issues/20923
            # we need to modify Yolo code not to expect a nested structure
            g.logger.Debug(1, 'You are using OpenCV >= 4.5.4, making sure we fix getUnconnectedOutLayers() API')
            self.is_get_unconnected_api_list = True
That will handle the case where your minor version is > 5 but your patch is < 4.
Pedulla
Posts: 167
Joined: Thu Nov 27, 2014 11:16 am
Location: Portland, Or

Re: zmeventnotification stopped working

Post by Pedulla »

On a 1.36.33 ZM server...
04/30/23 23:52:47 zmesdetect_m10[881377] ERR detect_sequence.py:713 [Error running model: invalid index to scalar variable.]
using

Code: Select all

>>> from cv2 import __version__
>>> __version__
'4.7.0'
Is this something else?

New install of ES, wiped out the models and reinstalled just in case and fix above runs, but does not solve.
Post Reply