ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Discussion topics related to mobile applications and ZoneMinder Event Server (including machine learning)
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

I removed those deps, they were in the mlapi_config file, rm -rf the mlapi folder and clone again.

I just pushed updates for all 3 repos so rm -rf all 3 folders and reclone then reinstall.

futures was left over from maybe doing a python 2.7 compat but pyzm is python3 anyways so I decided to just stick with python 3.6+ (f-strings)
juan11perez
Posts: 64
Joined: Tue Apr 27, 2021 3:41 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by juan11perez »

thank you. So I got the mlapi image running. I tried it with the zoneminder/zmes in my current production unit and got the following errors:

Code: Select all

07/31/21 16:23:26.292015 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '1' ]

07/31/21 16:23:26.292257 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'match_past_detections' is an unknown key, adding to monitor '1' config ]

07/31/21 16:23:26.292453 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'model_sequence' is an unknown key, adding to monitor '1' config ]

07/31/21 16:23:26.29264 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-1] overrides key 'object_detection_pattern' = '(person|monitor_doorbell)' ]

07/31/21 16:23:26.292817 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '2' ]

07/31/21 16:23:26.293012 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'match_past_detections' is an unknown key, adding to monitor '2' config ]

07/31/21 16:23:26.293197 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'model_sequence' is an unknown key, adding to monitor '2' config ]

07/31/21 16:23:26.293375 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-2] overrides key 'object_detection_pattern' = '(person)' ]

07/31/21 16:23:26.293549 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '3' ]

07/31/21 16:23:26.293748 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'match_past_detections' is an unknown key, adding to monitor '3' config ]

07/31/21 16:23:26.293932 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'model_sequence' is an unknown key, adding to monitor '3' config ]

07/31/21 16:23:26.29411 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-3] overrides key 'object_detection_pattern' = '(person)' ]

07/31/21 16:23:26.294284 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '4' ]

07/31/21 16:23:26.294477 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'match_past_detections' is an unknown key, adding to monitor '4' config ]

07/31/21 16:23:26.294671 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'model_sequence' is an unknown key, adding to monitor '4' config ]

07/31/21 16:23:26.294852 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-4] overrides key 'object_detection_pattern' = '(person)' ]

07/31/21 16:23:26.295026 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '5' ]

07/31/21 16:23:26.295227 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'match_past_detections' is an unknown key, adding to monitor '5' config ]

07/31/21 16:23:26.295415 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'model_sequence' is an unknown key, adding to monitor '5' config ]

07/31/21 16:23:26.295593 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-5] overrides key 'object_detection_pattern' = '(person)' ]

07/31/21 16:23:26.295771 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '6' ]

07/31/21 16:23:26.295963 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'match_past_detections' is an unknown key, adding to monitor '6' config ]

07/31/21 16:23:26.296147 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'model_sequence' is an unknown key, adding to monitor '6' config ]

07/31/21 16:23:26.29635 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-6] overrides key 'object_detection_pattern' = '(person|monitor_office)' ]

07/31/21 16:23:26.296548 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '9998' ]

07/31/21 16:23:26.296769 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'model_sequence' is an unknown key, adding to monitor '9998' config ]

07/31/21 16:23:26.296975 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-9998] overrides key 'object_detection_pattern' = '(person|monitor_doorbell)' ]

07/31/21 16:23:26.297174 CNSL[DBG2] mlapi_utils:198->[mlapi:util: adding polygon: 'valid_face_area' [184,235 1475,307 1523,1940 146,1940] ]

07/31/21 16:23:26.297417 CNSL[DBG2] mlapi_utils:200->[mlapi:util: 'match_past_detections' is an unknown key, adding to monitor '9998' config ]

07/31/21 16:23:26.297705 CNSL[DBG2] mlapi_utils:157->[mlapi:util: found monitor specific section for monitor '9999' ]

07/31/21 16:23:26.298001 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-9999] overrides key 'object_detection_pattern' = '(person|monitor_deck)' ]

07/31/21 16:23:26.298282 CNSL[DBG2] mlapi_utils:188->[mlapi:util: [monitor-9999] overrides key 'stream_sequence' = '{

'frame_strategy': 'most_models',

'frame_set': 'alarm',

'contig_frames_before_error': 5,

'max_attempts': 3,

'sleep_between_attempts': 4,

'resize':800


}' ]

07/31/21 16:23:26.29859 CNSL[DBG2] mlapi_utils:219->[mlapi:util: doing parameter substitution for globals ]

07/31/21 16:23:26.299392 CNSL[DBG2] mlapi_utils:253->[mlapi:util: doing parameter substitution for monitor specific entities ]

07/31/21 16:23:26.303254 CNSL[DBG1] db:14->[db: opening data base at /config/db/db.json ]

07/31/21 16:23:26.303657 CNSL[DBG1] db:18->[db: engine ready ]

07/31/21 16:23:26.306625 CNSL[DBG2] api:83->[pyzm:api: SSL certificate verification disabled (encryption still works, vuln to spoofing) ]

07/31/21 16:23:26.307014 CNSL[DBG1] api:199->[pyzm:api: no token found, using user/pass to login ]

07/31/21 16:23:26.393426 CNSL[DBG2] api:228->[pyzm:api: detected API ver 2.0+, using token system ]

07/31/21 16:23:26.393682 CNSL[DBG1] api:236->[pyzm:api: access token expires on: 2021-07-31 18:23:26.393462 (7200s) ]

07/31/21 16:23:26.393956 CNSL[DBG1] api:243->[pyzm:api: refresh token expires on: 2021-08-01 16:23:26.393696 (86400s) ]

07/31/21 16:23:26.394262 CNSL[DBG2] api:303->[pyzm:api:make_req: 'get'->http://192.168.1.100:8088/zm/api/zones.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]

07/31/21 16:23:26.398801 CNSL[DBG1] mlapi_utils:42->[mid = '1' g.config['import_zm_zones'] = 'no' g.monitor_config.get(mid,{}).get('import_zm_zones') = None ]

07/31/21 16:23:26.39903 CNSL[DBG1] mlapi_utils:42->[mid = '2' g.config['import_zm_zones'] = 'no' g.monitor_config.get(mid,{}).get('import_zm_zones') = None ]

07/31/21 16:23:26.399212 CNSL[DBG1] mlapi_utils:42->[mid = '3' g.config['import_zm_zones'] = 'no' g.monitor_config.get(mid,{}).get('import_zm_zones') = None ]

07/31/21 16:23:26.399392 CNSL[DBG1] mlapi_utils:42->[mid = '4' g.config['import_zm_zones'] = 'no' g.monitor_config.get(mid,{}).get('import_zm_zones') = None ]

07/31/21 16:23:26.399573 CNSL[DBG1] mlapi_utils:42->[mid = '5' g.config['import_zm_zones'] = 'no' g.monitor_config.get(mid,{}).get('import_zm_zones') = None ]

07/31/21 16:23:26.399755 CNSL[DBG1] mlapi_utils:42->[mid = '6' g.config['import_zm_zones'] = 'no' g.monitor_config.get(mid,{}).get('import_zm_zones') = None ]


07/31/21 16:23:26.400109 CNSL[DBG2] mlapi:424->[mlapi: preloading mlapiconfig.ini ml_sequence ]

07/31/21 16:23:26.400923 CNSL[DBG1] detect_sequence:221->[pyzm:ml:force_reload-> pre-loading models! ]

07/31/21 16:23:26.401372 CNSL[DBG2] detect_sequence:256->[pyzm:ml: loading 'object' sequence 'TPU object detection' (1 of 2) with options-> {'name': 'TPU object detection', 'enabled': 'yes', 'object_weights': '/var/lib/zmeventnotification/models/coral_edgetpu/ssdlite_mobiledet_coco_qat_postprocess_edgetpu.tflite', 'object_labels': '/var/lib/zmeventnotification/models/coral_edgetpu/coco_indexed.names', 'object_min_confidence': 0.6, 'object_framework': 'coral_edgetpu', 'tpu_max_processes': 3, 'tpu_max_lock_wait': 120, 'max_detection_size': '90%', 'show_models': 'no', 'disable_locks': 'no'} ]

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2464, in __call__

    return self.wsgi_app(environ, start_response)

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2450, in wsgi_app

    response = self.handle_exception(e)

  File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 271, in error_router

    return original_handler(e)

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1867, in handle_exception

    reraise(exc_type, exc_value, tb)

  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 38, in reraise

    raise value.with_traceback(tb)

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app

    response = self.full_dispatch_request()

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request

    rv = self.handle_user_exception(e)

  File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 271, in error_router

    return original_handler(e)

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception

    reraise(exc_type, exc_value, tb)

  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 38, in reraise

    raise value.with_traceback(tb)

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request

    rv = self.dispatch_request()

  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

  File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper

    resp = resource(*args, **kwargs)

  File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 89, in view

    return self.dispatch_request(*args, **kwargs)

  File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request

    resp = meth(*args, **kwargs)

  File "/usr/local/lib/python3.8/dist-packages/flask_jwt_extended/view_decorators.py", line 108, in wrapper

    return fn(*args, **kwargs)

  File "/var/lib/zmeventnotification/mlapi.py", line 225, in post

    stream_options = template_fill(input_str=stream_options, config=None, secrets=secrets_conf.get('secrets'))

  File "/usr/local/lib/python3.8/dist-packages/pyzm/helpers/pyzm_utils.py", line 366, in template_fill

    res = re.sub(p, lambda m: secrets.get(m.group(1).lower(), '!{}'.format(m.group(1).lower())), res)

  File "/usr/lib/python3.8/re.py", line 210, in sub

    return _compile(pattern, flags).sub(repl, string, count)

TypeError: expected string or bytes-like object
Not sure if this mlapi is designed to work only with its companion baudneo/zmes only. zmes is completing the inference via fallback (no tpu).

the other error I noticed was that when I first run the docker without connecting it to zmes, if I activated the coral it would immediately crash and I think it's because I have a pcie based Coral and your USB reset script is looking for a usb based coral. Below find the error

Code: Select all

ValueError: Failed to load delegate from libedgetpu.so.1



During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/var/lib/zmeventnotification/mlapi.py", line 430, in <module>

    m = DetectSequence(options=ml_options, global_config=g.config)

  File "/usr/local/lib/python3.8/dist-packages/pyzm/ml/detect_sequence.py", line 170, in __init__

    self.set_ml_options(options, force_reload=True) # processing

  File "/usr/local/lib/python3.8/dist-packages/pyzm/ml/detect_sequence.py", line 225, in set_ml_options

    self.models[model_name][idx].load_model()

  File "/usr/local/lib/python3.8/dist-packages/pyzm/ml/object.py", line 53, in load_model

    return self.model.load_model()

  File "/usr/local/lib/python3.8/dist-packages/pyzm/ml/coral_edgetpu.py", line 118, in load_model

    g.logger.Debug(f"coral: found tpu in USB FileSystem as -> {get_tpu()}")

  File "/usr/local/lib/python3.8/dist-packages/pyzm/helpers/usb_reset.py", line 30, in get_tpu

    proc = subprocess.Popen(['lsusb'], stdout=subprocess.PIPE)

  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__

    self._execute_child(args, executable, preexec_fn, close_fds,

  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child

    raise child_exception_type(errno_num, err_msg, err_filename)

FileNotFoundError: [Errno 2] No such file or directory: 'lsusb'

*** /etc/my_init.d/40_firstrun.sh failed with status 1


*** Killing all processes...

Jul 31 16:32:05 c3009129e448 syslog-ng[16]: syslog-ng shutting down; version='3.25.1'
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Yes there is an issue with mlapi I was up late trying to fix. It's the {{ var substitution }} in config and logic is screwey.

The usb error is happening because I haven't had a chance to catch the error you see at first "delegate error". That error usually means the system can't talk to the TPU properly. On usb TPU that error usually means a usb reset then a restart of mlapi. I will look into it asap but I don't have access to a PCIe TPU. One thing to check is to run the example that coral uses in their install.guide, the parrot detection test, and make sure everything is functioning as expected. That xdelefate error" makes me think there's an issue with the drivers or hardware config, BUT I am just guessing there

The first issue, I am just reading over your debug log snippet and will sort it out. Mlapi does have an issue with "parameter substitution" I am working at right now, I will push an update here shortly. There is a good chance mlapi is compatible with old zmes as long as using my forked pyzm, I haven't tested that as of yet.
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Just pushed, hopefully everything should work as expected.

Let me know about that delegate error plz.
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Working on a restart wrapper for mlapi, the idea behind it is if the USB TPU hits a delegate error, the script will reset the TPU port then restart mlapi and preload the weights so its automagically good to go after a crash. I am also working on yolo error restarting (cannot convert float *inf to int()). It is aware of if systemD .service started it and then allows systemD or whatever wrapper to restart it instead of it handling restarts itself, you just need to pass it the -fs or --from-service arg.

Also the usb_reset module now works as designed. I just need to see how it works when catching the delegate error, PCI devices should be unaffected by USB reset module. I will research if there is a way to reset a PCIe device and if there is I will work on adding that.

Here is some log output, the keyboard interrupt error is just so I can show the wrapper works, in production, keyboard interrupt will exit the script and not be restarted. Notice the 2nd loading of TPU weights is only 5-10ms instead of 3500ms+. :twisted:

Code: Select all

    ~/PycharmProjects/my_mlapi/mlapi ▓▒░                                                                                                             ░▒▓ INT ✘  44s   tyler@mlapi  08:19:13 PM  
❯ mlapi.debug
[1] 47526

    ~/PycharmProjects/my_mlapi/mlapi ▓▒░                                                                                                                         ░▒▓   tyler@mlapi  08:19:15 PM  
❯ 07/31/21 20:19:16.281579 CNSL[INF] mlapi_utils:129->[mlapi:util: config file found at '/home/tyler/mlapi/mlapiconfig.ini' ]
07/31/21 20:19:16.282645 CNSL[INF] mlapi_utils:134->[mlapi:util: reading secrets from: /home/tyler/mlapi/secrets.ini ]
07/31/21 20:19:16.284409 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_PORTAL ]
07/31/21 20:19:16.285501 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_USER ]
07/31/21 20:19:16.286547 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_PASSWORD ]
07/31/21 20:19:16.287597 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_API_PORTAL ]
07/31/21 20:19:16.288702 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !MLAPI_SECRET_KEY ]
07/31/21 20:19:16.290053 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !PLATEREC_ALPR_KEY ]
07/31/21 20:19:16.291063 CNSL[DBG2] mlapi_utils:167->[mlapi:util: found monitor specific section for monitor '1' ]
07/31/21 20:19:16.292062 CNSL[DBG2] mlapi_utils:209->[mlapi:util: adding polygon: 'front_yard' [0,427 1085,261 1075,200 1912,448 1912,1071 0,1079] ]
07/31/21 20:19:16.293053 CNSL[DBG2] mlapi_utils:193->[mlapi:util: found zone 'front_yard' specific pattern: (person|dog|cat) ]
07/31/21 20:19:16.293953 CNSL[DBG2] mlapi_utils:199->[mlapi:util: [Monitor-1] overrides key 'object_detection_pattern' = '(person|dog|cat)' ]
07/31/21 20:19:16.29489 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'frame_set' is an unknown key, adding to monitor '1' config ]
07/31/21 20:19:16.295802 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'person_min_confidence' is an unknown key, adding to monitor '1' config ]
07/31/21 20:19:16.296775 CNSL[DBG2] mlapi_utils:167->[mlapi:util: found monitor specific section for monitor '3' ]
07/31/21 20:19:16.297716 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'frame_set' is an unknown key, adding to monitor '3' config ]
07/31/21 20:19:16.298617 CNSL[DBG2] mlapi_utils:199->[mlapi:util: [Monitor-3] overrides key 'object_detection_pattern' = '(person|dog|cat)' ]
07/31/21 20:19:16.29952 CNSL[DBG2] mlapi_utils:209->[mlapi:util: adding polygon: 'parking_pad' [5,408 660,175 1695,100 1910,275 1910,1070 0,1080] ]
07/31/21 20:19:16.300439 CNSL[DBG2] mlapi_utils:193->[mlapi:util: found zone 'parking_pad' specific pattern: (person|dog|cat) ]
07/31/21 20:19:16.301345 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'person_min_confidence' is an unknown key, adding to monitor '3' config ]
07/31/21 20:19:16.302227 CNSL[DBG2] mlapi_utils:167->[mlapi:util: found monitor specific section for monitor '4' ]
07/31/21 20:19:16.303137 CNSL[DBG2] mlapi_utils:199->[mlapi:util: [Monitor-4] overrides key 'object_detection_pattern' = '(person)' ]
07/31/21 20:19:16.304035 CNSL[DBG2] mlapi_utils:193->[mlapi:util: found zone 'back_yard' specific pattern: (person) ]
07/31/21 20:19:16.304968 CNSL[DBG2] mlapi_utils:209->[mlapi:util: adding polygon: 'back_yard' [0,544 833,247 1697,140 1912,254 1919,1079 0,1079] ]
07/31/21 20:19:16.305886 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'person_min_confidence' is an unknown key, adding to monitor '4' config ]
07/31/21 20:19:16.306802 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'frame_set' is an unknown key, adding to monitor '4' config ]
07/31/21 20:19:16.307746 CNSL[DBG2] mlapi_utils:226->[mlapi:util: doing parameter substitution for globals ]
07/31/21 20:19:16.309467 CNSL[DBG2] mlapi_utils:252->[mlapi:util: doing parameter substitution for monitor specific entities ]
07/31/21 20:19:16.315113 CNSL[DBG1] db:17->[db: opening data base at /home/tyler/mlapi/db/db.json ]
07/31/21 20:19:16.316242 CNSL[DBG1] db:21->[db: engine ready ]
07/31/21 20:19:16.321235 CNSL[DBG2] api:82->[pyzm:api: SSL certificate verification disabled (encryption still works, vuln to spoofing) ]
07/31/21 20:19:16.322265 CNSL[DBG1] api:198->[pyzm:api: no token found, using user/pass to login ]
07/31/21 20:19:16.453623 CNSL[DBG2] api:227->[pyzm:api: detected API ver 2.0+, using token system ]
07/31/21 20:19:16.45556 CNSL[DBG1] api:235->[pyzm:api: access token expires on: 2021-08-01 02:19:16.453753 (21600s) ]
07/31/21 20:19:16.457072 CNSL[DBG1] api:242->[pyzm:api: refresh token expires on: 2021-08-01 20:19:16.455712 (86400s) ]
07/31/21 20:19:16.458175 CNSL[DBG2] api:302->[pyzm:api:make_req: 'get'->https://zm.baudneo.com/zm/api/zones.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]
07/31/21 20:19:16.471758 CNSL[DBG2] mlapi_utils:70->[mlapi:import: replacing match pattern for mid-1 zone 'front_yard' with: (person|dog|cat) ]
07/31/21 20:19:16.472663 CNSL[DBG2] mlapi_utils:70->[mlapi:import: replacing match pattern for mid-3 zone 'parking_pad' with: (person|dog|cat) ]
07/31/21 20:19:16.473538 CNSL[DBG2] mlapi_utils:70->[mlapi:import: replacing match pattern for mid-4 zone 'back_yard' with: (person) ]

07/31/21 20:19:16.475085 CNSL[DBG2] mlapi:432->[mlapi: loading initial ml_sequence from config file -> mlapiconfig.ini ]
07/31/21 20:19:16.510511 CNSL[DBG2] coral_edgetpu:39->[portalock: [max: 1] - [timeout: 120s] - [name: pyzm_uid1000_tpu_lock] ]
07/31/21 20:19:16.513607 CNSL[DBG2] yolo:34->[portalock: [max: 4] - [timeout: 120s] - [name: pyzm_uid1000_GPU_lock] ]
07/31/21 20:19:16.514824 CNSL[DBG1] coral_edgetpu:96->[coral: loading model data from 'pycoral yolov5'  ]
07/31/21 20:19:20.103987 CNSL[DBG1] coral_edgetpu:144->[perf:coral: initialization (loading /home/tyler/mlapi/models/coral_edgetpu/ssdlite_mobiledet_coco_qat_postprocess_edgetpu.tflite from disk) took: 3587.29 ms ]
07/31/21 20:19:20.105859 CNSL[DBG1] yolo:88->[yolo: loading model data from 'v4' ]
07/31/21 20:19:20.355056 CNSL[DBG1] yolo:93->[perf:yolo: 'v4' initialization (loading '/home/tyler/mlapi/models/yolov4/yolov4.weights') took: 247.45 ms ]
07/31/21 20:19:20.356719 CNSL[DBG1] yolo:106->[yolo: using GPU for detection, setting CUDA and cuDNN backend/target ]
07/31/21 20:19:20.358004 CNSL[INF] mlapi:450->[mlapi: version:2.20.18, pyzm version:0.30.1  ]
07/31/21 20:19:20.358952 CNSL[INF] mlapi:453->[mlapi: using bjoern as WSGI server @ 0.0.0.0:5000 ]
fg
[1]  + running    python3 /home/tyler/mlapi/mlapi.py --config /home/tyler/mlapi/mlapiconfig.ini
^C07/31/21 20:19:23.968933 CNSL[DBG1] mlapi:458->[mlapi: keyboard interrupt, exiting... ]
07/31/21 20:19:23.969979 CNSL[INF] mlapi:489->[MLAPI RESTART WRAPPER-> exception = KEYBOARD_INT ]
07/31/21 20:19:23.971037 CNSL[INF] mlapi:490->[MLAPI RESTART WRAPPER-> restarting (current restart/max) (2/10)  ]
07/31/21 20:19:23.977404 CNSL[INF] mlapi_utils:129->[mlapi:util: config file found at '/home/tyler/mlapi/mlapiconfig.ini' ]
07/31/21 20:19:23.978456 CNSL[INF] mlapi_utils:134->[mlapi:util: reading secrets from: /home/tyler/mlapi/secrets.ini ]
07/31/21 20:19:23.980356 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_PORTAL ]
07/31/21 20:19:23.981486 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_USER ]
07/31/21 20:19:23.982596 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_PASSWORD ]
07/31/21 20:19:23.9837 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !ZM_API_PORTAL ]
07/31/21 20:19:23.984891 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !MLAPI_SECRET_KEY ]
07/31/21 20:19:23.986478 CNSL[DBG1] mlapi_utils:105->[mlapi:proc_conf: secret token found in config: !PLATEREC_ALPR_KEY ]
07/31/21 20:19:23.987696 CNSL[DBG2] mlapi_utils:167->[mlapi:util: found monitor specific section for monitor '1' ]
07/31/21 20:19:23.988862 CNSL[DBG2] mlapi_utils:209->[mlapi:util: adding polygon: 'front_yard' [0,427 1085,261 1075,200 1912,448 1912,1071 0,1079] ]
07/31/21 20:19:23.989866 CNSL[DBG2] mlapi_utils:193->[mlapi:util: found zone 'front_yard' specific pattern: (person|dog|cat) ]
07/31/21 20:19:23.990903 CNSL[DBG2] mlapi_utils:199->[mlapi:util: [Monitor-1] overrides key 'object_detection_pattern' = '(person|dog|cat)' ]
07/31/21 20:19:23.992009 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'frame_set' is an unknown key, adding to monitor '1' config ]
07/31/21 20:19:23.993078 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'person_min_confidence' is an unknown key, adding to monitor '1' config ]
07/31/21 20:19:23.994047 CNSL[DBG2] mlapi_utils:167->[mlapi:util: found monitor specific section for monitor '3' ]
07/31/21 20:19:23.9951 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'frame_set' is an unknown key, adding to monitor '3' config ]
07/31/21 20:19:23.996096 CNSL[DBG2] mlapi_utils:199->[mlapi:util: [Monitor-3] overrides key 'object_detection_pattern' = '(person|dog|cat)' ]
07/31/21 20:19:23.997083 CNSL[DBG2] mlapi_utils:209->[mlapi:util: adding polygon: 'parking_pad' [5,408 660,175 1695,100 1910,275 1910,1070 0,1080] ]
07/31/21 20:19:23.998049 CNSL[DBG2] mlapi_utils:193->[mlapi:util: found zone 'parking_pad' specific pattern: (person|dog|cat) ]
07/31/21 20:19:23.999043 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'person_min_confidence' is an unknown key, adding to monitor '3' config ]
07/31/21 20:19:23.999995 CNSL[DBG2] mlapi_utils:167->[mlapi:util: found monitor specific section for monitor '4' ]
07/31/21 20:19:24.001 CNSL[DBG2] mlapi_utils:199->[mlapi:util: [Monitor-4] overrides key 'object_detection_pattern' = '(person)' ]
07/31/21 20:19:24.001967 CNSL[DBG2] mlapi_utils:193->[mlapi:util: found zone 'back_yard' specific pattern: (person) ]
07/31/21 20:19:24.002947 CNSL[DBG2] mlapi_utils:209->[mlapi:util: adding polygon: 'back_yard' [0,544 833,247 1697,140 1912,254 1919,1079 0,1079] ]
07/31/21 20:19:24.003937 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'person_min_confidence' is an unknown key, adding to monitor '4' config ]
07/31/21 20:19:24.004941 CNSL[DBG2] mlapi_utils:211->[mlapi:util: 'frame_set' is an unknown key, adding to monitor '4' config ]
07/31/21 20:19:24.005971 CNSL[DBG2] mlapi_utils:226->[mlapi:util: doing parameter substitution for globals ]
07/31/21 20:19:24.007643 CNSL[DBG2] mlapi_utils:252->[mlapi:util: doing parameter substitution for monitor specific entities ]
07/31/21 20:19:24.012145 CNSL[DBG1] db:17->[db: opening data base at /home/tyler/mlapi/db/db.json ]
07/31/21 20:19:24.013286 CNSL[DBG1] db:21->[db: engine ready ]
07/31/21 20:19:24.017658 CNSL[DBG2] api:82->[pyzm:api: SSL certificate verification disabled (encryption still works, vuln to spoofing) ]
07/31/21 20:19:24.018787 CNSL[DBG1] api:198->[pyzm:api: no token found, using user/pass to login ]
07/31/21 20:19:24.150081 CNSL[DBG2] api:227->[pyzm:api: detected API ver 2.0+, using token system ]
07/31/21 20:19:24.151622 CNSL[DBG1] api:235->[pyzm:api: access token expires on: 2021-08-01 02:19:24.150212 (21600s) ]
07/31/21 20:19:24.15266 CNSL[DBG1] api:242->[pyzm:api: refresh token expires on: 2021-08-01 20:19:24.151674 (86400s) ]
07/31/21 20:19:24.153683 CNSL[DBG2] api:302->[pyzm:api:make_req: 'get'->https://zm.baudneo.com/zm/api/zones.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]
07/31/21 20:19:24.166284 CNSL[DBG2] mlapi_utils:70->[mlapi:import: replacing match pattern for mid-1 zone 'front_yard' with: (person|dog|cat) ]
07/31/21 20:19:24.167268 CNSL[DBG2] mlapi_utils:70->[mlapi:import: replacing match pattern for mid-3 zone 'parking_pad' with: (person|dog|cat) ]
07/31/21 20:19:24.168157 CNSL[DBG2] mlapi_utils:70->[mlapi:import: replacing match pattern for mid-4 zone 'back_yard' with: (person) ]

07/31/21 20:19:24.169987 CNSL[DBG2] mlapi:432->[mlapi: loading initial ml_sequence from config file -> mlapiconfig.ini ]
07/31/21 20:19:24.172126 CNSL[DBG2] coral_edgetpu:39->[portalock: [max: 1] - [timeout: 120s] - [name: pyzm_uid1000_tpu_lock] ]
07/31/21 20:19:24.173747 CNSL[DBG2] yolo:34->[portalock: [max: 4] - [timeout: 120s] - [name: pyzm_uid1000_GPU_lock] ]
07/31/21 20:19:24.174975 CNSL[DBG1] coral_edgetpu:96->[coral: loading model data from 'pycoral yolov5'  ]
07/31/21 20:19:24.181271 CNSL[DBG1] coral_edgetpu:144->[perf:coral: initialization (loading /home/tyler/mlapi/models/coral_edgetpu/ssdlite_mobiledet_coco_qat_postprocess_edgetpu.tflite from disk) took: 4.84 ms ]
07/31/21 20:19:24.182596 CNSL[DBG1] yolo:88->[yolo: loading model data from 'v4' ]
07/31/21 20:19:24.382545 CNSL[DBG1] yolo:93->[perf:yolo: 'v4' initialization (loading '/home/tyler/mlapi/models/yolov4/yolov4.weights') took: 198.55 ms ]
07/31/21 20:19:24.383845 CNSL[DBG1] yolo:106->[yolo: using GPU for detection, setting CUDA and cuDNN backend/target ]
07/31/21 20:19:24.401762 CNSL[INF] mlapi:450->[mlapi: version:2.20.18, pyzm version:0.30.1  ]
07/31/21 20:19:24.40255 CNSL[INF] mlapi:453->[mlapi: using bjoern as WSGI server @ 0.0.0.0:5000 ]

tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Just pushed a new version of all 3 components. The creation of animations are threaded now so the animation notification sent off is quite a bit faster. Sending pushover notifications and mqtt are also threaded, I have noticed a large increase in speed. YMMV, but it should help regardless.

Added more configuration for pushover notifications, like different USER KEYS for jpg and/or gif. User keys are also group tokens in pushover, so you can send notifications to groups made up of certain devices instead of just 1 user key and specifting devices and working logic on that. My JPG pushover notification is now the fastest way to be notified of a detected object.

Added option for pushover notifications to not have a clickable link to view event in a browser. The clickable link allows devices without zmNinja to easily view the event in question in any modern browser. Even on my device with zmNinja its very convenient to click the link and open the event in the browser, IDK if I can make a URL that would open zmNinja to that certain event?
I recommend creating a user called pushover_view and only allowing VIEW of EVENTS and STREAM or even just EVENTS and configure that user for the URL link in pushover (push_user, push_pass).

Added per monitor option 'push_cooldown' to control cooldown between sending pushover notifications if you don't use home assistant for its sensor integrations.

Added a 'skip_mons' to objectconfig.ini, does the same thing as skipping a monitor in zmeventnotification. I put it here so you don't have to restart zmeventnotification.pl for it to take effect.

Added 'tpu_320' to [object] section -> this will resize the input image to 320x320 right before running the tpu model for object detection.

Upgrading ml_overrides logic, ml_overrides only applies to detections sent via mlapi -> right now ml_overrides only overrides the 'detection_sequence' (object,face,alpr) but it sends the pattern for each. I am also sending frame_set and thinking of a few other things. Should ml_overrides take precedence or should the local mlapiconfig options take precedence? I am unsure at the moment so I am keeping only the detection_sequence being overridden for now.

Added an option to receive a pushover notification whenever a GPU or TPU crash is detected. I am working on automated TPU/GPU resetting but there are several things to consider. This notification allows you to login and reset mlapi or whatever if the automated resetting processes aren't working. There is a good chance the automated resetters wont work as of yet, the TPU port resetter *should* work, the usb_controller reset script is the end all though. I included instructions on how to setup passwordless sudo access for setting up the es.debug.objdet shell function and laying the path for the USB Controller reset automation.

Forked pygifsicle to allow piping virtual file buffers into gifsicle standard input. I didn't end up using the functionality, I was testing out ways to speed up animations and thought instead of writing and reading files from disk using virt file buffer would be faster. The difference is negligible but at least I am working on polishing and P/R to add this functionality and the ability to pipe in raw byte strings to gifsicle std_in instead of just a path to a file on disk. Threading was the answer for speeding creating animations up

Several bug fixes, remember this is my personal blend so it changes a lot while I am trying to get it stable, I am always adding and modifying things.

REVIEW the mlapiconfig.ini, objectconfig.ini and secrets.ini files to see new options to configure and their details.

I put more work into catching the USB TPU errors and also the yolo GPU errors. For USB TPU if its the 'delegate' error and you can still see your device when using lsusb

Code: Select all

lsusb ->
Bus 004 Device 003: ID 18d1:9302 Google Inc.   <----- TPU
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub <-- USB Controller 
Bus 003 Device 002: ID 2109:3431 VIA Labs, Inc. Hub <-- USB Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub <-- USB Controller 
Then you can reset it by its USB address so only that specific USB port is reset. The usb_reset.py module attempts to do this automatically before it exits due to the error so when things are restarted, all is working.
If you lsusb and the TPU is no longer detected then you have 2 choices (I am working on automating this, but it has implications) you can either reboot or 'unbind' and 'bind' the USB Controller by its PCI address. USB Controllers have multiple port attached to them. For example in the output from above ->

Code: Select all

Bus 004 Device 003: ID 18d1:9302 Google Inc.   <----- TPU
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub <-- USB Controller 
The 3.0 root hub is bus 04:00.0, and the TPU is connected to that Controller as you can see its bus 004 device 003. In this setup it is a PCIe USB 3.1 add-on card and the TPU is the only thing connected to it. Since it is setup this way I am able to reset the USB Controller and it will reset all the ports on the PCIe Card so the TPU is accessible again to load the .tflite model. The TPU is only thing attached to my 3.1 card so I have no worries.

Sounds well and all right? Well as you can see there is a Hub upstream of the root hub on bus 003, so if you had your TPU connected to bus 003 then resetting bus 003 will take down everything connected to both of those devices. How bad could it be? Well if you are using a USB HDD and are transferring data, it could be disastrous. It will not wait and gracefully reset the ports, it's the equivalent of pulling the plug and re inserting.

This method, when you do not specify a PCI address will crawl all the available USB Controllers and reset them if it has the available permissions to do so. IDK if this will work out for mass distribution or if I will keep it for myself, yet. There is also a small c program you can use to reset the USB port if you can still see it in the output of lsusb but I am still testing the python method of resetting a single USB port.
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Pushed a new update for all components, a lot has changed and i've worked to get things stable. If anyone is using this please let me know of any errors.
  • I added support for half precision floating point (FP16) for yolo v4 target backend (may improve inference time on newer cards Compute 5.6+).
  • worked on frame_set being smarter, hard to explain but things work much better now, I will be adding an adaptive frameset that grabs frames based on total length of ongoing event (updates frame buffer length)
  • All sorts of tweaks, bug fixes, stability.
I am looking into adding different model frameworks and am still designing a server/host-client setup for a multi server ZM setup. Meaning 1 main 'server/host' ZMES to handle all clients and 'client' ZMES on multi server setups to send data to the main ZMES for notifications and possible db update (db for grafana data points to visualize)

I have threaded some things to try and speed up animation notifications, once the code is stable I will start refining it for more speed as well. Feed back would be appreciated.
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Been busy tinkering, Im gearing down to put out a fairly stable release that the general public should be able to install and use without any issues.

Some things I have been working on are frame_set logic, adding a n additional filtering method for detection 'matches' (its a 'high confidence' filter that is on top of 'most/first' same_model_sequence_strategy), multi server communication support (WIP) and bug fixes. I have the first stage of the frame_set logic working and what it basically does is handle frame grabbing errors with more in depth logic than before. For an example lets say you have a monitor that has an average event time of 25-35 seconds and a frame_set of 'snapshot,alarm,175,230,snapshot,350'. Frames 1-4 have been processed so only 'snapshot' and '350' are left, the last processed frame ID was 230. Frame ID 'snapshot' is called for (snapshot/alarm converted to frame ID is part of the default pipeline now) and is converted to a frame ID, lets say 245. 245 is processed so frame 350 is requested but the API returns a 404 or other message which is caught as a BAD_IMAGE exception. Here is where things differ, the old script had a configurable retry loop amount and a configurable set sleep time (which was a default of 15 seconds!) The new method lets you specify those values but also uses logic to override those configs to be more efficent or just skip the frame and move onto the next one. For insrtance there is a new setting for a 'threshold' and it is configured in seconds but calculated into fps based on the monitors fps.

So 350 is unavailable and the script keeps track of the length of the current and last frame buffer. So the script can compare the current frame buffer length to the last requests to see if the frame buffer has grown or if it hasnt (is the event still writing frames to disk or has it eneded, basically). It does different things based on if the frame buffer is still growing or not but with the default configured threshold of 8 seconds, we are calling frame 350 on a monitor running at 10fps and the current frame buffer is still writing frames to disk and is currently at frame 320. With the threshold of 8 seconds, calculated using the monitor fps of 10FPS works out to be a threshold of 80 frames. The frame buffer is at 320, we requested 350 so there is a difference of 30 frames or 3 seconds, since the threshold is 8 seconds the script will sit and wait for the frame buffer to hopefully catchup and run a few frame request loops until it exhausts its attempts and skips to the next frame ID or it grabs the frame it was waiting for. If the frame buffer does not catch up and lets say it stops writing frames at frame # 349, the script will try and grab the last available frame (minus 2 frames for allowing it to be written to disk so 347 in this case). This has the effect of grabbing frame ids you never called for but at least it grabs something and its not waiting for a set amoutn of time that doesnt change. The script also calculates its own times on the fly to try and make things more efficent.

Hard to explain so here is a snippet of debug logs of it in action, this isnt a great example so ill try and find a better one where it has to deal with waiting on frames and skipping / regrabbing different frame ids based on whats available in the frame buffer. ->

Code: Select all

10/02/21 22:26:15.20701 zm_mlapi[11690] DBG2 detect_sequence:1304 [ml: model: 'object' seq: 'DarkNet::v4 Pre-Trained' found 0 detections -> ]
  10/02/21 22:26:15.208936 zm_mlapi[11690] DBG2 detect_sequence:1593 [ml: no 'object' matches at all in frame: 210]
  10/02/21 22:26:15.210911 zm_mlapi[11690] DBG2 detect_sequence:1613 [perf:frame: 210 took 625.59 ms]
  10/02/21 22:26:15.213002 zm_mlapi[11690] DBG1 Media:470 [media:read:image: self.frames_processed=7 --- self.fids_processed=['s-160', '70', '110', 's-188', 'a-108', '210'] --- self.frame_set[self.frame_set_index]
  ='280' --- self.get_last_frame()='210' --- self.max_frames=8]
  10/02/21 22:26:15.215113 zm_mlapi[11690] DBG2 Media:487 [api:make_req: 'get'-><hidden>/api/events/34141.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'}]
  10/02/21 22:26:15.472206 zm_mlapi[11690] DBG1 Media:501 [media:read:image: DBG! fid=280 current frame buffer length '269' last API call frame buffer length was '258']
  10/02/21 22:26:15.47252 zm_mlapi[11690] ERR Media:559 [media:read:image: skip_all=False --- self.skip_all_count=0]
  10/02/21 22:26:15.476681 zm_mlapi[11690] DBG1 Media:579 [media:read:image: monitor 3 is running at 10 FPS - threshold of 8 converts to 80 frames. frames over by -> 11 / fps -> 1.1 seconds]
  10/02/21 22:26:15.478899 zm_mlapi[11690] DBG1 Media:590 [media:read:oob: the current requested frame ID '280' is out of bounds of the current frame buffer length '269' by -> 11 frames. '280' is within 8 seconds
  based on monitor 3 @ 10 fps. setting attempt loops at 3 with a wait period of 2.9966666666666666 seconds]
  10/02/21 22:26:15.480981 zm_mlapi[11690] DBG1 Media:602 [media:read:oob: sleeping for '1.1' sec (# of OOB frames / fps of monitor) before attempting to call for the frame ID, this will allow the frame buffer to
  catch up (if the event runs that long) and write the frame to disk]
  10/02/21 22:26:16.584521 zm_mlapi[11690] DBG2 Media:683 [api:make_req: 'get'-><hidden>/index.php?view=image&eid=34141&fid=280 query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'}]
  10/02/21 22:26:17.000157 zm_mlapi[11690] DBG4 Media:683 [api: raising BAD_IMAGE ValueError for a 404 (image doesnt exist)]
  10/02/21 22:26:17.002622 zm_mlapi[11690] DBG2 Media:693 [api:make_req: 'get'-><hidden>/api/events/34141.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'}]
  10/02/21 22:26:17.275925 zm_mlapi[11690] DBG2 Media:711 [media:read:image: failed attempt 1/3 of grabbing the image for frame ID '280' -> sleeping for 2.9966666666666666 seconds before retry]
  10/02/21 22:26:20.276735 zm_mlapi[11690] DBG2 Media:683 [api:make_req: 'get'-><hidden>/index.php?view=image&eid=34141&fid=280 query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'}]
  10/02/21 22:26:20.761084 zm_mlapi[11690] DBG1 Media:756 [media:read: API returned frame with dimensions -> (1080, 1920), using 'resize' of 800=(450, 800)]
  10/02/21 22:26:20.76387 zm_mlapi[11690] DBG2 detect_sequence:1227 [frame: 280 [strategy:'first'] (7 of 8) - model: 'object' [strategy:'first'] (1 of 1) - sequence: 'coral::MobileNETv2-SSD' [strategy:'most'] (1 o
  f 2)]
It isnt quite tuned all the ay in yet but the results so far are absolutly fantastic. I am starting to do the ground work of an adaptive frame set that will keep requesting frames if the frame buffer is growing and there are no detections as of yet. Adding a cache to Match Past Detections is also a WIP.

Expanding model frameowrk coverage is......... iffy. For instance adding in pytorch support means alot more dependancies and writing scripts to convert to models to usable formats for the TPU as well. There is face detection via a frame work called mediapipe but IDK about that. OpenCV has the ONNX framework and you can convert alot of models to ONNX format but that requires some knowledge and experiance and the proper environment to convert frameworks over.

As of now the only KNOWN bug that I cannot fix involves using yolo on gpu with the command line binary ALPR also using GPU while running on a mlapi instance. Somehow the yolo algo gets stuck always reporting its first detection for every detection request it receives. As an example lets say we pass mlapi an event using yolo and ALPR binary GPU models/sequences. Yolo finds a person with 83.47% confidence with bbox of [1,2 3,4 5,6 7,8] and ALPR finds nothing, EVERY SINGLE detection request after this, yolo will return the person with 83.47% confidence with bbox of [1,2 3,4 5,6 7,8] for every frame you send it regardless until you restart mlapi. Ive tested extensively with PDB and have basically lost my mind trying to debug it and have given up for the time being. Remember this may be limited to my setup somehow.

The high confidence filter is being worked on the most right now as I have the POC working I just need to polish it, the logic is.... wild lol. I will post more info about its workings and some of the log output from it but it does make a positive impact on detections.

I have alrteady integrated basic HASS 'helper' sensors for controlling the pushover notifications, you can control sending notifications at all and also specify a cooldown period between notifications per monitor. I have seen a few requests about presence detection and I can add that no problem as I work with HASS here at home but I dont use OpenHAB so if someone wants that functionality added I just need some logs and other details to make it work.

I am open to adding things as well! so if you have some edge use cases I can either include it into ZMES or help you figure out how to integrate it yourself. The HASS option opens alot of interesting things, my pushover add on is lightening fast at getting push notifications out to you and my python mqtt add on helps move data to HASS from ZMES and includes an image wrapper to send objdetect.jpg or the GIF to an MQTT Camera entity in HASS.

Grafana datapoints - I havnt even scratched the surface yet, but am still looking into in the future.
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Pushed earlier this morning, mostly prepared for the public beta if anyone is interested. There are so many changes I can not list them all but I will try and make a list and organize it into something I can document. For the time being, this will not be 'new user' friendly, You must have some understanding of how this software system works and how to install and troubleshoot some things.

My end goal is to make ZM, ZMES, MLAPI more accessible to the general public and to have some sort of active development on this subsystem of ZM where enhancements can be made/requested. I am trying to write a mlapi install script to make installing mlapi easier for the public. I am also hoping to write a new python module to integrate ZM with Home Assistant better, Frigate is getting all the love there because it's so easy to install and run via HA, we should be focusing some attention on that area. The people doing home automation will most definitely financially support a good project (ZoneMinder), we just need to make it far more accessible to the point and click crowd.


There are configuration-breaking changes! Do not expect the beta testing to be super smooth! The core logic and structures of the system may still change drastically. I am considering changing configuration files (objectconfig.ini and mlapiconfig.ini) to YAML syntax

!SECRETS have been changed to {[secrets]}. Before there was a very small chance of errors (if a password started with ! for example). This lays the groundwork to allow substituting {[secrets]} when that secret is embedded in a substring (or nested into per monitor overrides, like a per-monitor override of ml_sequence)

The 'pattern' for each model has been changed -> ml_sequence:object:general:pattern used to have a substitution var of {{pattern}}. I have changed it to {{<model>_detection_pattern}}. So object would have {{object_detection_pattern}}, face would have {{face_detection_pattern}}.

The newest (and absolutely clutch) addition I have made to ZMES is how MLAPI and ZMES communicate with each other. In the original version, you would set up mlapi and hard configure that mlapi instance to only receive detections from 1 ZMES host. Using the new system, 'ml_routes' in objectconfig.ini and 'zmes_keys' in mlapiconfig.ini, ZMES encrypts all the configuration info that mlapi needs to reply to ZMES in the detection request using symmetrical key encryption. Let us analyze the example configuration for a quick breakdown.


The URLs and credentials are encrypted for transit, only the 'name' is unencrypted so mlapi can look in zmes_keys for the key to decrypt.

Snippit from objectconfig.ini ->

Code: Select all

ml_routes = [
			# {[SECRETS]} work as long as they are the only value, they cannot be embedded in a substring ->
;			CORRECT -> 'name': '{[secret_name]}',
;			INCORRECT -> 'name': 'I have a really cool {[secret_name]}',

			# {{VARS}} also work in here if needed, they can be in a substring and be correctly replaced.
			{
				# weight of route, lower is more important.
				'weight': 1,
				'name': 'mlapi_two',
				'gateway': 'http://10.0.0.32:5000/api/v1',
				'user': '{[ML_USER]}',
				'pass': '{[ML_PASSWORD]}',
				'zm_user': '{[ZM_ML_USER]}',
				'zm_pass': '{[ZM_ML_PASSWORD]}',
				'zm_basic_user': '',
				'zm_basic_pass': '',
				'enc_key': '{[mlapi_one_key]}',
			},
			{
				# weight of route, lower is more important.
				'weight': 0,

				'name': 'mlapi_one',
				'gateway': 'http://10.0.0.32:5000/api/v1',
				# user/pass is the MLAPI user and pass
				'user': '{[ML_USER]}',
				'pass': '{[ML_PASSWORD]}',
				# ZM user/pass is for token (API 2.0+) auth, these are the credentials passed to the mlapi host
			    # that mlapi will use to login to this ZM hosts API. This makes mlapi dynamic!
				'zm_user': '{[ZM_ML_USER]}',
				'zm_pass': '{[ZM_ML_PASSWORD]}',
				# zm_basic is for using ZM basic auth
				'zm_basic_user': '',
				'zm_basic_pass': '',
				# encryption key here must match the encryption key in mlapiconfig.ini zmes_keys
				'enc_key': '{[mlapi_one_key]}',

		},

	]  # IMPORTANT to keep ending brace indented
	
- weight: the importance of the route, lower numbers are the higher priority so 0 is the top priority. If 2+ routes have the same weight, it's a gamble which one will be chosen
- name: The name of the route ** THIS IS IMPORTANT, it must match the name of the key in zmes_keys so mlapi knows which key to use
- gateway: this is the URL of the mlapi host (ml_gateway basically)
- user: This is the ML username to log in to mlapi (the username and password you created when adding a user to the mlapi 'DB')
- pass: The password for mlapi user
- zm_user: The ZM API user you want mlapi to login with (This can be the same user that ZMES uses or a new user specifically for mlapi)
- zm_pass: The ZM API username password
- zm_basic_user: ZM API basic auth user if using basic auth
- zm_basic_pass: ZM API username password if using basic auth
- enc_key: The encryption key that 'get_encryption_key.py' output ** IMPORTANT: This key must be the same here and in zmes_keys otherwise decryption WILL FAIL


Snippet from mlapiconfig.ini ->

Code: Select all

zmes_keys = {
		# Format -> 'name of key/route/zmes host': '<key>',
			'mlapi_one': 'tlYoF3YT2vgrFPt8SHmEkw8bX0ISG4x0TwOVnbMS4oA=',
	} # Always keep the closing brace indented
- 'mlapi_one': See how in ml_routes we named 1 route mlapi_one, this is how mlapi will know which encryption key to use
- 'NAME OF ROUTE': '<ENCRYPTION KEY>',

You can change the encryption key at any time by running get_encryption_key.py and copying the output key into ml_routes and zmes_keys for the specified route. Remember that this is symmetrical key encryption, the encryption key and decryption key are the same.

Here is some log output from the ZMES side ->

Code: Select all

10/19/21 13:46:56.961577 zmesdetect_m4[563468] DBG2 zm_detect:333 [**** 'http://10.0.0.32:5000/api/v1/detect/object?
  type=object' using auth_header={'Authorization': 'Bearer eyJ0eXAiOiJKV1QiLCJhbGc...'}
  *** params={'delete': True, 'response_format': 'zm_detect'}
  ****** JSON: {'version': '7.0.0', 'mid': '4', 'reason': 'Motion Back Fence,Walkway,Fire pit / Dog area', 'stream': '
  40689', 'stream_options': {'frame_strategy': 'first', 'frame_set': 'snapshot,70,110,160,snapshot,alarm,210,280', 'co
  ntig_frames_before_error': 2, 'max_attempts': 3, 'sleep_between_attempts': 2, 'sleep_between_frames': 0.04, 'sleep_b
  etween_snapshots': 1, 'smart_fs_thresh': '5', 'resize': '800', 'api': '<REMOVED FOR HTTP TRANSFER>', 'polygons': [{'
  name': 'back_yard', 'value': [(0, 547), (824, 246), (1511, 161), (1706, 1070), (0, 1080)], 'pattern': '(person)'}],
  'mid': '4', 'reply_data': {'name': 'mlapi_one', 'auth_type': b'gAAAAABhbyCwZBOBvrdDh8FhRbcpHKo8pejg3ybwttzwp-2fzPKLb
  q6xMgpATRuA1Iurx-m9pKkSq6VVUmp2uXbGU9UldzGYyg==', 'api_url': b'gAAAAABhbyCwsCpLKqhcBZVwA5FM2K-up7AhSVTVRIPss9TVMd-tc
  NrNSH6Ohi2k0n-fFbLUC3i5ef4uoLlRA8ZT8tIwlyodwQiiA-om15AQElRSFnsJv54=', 'portal_url': b'gAAAAABhbyCw-xemxEkbKlf79q-amA
  t1uFlDJ8aWEC-TKtM8jQNxglzYKbjc3s6rv_9DRxyUK_d-WjRLInEqdsad6BdBRoTv8wnsRPcOmdlaFgSlykdGa0U=', 'user': b'gAAAAABhbyCwC
  o_KSkXiSgshEpLPc0yKQFl2xfhSrM5yCj6CLJIgEMrdaumAE4mndHOg5yF_gKKmGqCOOmp7uyT0SP6onKN1Kg==', 'pass': b'gAAAAABhbyCwQz0k
  lECz-wvJTvfDAlm0G-npsOSGhI-MwigkQbjQCjHykXVEUOitVNtpDTTMQokQxEGVNm6aP5_L0Rb6JXiKTMkniDkLfXLzo36MhA0HamY2dLMGnbnbDCWq
  3sRZUwPM1u6mK1sLIr7dnE5rQAAnxw==', 'b_user': None, 'b_pass': None}}, 'ml_overrides': {'enable': False, 'model_sequen
  ce': 'object', 'object': {'pattern': '(person|car|motorbike|bus|truck|boat|dog|cat)'}, 'face': {'pattern': '.*'}, 'a
  lpr': {'pattern': '.*'}}, 'sub_options': {'push_errors': 'yes', 'push_options': {'token': '<redacted>', 'user': <redacted>', 'sound': None}}}
  ]
And here are the logs from MLAPI side of that request ->

Code: Select all

10/19/21 13:46:57.057547 zm_mlapi[17070] DBG1 mlapi:165 [mlapi: ZMES sent encrypted data! Checking the keystore for '
  mlapi_one']
  10/19/21 13:46:57.060633 zm_mlapi[17070] DBG1 mlapi:180 [decrypted={'auth_type': 'token', 'api_url': 'https://<redacted>.com/zm/api', 'portal_url': 'https://<redacted>.com/zm', 'user': '<redacted>', 'pass': '<redacted>'}]
  
MLAPI uses that encrypted info to reply to that specific detection request and access that ZM API for information about the event!
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Pushed earlier this morning, mostly prepared for the public beta if anyone is interested. There are so many changes I can not list them all but I will try and make a list and organize it into something I can document. For the time being, this will not be 'new user' friendly, You must have some understanding of how this software system works and how to install and troubleshoot some things.

My end goal is to make ZM, ZMES, MLAPI more accessible to the general public and to have some sort of active development on this subsystem of ZM where enhancements can be made/requested. I am trying to write a mlapi install script to make installing mlapi easier for the public. I am also hoping to write a new python module to integrate ZM with Home Assistant better, Frigate is getting all the love there because it's so easy to install and run via HA, we should be focusing some attention on that area. The people doing home automation will most definitely financially support a good project (ZoneMinder), we just need to make it far more accessible to the point and click crowd.


There are configuration-breaking changes! Do not expect the beta testing to be super smooth! The core logic and structures of the system may still change drastically. I am considering changing configuration files (objectconfig.ini and mlapiconfig.ini) to YAML syntax

!SECRETS have been changed to {[secrets]}. Before there was a very small chance of errors (if a password started with ! for example). This lays the groundwork to allow substituting {[secrets]} when that secret is embedded in a substring (or nested into per monitor overrides, like a per-monitor override of ml_sequence)

The 'pattern' for each model has been changed -> ml_sequence:object:general:pattern used to have a substitution var of {{pattern}}. I have changed it to {{<model>_detection_pattern}}. So object would have {{object_detection_pattern}}, face would have {{face_detection_pattern}}.

The newest (and absolutely clutch) addition I have made to ZMES is how MLAPI and ZMES communicate with each other. In the original version, you would set up mlapi and hard configure that mlapi instance to only receive detections from 1 ZMES host. Using the new system, 'ml_routes' in objectconfig.ini and 'zmes_keys' in mlapiconfig.ini, ZMES encrypts all the configuration info that mlapi needs to reply to ZMES in the detection request using symmetrical key encryption. Let us analyze the example configuration for a quick breakdown.


The URLs and credentials are encrypted for transit, only the 'name' is unencrypted so mlapi can look in zmes_keys for the key to decrypt.

Snippit from objectconfig.ini ->

Code: Select all

ml_routes = [
			# {[SECRETS]} work as long as they are the only value, they cannot be embedded in a substring ->
;			CORRECT -> 'name': '{[secret_name]}',
;			INCORRECT -> 'name': 'I have a really cool {[secret_name]}',

			# {{VARS}} also work in here if needed, they can be in a substring and be correctly replaced.
			{
				# weight of route, lower is more important.
				'weight': 1,
				'name': 'mlapi_two',
				'gateway': 'http://10.0.0.32:5000/api/v1',
				'user': '{[ML_USER]}',
				'pass': '{[ML_PASSWORD]}',
				'zm_user': '{[ZM_ML_USER]}',
				'zm_pass': '{[ZM_ML_PASSWORD]}',
				'zm_basic_user': '',
				'zm_basic_pass': '',
				'enc_key': '{[mlapi_one_key]}',
			},
			{
				# weight of route, lower is more important.
				'weight': 0,

				'name': 'mlapi_one',
				'gateway': 'http://10.0.0.32:5000/api/v1',
				# user/pass is the MLAPI user and pass
				'user': '{[ML_USER]}',
				'pass': '{[ML_PASSWORD]}',
				# ZM user/pass is for token (API 2.0+) auth, these are the credentials passed to the mlapi host
			    # that mlapi will use to login to this ZM hosts API. This makes mlapi dynamic!
				'zm_user': '{[ZM_ML_USER]}',
				'zm_pass': '{[ZM_ML_PASSWORD]}',
				# zm_basic is for using ZM basic auth
				'zm_basic_user': '',
				'zm_basic_pass': '',
				# encryption key here must match the encryption key in mlapiconfig.ini zmes_keys
				'enc_key': '{[mlapi_one_key]}',

		},

	]  # IMPORTANT to keep ending brace indented
	
- weight: the importance of the route, lower numbers are the higher priority so 0 is the top priority. If 2+ routes have the same weight, it's a gamble which one will be chosen
- name: The name of the route ** THIS IS IMPORTANT, it must match the name of the key in zmes_keys so mlapi knows which key to use
- gateway: this is the URL of the mlapi host (ml_gateway basically)
- user: This is the ML username to log in to mlapi (the username and password you created when adding a user to the mlapi 'DB')
- pass: The password for mlapi user
- zm_user: The ZM API user you want mlapi to login with (This can be the same user that ZMES uses or a new user specifically for mlapi)
- zm_pass: The ZM API username password
- zm_basic_user: ZM API basic auth user if using basic auth
- zm_basic_pass: ZM API username password if using basic auth
- enc_key: The encryption key that 'get_encryption_key.py' output ** IMPORTANT: This key must be the same here and in zmes_keys otherwise decryption WILL FAIL


Snippet from mlapiconfig.ini ->

Code: Select all

zmes_keys = {
		# Format -> 'name of key/route/zmes host': '<key>',
			'mlapi_one': 'tlYoF3YT2vgrFPt8SHmEkw8bX0ISG4x0TwOVnbMS4oA=',
	} # Always keep the closing brace indented
- 'mlapi_one': See how in ml_routes we named 1 route mlapi_one, this is how mlapi will know which encryption key to use
- 'NAME OF ROUTE': '<ENCRYPTION KEY>',

You can change the encryption key at any time by running get_encryption_key.py and copying the output key into ml_routes and zmes_keys for the specified route. Remember that this is symmetrical key encryption, the encryption key and decryption key are the same.

Here is some log output from the ZMES side ->

Code: Select all

10/19/21 13:46:56.961577 zmesdetect_m4[563468] DBG2 zm_detect:333 [**** 'http://10.0.0.32:5000/api/v1/detect/object?
  type=object' using auth_header={'Authorization': 'Bearer eyJ0eXAiOiJKV1QiLCJhbGc...'}
  *** params={'delete': True, 'response_format': 'zm_detect'}
  ****** JSON: {'version': '7.0.0', 'mid': '4', 'reason': 'Motion Back Fence,Walkway,Fire pit / Dog area', 'stream': '
  40689', 'stream_options': {'frame_strategy': 'first', 'frame_set': 'snapshot,70,110,160,snapshot,alarm,210,280', 'co
  ntig_frames_before_error': 2, 'max_attempts': 3, 'sleep_between_attempts': 2, 'sleep_between_frames': 0.04, 'sleep_b
  etween_snapshots': 1, 'smart_fs_thresh': '5', 'resize': '800', 'api': '<REMOVED FOR HTTP TRANSFER>', 'polygons': [{'
  name': 'back_yard', 'value': [(0, 547), (824, 246), (1511, 161), (1706, 1070), (0, 1080)], 'pattern': '(person)'}],
  'mid': '4', 'reply_data': {'name': 'mlapi_one', 'auth_type': b'gAAAAABhbyCwZBOBvrdDh8FhRbcpHKo8pejg3ybwttzwp-2fzPKLb
  q6xMgpATRuA1Iurx-m9pKkSq6VVUmp2uXbGU9UldzGYyg==', 'api_url': b'gAAAAABhbyCwsCpLKqhcBZVwA5FM2K-up7AhSVTVRIPss9TVMd-tc
  NrNSH6Ohi2k0n-fFbLUC3i5ef4uoLlRA8ZT8tIwlyodwQiiA-om15AQElRSFnsJv54=', 'portal_url': b'gAAAAABhbyCw-xemxEkbKlf79q-amA
  t1uFlDJ8aWEC-TKtM8jQNxglzYKbjc3s6rv_9DRxyUK_d-WjRLInEqdsad6BdBRoTv8wnsRPcOmdlaFgSlykdGa0U=', 'user': b'gAAAAABhbyCwC
  o_KSkXiSgshEpLPc0yKQFl2xfhSrM5yCj6CLJIgEMrdaumAE4mndHOg5yF_gKKmGqCOOmp7uyT0SP6onKN1Kg==', 'pass': b'gAAAAABhbyCwQz0k
  lECz-wvJTvfDAlm0G-npsOSGhI-MwigkQbjQCjHykXVEUOitVNtpDTTMQokQxEGVNm6aP5_L0Rb6JXiKTMkniDkLfXLzo36MhA0HamY2dLMGnbnbDCWq
  3sRZUwPM1u6mK1sLIr7dnE5rQAAnxw==', 'b_user': None, 'b_pass': None}}, 'ml_overrides': {'enable': False, 'model_sequen
  ce': 'object', 'object': {'pattern': '(person|car|motorbike|bus|truck|boat|dog|cat)'}, 'face': {'pattern': '.*'}, 'a
  lpr': {'pattern': '.*'}}, 'sub_options': {'push_errors': 'yes', 'push_options': {'token': '<redacted>', 'user': <redacted>', 'sound': None}}}
  ]
And here are the logs from MLAPI side of that request ->

Code: Select all

10/19/21 13:46:57.057547 zm_mlapi[17070] DBG1 mlapi:165 [mlapi: ZMES sent encrypted data! Checking the keystore for '
  mlapi_one']
  10/19/21 13:46:57.060633 zm_mlapi[17070] DBG1 mlapi:180 [decrypted={'auth_type': 'token', 'api_url': 'https://<redacted>.com/zm/api', 'portal_url': 'https://<redacted>.com/zm', 'user': '<redacted>', 'pass': '<redacted>'}]
  
MLAPI uses that encrypted info to reply to that specific detection request and access that ZM API for information about the event!

I am currently spinning up LXC's to test installing ZoneMinder and then my forked ZMES and MLAPI (dubbed neo-zmeventnotfication, neo-mlapi, neo-pyzm-> neo_ZMES - to differentiate from source). Once I iron out the bugs there, things should be ready to beta test. I am writing up a tutorial on Medium for a full ZM 1.36.x branch install onto Ubuntu 21.04 using neo-zmeventnotification with GPU/TPU support and MLAPI. There will be a section on the new config options.

** I think I have found a change in OpenCV code that causes breaking changes when using GPU and yolov4 for detections. If using OpenCV 4.5.3-dev + yolo will throw a scalar error. There is a post on the forums here about it and the OP fixed it. I ran into this problem when testing the neo-ZMES system in the new LXC I am using for testing. Forum post -> viewtopic.php?f=43&t=31129&sid=c987d358 ... 8f363c62b2

My stable LXC uses OpenCV 4.5.1 and the scalar error is not there, if I use the fix on the instance using OpenCV 4.5.1 it throws a different error all together -> only integer scalar arrays can be converted to a scalar index.

I have implemented a new fix for now that checks the OpenCV version and uses the fix based on if the version of opencv is 4.5.2 or above. I am unsure which version causes the breaking change so some testing will need to be done. Right now the fix is working for the LXC using OpenCV 4.5.4 and the 4.5.1 version instance uses the old code.
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Testing of the new system is going well, I have found some bugs and am fixing them as I go. I have spun up a fresh LXC and am installing ZM with GPU and TPU support with MLAPI set up. I am documenting the install and all the weird edge case errors I ran into (like using CUDA 10.2 and needing to use an older version of GCC) and stumbling upon this OpenCV version code breaking change by accident, I have already issued a fix but I am expecting errors until we nail down exactly which version of OpenCV changed to this new breaking code.

I am at the point of testing the new neo-zmeventnotification install with CPU/GPU/TPU detections and a virgin config file/install. I should catch most of the bugs testing it this way. Tmrw I will start installing neo-mlapi on the same host as neo-zmes and test, then instal neo-mlapi into its own LXC and test it (as a remote mlapi install). As it sits right now most of the code is working, there are small bugs I am ironing out and the repo should be ready for public testing in the next days.

I am excited to see people using the new encrypted credentials system between zmes and mlapi and start hammering mlapi installs with multiple zmes instances worth of events. My plan is to rewrite a lot of the system to optimize things, I made a lot of band aids to interface to @plaiblepixels original code and I think writing my own system would allow me to really get some of the things I would like to add working.

If anyone has any requests for enhancements, fire away. Someone made a post about mlapi not being able to do detections for more than 1 zmes host and I managed to hammer that out in a cpl days. Now things just need testing!

I will post the Medium article with the Install guide probably tmrw, it is very thorough! I will try and write up some docs on the changes honestly I have changed so much it's hard to remember!
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

Repos to watch ->

https://github.com/baudneo/zmeventnotification
https://github.com/baudneo/pyzm
https://github.com/baudneo/mlapi

The commits are trash and the plan is to move this code to a separate repo and then re-fork the ZM original repos and do proper commits so upstream can merge whatever they like. I literally taught myself python to work on this project, so the code might not be the prettiest, lol.

Medium article - Warning - It is a work in progress. There is some pretty good info in there might now. it is long as it is being worked on actively, in the end I will split the article up into sections/pages.

https://medium.com/@baudneo/install-zon ... fab7d7afe7
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 »

I have just pushed some recent bug fixes and the OpenCV scalar hotfix to zmes and pyzm repos. I am testing zmeventnotification on live events to see if anything happens and will be installing mlapi today to test. In theory, the repos are ready to start beta testing. I will make a new post later on when I feel like they are ready, ready. It should work at the moment though, so browse the Medium article, the example configs and start tinkering!
Post Reply