ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Discussion topics related to mobile applications and ZoneMinder Event Server (including machine learning)
Post Reply
tsp84
Posts: 87
Joined: Thu Dec 24, 2020 4:04 am

ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 » Sun Jul 18, 2021 2:22 am

Image

Just seeing if anyone would be interested? I tried doing it with the mp4 as well but for some reason the color palette is off for the bounding box frames no matter what color scheme i convert to using openCV, I'll keep working on that though.[completed]

There is Past/Live event logic and some things are optimized/skipped/forced based on if its a LIVE or PAST/DEBUG event, also takes into consideration if this is an event START or event END.

I've been busy adding all sorts of stuff to zmeventnotification, pyzm, mlapi. I have made a python mqtt add on (TLS,mTLS) for zmes to send the gif/jpg to HA by mqtt camera and am currently working on configurable topics with configurable data and attributes.
Added PushOver logic to the hooks scripts for better control. I also have a HA add on that integrates sensors to control sending PushOver notifications at all and also controls the cool down period between PushOver notifications, per monitor (no AppDaemon scripts, all ZMES side, just need to activate HA API and create a token). You could use a Bayesian sensor for presence detection (working on adding presence logic to pushover send), I currently use an input boolean for sending on/off and an input.number for cooldown time in seconds. I am also adding a counter so if you are home/away and only want to get notified x in a set amount of time, per monitor.

Currently working on a Zone python module to control zones to make it easy to have 'Night' or 'Day' zones switched based on time/sunset/sunrise. Before the method was to create duplicate monitors and setup the Day zones on 1 set , save as a run state and then same for Night on the other duplicate set. Then you needed to make API calls to switch run states........ I did this and made a bash script to manage auth and refresh tokens and also used sunwait to do all the sunrise/sunset stuff with date wrapping, While it does work, it was much clunkier as it restarts all streams, I noticed a few issues cropping from changing run states so decided to go this route.
My way will use Zones and switch them using API (not just Active/Inactive either), no more duplicate monitors that zmNinja has issues displaying the events list for. It will be expandable for not just Night/Day but certain times and you can make groups of Zones to control, utilizes the same config structure as zmes with !secrets and {{ template_replacement }}. It accepts strings like 'tomorrow at noon", "07/19/2021 14:15:05.123456". I haven't tested the human readability string input to the extreme but it's fairly good so far. You setup your LAT, LONG, timezone, city/region data in config file and all the time zone stuff is taken care of in background (DST aware).

I noticed Linked monitors stop triggering after awhile and the only cure is to restart the linked and linked from streams, so there is another add on for zmes to restart specified monitors every x sec/min/hour to avoid missing linked events or just let it run and it will automatically find the linked/linked from monitors and restart them every 3 hours.

I upgraded the animation logic for better error handling, it will drop the anchor frame by fps if certain conditions are met on failure to get the correct frame buffer. To completely annotate the gif would increase the notification time even more, so I only added timestamp (based on event API calls - So when ZM recorded them) and the first few frames of the gif are the objdet.jpg frame that is annotated (draw zone/polygon and filtered objects configurable).

Changed almost ALL requests to ZM API class calls (except for get started and backup calls; if API class isn't working correctly somehow). Did a deep dive on error handling and fallback from remote to local, added several new data points for correlation/graphing in future.

I am looking into feasibility of adding host/client ability to zmes so 1 zmes can be the central notification authority for several other zmes/ZM installs and they communicate amongst themselves, this isn't a small thing so I am just in the research part of it right now. I am heavily leaning towards MQTT (TCP and or WS - SSL/noSSL) or just websockets for protocol (SSL/noSSL) with auth. Also looking into feasibility of TOTP integration with some things (would also like to see ZM add this and am researching PHP and how ZM stores user data to add TOTP secret to user DB)

Includes alias and bash functions for debugging and easier control, If you pass it only event ID (even if you pass it a monitor ID it confirms it with an API call) it will immediately figure out the monitor ID and apply all masking options from config file. Syntax for some of the alias' and functions ->
  • es.start.debug = start zmeventnotification.pl manually with --debug and set to background
    es.start = calls zmdc.pl to start
    es.restart = calls zmdc to restart
    es.stop = calls zmdc to stop
    es.status = calls zmdc to get status
    FUNC: es.objdet.debug <event ID - required> <monitorID - optional> = run zm_detect.sh on a PAST event for debugging or gif/jpeg recreation/force
    FUNC: es.debug.event_start <flags> = same as above but starts it off with the event_start.sh script
  • mlapi.debug = start mlapi with --debug and --config in background
    mlapi.init.start/stop/restart/status = alias linked to mlapi user to control systemctl events
    mlapi.start = start mlapi manually in foreground
    mlapi.stop/restart = manual controls

Completely reworked the filtering logic and added several filtering options for tighter control of detection filtering; per object confidence filtering, per object AREA inside of zone/polygon threshold filtering, most filtering options are per object now instead of global. I am working on the logic for picking the best match, tighter control using sum of all confidences or mean average, etc.
Currently working on buffer for match_past_detections that is time aware, this is actually tricky for me, I have a basic buffer implemented but it needs more work.

Adding most of the ZM API calls to pyzm and classes to interface with/functions to call. Haven't figured out how the index.php?view=video option works yet, everytime ive tried it, it brings me to a video encoding page I've never seen to encode the video into .avi or .3gp or .xxx instead of feeding me a video. Then again I haven't learned much HTML so maybe this is some sort of HTML interface you call strictly from HTML? IDK.

zmMagik has also been updated to use pyzm API calls (meaning mp4 and jpg support) while still retaining ability to feed it urls or a file. It's still a work in progress though (converting ML from in-house to pyzm but also leaving in-house functionality available to call) and also trying to find additional template matching algo's for the find/search function (only added 1 extra so far with roughly same results as original function, honestly the better solution is to train a yolo model to recognize certain things in the frame and then setup zmes to alert on it missing/moved/interacted with).
I am adding the zmMagik functionality and calls into pyzm and thinking of ways to add the blend or annotation features into a daily task that won't affect performance or be intrusive. Also means adding an interface to call from a script to run batches of blend or annotations based on a config file.
The blend logic has been upgraded for tighter control over how blending 2 different monitors should work (if you have duplicate monitors and want to blend them together based on start time or event ID, or make a blended video of monitor x events for the first bit of video and then move onto the next monitor for rest of video). Added support for more then 1 mask.

Added an option to store darknet 'training' images and also 'compare' frames. The training image is the base/raw frame that the object was detected in, no bounding boxes,zones/polygons,timestamp or annotation that you can label to train/test a model. The 'compare' image is a debug frame to compare with that has all the bounding boxes. You have the option to remove/add zone/polygon outline all together and error box's (filtered out objects) outlined in red.

The mqtt and HA add on can be extended for all sorts of other things if needed.

Adding ability to mlapi to reload config file/ feed it a config file via API and also to preload 1st sequence into memory on start (means being able to receive data from ZM host with info about its ML-overrides and sequences)[completed], adding Error handling to better catch weird Errors -> OverFlowError: cannot convert float * infinite to int() - is a weird one i've noticed using yolo and gpu, sometimes it takes 10 days of detections, sometimes its 2 hours before it starts hitting that error and then it needs a restart. Also the USB tpu cable error, I haven't experienced it yet so I haven't been able to setup error catching for that. Basically it catches those weird errors logs them and restarts mlapi, then it has logic for notifications based on severity/occurrence of errors.

I haven't published anything yet, just seeing if there's any outside interest at all? Ill keep this going for my own uses anyways but maybe someone would like some of these add ons. Next big add on is a better HA ZM integration then HA currently uses. I would like to have better history/state data to correlate and graph better then just current events or events in last hour/day/week.

Also hoping to add more data points to ZMES/HA for something to read like grafana or maybe make mqtt sensors in HA vor visualization, but I haven't moved onto actively setting things up for that yet. Would be nice to see data on hits/misses between tpu/yolo detections and correlate it, among several other data points. HA could send daily stats via email/push/whatever. Options are endless!

I'll start working on python 2.7 compatibility and get it somewhat ready for testing if anyone's interested. I will post repos when available and see if anything is PR worthy.


Code: Select all

07/17/21 19:52:03.739366 zmesdetect_m1[1156656] INF zm_detect:661 [zm_detect:prediction: '[70] person(75%),']
07/17/21 19:52:03.740479 zmesdetect_m1[1156656] DBG1 zm_detect:664 [zm_detect:prediction:JSON: {"labels": ["person", "person"], "boxes": [[277, 98, 317, 195], [430, 181, 479, 277]], "frame_id": "70", "confidences": [0.75390625, 0.71484375], "image_dimensions": {"original": [1080, 1920], "resized": [450, 800]}}]
07/17/21 19:52:03.741415 zmesdetect_m1[1156656] DBG2 api:303 [pyzm:api:make_req: 'get'-><hidden>/api/events/17685.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'}]
07/17/21 19:52:04.264553 zmesdetect_m1[1156656] DBG1 zm_detect:697 [zm_detect: old_notes='Linked: Front - Sub' new_notes='[70] detected:person(75%),  Linked:Front - Sub' notes_zone=' Front - Sub' old_notes_split=['Linked', ' Front - Sub'] notes_cause='Linked' g.config.get('api_cause') = 'Linked']
07/17/21 19:52:04.27475 zmesdetect_m1[1156656] DBG1 zm_detect:752 [zm_detect: writing objects.json and objdetect.jpg to /nvr/1/2021-07-17/17685]
07/17/21 19:52:04.315504 zmesdetect_m1[1156656] DBG1 zm_detect:828 [zm_detect:pushover:hass: input_boolean.front_pushover state is {'entity_id': 'input_boolean.front_pushover', 'state': 'on', 'attributes': {'editable': True, 'friendly_name': 'Front Pushover', 'icon': 'mdi:motion'}, 'last_changed': '2021-07-18T01:15:41.314310+00:00', 'last_updated': '2021-07-18T01:15:41.314310+00:00', 'context': {'id': 'f6020a5ad0aa8c9ef3280478c7d05972', 'parent_id': None, 'user_id': None}}]
07/17/21 19:52:04.341597 zmesdetect_m1[1156656] DBG1 zm_detect:842 [zm_detect:pushover:hass: input_number.pushover_cooldown_front state is {'entity_id': 'input_number.pushover_cooldown_front', 'state': '1.0', 'attributes': {'initial': None, 'editable': True, 'min': 0.0, 'max': 3600.0, 'step': 1.0, 'mode': 'box', 'unit_of_measurement': 'seconds', 'friendly_name': 'pushover_cooldown_front', 'icon': 'mdi:clock'}, 'last_changed': '2021-07-18T01:15:41.311723+00:00', 'last_updated': '2021-07-18T01:15:41.311723+00:00', 'context': {'id': 'b993d676124eaa4d72365b103a4a6819', 'parent_id': None, 'user_id': None}}]
07/17/21 19:52:04.342629 zmesdetect_m1[1156656] DBG2 pyzm_utils:794 [push:pkl:trying to load /var/lib/zmeventnotification/push/mon-1-pushover.pkl]
07/17/21 19:52:04.343596 zmesdetect_m1[1156656] DBG2 zm_detect:880 [going to save model training and compare images to: /nas/images/zm-ml/1/17685-training-frame-70.jpg /nas/images/zm-ml/1/17685-compare-frame-70.jpg]
07/17/21 19:52:04.359816 zmesdetect_m1[1156656] DBG1 zm_detect:898 [zm_detect: replacing old note: 'Linked: Front - Sub' with new note: '[70] detected:person(75%),  Linked:Front - Sub']
07/17/21 19:52:04.360785 zmesdetect_m1[1156656] DBG2 api:303 [pyzm:api:make_req: 'put'-><hidden>/api/events/17685.json payload={'Event[Notes]': '[70] detected:person(75%),  Linked:Front - Sub'} query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'}]
07/17/21 19:52:04.392479 zmesdetect_m1[1156656] DBG1 zm_detect:910 [zm_detect:animation: gathering data...]
07/17/21 19:52:04.41378 zmesdetect_m1[1156656] DBG1 pyzm_utils:128 [r_frame_len=213 | fid+fps*buffer_seconds=120 | fid=70 | fps=10 | buffer_seconds=5 | total_time=21 | target_fps=2 ]
07/17/21 19:52:04.414769 zmesdetect_m1[1156656] DBG1 pyzm_utils:171 [animation: 213 frames to work with->anchor_frame=70 start_frame=20 end_frame=120 skip_frames=5 fps=10]
07/17/21 19:52:04.415696 zmesdetect_m1[1156656] DBG1 pyzm_utils:177 [animation: grabbing frames.....]
07/17/21 19:52:13.973416 zmesdetect_m1[1156656] DBG2 pyzm_utils:214 [animation: grabbed frames: [20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120]]
07/17/21 19:52:13.976537 zmesdetect_m1[1156656] DBG1 pyzm_utils:238 [animation: fast GIF requested....]
07/17/21 19:52:13.977841 zmesdetect_m1[1156656] DBG1 pyzm_utils:258 [gif_buffer_seconds=3.0 | target_fps=4 | gif_start_frame=40 | gif_end_frame=100 | slicing from s1=4 | negative s2=4]
07/17/21 19:52:13.978994 zmesdetect_m1[1156656] DBG1 pyzm_utils:260 [animation:gif: fast gif slicing 4 to -4 from a total of 21]
07/17/21 19:52:18.633982 zmesdetect_m1[1156656] DBG1 pyzm_utils:262 [animation:gif: optimizing...]
07/17/21 19:52:18.930596 zmesdetect_m1[1156656] DBG1 pyzm_utils:265 [animation:gif: saved to /nvr/1/2021-07-17/17685/objdetect.gif, size: 1.37MB, frames: 15]
07/17/21 19:52:18.931662 zmesdetect_m1[1156656] DBG1 zm_detect:935 [zm_detect:pushover:gif: data={'token': '<hidden>', 'user': '<hidden>', 'title': '(17685) Front:Linked->Front - Sub', 'message': '[70] detected:person(75%),  at 07:52 PM, Jul-17', 'url': '<hidden>/zm/cgi-bin/nph-zms?mode=jpeg&scale=50maxfps=5&buffer=1000&replay=single&monitor=1&event=17685&connkey=1288728048174780&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNjI2NTczMTE1LCJleHAiOjE2MjY1ODAzMTUsInVXIiOiJ6bWVzIiwidHlwZSI6ImFjY2VzcyJ9.hIWXdh-L0_KgAX-FZA-HLQzQcOeEYTE01Q7LR3WA3ug', 'url_title': 'View event in browser'} files={'attachment': ('1-objdet.gif', <_io.BufferedReader name='/nvr/1/2021-07-17/17685/objdetect.gif'>, 'image/jpeg')} ]
07/17/21 19:52:22.601827 zmesdetect_m1[1156656] DBG4 pyzm_utils:816 [push:pkl:time since sent:2021-07-17 19:52:22.600034 to /var/lib/zmeventnotification/push/mon-1-pushover.pkl]
07/17/21 19:52:22.618846 zmesdetect_m1[1156656] DBG1 zm_detect:962 [zm_detect:mqtt: enabled and initilising...]
07/17/21 19:52:22.634508 zmesdetect_m1[1156656] DBG1 mqtt:167 [mqtt:connect: <hidden>:8883 trying mTLS -> tls_ca: /nas/mqtt_certs/ca.crt tls_client_key: /nas/mqtt_certs/client-zm.key tls_client_cert: /nas/mqtt_certs/client-zm.crt]
07/17/21 19:52:22.637064 zmesdetect_m1[1156656] DBG2 mqtt:203 [mqtt:connect: waiting for broker to reply (timeout: 2]
07/17/21 19:52:23.232285 zmesdetect_m1[1156656] DBG1 mqtt:54 [mqtt:paho_log: Sending CONNECT (u1, p1, wr0, wq0, wf0, c1, k60) client_id=zmes-mTLS-WPyDvQ0lJw8LwCvi']
07/17/21 19:52:23.718461 zmesdetect_m1[1156656] DBG1 mqtt:54 [mqtt:paho_log: Received CONNACK (0, 0)]
07/17/21 19:52:24.136076 zmesdetect_m1[1156656] DBG1 mqtt:58 [mqtt:connect: connected to broker with flags-> {'session present': 0} result code-> 0]
07/17/21 19:52:24.142466 zmesdetect_m1[1156656] DBG1 mqtt:128 [mqtt:grab_image: gif to be used is: /nvr/1/2021-07-17/17685/objdetect.gif, converting to byte array]
07/17/21 19:52:24.146044 zmesdetect_m1[1156656] DBG2 mqtt:232 [mqtt:publish: sending -> topic: zmes/picture/1  data: <serialized bytearray>  size=1.37 MB]
07/17/21 19:52:24.147679 zmesdetect_m1[1156656] DBG1 mqtt:54 [mqtt:paho_log: Sending PUBLISH (d0, q0, r0, m1), 'b'zmes/picture/1'', ... (1439998 bytes)]
07/17/21 19:52:24.150125 zmesdetect_m1[1156656] DBG2 mqtt:230 [mqtt:publish: sending -> topic: 'zmes/data/1' data: '{"mid": "1", "name": "Front", "reason": "[70] detected:person(75%),", "eid": "17685", "zone": "Front - Sub", "cause": "Linked"}']
07/17/21 19:52:24.151541 zmesdetect_m1[1156656] DBG1 mqtt:54 [mqtt:paho_log: Sending PUBLISH (d0, q0, r0, m2), 'b'zmes/data/1'', ... (127 bytes)]
07/17/21 19:52:24.152438 zmesdetect_m1[1156656] DBG2 mqtt:230 [mqtt:publish: sending -> topic: 'zmes/detection/1' data: '{"labels": ["person", "person"], "boxes": [[277, 98, 317, 195], [430, 181, 479, 277]], "frame_id": "70", "confidences": [0.75390625, 0.71484375], "image_dimensions": {"original": [1080, 1920], "resized": [450, 800]}}']
07/17/21 19:52:24.154036 zmesdetect_m1[1156656] DBG1 mqtt:54 [mqtt:paho_log: Sending PUBLISH (d0, q0, r0, m3), 'b'zmes/detection/1'', ... (216 bytes)]
07/17/21 19:52:24.155096 zmesdetect_m1[1156656] DBG2 mqtt:244 [mqtt:close: disconnecting from mqtt broker: <hidden>:8883]
07/17/21 19:52:24.156431 zmesdetect_m1[1156656] DBG1 mqtt:54 [mqtt:paho_log: Sending DISCONNECT]
07/17/21 19:52:24.157586 zmesdetect_m1[1156656] DBG2 zm_detect:988 [perf: 1-->17685: [total:27.534651 sec] [detection:7.081833 sec] [processing objdetect.jpg and .gif:20.452842 sec] [mqtt:1.554616 sec]]
07/17/21 19:52:24.159012 zmesdetect_m1[1156656] DBG1 zm_detect:999 [zm_detect: closing logs]
Last edited by tsp84 on Mon Jul 26, 2021 5:17 am, edited 6 times in total.

SteveGilvarry
Posts: 475
Joined: Sun Jun 29, 2014 1:12 pm
Location: Melbourne, AU

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by SteveGilvarry » Sun Jul 18, 2021 1:58 pm

Nice.
I am sure Asker/PliablePixels, who is still supporting at the moment will be happy to accept PR's to the repos that we just moved under ZoneMinder organisation in Github. And we are always happy to welcome people willing to develop new features, clean up existing or just generally help out.
It is safe to assume that if you found it worthwhile to develop others will also value it.
Production Zoneminder 1.36.x
Random Selection of Cameras (Dahua and Hikvision)
More Dev VM's than I care to count.

tsp84
Posts: 87
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 » Sun Jul 18, 2021 6:47 pm

I think in the future I would also like to add remote backup to mlapi and ability to smartly queue events amongst different processors based on queue/load. This is only an idea at the moment though, nothing more. I haven't experienced the TPU overload i've seen mentioned places online but I think that also has a lot to do with zone tuning if you don't have a large amount of monitors creating events. If you do have a lot of monitors making events IDK how mlapi/zmes responds and utilizes its Semaphore locks and how the memory/model is passed around as I haven't set up an environment to test like that yet but it sure sounds fun.

yannlieb
Posts: 12
Joined: Fri Mar 19, 2021 9:40 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by yannlieb » Thu Jul 22, 2021 7:29 am

This is going too deep technical for me to understand everything, but it sounds promising. Thanks for taking the time to detail your plans!

tsp84
Posts: 87
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 » Mon Jul 26, 2021 7:34 pm

The linked script is basically ready, there is more polishing to be done but at the moment it works.

Its python3, and needs pyzm to work at the moment, also only logs to console (I will add logging and fall-back not using pyzm soon). Currently only automatically finds linked to and linked from monitors but am planning to also let you over ride that if you want in the config file (idk if this is really needed though). Config file is just path to secrets file, portal/portal_api, user, password, allow self signed cert options at the moment.

Script needs to be setup to run as a cron job every x mins/hours/etc.

So, example being, you have a monitor with id: 2 set to Mocord and a monitor id:1 set to Nodect. Both of these monitors need to be 'enabled'. Monitor 1 is the 'linked to' and 2 is the 'linked from'.
The script checks both of those monitors current function and saves it, sets the monitors stream to None waits 5 seconds and then sets the monitors back to whatever function they were before setting to None.

Code: Select all

07/26/21 13:19:36.974676 CNSL[DBG1] linked_reset:34->[util: secret filename: ./secrets.ini ]
07/26/21 13:19:36.975688 CNSL[DBG2] linked_reset:69->[util: secret token found in config: !ZMUSER ]
07/26/21 13:19:36.976247 CNSL[DBG2] linked_reset:69->[util: secret token found in config: !ZMPASS ]
07/26/21 13:19:36.976766 CNSL[DBG2] linked_reset:69->[util: secret token found in config: !PORTAL ]
07/26/21 13:19:36.977291 CNSL[DBG2] linked_reset:69->[util: secret token found in config: !API_PORTAL ]
07/26/21 13:19:36.977955 CNSL[DBG2] linked_reset:94->[util: parameter substitution ({{ <var> }}) ]
07/26/21 13:19:36.979292 CNSL[DBG2] api:83->[pyzm:api: SSL certificate verification disabled (encryption still works, vuln to spoofing) ]
07/26/21 13:19:36.980008 CNSL[DBG1] api:199->[pyzm:api: no token found, using user/pass to login ]
07/26/21 13:19:37.133449 CNSL[DBG2] api:228->[pyzm:api: detected API ver 2.0+, using token system ]
07/26/21 13:19:37.134183 CNSL[DBG1] api:236->[pyzm:api: access token expires on: 2021-07-26 19:19:37.133546 (21600s) ]
07/26/21 13:19:37.134847 CNSL[DBG1] api:243->[pyzm:api: refresh token expires on: 2021-07-27 13:19:37.134228 (86400s) ]
07/26/21 13:19:37.135726 CNSL[DBG2] Monitors:22->[Retrieving monitors via API ]
07/26/21 13:19:37.13656 CNSL[DBG2] api:303->[pyzm:api:make_req: 'get'->https://<hidden>/zm/api/monitors.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]
mid: 1 name: Front function: Nodect linked monitors: 2 enabled:True
mid: 2 name: Front - Sub function: Mocord linked monitors: None enabled:True
mid: 3 name: Back Alley function: Modect linked monitors: None enabled:True
mid: 4 name: Back Yard function: Monitor linked monitors: None enabled:True
inside linked_from loop x=2 mon_ = <pyzm.helpers.Monitor.Monitor object at 0x7fe58c7b79d0>
has_linked = [1]  ----  linked_from = [2] ---- tot_mons = [2, 1] --- tot = {1: 'Nodect', 2: 'Mocord'}
07/26/21 13:19:37.156598 CNSL[DBG2] api:303->[pyzm:api:make_req: 'post'->https://<hidden>/zm/api/monitors/1.json payload={'Monitor[Function]': 'None'} query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]
07/26/21 13:19:37.413014 CNSL[DBG2] api:303->[pyzm:api:make_req: 'post'->https://<hidden>/zm/api/monitors/2.json payload={'Monitor[Function]': 'None'} query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]
sleeping for 5 seconds and then returning monitors to previous function
07/26/21 13:19:47.725932 CNSL[DBG2] api:303->[pyzm:api:make_req: 'post'->https://<hidden>/zm/api/monitors/1.json payload={'Monitor[Function]': 'Nodect'} query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]
07/26/21 13:19:48.186538 CNSL[DBG2] api:303->[pyzm:api:make_req: 'post'->https://<hidden>/zm/api/monitors/2.json payload={'Monitor[Function]': 'Mocord'} query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]

Process finished with exit code 0

tsp84
Posts: 87
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 » Mon Jul 26, 2021 7:41 pm

The zone switcher script (module if daemonized) is almost done as well I just don't have a use case for it at the moment So i can't test it out thoroughly. I am unsure if it should be a standalone script for a cron job or to daemonize it and let it compare times every x seconds/minutes.

Most of the other stuff is ready as well i'm just figuring out how to roll it all out and get it installed. It is very different from the core code and I added and moved a lot of things into pyzm. So it would be a forked pyzm install, forked mlapi and forkled zmeventnotification needed to run in a venv or however you set up your python env. For now its going to be a python3 and zm 1.34+ minimum and there may be a cpl more dependencies to download.

I have added a python based USB port resetting tool for the USB TPU delegate error but haven't been able to test it out or automate a USB port reset when its actually needed yet.

Some log output ->

Code: Select all

07/26/21 13:51:28.377207 CNSL[DBG1] zsone:38->[util: secret filename: ./secrets.ini ]
07/26/21 13:51:28.378226 CNSL[DBG2] zsone:73->[util: secret token found in config: !ZMUSER ]
07/26/21 13:51:28.378775 CNSL[DBG2] zsone:73->[util: secret token found in config: !ZMPASS ]
07/26/21 13:51:28.379298 CNSL[DBG2] zsone:73->[util: secret token found in config: !PORTAL ]
07/26/21 13:51:28.379818 CNSL[DBG2] zsone:73->[util: secret token found in config: !API_PORTAL ]
07/26/21 13:51:28.380369 CNSL[DBG2] zsone:73->[util: secret token found in config: !LAT ]
07/26/21 13:51:28.38089 CNSL[DBG2] zsone:73->[util: secret token found in config: !LONG ]
07/26/21 13:51:28.381842 CNSL[DBG2] zsone:98->[util: parameter substitution ({{ <var> }}) ]
07/26/21 13:51:28.383026 CNSL[DBG2] api:83->[pyzm:api: SSL certificate verification disabled (encryption still works, vuln to spoofing) ]
07/26/21 13:51:28.383715 CNSL[DBG1] api:199->[pyzm:api: no token found, using user/pass to login ]
07/26/21 13:51:28.54551 CNSL[DBG2] api:228->[pyzm:api: detected API ver 2.0+, using token system ]
07/26/21 13:51:28.546252 CNSL[DBG1] api:236->[pyzm:api: access token expires on: 2021-07-26 19:51:28.545611 (21600s) ]
07/26/21 13:51:28.546924 CNSL[DBG1] api:243->[pyzm:api: refresh token expires on: 2021-07-27 13:51:28.546298 (86400s) ]
07/26/21 13:51:28.547953 CNSL[DBG2] Zones:48->[Retrieving zones via API ]
07/26/21 13:51:28.548806 CNSL[DBG2] api:303->[pyzm:api:make_req: 'get'->https://<hidden>/zm/api/zones/index.json query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciO...'} ]

Information for <CITY>/<REGION>
Timezone: America/<TZ NAME>
Latitude: xx.xx; Longitude: -xxx.xx

just after midnight -> 2021-07-26 00:00:00.000001-06:00
now -> 2021-07-26 13:51:28.599612-06:00
sunrise -> 2021-07-26 05:40:45.884823-06:00
noon -> 2021-07-26 13:40:30-06:00
sunset -> 2021-07-26 21:39:12.639746-06:00
just before midnight -> 2021-07-26 23:59:59.999999-06:00
midnight -> 2021-07-26 00:00:00-06:00

currently between noon and sunset

trigger = {'trigger_1': {'name': 'Shift Change (Closing) - 25 mins before', 'offset': '25', 'time': 'today at 10:30PM', 'zones': {'1': {'name': 'cash room door', 'type': 'Active'}, '2': {'name': 'cash room door window', 'type': 'Active'}, '3': {'name': 'cash room door handle', 'type': 'Active'}, '4': {'name': 'staff room', 'type': 'Privacy'}, '5': {'name': 'staff room', 'type': 'Privacy'}}}}

converting: trigger_1:time 'today at 10:30PM' -> 2021-07-26 22:30:00-06:00
offset for trigger_1 -> 25
offset applied! 2021-07-26 22:05:00-06:00
currently before trigger_1:time

trigger = {'trigger_2': {'name': 'Shift Change (Opening) - 30 mins After', 'offset': '-30', 'time': 'today at 10:30PM', 'zones': {'1': {'name': 'staff room', 'type': 'Active'}, '2': {'name': 'staff room', 'type': 'Active'}}}}

converting: trigger_2:time 'today at 10:30PM' -> 2021-07-26 22:30:00-06:00
offset for trigger_2 -> -30
offset applied! 2021-07-26 23:00:00-06:00
currently before trigger_2:time

trigger = {'trigger_3': {'name': 'Management Meeting', 'time': 'today at noon', 'zones': {'1': {'name': 'walkway', 'type': 'Active'}, '2': {'name': 'cash room door window', 'type': 'Inactive'}, '3': {'name': 'cash room door handle', 'type': 'Inactive'}}}}

converting: trigger_3:time 'today at noon' -> 2021-07-26 12:00:00-06:00
currently after trigger_3:time, TRIGGERING!

match has been found!
No need to switch Zone 'type', already set as specified -> Active

Process finished with exit code 0

tsp84
Posts: 87
Joined: Thu Dec 24, 2020 4:04 am

Re: ZMES upgrades, GIF annotated for android pushover notifications, MQTT, etc.

Post by tsp84 » Thu Jul 29, 2021 5:59 am

Repos are ready to go, test away.

The current zmes_hook_helpers and pyzm from python library will be removed during new installs or start a new virtual env for this setup. This new ZMES requires my forked version of pyzm, I also updated mlapi. I updated config files as well so take a look in there to see the new options to configure.

Order to install:

optional: remove pyzm and zmes_hook_helpers
clone pyzm and then run ./install, it will ask you to uninstall old pyzm first (do not install pyzm using pip or it will grab the original)
clone zmeventnotification and run install, it will ask you to remove zmes_hook_helpers
clone mlapi, install like original


ZMES:
https://github.com/baudneo/zmeventnotification

PYZM:
https://github.com/baudneo/pyzm

MLAPI:
https://github.com/baudneo/mlapi

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests