ZMEventServer config and speed questions

[ibrewster]

Two general questions about ZM Event Server:

1) It seems like every time I go to update my ZM Event Server, there have been significant changes to the config files, meaning that I have to go through my existing configs line-by-line, comparing them to the new ones, and figuring out what I need to change to make it work. Is there a better way to do this? For example, some sort of "override" config file, where I can put just the things I need to change, and they will override what is in the main config file?

2) My primary use for the event server is to trigger an automatic video doorbell system. That is, when motion and a person/car/truck/etc is detected, ZM Event Monitor posts a message, which the digital photo frame receives to sound a chime and show the live video. At the moment it is taking several seconds for the event monitor to process any given image and send a message. I realize that this is hugely hardware dependent, and arguably impressive that it can manage things that fast (I am NOT knocking the speed!), but are there any settings I could tweak to make it go faster? Also, if I was to update *one* piece of hardware to improve things (every part of my current system is several years old), what upgrade would give me the best bang for my buck?

[asker]

1. Always look at breaking changes (https://zmeventnotification.readthedocs ... aking.html with every release - it documents what has changed. You shouldn't have to compare old config to new line by line. breaking.md tells you what has changed. But beyond that is is manual. And unfortunately, I'll be making some heavy changes with the next release in the hope I have to make less breaking changes in the future. That being said, the machine learning stuff grew in an ad-hoc fashion, so it will continue to have breaking changes in the near future.

2. make sure you have read the documentation on hooks and key principles (https://zmeventnotification.readthedocs ... iples.html) - they give you a good flow on how thing work which will help you break down parts better. Specific to your question:
a) Look at mlapi - a big part of the delay is the model load time. MLAPI only loads the model once, but the downside is it keeps it in memory
b) Look at using a low overhead model (Tiny Yolov4)

Bottom line, in the ML part, the more you read the docs as well as research on areas you can improve, the better it will get. Don't think of the ES ML hooks to be a very user friendly system - it's more like a raw platform with some basic functionality that works on auto mode but a lot depends on what you tweak (and that is a discovery process.)

Also, if I was to update *one* piece of hardware to improve things (every part of my current system is several years old), what upgrade would give me the best bang for my buck?

As of today, get a GPU. In the next release, I'll also support Google Coral USB Accelerator

[ibrewster]

Thanks for the suggestion of using MLAPI - that improved detection speed from around 5.2 seconds to only 2-3 seconds, so that's a significant improvement! I haven't had any luck using Tiny YoloV3 - it never seems to detect anything, even when the full model has detections in the high 90% confidence (and is correct about that). Is switching to Tiny YoloV4 just a matter of changing the paths, and would it be more likely to work? Of course, I can certainly just try it...

yeah, I look at breaking changes, I just didn't think that it was necessarily a comprehensive list of additions/changes/deletions to the config file. If it is, that makes things easier. Thanks!

One follow-up question: logic seems to indicate that if I limit the area of the image in which to do recognition (by importing my ZM zones and limiting to zones where motion was detected) that should improve speed at least somewhat. Unfortunately, in my testing turning on the import zones features breaks detection completely - I get no detections on events where previously it did detect something within a ZM zone (even without the limiting to motion detected zones on). Any tips for how to troubleshoot this?

Thanks again, and BTW - great software!

[asker]

Thanks for the suggestion of using MLAPI - that improved detection speed from around 5.2 seconds to only 2-3 seconds, so that's a significant improvement! I haven't had any luck using Tiny YoloV3 - it never seems to detect anything, even when the full model has detections in the high 90% confidence (and is correct about that). Is switching to Tiny YoloV4 just a matter of changing the paths, and would it be more likely to work? Of course, I can certainly just try it...

I am going to go back to my suggestion of encouraging you to dive into the docs and ini comments.
Specifically, https://zmeventnotification.readthedocs ... -6-onwards, see bullet on using v4.

One follow-up question: logic seems to indicate that if I limit the area of the image in which to do recognition (by importing my ZM zones and limiting to zones where motion was detected) that should improve speed at least somewhat. Unfortunately, in my testing turning on the import zones features breaks detection completely - I get no detections on events where previously it did detect something within a ZM zone (even without the limiting to motion detected zones on). Any tips for how to troubleshoot this?

See debug logs. See docs on how to make sure debug levels are set correctly, then look at why you get no detections. The

Thanks again, and BTW - great software!

Glad you liked it.

[ibrewster]

One follow-up question: logic seems to indicate that if I limit the area of the image in which to do recognition (by importing my ZM zones and limiting to zones where motion was detected) that should improve speed at least somewhat. Unfortunately, in my testing turning on the import zones features breaks detection completely - I get no detections on events where previously it did detect something within a ZM zone (even without the limiting to motion detected zones on). Any tips for how to troubleshoot this?

See debug logs. See docs on how to make sure debug levels are set correctly, then look at why you get no detections. The

Ok, after some hunting through half a dozen logs, I found that when I tried to run `zm_event_start` with import zm zones on, I was getting a HTTP 500 error:

Code: Select all

07/27/20 13:09:58 zmesdetect_m3[6994] INF zm_detect.py:181 [---------| hook version: 5.15.6, ES version: 5.15 , OpenCV version: 4.4.0|------------]
07/27/20 13:09:58 zmesdetect_m3[6994] ERR utils.py:87 [HTTP Error in import_zm_zones:HTTP Error 500: Internal Server Error]
07/27/20 13:09:58 zmesdetect_m3[6994] ERR utils.py:354 [Error parsing config:/etc/zm/objectconfig.ini]
07/27/20 13:09:58 zmesdetect_m3[6994] ERR utils.py:355 [Error was:HTTP Error 500: Internal Server Error]
07/27/20 13:09:58 zmesdetect_m3[6994] FAT utils.py:356 [error: Traceback:Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/zmes_hook_helpers/utils.py", line 347, in process_config
    import_zm_zones(args.get('monitorid'), args.get('reason'))
  File "/usr/local/lib/python3.6/site-packages/zmes_hook_helpers/utils.py", line 85, in import_zm_zones
    input_file = opener.open(url)
  File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

So I then went looking for the error log, and found both error.log and cake_error.log, both of which seemed to have the same content. This content appears to indicate that it can't connect to the MySQL server, even though Zoneminder is running (and connecting to the mysql server) just fine:

Code: Select all

2020-07-27 13:13:02 Warning: Warning (2): Use of undefined constant ZM_DB_SSL_CA_CERT - assumed 'ZM_DB_SSL_CA_CERT' (this will throw an Error in a future version of PHP) in [/usr/share/zoneminder/www/api/lib/Cake/Model/ConnectionManager.php, line 69]
Trace:
ErrorHandler::handleError() - CORE/Cake/Error/ErrorHandler.php, line 230
ConnectionManager::_init() - CORE/Cake/Model/ConnectionManager.php, line 69
ConnectionManager::getDataSource() - CORE/Cake/Model/ConnectionManager.php, line 83
Model::setDataSource() - CORE/Cake/Model/Model.php, line 3646
Model::setSource() - CORE/Cake/Model/Model.php, line 1162
Model::getDataSource() - CORE/Cake/Model/Model.php, line 3673
Model::schema() - CORE/Cake/Model/Model.php, line 1394
EnumBehavior::setup() - APP/Plugin/CakePHP-Enum-Behavior/Model/Behavior/EnumBehavior.php, line 34
BehaviorCollection::load() - CORE/Cake/Model/BehaviorCollection.php, line 145
BehaviorCollection::init() - CORE/Cake/Model/BehaviorCollection.php, line 66
Model::__construct() - CORE/Cake/Model/Model.php, line 781
ReflectionClass::newInstance() - [internal], line ??
ClassRegistry::init() - CORE/Cake/Utility/ClassRegistry.php, line 169
Controller::loadModel() - CORE/Cake/Controller/Controller.php, line 741
ZonesController::forMonitor() - APP/Controller/ZonesController.php, line 30
ReflectionMethod::invokeArgs() - [internal], line ??
Controller::invokeAction() - CORE/Cake/Controller/Controller.php, line 491
AppController::invokeAction() - APP/Plugin/Crud/Lib/CrudControllerTrait.php, line 31
Dispatcher::_invoke() - CORE/Cake/Routing/Dispatcher.php, line 193
Dispatcher::dispatch() - CORE/Cake/Routing/Dispatcher.php, line 167
[main] - APP/webroot/index.php, line 107

2020-07-27 13:13:02 Warning: Warning (2): Use of undefined constant ZM_DB_SSL_CLIENT_KEY - assumed 'ZM_DB_SSL_CLIENT_KEY' (this will throw an Error in a future version of PHP) in [/usr/share/zoneminder/www/api/lib/Cake/Model/ConnectionManager.php, line 69]
Trace:
ErrorHandler::handleError() - CORE/Cake/Error/ErrorHandler.php, line 230
ConnectionManager::_init() - CORE/Cake/Model/ConnectionManager.php, line 69
ConnectionManager::getDataSource() - CORE/Cake/Model/ConnectionManager.php, line 83
Model::setDataSource() - CORE/Cake/Model/Model.php, line 3646
Model::setSource() - CORE/Cake/Model/Model.php, line 1162
Model::getDataSource() - CORE/Cake/Model/Model.php, line 3673
Model::schema() - CORE/Cake/Model/Model.php, line 1394
EnumBehavior::setup() - APP/Plugin/CakePHP-Enum-Behavior/Model/Behavior/EnumBehavior.php, line 34
BehaviorCollection::load() - CORE/Cake/Model/BehaviorCollection.php, line 145
BehaviorCollection::init() - CORE/Cake/Model/BehaviorCollection.php, line 66
Model::__construct() - CORE/Cake/Model/Model.php, line 781
ReflectionClass::newInstance() - [internal], line ??
ClassRegistry::init() - CORE/Cake/Utility/ClassRegistry.php, line 169
Controller::loadModel() - CORE/Cake/Controller/Controller.php, line 741
ZonesController::forMonitor() - APP/Controller/ZonesController.php, line 30
ReflectionMethod::invokeArgs() - [internal], line ??
Controller::invokeAction() - CORE/Cake/Controller/Controller.php, line 491
AppController::invokeAction() - APP/Plugin/Crud/Lib/CrudControllerTrait.php, line 31
Dispatcher::_invoke() - CORE/Cake/Routing/Dispatcher.php, line 193
Dispatcher::dispatch() - CORE/Cake/Routing/Dispatcher.php, line 167
[main] - APP/webroot/index.php, line 107

2020-07-27 13:13:02 Warning: Warning (2): Use of undefined constant ZM_DB_SSL_CLIENT_CERT - assumed 'ZM_DB_SSL_CLIENT_CERT' (this will throw an Error in a future version of PHP) in [/usr/share/zoneminder/www/api/lib/Cake/Model/ConnectionManager.php, line 69]
Trace:
ErrorHandler::handleError() - CORE/Cake/Error/ErrorHandler.php, line 230
ConnectionManager::_init() - CORE/Cake/Model/ConnectionManager.php, line 69
ConnectionManager::getDataSource() - CORE/Cake/Model/ConnectionManager.php, line 83
Model::setDataSource() - CORE/Cake/Model/Model.php, line 3646
Model::setSource() - CORE/Cake/Model/Model.php, line 1162
Model::getDataSource() - CORE/Cake/Model/Model.php, line 3673
Model::schema() - CORE/Cake/Model/Model.php, line 1394
EnumBehavior::setup() - APP/Plugin/CakePHP-Enum-Behavior/Model/Behavior/EnumBehavior.php, line 34
BehaviorCollection::load() - CORE/Cake/Model/BehaviorCollection.php, line 145
BehaviorCollection::init() - CORE/Cake/Model/BehaviorCollection.php, line 66
Model::__construct() - CORE/Cake/Model/Model.php, line 781
ReflectionClass::newInstance() - [internal], line ??
ClassRegistry::init() - CORE/Cake/Utility/ClassRegistry.php, line 169
Controller::loadModel() - CORE/Cake/Controller/Controller.php, line 741
ZonesController::forMonitor() - APP/Controller/ZonesController.php, line 30
ReflectionMethod::invokeArgs() - [internal], line ??
Controller::invokeAction() - CORE/Cake/Controller/Controller.php, line 491
AppController::invokeAction() - APP/Plugin/Crud/Lib/CrudControllerTrait.php, line 31
Dispatcher::_invoke() - CORE/Cake/Routing/Dispatcher.php, line 193
Dispatcher::dispatch() - CORE/Cake/Routing/Dispatcher.php, line 167
[main] - APP/webroot/index.php, line 107

2020-07-27 13:13:02 Error: [MissingConnectionException] Database connection "Mysql" is missing, or could not be created.
Exception Attributes: array (
  'class' => 'Mysql',
  'message' => 'SQLSTATE[HY000] [2006] MySQL server has gone away',
  'enabled' => true,
)
Request URL: /zm/api/zones/forMonitor/3.json?username=<my_user>&password=<my_password>&user=<my_user>&pass=<my_password>
Stack Trace:
#0 /usr/share/zoneminder/www/api/lib/Cake/Model/Datasource/DboSource.php(271): Mysql->connect()
#1 /usr/share/zoneminder/www/api/lib/Cake/Model/ConnectionManager.php(105): DboSource->__construct(Array)
#2 /usr/share/zoneminder/www/api/lib/Cake/Model/Model.php(3646): ConnectionManager::getDataSource('default')
#3 /usr/share/zoneminder/www/api/lib/Cake/Model/Model.php(1162): Model->setDataSource('default')
#4 /usr/share/zoneminder/www/api/lib/Cake/Model/Model.php(3673): Model->setSource('Monitors')
#5 /usr/share/zoneminder/www/api/lib/Cake/Model/Model.php(1394): Model->getDataSource()
#6 /usr/share/zoneminder/www/api/app/Plugin/CakePHP-Enum-Behavior/Model/Behavior/EnumBehavior.php(34): Model->schema()
#7 /usr/share/zoneminder/www/api/lib/Cake/Model/BehaviorCollection.php(145): EnumBehavior->setup(Object(Monitor), Array)
#8 /usr/share/zoneminder/www/api/lib/Cake/Model/BehaviorCollection.php(66): BehaviorCollection->load('CakePHP-Enum-Be...', Array)
#9 /usr/share/zoneminder/www/api/lib/Cake/Model/Model.php(781): BehaviorCollection->init('Monitor', Array)
#10 [internal function]: Model->__construct(NULL)
#11 /usr/share/zoneminder/www/api/lib/Cake/Utility/ClassRegistry.php(169): ReflectionClass->newInstance(Array)
#12 /usr/share/zoneminder/www/api/lib/Cake/Controller/Controller.php(741): ClassRegistry::init('Monitor')
#13 /usr/share/zoneminder/www/api/app/Controller/ZonesController.php(30): Controller->loadModel('Monitor')
#14 [internal function]: ZonesController->forMonitor('3')
#15 /usr/share/zoneminder/www/api/lib/Cake/Controller/Controller.php(491): ReflectionMethod->invokeArgs(Object(ZonesController), Array)
#16 /usr/share/zoneminder/www/api/app/Plugin/Crud/Lib/CrudControllerTrait.php(31): Controller->invokeAction(Object(CakeRequest))
#17 /usr/share/zoneminder/www/api/lib/Cake/Routing/Dispatcher.php(193): AppController->invokeAction(Object(CakeRequest))
#18 /usr/share/zoneminder/www/api/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke(Object(ZonesController), Object(CakeRequest))
#19 /usr/share/zoneminder/www/api/app/webroot/index.php(107): Dispatcher->dispatch(Object(CakeRequest), Object(CakeResponse))
#20 {main}

So now I'm a bit lost. Zoneminder can obviously connect, since it uses the database to store events, and that's working fine. But this seems to indicate that the zone minder API *can't* connect to the DB. Is it a different config or something? I guess I'll keep digging, see if I can find anything else to indicate why it can't connect...

[ibrewster]

I guess I'll keep digging, see if I can find anything else to indicate why it can't connect...

Found it, in an OLD forum thread: viewtopic.php?t=27518#p108390 Apparently my zm.conf was missing a few required parameters.

For what it's worth, importing the ZM zones doesn't seem to have sped things up any, but at least it works now, so that's a problem solved.