Odd MLAPI detection behavior after upgrade

[ibrewster]

I just completed a wholesale update of my zoneminder/zmeventnotification/mlapi system (I was several versions behind), and now MLAPI is detecting objects outside of my zones, even though I have both the `import_zm_zones` and `only_triggered_zm_zones` options set to `yes` in the mlapiconfig.ini file that MLAPI is pointed to. To make things even odder, this behavior doesn't show up on the first detection attempt - only after several detections. Turning off MLAPI and just using the "onboard" detection seems to work properly.

See this test output both with MLAPI running and with it disabled. Note that with MLAPI enabled at first it looks like it is working, but then on run #4 it starts detecting a "truck", even though it is still looking at the same event. There is a truck in the image, so that much is correct, however it is outside *any* of my detection zones, much less the one that was triggered to create this event. Once I switch off mlapi, I stop getting this false detection:

Code: Select all

root@watchman:~# systemctl start mlapi
root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3
[a] detected:truck:52% --SPLIT--{"labels": ["truck"], "boxes": [[-2, -1, 396, 121]], "frame_id": "alarm", "confidences": [0.5240153074264526], "image_dimensions": {"original": [1080, 1920], "resized": [1080, 1920]}}
root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3
[a] detected:truck:52% --SPLIT--{"labels": ["truck"], "boxes": [[-2, -1, 396, 121]], "frame_id": "alarm", "confidences": [0.5240153074264526], "image_dimensions": {"original": [1080, 1920], "resized": [1080, 1920]}}
root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3
[a] detected:truck:52% --SPLIT--{"labels": ["truck"], "boxes": [[-2, -1, 396, 121]], "frame_id": "alarm", "confidences": [0.5240153074264526], "image_dimensions": {"original": [1080, 1920], "resized": [1080, 1920]}}


root@watchman:~# systemctl stop mlapi
root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 674735 3

root@watchman:~# 

Did I do something wrong/miss something with the new config formats? Or is this a bug in MLAPI?

[asker]

Compare mlapi debug logs between the version that reports nothing vs. the one that does.

[ibrewster]

Ok, now I'm really confused. I re-started MLAPI to get the logs from when it gives the incorrect results, only now it's working perfectly. I guess I'll keep an eye on it, and update this thread if/when it mis-behaves again. Thanks anyway!

[ibrewster]

Ok, it started acting up again last night. Looking at the logs generated when running the test shows this line in the "bad" run that is not present in the "good" run:

Code: Select all

May 26 05:53:29 watchman mlapi[2305103]: May 26 2021 05:53:29.135784 [DBG 2] No polygons, adding full image polygon: {'name': 'full_image', 'value': [(0, 0), (1920, 0), (1920, 1080), (0, 1080)], 'pattern': None}

Which, of course, explains *what* is happening (it's looking at the full image, rather than using my ZM defined zones), but not why.

Of course, further down we see the effects of this. In the "good" run we get:

Code: Select all

intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:driveway-back,POLYGON ((0 602, 1193 562, 1522 833, 1371 1076, 0 1076, 0 602))
 intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:porch,POLYGON ((1370 707, 1813 521, 1916 574, 1916 615, 1857 750, 1850 759, 1833 785, 1829 752, 1732 656, 1601 809, 1520 830, 1370 707))
 intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:driveway-front,POLYGON ((2 183, 131 178, 131 161, 680 167, 1188 561, 333 588, 314 232, 0 236, 2 183))
 intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:fireweed,POLYGON ((1732 656, 1827 746, 1834 761, 1833 785, 1859 748, 1916 1076, 1375 1076, 1521 830, 1600 812, 1732 656))
 intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:yard,POLYGON ((1262 202, 1814 519, 1372 706, 676 168, 950 193, 1151 377, 1320 322, 1262 202))
 intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:front_bush1,POLYGON ((949 191, 1255 196, 1320 325, 1151 377, 949 191))
 intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:side_bushes,POLYGON ((1275 118, 1851 438, 1859 544, 1258 198, 1275 118))
 object:truck at POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) does not fall into any polygons, removing...
 We did not find any object matches in frame: alarm
 ...
 ...
 Returning {'matched_data': {'boxes': [], 'error_boxes': [], 'labels': [], 'confidences': [], 'frame_id': None, 'model_names': [], 'image_dimensions': {'original': (1080, 1920), 'resized': (1080, 1920)}, 'image': None, 'polygons': [{'name': 'driveway-back', 'value': [(0, 602), (1193, 562), (1522, 833), (1371, 1076), (0, 1076)], 'pattern': None}, {'name': 'porch', 'value': [(1370, 707), (1813, 521), (1916, 574), (1916, 615), (1857, 750), (1850, 759), (1833, 785), (1829, 752), (1732, 656), (1601, 809), (1520, 830)], 'pattern': None}, {'name': 'driveway-front', 'value': [(2, 183), (131, 178), (131, 161), (680, 167), (1188, 561), (333, 588), (314, 232), (0, 236)], 'pattern': None}, {'name': 'fireweed', 'value': [(1732, 656), (1827, 746), (1834, 761), (1833, 785), (1859, 748), (1916, 1076), (1375, 1076), (1521, 830), (1600, 812)], 'pattern': None}, {'name': 'yard', 'value': [(1262, 202), (1814, 519), (1372, 706), (676, 168), (950, 193), (1151, 377), (1320, 322)], 'pattern': None}, {'name': 'front_bush1', 'value': [(949, 191), (1255, 196), (1320, 325), (1151, 377)], 'pattern': None}, {'name': 'side_bushes', 'value': [(1275, 118), (1851, 438), (1859, 544), (1258, 198)], 'pattern': None}]}, 'all_matches': []}

 

while in the bad run it shows (yes, I know my match pattern is bad, I'll need to fix that):

Code: Select all

intersection: comparing object:truck,POLYGON ((-2 -1, 396 -1, 396 121, -2 121, -2 -1)) to polygon:full_image,POLYGON ((0 0, 1920 0, 1920 1080, 0 1080, 0 0))
 Using global match pattern: ^(?!bird,bench)
 full_image intersects object:truck[[(-2, -1), (396, -1), (396, 121), (-2, 121)]]
 ...
 ...
 Returning {'matched_data': {'boxes': [[-2, -1, 396, 121]], 'error_boxes': [], 'labels': ['truck'], 'confidences': [0.5240153074264526], 'frame_id': 'alarm', 'model_names': ['yolo'], 'image_dimensions': {'original': (1080, 1920), 'resized': (1080, 1920)}, 'image': None, 'polygons': [{'name': 'full_image', 'value': [(0, 0), (1920, 0), (1920, 1080), (0, 1080)], 'pattern': None}]}, 'all_matches': [{'frame_id': 'alarm', 'boxes': [[-2, -1, 396, 121]], 'error_boxes': [], 'labels': ['truck'], 'confidences': [0.5240153074264526], 'detection_types': ['object'], 'model_names': ['yolo']}]}
 

Other than that, and some inconsequential differences (such as lock id's and timings, which would be expected), the two log sections are identical according to a diff. Looking back further, I find this sequence of entries that looks like it may be relevant. I don't see similar entries to the final three lines prior to the good runs:

Code: Select all

May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502751 [DBG 1] Monitor ID 3 provided & matching config found in mlapi, ignoring objectconfig.ini
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502818 [DBG 2] Overriding global ml_sequence with {
May 25 10:05:06 watchman mlapi[484977]: 'general': {
May 25 10:05:06 watchman mlapi[484977]: 'model_sequence...
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502848 [DBG 2] Overriding global stream_sequence with {
May 25 10:05:06 watchman mlapi[484977]: 'frame_strategy': 'most_mode...
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502871 [DBG 2] Overriding global object_detection_pattern with ^(?!bird,bench)...
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502886 [DBG 2] Overriding global model_sequence with object,alpr,face...
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502900 [DBG 2] Overriding global resize with no...
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502915 [DBG 2] Only filtering polygon names that have Motion Porch
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.502992 [DBG 2] Original polygons being used: [{'name': 'driveway-back', 'value': [(0, 602), (1193, 562), (1522, 833), (1371, 1076), (0, 1076)], 'pattern': None}, {'n
ame': 'porch', 'value': [(1370, 707), (1813, 521), (1916, 574), (1916, 615), (1857, 750), (1850, 759), (1833, 785), (1829, 752), (1732, 656), (1601, 809), (1520, 830)], 'pattern': None}, {'name': 'driveway-front', 'value': [(2
, 183), (131, 178), (131, 161), (680, 167), (1188, 561), (333, 588), (314, 232), (0, 236)], 'pattern': None}, {'name': 'fireweed', 'value': [(1732, 656), (1827, 746), (1834, 761), (1833, 785), (1859, 748), (1916, 1076), (1375,
 1076), (1521, 830), (1600, 812)], 'pattern': None}, {'name': 'yard', 'value': [(1262, 202), (1814, 519), (1372, 706), (676, 168), (950, 193), (1151, 377), (1320, 322)], 'pattern': None}, {'name': 'front_bush1', 'value': [(949
, 191), (1255, 196), (1320, 325), (1151, 377)], 'pattern': None}, {'name': 'side_bushes', 'value': [(1275, 118), (1851, 438), (1859, 544), (1258, 198)], 'pattern': None}]
May 25 10:05:06 watchman mlapi[484977]: May 25 2021 10:05:06.503549 [DBG 2] Final polygons being used: []

POSIT (forgive me if this is nuts): when I restart MLAPI and run the test per the command line shown in the initial post, there is no motion zone specified, so the "only_triggered_zm_zones" option isn't used, instead it simply uses all defined zones and works. Once MLAPI is triggered by an actual motion event, however, such as the above logged "Motion Porch", the "only_triggered_zm_zones" option kicks in, filtering the list of polygons down - but for some reason instead of being left with only the "Porch" polygon, I'm left with nothing, leading to the "No Polygons" issue. Additionally - and crucially - after this filtering "failure", the original zones are not restored unless I restart MLAPI, meaning that from that point on, all detections are done using the full image polygon.

Like I said, maybe that posit is completely nuts, if so please disregard it. But it *would* seem to be a logical explanation for the behavior I am seeing...

I still need to dig up the zmeventnotification logs (I may need to tweak logging levels or something to get them, as all I'm finding at the moment are ERR and above) and see if it has similar entries.

[ibrewster]

For what it's worth, disabling the "only_triggered_zm_zones" option (but leaving the "import_zm_zones" option enabled) appears to resolve the issue - with the obvious caveat that it will detect objects in any of my defined zones, not just the alarmed zone. Still better than detecting the cars driving by in the street though!

[asker]

Thanks for the detailed logs.
I think I know what is going on. When your reason is "Motion porch" and your zone name is "porch", I am replacing the reason with "Motion_porch" and then looking for a whole word "porch" in it, which will fail (as I just removed the word delimiter). I removed that with https://github.com/pliablepixels/mlapi/ ... 67f063541b

Please try only_triggered_zm_zones again and let me know

[ibrewster]

It looks like that did it about halfway. The first time looks correct:

Code: Select all

May 28 2021 11:25:26.073179 [DBG 2] Only filtering polygon names that have Motion Driveway-Front
May 28 2021 11:25:26.073258 [DBG 2] Original polygons being used: [{'name': 'driveway-back', 'value': [(0, 602), (1193, 562), (1522, 833), (1371, 1076), (0, 1076)], 'pattern': None}, {'name': 'porch', 'value': [(1370, 707), (1813, 521), (1916, 574), (1916, 615), (1857, 750), (1850, 759), (1833, 785), (1829, 752), (1732, 656), (1601, 809), (1520, 830)], 'pattern': None}, {'name': 'driveway-front', 'value': [(2, 183), (131, 178), (131, 161), (680, 167), (1188, 561), (333, 588), (314, 232), (0, 236)], 'pattern': None}, {'name': 'fireweed', 'value': [(1732, 656), (1827, 746), (1834, 761), (1833, 785), (1859, 748), (1916, 1076), (1375, 1076), (1521, 830), (1600, 812)], 'pattern': None}, {'name': 'yard', 'value': [(1262, 202), (1814, 519), (1372, 706), (676, 168), (950, 193), (1151, 377), (1320, 322)], 'pattern': None}, {'name': 'front_bush1', 'value': [(949, 191), (1255, 196), (1320, 325), (1151, 377)], 'pattern': None}, {'name': 'side_bushes', 'value': [(1275, 118), (1851, 438), (1859, 544), (1258, 198)], 'pattern': None}]
May 28 11:25:26 watchman mlapi[1655344]: May 28 2021 11:25:26.073923 [DBG 2] Final polygons being used: [{'name': 'driveway-front', 'value': [(2, 183), (131, 178), (131, 161), (680, 167), (1188, 561), (333, 588), (314, 232), (0, 236)], 'pattern': None}]

So it is now filtering correctly, giving us the driveway-front polygon that we would expect. Unfortunately, it looks like the full list of polygons is still not being restored after the detection, because shortly thereafter when there is motion detected in a different zone I get this:

Code: Select all

May 28 2021 11:28:50.941516 [DBG 2] Only filtering polygon names that have Motion Porch
May 28 2021 11:28:50.941555 [DBG 2] Original polygons being used: [{'name': 'driveway-front', 'value': [(2, 183), (131, 178), (131, 161), (680, 167), (1188, 561), (333, 588), (314, 232), (0, 236)], 'pattern': None}]
May 28 11:28:50 watchman mlapi[1655344]: May 28 2021 11:28:50.941597 [DBG 2] Final polygons being used: []

Where we see that the original polygons list is now only the one polygon that was filtered earlier (rather than the full list), and that as such we are now left with nothing after filtering for the new zone. Which, of course, leaves us right back where we started. Still, at least one step in the right direction! Thanks for your hard work, and the excellent software!

[asker]

Okay, I think I've fixed it now - I did a few runs by invoking zm_detect with and without '--reason' as well as different values - and looks like the original polygons are being reset now at each invocations on the mlapi side.

(https://github.com/pliablepixels/mlapi/ ... 84ebe2e5fe)

Please give it another shot.

[ibrewster]

Looks good now. Even after filtering, on the next run the original polygon list showed the full list, and as such filtered correctly. Thanks!