I wrote an object-detection add-on for ZM

Forum for questions and support relating to the 1.30.x releases only.
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: I wrote an object-detection add-on for ZM

Post by rockedge »

is there a way to specify a certain camera or more than one in the config file?. Not sure if it will be necessary when setting up filters.
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: I wrote an object-detection add-on for ZM

Post by rockedge »

I know the project well. I have a dev and master branch running of this CCTV NVR system. I actually us ZM as the supplier of streams for it. So I kind of get the best of both when they are combined. I urged certain developers to perhaps communicate better to the benefit of both and especially the end user...but humans being what they are (generally short sighted)......... Any way I think the example the link shows is a good indicator of how good Shin* is becoming. Me, I like to use them (ZM & Shin*) together with my X10 remote sensors....makes me feel all tricky and sh#$.
aindilis
Posts: 4
Joined: Fri May 11, 2018 9:49 pm

Re: I wrote an object-detection add-on for ZM

Post by aindilis »

Hi,

I have dabbled with ZoneMinder in the distant past, but at the time I had difficulty installing it. For now, I'm using the default TrendNet surveillance software, but am planning to make the jump to ZoneMinder. I am a programmer and very interested in AI. I am sure many of the ideas I'm going to mention have been thought of or planned to integrate. I am a systems integrator by nature, and if I could find someone to work with, I would be interested in trying to integrate these capabilities.

My question is, can anyone integrate / help me integrate the following tools into ZoneMinder? First, this integration of YOLO is awesome! I hope it is the latest version of YOLO. Object detection is great - I don't know what kind of inference mechanism for sensor integration ZoneMinder uses, but being able to tell if a person is in view of the camera and triggering a callback is a great idea. For instance, if you lived in a neighborhood that had a high crime rate and you were concerned about burglaries - this could transform ZoneMinder from a surveillance system into an intrusion prevention system where motion detection alone fails, for instance by disambiguating people from pets. Another use of object detection is inventory management, for instance, if you brought in a particular item, the system could mark the route the item took (which cameras, or even spatial information). Another application would be monitoring the health of pets, to detect anomalies in their daily routines.

Anomaly detection also have obvious security applications. For instance, if someone is casing your house, or you have a particular neighbor you're trying to avoid, or such. For such, I've heard that gait recognition is invaluable. There is a gate recognition system that I imagine may be able to work here:

https://github.com/marian-margeta/gait-recognition

This seems to be the best option available on GitHub, from a cursory glance. I briefly tried setting it up but I am not a Python programmer and still have to figure out how to get the frames variable set. I am not sure how this would be integrated, but when I found this thread here about ZoneMinder YOLO integration, I became more optimistic seeing as you have managed to get them to work together. I do not know what kind of inborn intelligence comes with this Gait Recognition repository. I wonder if it can handle previously unseen gaits, and create new identities based on newly observed gaits, which could be ordered by frequency of occurrence and a GUI given to the user to add any other information such as names or such.

While gait recognition in itself is useful in providing identification, what would really help is integration of anomaly detection with gait recognition. This way, for instance, people walking to and from work or school would quickly be normalized, whereas the same individual appearing three times throughout the day at irregular hours would be anomalous and trigger a warning. It can also help with recording for the user their own movements both in and outside of the house. By recording these times, it can help the user to avoid predictable movements, unpredictability being a key factor in security, such as leaving for the store at the same time each week, etc, which could be exploited, such as by an ambush or a break in. Perhaps it could trigger an audible warning if the user went for the door at a predictable time. Things like that. There are many anomaly detection repositories on GitHub and many books on how to use them. And I know someone who works in that area who I could ask for help if I got stuck. Some questions are what kind of accuracy do you get at long distances and/or lower resolutions? Do the machine learning models have to be trained on the same individuals the classifiers run against?

I will just mention briefly that OpenFace face recognition could be integrated, and again I do not know how to get it to create new identities for previously unseen faces. It sets up rather quickly, but it is unclear to me after a quick glance how exactly to integrate it. This would be useful for cameras that get close ups, such as a front door camera or a camera guarding a hallway or such. Again, identification information could be used to help randomize the users movements and also for multi-factor authentication such as of a doorway. Also, it could contribute to activity recognition and also goal recognition. For instance in a smart home environment, a goal recognition system could be used to determine the purpose for a person going somewhere or doing something, and then, having figured out the most likely motives, a smart home could think about whether these are good ideas and interrupt, or suggest something like don't forget to bring the package when you drive off, etc. Here is such a system:

https://github.com/aindilis/optimal-plan-recognition

Lastly, activity detection is a good feature, if for no other reason than to add to the anomaly detection inputs. There seem to be at least a few good activity recognition systems on GitHub. Activity recognition is also important for monitoring sick or elderly patients. Such features in ZoneMinder may extend and enhance it for such other purposes.

I am not sure as to what extent existing smart home systems perform these kinds of analyses. For instance, Home-Assistant. I have been working on such a system, which can be seen here:

https://frdcsa.org/~andrewdo

Some unanswered questions remain - can a video card run multiple models at the same time, how much power does it take and is it problematic to run a video card 24x7?

Anyway, I know this is a lot to suggest, however, by adding these features, it could increase the marketability of ZoneMinder for a few reasons. Smart home technology is rapidly accelerating into the mainstream, and thus provides ample surface area for integration. Having ZoneMinder more compatible with smart home technology through the integration of computer vision capabilities could help to future proof it and market it. The availability of these computer vision tools and other AI tools such as plan and anomaly recognition now is something that simply didn't exist even 5 years ago, so there is really an explosion of options here. On the other hand, it may prove more prudent to wait it out in order for some particular combination of technologies to win out before investing effort in preliminary integrations. Whether these capabilities will be extended or whether they are one off research attempts affects the value of integrating them. Perhaps you already have APIs defined which would allow CV modules to be integrated quicker.

If the choice to integrate one or more of these technologies were made, probably best would be to do an SMEA (Success Modes Effects Analysis) to see which techniques provide the greatest benefit to the most use cases, and integrate those first.

As for what part I play, I am proficient in Perl and moderately skilled with Prolog. I have written a life planning system which combines these. In particular, I am versed in systems integration and stove piping various software systems together. I am also skilled in making operating systems packages. So I could package these tools for Debian and use Alien to help bootstrap other package formats. I think that adding these capabilities would be a force multiplier for the project.

I have many projects I am working on, but if someone were to work with me on this it would increase my motivation and I might be able to get further along - particularly someone already familiar with ZoneMinder and video programming. I would like to point out however that I have a tendency to fail to complete projects on time if at all.

Best,
Andrew
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: I wrote an object-detection add-on for ZM

Post by iconnor »

Hi Andrew

I'm happy to try to work with you on this. My time is limited, but perhaps I have enough to keep you motivated.
snickers2k
Posts: 2
Joined: Sun Jun 10, 2018 10:11 pm

Re: I wrote an object-detection add-on for ZM

Post by snickers2k »

just found this: https://github.com/goruck/smart-zoneminder
maybe not so real-time, but good enough for me, since i only want to avoid false motion detections by wind/tree's
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: I wrote an object-detection add-on for ZM

Post by rockedge »

wow that project looks interesting.
Maybe a object detection routine that acts as an 2nd level advance filter and processes events and analyzes their alarm frames to determine if the event and alarm is false, and if no object is detected then delete those false events. It wouldn't be quite real time but more a background function that checks events that ZM is creating. This little program worked enough to put something together as a proof of concept. I may try this out.
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: I wrote an object-detection add-on for ZM

Post by rockedge »

this is also very interesting and I am going to test some of possibilities
viewtopic.php?p=95069#p95069
https://medium.com/ozone-security/shap ... 2908a8a7fe
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: I wrote an object-detection add-on for ZM

Post by asker »

FYI, another option for folks using the notification server. YOLOv3 is significantly faster than YOLOv2. I've added 2 options
for detection at "semi real time" (that is, while the event is occurring)

blurb: https://medium.com/zmninja/hooked-on-hooks-a2210e9f95cf
code: https://github.com/pliablepixels/zmeven ... ok_example

Note that these detection hooks are called after ZM detects motion and triggers an alarm, so its not like they will be called for each frame buffer of ZM.

This has been integrated into the event server.

For anyone that wants to play with it:
a) Only tested on ZM 1.32
b) Needs this PR to fetch real time images

I run ZM on a Xeon 3GHz 4core machine with no GPU. YOLOv3 takes approx 2.4 seconds which includes model load time. The HOG approach takes 0.2 seconds. Note that YOLOv3 takes up a lot of memory when it loads the model. So if you are low on memory, stick with HOG.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Locked