zmfilter polling slow.

Forum for questions and support relating to the 1.28.x releases only.
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: zmfilter polling slow.

Post by asker »

Whoops looks like there was a typo in line 1175
dbh needs to be $dbh

Please make that change - I don't know how that $ got removed

That explains why the new zmfilter crashed the moment it matched first - this is in the "execute" subroutine which gets called on match
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
segordon
Posts: 37
Joined: Tue Aug 11, 2015 10:04 am

Re: zmfilter polling slow.

Post by segordon »

zmfilter.pl is back to being stable. Getting SQL queries every 20 seconds, but same batch processing as far as filters go; anywhere from 2 to 5 minutes until filters will see an event.

i'll re-run the scenario and gather logs in a little bit.
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: zmfilter polling slow.

Post by asker »

segordon wrote:zmfilter.pl is back to being stable. Getting SQL queries every 20 seconds, but same batch processing as far as filters go; anywhere from 2 to 5 minutes until filters will see an event.

i'll re-run the scenario and gather logs in a little bit.
Okay. Look forward to the uncut logs
Please sort them by time.

The only other thing I can think of is the filter condition is not matching - that now keyword is causing issues.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: zmfilter polling slow.

Post by asker »

I just noticed you have Asc in the filter match. Can you please change it to descending just for now?

Also, please use this latest zmfilter.pl https://gist.github.com/pliablepixels/5 ... f595cbd9d7
I've added more logs so we can see what is going on
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
segordon
Posts: 37
Joined: Tue Aug 11, 2015 10:04 am

Re: zmfilter polling slow.

Post by segordon »

Changed filter to Desc, same behavior. Also switched back to the email filter. Let me know if you want me to switch back to the script.

Here's the big log. gist github com segordon 9dafb509cb6866e89441

Consists of three forced events, 5:23:51, 5:27:38, and 5:32:43. I started each new alarm as soon as I received the email from the previous one; so that is to say that the difference between those times is the delay i'm seeing for bulk mailing - from 3m 47s to 5m 5s.

I removed lines from the log having to do with blend/deinterlace/delta. I don't believe they are relevant to our investigation, and they contributed 7mb out of the 10mb log. I left the 'Image buffer copy' lines, which mark where the other lines were more-or-less. If you need a totally unmodified log file let me know, big files are just a pain to share.
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: zmfilter polling slow.

Post by asker »

1) Can you please use the updated zmfilter script I posted above?
2) Looks like you have enabled Frame Server - just for now, please disable it
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
segordon
Posts: 37
Joined: Tue Aug 11, 2015 10:04 am

Re: zmfilter polling slow.

Post by segordon »

same scenario , 3 forced alarms.

gist github com segordon 27b52b361d61807c0538

reduced logging verbosity. deactivated frame server. installed new zmfilter.pl.

let me know if I need to crank logging level back up; trying to keep logs tidy/small.
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: zmfilter polling slow.

Post by asker »

Well, this brings out the real problem!!

The SQL query being formed is not correct.
Look at the logs which are displaying the SQL queries:

You will notice irrespective of the current time, the SQL Event Start Time does not change. For approximately 5 minutes it is saying "Give me events with a start time less than X" and that X does not change till after approximately 5 minutes. Which means, for around that time, the SQL query will not return any new events because the time has not advanced in the query.

Can you re-inspect your filter condition and play around a bit?
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
segordon
Posts: 37
Joined: Tue Aug 11, 2015 10:04 am

Re: zmfilter polling slow.

Post by segordon »

changing filter from the original conditions to this has solved my problem.

imgur com g5JIlG3

Now cooldown tick is dependent on time set in options. (Weird!)

Thanks for all your help. Could a malformed SQL check be imposed on filters before they are saved? (Cannot save : invalid)

I used 'not equal to 'xx' ' as a field ,because a blank field ('not equal to ' ' ) also fails similarly to our original filter.
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: zmfilter polling slow.

Post by asker »

segordon wrote:changing filter from the original conditions to this has solved my problem.

imgur com g5JIlG3

Now cooldown tick is dependent on time set in options. (Weird!)

Thanks for all your help. Could a malformed SQL check be imposed on filters before they are saved? (Cannot save : invalid)

I used 'not equal to 'xx' ' as a field ,because a blank field also fails similarly to our original filter.
Great. So what I think was happening was that during 'getFilter' it computed what 'now' means and coverted it to a time. That time never changed unless getFilter was called again. I'm somewhat sure if you changed this line https://gist.github.com/pliablepixels/5 ... er-pl-L197 to just "if (1)" it would also start working with your old filter setting.

But glad you got it sorted. Please make sure you revert to the old zmfilter.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Locked