Weird Issues Deleting Old Database Events - Causing High System Resource Usage

[ZMNoobja]

Hi there,

We are running ZM 1.36.12 on a Dell R710 server with 120GB RAM, 30TB HDD space, and 56 cameras.

We initially had worked through some issues that required us to scrap the events that had been created up to that point. We simply ran a script that ripped through the events' storage location and deleted all of them.

HOWEVER - We expected that zmaudit would come behind and clean up the database to match the lack of those events. That does not seem to happening - or at least something seems bugged. We frequently are seeing a complete bogging down of the server when a set of certain error logs show, attached below. The errors seem to reference events that do not exist in their expected location, however these errors keep referencing the same exact events. I would expect it to be crunched, deleted as they don't exist, and then never spoken of again (lol).

To add to this, we notice that the ram usage and CPU load go nuts when this is happening. We've lowered all cameras to a 3FPS, VGA quality to attempt to offset the high usage, but something just doesn't seem right. It tends to lock the entire machine up to a point of needing a hard restart.

(Also, I'm aware that the file and component seem to be pointing to the filter, but this is the default DiskSpace Update filter that comes with the distro. Not sure why it would be acting like this or how to fix it.)

Thank you in advance for any advice/direction you're able to give!
ZMN

ZM-WarMessages-Diskspace.png (231.75 KiB) Viewed 6110 times

ZM-Update DiskSpace Filter Settings.png (46.46 KiB) Viewed 6105 times

[ZMNoobja]

What's stumping me, and forgive my lack of familiarity, is the wording in the WAR message regarding the "Cause => Continuous EndDateTime" and the "Notes => Recovered". I'm just not sure I'm understanding what the error message is stating/doing. Perhaps it's just information, but I wouldn't expect these same exact errors to repeatedly loop.

Thanks,
ZMN

[Magic919]

I suppose it's a bit late to suggest better ways to achieve this. Like running a filter.

Is zmaudit actually running? Worth a look.

Might be worth stopping ZM whilst you sort it out.

[ZMNoobja]

The filters weren't acting as expected and deleting the old footage. We have an older build running on 1.30.X that is stable and hasn't had any issues in over a year, so we were matching a lot of settings to that as well as combining the newer suggested methods. Disk space would fill up and then wouldn't delete, or at the very least, the system would clog up when trying to delete and cause an even more debilitating lock-up.

We made some progress, however... I was able to get into the filter and use the List Matches function for the 'Update DiskSpace' filter and then simply delete the old entries from that page. THis immediately quieted the logs down and I haven't seen this error since.

With that said, we are now seeing a different error that seems to be looking at old entries in a database or something to that affect. I've attached the screenshot showing the error.

Thoughts?

Thank you, again!
ZMN

ZM-New Errors in Log2.png (194.46 KiB) Viewed 6086 times

[Magic919]

As before - is zmaudit actually running?

It’s the proper way to sort this situation.

[ZMNoobja]

My apologies - I meant to include that zmaudit is and has been running in my reply

Seems to run, chew some resources for a short while, then settle back down. It just doesn't seem to be clearing out some entries from before we deleted the storage events.

I should also add that these events had been created before we set the storage area specifically on the monitors. Previously, though the default location was the same, they were set to default. Not sure if that has anything to do with it, but thought I'd mention it.

Thanks for your replies, Magic!

ZMN

[Magic919]

Ok, gotcha.

The default ZM_AUDIT_CHECK_INTERVAL seems to be 900 seconds. That'll control how often it checks.

It's likely it'll get the job done, but it will take time.

I'd consider stopping ZM and dropping the interval.

The quick and nasty method would be to prune the DB to match the filesystem. It's not recommended.

[iconnor]

I suggest creating a filter with a rule for ExistsInFileSystem. Use it to delete the events. Much lighter than zmaudit.