Something has gone wrong with my ZM since last week as I noticed over the weekend that all events are missing in the Console page. The Events table in mysql has zero rows but there are events stored in the storage area and, strangely, new events are being recorded on the disk but not in the DB.
Is this a known issue with a recent release? I've just run an update on Ubuntu and it's found a later release but still no joy recording events in the DB.
In the logs I'm seeing lots of entries like this:
04/12/2023, 21:44:07 GMT zmaudit 98436 ERR Error deleting event directory at /zmdata/events/2/2023-11-30/275617 using -rf /zmdata/events/2/2023-11-30/275617 2>&1 zmaudit.pl -
04/12/2023, 21:44:07 GMT zmaudit 98436 WAR Failed Command: -rf /zmdata/events/2/2023-11-30/275617 2>&1 Status: 72057594037927935 Output: zmaudit.pl -
04/12/2023, 21:44:07 GMT zmaudit 98436 INF deleting zmaudit.pl -
04/12/2023, 21:44:07 GMT zmaudit 98436 INF Filesystem event 275617 at /zmdata/events/2/2023-11-30/275617 does not exist in database and is 378637 seconds old zmaudit.pl -
Any ideas what might be causing this and how to recover the recording of events to the DB please?
Events all gone from DB but still on disk
Re: Events all gone from DB but still on disk
In the past we hard coded the path to rm. This isn't so good as on some systems it can be elsewhere. Same with some other utilities. So now we detect it on build.
For some reason your systems has not picked up the new conf entries. Check in /etc/zm/conf.d/01-system-paths.conf. There should be lines in it like:
# Full path to optional rm binary or just rm to use PATH
# ZoneMinder will find the rm binary automatically on most systems
ZM_PATH_RM="/usr/bin/rm"
It occurs to me that we should more gracefully handle this not being set.
Also, you can turn off zmaudit. It is no longer intended to run all the time.
For some reason your systems has not picked up the new conf entries. Check in /etc/zm/conf.d/01-system-paths.conf. There should be lines in it like:
# Full path to optional rm binary or just rm to use PATH
# ZoneMinder will find the rm binary automatically on most systems
ZM_PATH_RM="/usr/bin/rm"
It occurs to me that we should more gracefully handle this not being set.
Also, you can turn off zmaudit. It is no longer intended to run all the time.
Re: Events all gone from DB but still on disk
That section is missing from the 01-system-paths.conf file. I upgraded from 1.36 so could the old version have been retained during the upgrade?
Should I add the section you describe?
After the upgrade the system appeared to be working normally but now it seems to be recording events on the disk but not in the DB. How do I figure out what might be causing this?
Am I right in assuming the if the path to rm was found, the events would have been removed from the disk? There is no way to repopulate the DB with the data stored in the events on the disk?
I've disabled the option for RUN_AUDIT and restarted ZM and now the log isn't showing these errors. I tried forcing an alarm to check it would display the event in the Console view, and it didn't, plus I have the following in the logs:
07/12/2023, 22:09:18 GMT zmfilter_1 290329 ERR Error deleting event directory at /zmdata/events/2/2023-12-07/276355 using -rf /zmdata/events/2/2023-12-07/276355 2>&1 zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 WAR Failed Command: -rf /zmdata/events/2/2023-12-07/276355 2>&1 Status: 72057594037927935 Output: zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 INF Deleting event 276355 from Monitor 2 StartDateTime:2023-12-07 22:09:06 from /zmdata/events/2/2023-12-07/276355 zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 ERR Error deleting event directory at /zmdata/events/3/2023-12-07/276354 using -rf /zmdata/events/3/2023-12-07/276354 2>&1 zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 WAR Failed Command: -rf /zmdata/events/3/2023-12-07/276354 2>&1 Status: 72057594037927935 Output: zmfilter.pl -
07/12/2023, 22:09:16 GMT zmfilter_1 290329 INF Deleting event 276354 from Monitor 3 StartDateTime:2023-12-07 22:09:09 from /zmdata/events/3/2023-12-07/276354 zmfilter.pl -
Is this another missing section in a config file?
Should I add the section you describe?
After the upgrade the system appeared to be working normally but now it seems to be recording events on the disk but not in the DB. How do I figure out what might be causing this?
Am I right in assuming the if the path to rm was found, the events would have been removed from the disk? There is no way to repopulate the DB with the data stored in the events on the disk?
I've disabled the option for RUN_AUDIT and restarted ZM and now the log isn't showing these errors. I tried forcing an alarm to check it would display the event in the Console view, and it didn't, plus I have the following in the logs:
07/12/2023, 22:09:18 GMT zmfilter_1 290329 ERR Error deleting event directory at /zmdata/events/2/2023-12-07/276355 using -rf /zmdata/events/2/2023-12-07/276355 2>&1 zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 WAR Failed Command: -rf /zmdata/events/2/2023-12-07/276355 2>&1 Status: 72057594037927935 Output: zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 INF Deleting event 276355 from Monitor 2 StartDateTime:2023-12-07 22:09:06 from /zmdata/events/2/2023-12-07/276355 zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 ERR Error deleting event directory at /zmdata/events/3/2023-12-07/276354 using -rf /zmdata/events/3/2023-12-07/276354 2>&1 zmfilter.pl -
07/12/2023, 22:09:18 GMT zmfilter_1 290329 WAR Failed Command: -rf /zmdata/events/3/2023-12-07/276354 2>&1 Status: 72057594037927935 Output: zmfilter.pl -
07/12/2023, 22:09:16 GMT zmfilter_1 290329 INF Deleting event 276354 from Monitor 3 StartDateTime:2023-12-07 22:09:09 from /zmdata/events/3/2023-12-07/276354 zmfilter.pl -
Is this another missing section in a config file?
Re: Events all gone from DB but still on disk
If I force a capture the event is recorded but then it is removed from the DB (and the Console) and the following is logged in Logs:
10/12/2023, 15:50:48 GMT zmc_m2 40140 INF Front: 7900 - Capturing at 9.99 fps, capturing bandwidth 590298bytes/sec Analysing at 0.00 fps zm_monitor.cpp 1781
10/12/2023, 15:50:44 GMT zmc_m3 40229 INF Front-lo: 7700 - Capturing at 9.97 fps, capturing bandwidth 52582bytes/sec Analysing at 2.49 fps zm_monitor.cpp 1781
10/12/2023, 15:50:42 GMT zmfilter_1 40053 INF Deleting event 276666 from Monitor 2 StartDateTime:2023-12-10 15:49:44 from /zmdata/events/2/2023-12-10/276666 zmfilter.pl -
10/12/2023, 15:50:41 GMT zmfilter_1 40053 INF Deleting event 276665 from Monitor 3 StartDateTime:2023-12-10 15:49:45 from /zmdata/events/3/2023-12-10/276665 zmfilter.pl -
10/12/2023, 15:50:38 GMT zmc_m2 40140 INF Front: 7800 - Capturing at 9.99 fps, capturing bandwidth 601648bytes/sec Analysing at 0.00 fps zm_monitor.cpp 1781
10/12/2023, 15:50:34 GMT zmc_m3 40229 INF Front-lo: 7600 - Capturing at 9.99 fps, capturing bandwidth 54848bytes/sec Analysing at 2.50 fps zm_monitor.cpp 1781
Not sure why zmfilter is deleting the event straight after it is recorded. I created a custom filter but it was to delete events from a specific camera one hour after they occurred and it was working before upgrade. I have removed it now and will check the default PurgeWhenFull filter.
10/12/2023, 15:50:48 GMT zmc_m2 40140 INF Front: 7900 - Capturing at 9.99 fps, capturing bandwidth 590298bytes/sec Analysing at 0.00 fps zm_monitor.cpp 1781
10/12/2023, 15:50:44 GMT zmc_m3 40229 INF Front-lo: 7700 - Capturing at 9.97 fps, capturing bandwidth 52582bytes/sec Analysing at 2.49 fps zm_monitor.cpp 1781
10/12/2023, 15:50:42 GMT zmfilter_1 40053 INF Deleting event 276666 from Monitor 2 StartDateTime:2023-12-10 15:49:44 from /zmdata/events/2/2023-12-10/276666 zmfilter.pl -
10/12/2023, 15:50:41 GMT zmfilter_1 40053 INF Deleting event 276665 from Monitor 3 StartDateTime:2023-12-10 15:49:45 from /zmdata/events/3/2023-12-10/276665 zmfilter.pl -
10/12/2023, 15:50:38 GMT zmc_m2 40140 INF Front: 7800 - Capturing at 9.99 fps, capturing bandwidth 601648bytes/sec Analysing at 0.00 fps zm_monitor.cpp 1781
10/12/2023, 15:50:34 GMT zmc_m3 40229 INF Front-lo: 7600 - Capturing at 9.99 fps, capturing bandwidth 54848bytes/sec Analysing at 2.50 fps zm_monitor.cpp 1781
Not sure why zmfilter is deleting the event straight after it is recorded. I created a custom filter but it was to delete events from a specific camera one hour after they occurred and it was working before upgrade. I have removed it now and will check the default PurgeWhenFull filter.
Re: Events all gone from DB but still on disk
zmrecover.pl will do it's best to restore db entries.
Your filter likely needs editing. Post a screenshot here and we can guide you.
01-system-paths.conf should be updated to use the new one. Very rarely do you need to edit it, and if you do, then it is best to create an 03custom.conf which will override anything in it instead.
Your filter likely needs editing. Post a screenshot here and we can guide you.
01-system-paths.conf should be updated to use the new one. Very rarely do you need to edit it, and if you do, then it is best to create an 03custom.conf which will override anything in it instead.
Re: Events all gone from DB but still on disk
zmrecover is excellent, is there a Yes To All option to save me pressing enter for each event to recover? The only documentation I found didn't suggest there was unfortunately.
A recent update wanted to overwrite the 01-system-paths.conf so I allowed it and now the entry exists.
I think the PurgeWhenFull filter was causing the issue. The used disk space was over 95% so it was deleting events even though there weren't any until one was created and it got deleted.
I stopped the filter running the background and cleared some old events from the storage to bring used space below 95%. I'm just going through zmrecover.pl to get the DB events back in sync before reenabling the filter.
Thanks for your guidance, much appreciated.
A recent update wanted to overwrite the 01-system-paths.conf so I allowed it and now the entry exists.
I think the PurgeWhenFull filter was causing the issue. The used disk space was over 95% so it was deleting events even though there weren't any until one was created and it got deleted.
I stopped the filter running the background and cleared some old events from the storage to bring used space below 95%. I'm just going through zmrecover.pl to get the DB events back in sync before reenabling the filter.
Thanks for your guidance, much appreciated.