Yeah... Caught this just before ZM crashed, again. Monitor 10 is configured identically to monitors 11-13 and 18. Different cameras, same config, same colordepth, same resolution (these are the Hi-Res nocord that get triggered by their Lo-Res counterparts), same ZM config.
I cannot, for the life of me, figure why this issue remains after so many config changes.
Please let me know what you need in order to troubleshoot this, because if I'm getting this with the simplest of setups, I'm certain more complex/enterprise setups are also encountering similar issues.
The VM this is running on is assigned 12 vCPU and 64GB RAM. The underlying CPU on the machine is a pair of E5-2640v4s. I've got plenty of RAM and plenty of CPU assigned to them, and yet something doesn't keep up.
Please help
1.36.31 retains memory leak
1.36.31 retains memory leak
- Attachments
-
- Monitor10OverflowAgain.png (226.31 KiB) Viewed 1287 times
Re: 1.36.31 retains memory leak
We need debug level 4 logs from the affected monitor.
Please note that slow disk or slow db can be the culprit. Specifically if there is a filter that is impacting the events table it can slow insertions and updates to the event table... causing everything to back up. Setting a MaxImageBuffer will limit ram consumption, but will fill the logs with warnings.
Please note that slow disk or slow db can be the culprit. Specifically if there is a filter that is impacting the events table it can slow insertions and updates to the event table... causing everything to back up. Setting a MaxImageBuffer will limit ram consumption, but will fill the logs with warnings.
Re: 1.36.31 retains memory leak
All monitors between numbers 10 and 18 share the same network storage, which is a RAID-5 array on an LSI 2308 based controller. If it were slow disk access, all monitors 10 thru 18 would be showing the "backup"
The database is running on the VM itself. I do not have a centralized MySQL installation, yet. If it were database slowness, I would think all monitors would show the same issues, yet they don't.
It's not monitor 10 every time. Sometimes it's 12, or 14 (oddly, it's never been 13....), or 18.
Monitors 10, 11, 12, 13, and 18 are 5MP cams. These are all set up with identical settings.
All monitors from 10 to 18 are the hi-rez streams from the cams, 1 through 9 are the low-rez streams upon which all motion detection is done.
Analysis on or off on the hi-rez monitors seems to make zero difference - one will randomly start building up like the one shown until the VM crashes.
My only "solution" has been to "randomly" restart zoneminder via cron a couple times a day.
When it was doing this before on 1.36.26, I was doing motion detect and mocord only on the hi-rez streams, and the low-rez streams weren't touched - I was told here on this forum that the machine didn't have the CPU to handle the load, and that I should split it and do the motion detect on the low-rez streams and trigger the hi-rez to record the events. I've done that, and upgraded the system it's running on (from Xeon X5690s to Xeon E5-2640v4s)..
So.. if you've got a set of instructions that you'd like me to follow to gather the logs you'd need to tell me what's happening here, I'm ready and willing to help figure out where this memory leak is coming from... just point me at what you need and I'll pull it.
The database is running on the VM itself. I do not have a centralized MySQL installation, yet. If it were database slowness, I would think all monitors would show the same issues, yet they don't.
It's not monitor 10 every time. Sometimes it's 12, or 14 (oddly, it's never been 13....), or 18.
Monitors 10, 11, 12, 13, and 18 are 5MP cams. These are all set up with identical settings.
All monitors from 10 to 18 are the hi-rez streams from the cams, 1 through 9 are the low-rez streams upon which all motion detection is done.
Analysis on or off on the hi-rez monitors seems to make zero difference - one will randomly start building up like the one shown until the VM crashes.
My only "solution" has been to "randomly" restart zoneminder via cron a couple times a day.
When it was doing this before on 1.36.26, I was doing motion detect and mocord only on the hi-rez streams, and the low-rez streams weren't touched - I was told here on this forum that the machine didn't have the CPU to handle the load, and that I should split it and do the motion detect on the low-rez streams and trigger the hi-rez to record the events. I've done that, and upgraded the system it's running on (from Xeon X5690s to Xeon E5-2640v4s)..
So.. if you've got a set of instructions that you'd like me to follow to gather the logs you'd need to tell me what's happening here, I'm ready and willing to help figure out where this memory leak is coming from... just point me at what you need and I'll pull it.
CPU(s) 40 x Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz (2 Sockets)
Kernel Version Linux 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200)
PVE Manager Version pve-manager/7.2-11/b76d3178
Re: 1.36.31 retains memory leak
+1 Memory Leak affects me too. Running on bare metal 2 x 6 Core 3Ghz CPUs with 24GB Ram.
Re: 1.36.31 retains memory leak
So, now that I've "brought the car to the mechanic" it's "stopped making that funny clunking noise".
I haven't had a single cam go over 3G in RAM usage since Monday morning....
I haven't changed anything except the logging levels (moved them all from INFO to WARNING in the config in order to prepare for gathering the L4 logs)... That's the only change - everything else is identical.
Monitors 11, 12, and 18 (all 5MP cams) are using, max, 2.6GB RAM for each zmc instance. 10 and 13 (also 5MP cams...) are showing 1.6GB RAM per zmc instance. 14, 15, and 16 (2MP cams) are at around 1GB per zmc...
Not sure why the two 5MP cams are using much less than the other three, but at least they're not filling up RAM and swap, then crashing the VM.. though that begs the question: what is in the log levels that might have been causing this? I would think the two would be unrelated.
At any rate, when I start to see them climbing again, I'll set the Debug LL to 4 and turn it on and see what I can gather.
I haven't had a single cam go over 3G in RAM usage since Monday morning....
I haven't changed anything except the logging levels (moved them all from INFO to WARNING in the config in order to prepare for gathering the L4 logs)... That's the only change - everything else is identical.
Monitors 11, 12, and 18 (all 5MP cams) are using, max, 2.6GB RAM for each zmc instance. 10 and 13 (also 5MP cams...) are showing 1.6GB RAM per zmc instance. 14, 15, and 16 (2MP cams) are at around 1GB per zmc...
Not sure why the two 5MP cams are using much less than the other three, but at least they're not filling up RAM and swap, then crashing the VM.. though that begs the question: what is in the log levels that might have been causing this? I would think the two would be unrelated.
At any rate, when I start to see them climbing again, I'll set the Debug LL to 4 and turn it on and see what I can gather.
Re: 1.36.31 retains memory leak
It could be that the extra load of logging to the db/fs etc is slowing things down enough. Would be interesting to switch it back and see if the ram usage comes back.
Ram usage is dependent not only on resolution, but also keyframe interval.
Ram usage is dependent not only on resolution, but also keyframe interval.
Re: 1.36.31 retains memory leak
Keyframes are set for every two seconds.