Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Discussions related to the 1.36.x series of ZoneMinder
theogre
Posts: 12
Joined: Tue May 25, 2021 4:51 am

Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by theogre »

I recently upgraded from 1.34 to 1.36.1, then to 1.36.2 and have had numerous issues related to Image Buffers.

Cameras which are giving issues are 2xRLC-520's at 6fps.
2 streams at full resolution, in record mode, and 2 at 1/2 resolution for motion detection, both suffer the issue it seems.

My first issue was related to choppy playback, which was resolved by removing "Maximum Image Buffer Size" from 51, to nothing to allow for automatic balancing. This however would, over time, cause the server to run out of RAM as if a memory leak was present.

I then set my monitors to "Maximum Image Buffer Size" of 51, and the playback seems fine, but the events are now very choppy and completely pause once in a while for many seconds before picking back up, this is in the actual mp4 files generated. Even with this change, there still appears to be a memory leak, as with 1.34 the RAM usage sat around 76%, and now, not even 24 hours later, the usage is up to 93% and climbing.

No other settings have been changed, I have tried all different combinations of image buffer settings, and nothing seems to be stable. Not running hardware acceleration on any of these monitors, the only difference seems to be the buffer settings. A small increase in RAM usage, albeit annoying, is acceptable, but now events are choppy and the RAM just keeps increasing.

Logging into the Webui and setting the state to "restart", drops the RAM usage back to 75% and it starts climbing again. Can run valgrind and post results if necessary.

Any help would be greatly appreciated!

UPDATE: Since I restarted the zoneminder process at 12:38, now 3 hours later, RAM usage is at 98%, with 2 monitors each consuming almost 1gb of RAM each, events still choppy.
theogre
Posts: 12
Joined: Tue May 25, 2021 4:51 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by theogre »

Is anyone else experiencing a similar issue? I cannot downgrade to 1.34 due to database issues, and this is preventing the security system from functioning properly.

The only error I can find, show up in the log as

Code: Select all

2021-05-30 06:44:05	zmc_m9	raid	2364	ERR	Unable to free up older packets. Not queueing this video packet.	zm_packetqueue.cpp	135
2021-05-30 06:44:05	zmc_m9	raid	2364	WAR	You have set the max video packets in the queue to 51. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.	zm_packetqueue.cpp	92
51 was more than enough for 1.34, but now not so much. I still don't understand why after the upgrade, the monitor needs almost a full gb of RAM and is now choppy when it was fine before. This is happening to multiple monitors. If I remove the limit of 51, it will cause the server to run out of RAM and freeze, for now, choppy events are better than a completely non functioning system

UPDATE: Also, the issue seems to also be affecting different 1080p cameras I have, these have a super low bitrate of 83kb/s and 6fps, but are maxing out an imagebuffer of 51 and consuming the full 443mb allowable by the buffer. Insanity!
SteveGilvarry
Posts: 494
Joined: Sun Jun 29, 2014 1:12 pm
Location: Melbourne, AU

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by SteveGilvarry »

I have seen something similar check keyframe interval on the cameras, I lowered mine. The buffer is keeping the frames until it gets a new keyframe, but mine was getting into 25 frames even though interval was 10. Eventually tweaking that and max buffer I got it happy. But some work to go in here, need to look what is in the buffer. I think it should be compressed frames, but feels like we have uncompressed frames hanging around waiting for zma.
Production Zoneminder 1.37.x (Living dangerously)
Random Selection of Cameras (Dahua and Hikvision)
zarkov1994
Posts: 12
Joined: Mon Jul 14, 2014 9:14 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by zarkov1994 »

Hi,

I'm seeing the same behaviour in 1.36.1 on Fedora 34.

I'm running 4 RLC-410-5MP at 10FPS in Modetect which have been fine (at 15FPS) for months, but following an upgrade 2 days ago I've had issues with what appears to be a memory leak, just as theogre describes.

I've got my cams writting video in pass through using FFMPEG, "Maximum Image Buffer Size" set to zero following initial errors following the upgrade, these have gone now and video playback seems fine. It's just the memory usage that's the issue.

I've got 16Gb in my system and that slowly gets eaten up by the ZMC processes each one using a differing amount of RAM and creeping up at different speeds.

The only error I can find in any logs is from zm_videostore.cpp on line 642 which seem to appear when some events fire and looking at the system journal these appear to relate to writing tail of the video stream ("[zmc_m13] [Error writing trailer Error number 300 occurred]"). I'm wondering if there's some clean up code that is not being fired as a result of this exception and that's why the memory isn't being released after an event take place?

I hope there's a fix for this as it's going to make ZM unusable in my set up at it stands :(

Is anyone else seeing the same issue?
Zoneminder 1.36.15 running under:
Fedora 35, Ryzen 5 3600x, 24Gb RAM, 512Gb WD Blue SN570 System SDD, 4TB Seagate Skyhawk Event HDD
Numerous Reolink RLC-410 (Wired IP)
SteveGilvarry
Posts: 494
Joined: Sun Jun 29, 2014 1:12 pm
Location: Melbourne, AU

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by SteveGilvarry »

Maxbuffer 0 is saying get as big as you need, try cameras key frame interval x 2 + 1. But yes something is causing the auto setting to keep consuming buffer space.
Production Zoneminder 1.37.x (Living dangerously)
Random Selection of Cameras (Dahua and Hikvision)
zarkov1994
Posts: 12
Joined: Mon Jul 14, 2014 9:14 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by zarkov1994 »

Hi Steve,

I had tried various Maxbuffer sizes before having to leave it at 0. Even with a trim Maxbuffer value (it still threw the odd packet overrun but generally worked) I still saw the RAM creeping up. I was hoping the maxbuffer would have capped the RAM usage but it didn't seem to help, which is why I'm wondering if the "leak" is from elsewhere.

Matt
Zoneminder 1.36.15 running under:
Fedora 35, Ryzen 5 3600x, 24Gb RAM, 512Gb WD Blue SN570 System SDD, 4TB Seagate Skyhawk Event HDD
Numerous Reolink RLC-410 (Wired IP)
zarkov1994
Posts: 12
Joined: Mon Jul 14, 2014 9:14 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by zarkov1994 »

Just following up... I reset my Maxbuffer to 60 for each of my 4 cams and received no packet queue warnings for about 30 mins. The memory each ZMC service used slowly increased but did seem to level off (at around 2.2Gb each) so as Steve and the documentation mentions the cap does seem to work, but after that initial 30 mins I started to see...

Code: Select all

ERR [zmc_m18] [Unable to free up older packets.  Not queueing this video packet.]
WAR [zmc_m18] [You have set the max video packets in the queue to 60. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.
... in the journal again, so i'm guessing it starts throwing this error because it's unable to free up the queue and it's associated memory.

Matt
Zoneminder 1.36.15 running under:
Fedora 35, Ryzen 5 3600x, 24Gb RAM, 512Gb WD Blue SN570 System SDD, 4TB Seagate Skyhawk Event HDD
Numerous Reolink RLC-410 (Wired IP)
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by iconnor »

Please try something: Options->Config->OPT_ADAPTIVE_SKIP
Turn it off.

I think I will be removing this as it no longer makes sense. It caused the analysis thread to sleep instead of keeping up with capture.
zarkov1994
Posts: 12
Joined: Mon Jul 14, 2014 9:14 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by zarkov1994 »

Thanks, I've turned that off so I'll let it run for a while see how it goes. I'm still getting the Error Writing Trailer messages but no packet errors as yet.

I'll drop an update after its been running for a few hours.

Thanks
Zoneminder 1.36.15 running under:
Fedora 35, Ryzen 5 3600x, 24Gb RAM, 512Gb WD Blue SN570 System SDD, 4TB Seagate Skyhawk Event HDD
Numerous Reolink RLC-410 (Wired IP)
zarkov1994
Posts: 12
Joined: Mon Jul 14, 2014 9:14 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by zarkov1994 »

* UPDATE*

Unfortunately the same error has started across all monitors after been left for a short while, even though Options->Config->OPT_ADAPTIVE_SKIP is unticked.

Code: Select all

May 31 19:16:08 dixie.local zmc_m18[19537]: WAR [zmc_m18] [You have set the max video packets in the queue to 60. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.]
May 31 19:16:08 dixie.local zmc_m18[19537]: ERR [zmc_m18] [Unable to free up older packets.  Not queueing this video packet.]
May 31 19:16:08 dixie.local zmc_m14[19533]: WAR [zmc_m14] [You have set the max video packets in the queue to 60. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.]
May 31 19:16:08 dixie.local zmc_m14[19533]: ERR [zmc_m14] [Unable to free up older packets.  Not queueing this video packet.]
May 31 19:16:08 dixie.local zmc_m14[19533]: WAR [zmc_m14] [You have set the max video packets in the queue to 60. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.]
Zoneminder 1.36.15 running under:
Fedora 35, Ryzen 5 3600x, 24Gb RAM, 512Gb WD Blue SN570 System SDD, 4TB Seagate Skyhawk Event HDD
Numerous Reolink RLC-410 (Wired IP)
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by iconnor »

What is your keyframe interval?

Also, you have analysis fps set?
zarkov1994
Posts: 12
Joined: Mon Jul 14, 2014 9:14 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by zarkov1994 »

Unfortunately the the Reolink 410's don't allow you set a keyframe interval so I've run ffprobe on one of the recorded events (as I'm using pass thru) and they look to have a 2 second Keyframe interval. That's with a video recorded at 10FPS so I'm assuming that's every 20 frames.

I don't have an analysis FPS set at the moment (it's blank) but I was experimenting with that yesterday in order to try and reduce CPU load, it didn't seem to help with the memory ballooning though.
Zoneminder 1.36.15 running under:
Fedora 35, Ryzen 5 3600x, 24Gb RAM, 512Gb WD Blue SN570 System SDD, 4TB Seagate Skyhawk Event HDD
Numerous Reolink RLC-410 (Wired IP)
theogre
Posts: 12
Joined: Tue May 25, 2021 4:51 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by theogre »

Looking through the responses to my original post, as with zarkov1994, I am using Reolink camera's, however mine are RLC-520 but still cannot set the keyframe interval, this is also affection an rtsp stream I am pulling from an Annke DVR system.

Setting a buffer limit to something extremely high like 150, will allow the RAM to creep up until it hits that limit, then errors are thrown, or the server runs out of RAM.

I have just now disabled OPT_ADAPTIVE_SKIP to see if that resolves my issues as it seems promising, and restarted the container to have zoneminder updated to 1.36.3.

I have changed so many settings, pre and post-event frames, buffers, even source type from rtmp to rtsp with no result. Hopefully that settings resolves it, I will update over time as the RAM creeps up. With my current settings, RAM usage is very high, but the events aren't choppy like before, however I would like RAM use to be reduced like it was in 1.34

No analysis FPS set for any of the monitors.
theogre
Posts: 12
Joined: Tue May 25, 2021 4:51 am

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by theogre »

After my post last night of disabling the OPT_ADAPTIVE_SKIP and updating to 1.36.3, I am still unfortunately having the RAM increasing issue. RAM is up to 88% usage with ALL monitors climbing in usage.

System log has no errors yet.
tommisgr
Posts: 23
Joined: Mon Aug 31, 2020 6:28 pm

Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

Post by tommisgr »

Running
zm 1.36.3 under linuxMint 19.3
16 cameras mixed resolution from 640x480 - 2k)

Having same issues as above.
Have set the buffer high enough that I don't get errors but then ram usage rises to the point it starts using swap.
My workaround was a script to monitor ram and swap usage and over a limit to STOP zm - Clear swap - RESTART zm
Post Reply