Page 1 of 1

Another /dev/shm post - why the creep?

Posted: Tue Feb 09, 2021 4:41 pm
by ElectroStrong
Hello everyone - hopefully this is a question that's been answered before in which doesn't show up in the search (as usual, I tried search first and tried to identify a posting that had a similar experience).

I have ZoneMinder 1.34.23-focal1 installed on my Ubuntu 20.04 machine. I'm running it on base hardware (no docker) and I have around 13 cameras monitoring with different setups (3 mocord, remainder modect). All of the cameras use RTSP and are configured appropriately.

My /dev/shm upon startup is around 42% full. I have verified that I have an 8GB /dev/shm that is being generated upon boot and I technically have room to expand if needed fairly easily.

My cameras are setup primarily to run 10 fps. My Image Buffer Size (frames) is set to 20, and my Pre-Event Image Count is set to 10 and Post Image Event is set to 300. I want the system to start saving a second before motion occurs and to minimally track for 30 seconds after the event has occurred. I believe this is the correct setup to accomplish that goal (and also may be part of the issue).

General operating - I don't see /dev/shm growing beyond 42% (or rarely so). Eventually though, for some reason, /dev/shm will take up 100% of the space and I'll usually see a single camera that is not longer being rendered in my Montage Review. Going to the server, stopping the zoneminder service, and then starting it again cleans out /dev/shm (0% space used) and then it starts back up at 42% usage. This seemed to start happening more in the last package update to .23 but I'm not sure if that is the root cause.

Is there a reason why a particular camera would cause /dev/shm to become full if the settings are all the same as what I described above? Is there some type of safety logic built into Zoneminder where /dev/shm can be automated to "cleanup" or "dump" data when it starts getting full? Are there conditions in which /dev/shm has files that are orphaned because the analysis services fail due to some camera/protocol centric issue?

How do I go about debugging the root cause of this issue? It seems to be happening every day and a quick restart of the service tackles it - but I'm not fond of setting up a restart every night to be quite honest...thoughts?

Re: Another /dev/shm post - why the creep?

Posted: Tue Feb 09, 2021 4:50 pm
by ElectroStrong
Also - as another edit to this post - I have also done the memory "calculator" using the following formula for /dev/shm:

Min Memory = 1.2 * (image-width*image-height*image buffer size*target color space*number of cameras/8/1024/1024)

In my situation:

2467.96875 Mb = 1.2 * (1920 * 1080 * 20 * 32 * 13 / 8 / 1024 / 1024)

Taking this and multiplying it by 2 gives me a memory demand of around 5 GB. I have 8 GB allocated (almost 4 times as much). I believe off of these facts I technically have enough allocated, but I'm hoping someone with more knowledge on the internals of /dev/shm and how Zoneminder uses it can comment on how to determine if this is an inaccurate assumption.

Re: Another /dev/shm post - why the creep?

Posted: Tue Feb 09, 2021 7:57 pm
by jperkins
just wondering if the problematic camera is a different model than the rest ? or is it the same as other cameras in your system and configured the same ?

Re: Another /dev/shm post - why the creep?

Posted: Tue Feb 09, 2021 8:26 pm
by ElectroStrong
jperkins wrote: Tue Feb 09, 2021 7:57 pm just wondering if the problematic camera is a different model than the rest ? or is it the same as other cameras in your system and configured the same ?
Great question - they're all the same. They are all Wyze Cam v2's that have all been converted to DaFang CFW. Every one of them is setup the same for the configuration in the buffers other then the Mocord/Modect.

Lastly - when /dev/shm fills it's not always the same camera that fails. It seems to be random, and as you possibly already know, when /dev/shm fills sometimes I'll just have one camera that goes offline or I'll have cycling of cameras failing until I restart the zoneminder service.

When I restart the service, /dev/shm always clears, and then it always goes back up to 42% usage after zoneminder starts back up. I don't need to restart the service (which is good) just the service.

Re: Another /dev/shm post - why the creep?

Posted: Tue Feb 09, 2021 11:43 pm
by jperkins
Cool. Never heard of defang CFW before. Sorry but cant be much more help.

Maybe reboot the cameras weekly ?
See if they do this with stock firmware ?
set them on record and see what a bad actor is streaming before it fails. ?

I know nothing about using zm debug facilities. Maybe that could help out. Good luck

Re: Another /dev/shm post - why the creep?

Posted: Mon Feb 22, 2021 3:29 pm
by Acewiza
I see the same issue with camera hang, it's consistent and not related to tmpfs. This machine runs 16Gb, and drifts around 20% of that /dev/shm allocation, with 8 cameras running. But a camera (usually the same one) ALWAYS stops some short period of time after launching a montage. Often more than 1. I tried upping some of the numbers, pre/post, buffers, etc. like you described, with no obvious change in the behavior.

Promoting to HTTP version to 1.1 doesn't help.

Interesting observations: Refreshing the montage page always brings it back, temporarily.

Clicking a dead montage camera brings up a live view that starts normally.

So everything's working, sort of. It just keeps failing to continue doing so.

Lots of stuff uses /dev/shm, but ZM is not abusing it here, as far as I can tell. This one just seems to have a very erratic montage display for some reason.

Image

That was over a couple hours, so I'll watch it for awhile, but I doubt ZM is the issue with /dev/shm. This is starting to feel like a browser issue. The weird thing is, I'll go back to it after doing nothing an hour later and everything is flowing just fine.

Re: Another /dev/shm post - why the creep?

Posted: Mon Feb 22, 2021 8:13 pm
by Acewiza
It's always the same 1 or 2 cameras (occasionally more) and there is no common denominator. In fact, they are different camera models on different networks. 2 physically different displays on the same PC running their own montage sessions consistently show the same. One Chrome, 1 Chromium. Maybe I'll see what Firefox thinks about it. It looks like whatever it's doing, ZM is doing it. :?