[SOLVED] /dev/shm keeps filling up when ZoneMinder is stopped and started

[montagdude]

Ugh, more issues with my new server. These ones are actually serious, unlike the load and disk usage display issues in my other thread. I noticed that /dev/shm kept getting more and more full every time I started and stopped ZoneMinder (changing the run state). After about 5-10 of these, it filled up completely and of course caused all sorts of problems. Doing an ls in /dev/shm showed 0 files, but then I'm not sure if it is supposed to show anything.

Here are my system details:
ZoneMinder 1.30.4 on Slackware
Linux kernel 4.4.88
Zotac ZBox BI325 with quad-core Intel Celeron N3160 processor
4 GB ram

Any idea where to look? This processor was released in Q1 2016, which is prior to the initial release of the 4.4 kernel, but maybe I should try a newer one?

[bbunge]

With 4 gig of RAM that would make the tmpfs 2 gig if Slackware is like other distros. May not be enough if you are pushing high res cameras. The files Zoneminder creates should be deleated on a restart. As a test you could relocate the settings that use tmpfs to the hard drive but you will likely take a performance hit. Rebooting is another option.

[bbunge]

Another option is to switch to Ubuntu or Centos... just saying! Still trymore RAM. Your processor is not the issue unless you can only run a 32 bit OS.

[montagdude]

I'm certain that this isn't simply an issue of not having enough RAM. I am using a single 1280x960 camera at 3FPS, with the standard 50 frame buffer. The /dev/shm usage increases predictably by about 12% each time I restart ZoneMinder. You are correct that the total size of /dev/shm is 2 GB. I am going to try a new kernel first before switching to a different distro.

Something that I forgot to mention in the original post is that I had been testing the exact same setup on my Core i5 laptop for weeks before migrating to the new server. It has 8 GB of RAM instead of 4, but I certainly would have filled up /dev/shm within that time if it had the same problem of increasing the 240 MB every time ZoneMinder was started. Anyway, thank you for the advice.

Edit: nevertheless, I will definitely try running it on the laptop again to make sure /dev/shm is not growing and report back.

[montagdude]

Well, I believe I figured out my problem with /dev/shm. I have a script that sends automatic notifications when motion is detected. (I know there is a native way to do this in ZM, but it is not instantaneous. See my post in the User Contributions forum.) This keeps running even when ZoneMinder itself is stopped. Apparently, this process holds onto the memory map, and then a new one is created when ZoneMinder starts again. Output of `lsof | grep /dev/shm` after stopping ZoneMinder:

Code: Select all

zm_event_  1367             root  DEL       REG               0,17              114697 /dev/shm/zm.mmap.4

My script already checks for when ZoneMinder is stopped, so I just have to figure out how to make it release the memory map. Just for informational purposes, after rebooting and starting ZoneMinder, /dev/shm is only 235 MB.

I will mark this as solved. Sorry for the noise.

[rockedge]

question.... are you using a script similar to zm-alarm.pl to detect events and send the email?? I do the same thing but I do not have the memory map being retained issue that you mention. My scripts for ZM event detection are based on zm-alarm.pl

[montagdude]

question.... are you using a script similar to zm-alarm.pl to detect events and send the email?? I do the same thing but I do not have the memory map being retained issue that you mention. My scripts for ZM event detection are based on zm-alarm.pl

Yes, it's similar to that but a bit more complicated. I'm not sure what is causing it to hold onto the memory map files. Here is my script:

Code: Select all

#!/usr/bin/perl -w

# This script has been adapted from utils/zm-alarm.pl to send a notification
# within 3 seconds of a new event being generated for a monitor, which occurs
# as soon as motion is detected. The notification includes a link to the frame
# with the highest score, and filters out events caused by signal loss. For
# reference, see:
# scripts/ZoneMinder/lib/ZoneMinder/Memory.pm
#     and
# http://zoneminder.readthedocs.io/en/latest/faq.html#how-can-i-use-zoneminder-to-trigger-something-else-when-there-is-an-alarm
#     and
# https://forums.zoneminder.com/viewtopic.php?t=21781

use strict;
use warnings;
use ZoneMinder;
use DBI;
require MIME::Entity;
use List::Util qw[min max];

$| = 1;

# Interval between monitor checking cycles
my $timeout = 3;

# Pause after sending a message (to reduce spam)
our $message_timeout = 1;

# Min number of cycles after signal loss to send event message. Goal is to
# filter out any events caused by signal loss.
my $signal_overload_count = 5;

# Email parameters. Fill these in! Note that "@" has to be escaped like "\@"
our @toaddrs = ("");
our $fromaddr = "";
our $url = "";

my $driver = "mysql";
my $database = "zm";
my $user = "zmuser"; 
my $password = "zmpass";

my @monitors;
my $rebuild_monitors = 1;
while (1) {
        sleep $timeout;

        # Have to exit when ZoneMinder stops running to make sure mapped memory is freed
        my $status = runCommand( "zmdc.pl check" );
        if ( $status ne "running" ) {
                Info( "zm_event_alert.pl is exiting" );
                last;
        }

        # Initialize or re-initialize array of monitors
        if ( $rebuild_monitors ) {
                my $dbh = DBI->connect("DBI:$driver:$database", $user,
                                       $password) or die $DBI::errstr;
                
                my $sql = "select M.*, max(E.Id) as LastEventId from Monitors as M left join Events as E on M.Id = E.MonitorId where M.Function != 'None' group by (M.Id)";
                my $sth = $dbh->prepare_cached( $sql ) or die( "Can't prepare '$sql': ".$dbh->errstr() );
                my $res = $sth->execute() or die( "Can't execute '$sql': ".$sth->errstr() );

                @monitors = ();
                while ( my $monitor = $sth->fetchrow_hashref() ) {
                    $monitor->{LastEventId} = zmGetLastEvent( $monitor );
                    $monitor->{LastSignal} = $signal_overload_count + 1;
                    push( @monitors, $monitor );
                }
        }
        $rebuild_monitors = 0;

        # Loop over monitors
        foreach my $monitor ( @monitors ) {
                if ( !zmMemVerify( $monitor ) ) {
                        $rebuild_monitors = 1;
                        next;
                } 

                # Skip any monitor that is inactive
                if ( !getMonitorActive( $monitor ) ) {
                        next;
                } 

                # Skip any monitor that is not receiving a signal
                my $signal = getMonitorSignal( $monitor );
                if ( !$signal ) {
                        $monitor->{LastSignal} = 1;
                        Info( "$monitor->{Name} is not sending a signal." );
                        next;
                } else {
                        $monitor->{LastSignal} = min( $monitor->{LastSignal} + 1,
                                                      $signal_overload_count + 1 );
                }

                # Send a message if a new event is available. Try to filter out
                # any events caused by signal loss.
                my $last_event_id = zmGetLastEvent( $monitor );
                if ( $last_event_id != $monitor->{LastEventId} ) {
                        if ( $monitor->{LastSignal} <= $signal_overload_count ) {
                                Info( "$monitor->{Name} signal overload count ".
                                      "$monitor->{LastSignal}" );
                        } else {
                                notifyNewEvent( $monitor->{Name},
                                                $last_event_id );
                                Info( "Sent new event $last_event_id alert ".
                                      "for $monitor->{Name}" );
                        }
                        $monitor->{LastEventId} = $last_event_id;
                }        
        }
}

sub getMonitorActive {
        my $monitor = shift;

        return( zmMemRead( $monitor, 'shared_data:active' ) );
}

sub getMonitorSignal {
        my $monitor = shift;

        return( zmMemRead( $monitor, 'shared_data:signal' ) );
}

sub notifyNewEvent {
        my $monitor_name = shift;
        my $last_event_id = shift;

        my $subject = "ZoneMinder alarm";
        my $maxscore_url = $url . "/index.php?view=frame&eid=".
                           $last_event_id . "&fid=0";
        my $message = "New event $last_event_id for monitor $monitor_name.\n".
                      "URL: $maxscore_url";
        foreach my $toaddr ( @toaddrs ) {
                sendMessage($toaddr, $subject, $message);
        }
}

sub sendMessage {
        my $toaddr = shift;
        my $subject = shift;
        my $message = shift;
 
        my $mail = MIME::Entity->build(
                From => $fromaddr,
                To => $toaddr,
                Subject => $subject,
                Type => (($message=~/<html>/)?'text/html':'text/plain'),
                Data => $message
        );
        $mail->smtpsend( Host => "localhost",
                         MailFrom => $fromaddr,
        );
        sleep $message_timeout; 
}

The full thread is here:

viewtopic.php?f=9&t=26758

For now, I've just modified the script to stop when ZoneMinder stops, and I have a second "watcher" script that starts it again when ZoneMinder starts.

[rockedge]

I see....nice. I made a daemon using the zm-alarm.php script which I use as a base to modify too do various functions according to the state of the monitors. This example is to send one email only for a set period of time.....

viewtopic.php?p=103057#p103057

Can I try out your version?

[rockedge]

I need to test out if the daemon in some cases does the same thing as you found out when ZM stops and the daemon continues to run...I haven't noticed anything but I did not look specifically....

[montagdude]

I see....nice. I made a daemon using the zm-alarm.php script which I use as a base to modify too do various functions according to the state of the monitors. This example is to send one email only for a set period of time.....

viewtopic.php?p=103057#p103057

Can I try out your version?

Feel free to. If you want to test out the old behavior that was causing problems for me, you have to change the script so it doesn't exit when ZoneMinder is stopped. Change these lines:

Code: Select all

        # If zm is not running, have to stop to clear memory
        my $status = runCommand( "zmdc.pl check" );
        if ( $status ne "running" ) {
                Info( "zm_event_alert.pl is exiting" );
                last;
        }

to this:

Code: Select all

        # If zm is not running, just go to the next cycle
        my $status = runCommand( "zmdc.pl check" );
        if ( $status ne "running" ) {
                if ( !$rebuild_monitors ) {
                        $rebuild_monitors = 1;
                }
                next;
        }

[rockedge]

Have you run across this error at all using these types of queries to ZM database?? (before the API itself or the zmTrigger.pl this was the "API").

Code: Select all

no memkey in zmMemInvalidate

or

Code: Select all

Can't read from mapped memory for monitor '1', gone away?

these errors occur running zm-alarm.pl based scripts with ZM 1.31.1 +

thanks! I am messing around with your code and it works nicely on my ZM 1.30.4 running on Puppy Linux Xenial 7.0.6 32 bit operating system (ubuntu 16.04 binaries, built with Woof-CE )
Linux Kernel: 4.9.13 (i686)
Kernel Version: #1 SMP PREEMPT Thu Mar 9 09:27:40 GMT 2017
PAE Enabled: No

PHP 7.0
mariadb
Apache 2.4

development:

Bash: 4.3.29
Gtkdialog: 0.8.4
Perl: 5.22.1
Python: 2.7.12
Yad: 0.12.4

[montagdude]

Hm, no, I haven't seen errors like that. I am running 1.30.4, though. It would be worth submitting a bug report if the same script worked fine on 1.30.x. If you haven't done so already, also check whether zm-alarm.pl has been changed in 1.31.x.

[rockedge]

zm-alarm.php looks the same in v 1.31.x..... I will test again to see if I can track down the problem.