unstable zoneminder - ceases to capture video after some time

[marianh]

Hi,

I'm running Zoneminder 1.30.4 in (privileged) LXC container with Debian 9 on a server-grade hardware (lots of memory, enough disk space, ...).

Unfortunately it's not stable.
It runs fine for some time (hours, up to few days) but then suddenly it won't connect to camera anymore. Cameras are still green on the initial screen but there are no events detected and when I click on camera to view the feed, there's no image.

Error when attempting to view feed is "Terminating, last frame sent time 1514885533.985546 secs more than maximum of 10.000000".
Last previous errors (which likely show why Zoneminder crashed) are "Got signal 11 (Segmentation fault), crashing" in zm_signal.cpp, many backtraces and "'zmc -m 5' exited abnormally, exit status 11" (for all cams).
Restart from the web interface restores the functionality.

Any advice how to investigate?

[rockedge]

I get this with Debian 9 variants also. I have bugs appear in ZM running in Debian that don't appear in other OS's. I still have not tracked down what causes this after running okay and then at random times throwing those same errors.

[marianh]

Thank you.

Would I get the same problems if I switch to Ubuntu (as it's Debian based) or it's stable on Ubuntu?
If Ubuntu is not ok, what other Linux distribution would be best? I'd still want to run it in LXC container (Proxmox virtualization => Debian-based host), if it can run stable in container.

[rockedge]

I find that Ubuntu runs ZM the best. I use a Ubuntu based Puppy Linux that works very well but is tricky to install ZM. Ubuntu is easy to work with ZM and well documented. Also I believe most developers use Ubuntu server OS's but I can say I have had great success with Ubuntu and it's different offspring installing, configuring and using zoneminder. Debian 9 zoneminder installs seem to have some little bugs and some functions just won't work correctly...but is soooo close to being okay.

I have used a OS called XenialDog......a very minimal Ubuntu that I used the ZM install script to automatically install ZM. Worked perfectly and surprisingly ZM works almost error free....log is green most of the time......entire OS with ZM is around 250 megs in RAM

XenialDog 64 bit > https://debiandog.github.io/doglinux/zz ... aldog.html

XenialDog 32 bit > https://debiandog.github.io/doglinux/zz05xenialdog.html

install script > https://drive.google.com/open?id=0BwH00 ... 0g2ZVFmc2M
all described here : https://wiki.zoneminder.com/Ubuntu_Serv ... e_easy_way (works for 32 bit install as well)

[marianh]

Thnks.
Just installed Ubuntu 16.04 container. Hopefully it will be stable.
I'll check XenialDog if not.

[marianh]

So far running, but there is one occurrence of error "Got empty memory map file size 0, is the zmc process for this monitor running?".
Zoneminder recovered from that by itself so it's not a showstopper, but anyway it's strange that such "random" errors happen.

[bbunge]

My recomendation is to dump the container or vm! Just run zm on an old pc with Ubuntu 16.04 basic install, no desktop.

[marianh]

Well, in theory possible but that would be outside my standard setup and I would highly dislike it.

I have small server cluster at home (so it's easy to add VMs / containers) built on old but server-grade HW, with RAID, all that stuff.
It's replicated (few clicks to start VM/container on another node should HW fail on current one or if load needs to be balanced), backed up automatically, ... .
I'm not that keen to use separate HW, I definitely prefer to test/fix if needed the current setup.
I'll run it for few days.

[ScuttleSE]

I'm having the same problem with my setup running under Debian 9

After a while a few of my cameras randomly drops out. the cameras themselves work fine, the RTSP-stream is viewable from VLC, so the problem is definitely with ZM.

I created a simple script that checks the cameras every 10 minutes and restarts them if neccesary

Code: Select all

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

CAMERAS="2 4 5 6 10 11 12"

for i in $CAMERAS
do
        MEAN=`curl -s "http://zoneminder/cgi-bin/nph-zms?mode=single&monitor=$i" | convert - -format "%[mean]" info:`
        if [ $MEAN = 0 ]; then
                zmdc.pl stop zmc -m $i
                zmdc.pl stop zma -m $i
                zmdc.pl start zmc -m $i
                zmdc.pl start zma -m $i
        else
                echo Ok
        fi
done

The script downloads a jpeg from the camera from ZM, runs it through imagemagick to check the mean color, if the mean value is 0, it means the image is all black, i.e, the camera has died, and restarts the camera

[ScuttleSE]

Thnks.
Just installed Ubuntu 16.04 container. Hopefully it will be stable.
I'll check XenialDog if not.

Would be very interested to hear if switching to Ubuntu helps

[marianh]

It's too early to say as it's less than 24h so far. There were 2x "memory map file size" errors but Zoneminder recovered from that by itself and I do not know what was the gap (if any). But it requires more time to evaluate (I was running it on Debian for several weeks).
I'll give it few days of testing and report back.

Thanks for the script. I was thinking of adding small watchdog and this script is a good starting point.