Zoneminder processes being killed by OOM and OOMD on Fedora

Support and queries relating to third party Live CDs and DVD, VM images and other packaged distributions
Post Reply
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Hello, I recently installed Zoneminder on Fedora. I was having some issues with Zoneminder being killed by the system OOMD and OOM processes. I changed my settings for both and so far all seems ok. I'm hoping that what I post here can help someone. Or perhaps the powers that be can suggest better settings than I have used.

Install details:

Fedora 34 (server edition, very minimal install)
Zoneminder 1.36.12
Number of cameras: 2
fcgiwrap daemon procs: 6
Zoneminder Machine RAM: 8 GB

Files added/updated:

/etc/systemd/oomd.conf

Code: Select all

[OOM]
SwapUsedLimit=99%
DefaultMemoryPressureLimit=100%
DefaultMemoryPressureDurationSec=30s
/etc/sysctl.d/00-reserve-ratio.conf

Code: Select all

vm.lowmem_reserve_ratio = 128 128 32

I tried to update the OOMD configuration (oomd.conf) so that OOMD would not kill any processes. This seemed to work, but then to my surprise the old OOM process kicked in and would kill my Zoneminder processes.

To prevent OOM from doing this, I followed the advice from this link: https://blog.kamens.us/2016/11/10/linux ... of-memory/

The above link suggests changing the defaults from "256 256 32" to "128 128 32", which is what I did in the file 00-reserve-ratio.conf.

As I said, so far Zoneminder seems to be running well. I'll post an update here if that changes.
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Forgot to mention Zoneminder is running under nginx, and not Apache. It's probably not relevant, but thought I'd include this.
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Spoke too soon... Zoneminder has been killed. I will keep on playing with the values for the reserve ratio and see what happens.

Here's the service log, in case anyone is interested:

Code: Select all

Dec 30 16:53:03 zmstats[868]: INF [Stats Daemon exiting]
Dec 30 16:53:03 zmdc[805]: INF ['zmc -m 1' exited normally]
Dec 30 16:53:03 zmdc[805]: INF ['zmwatch.pl' exited normally]
Dec 30 16:53:03 zmdc[805]: INF ['zmstats.pl' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmfilter.pl --filter_id=1 --daemon' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmfilter.pl --filter_id=2 --daemon' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmc -m 2' exited normally]
Dec 30 16:53:04 zmdc[805]: INF [Server shutdown at 21/12/30 16:53:04]
Dec 30 16:53:04 systemd[1]: zoneminder.service: Failed with result 'oom-kill'.
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Adjusting the reserve ratio did not work. I kept halving the numbers until I reached the minimum. In other words, I tried these values for vm.lowmem_reserve_ratio:

Code: Select all

128 128 32
64 64 32
32 32 32
16 16 32
8 8 32
4 4 32
2 2 32
1 1 32
However, after each adjustment OOM always kicked in and killed ZM.

I then did a fresh install on Fedora 35 (not 34), but included 8 GB of old style disk swap space. By default Fedora 34 (?) and later do not include this. They include a compressed swap (zram) that exists in RAM only.

In this install I did not do anything to adjust OOMD or OOM.

In any case, Zoneminder seems much happier with the included on disk swap. It ran for 24 hours and wasn't killed, whereas without the disk swap ZM would be killed within 12 hours.

As an extra test, I removed the swap from the working install. Immediately, ZM was killed by OOMD.

For now, I'll add the swap back in and run with it. Hopefully this is solved.
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Not solved. ZM keeps getting killed. It looks like ZM is hanging onto too much memory. If there's anything else I can post here that can be of help, let me know. Here's a "top" of the system after running for a while:

f35sec3-top-w-virt-mem.JPG
f35sec3-top-w-virt-mem.JPG (97.86 KiB) Viewed 36816 times
Magic919
Posts: 865
Joined: Wed Sep 18, 2013 6:56 am

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by Magic919 »

Try disable saving JPEGs if you haven't already.
-----------------------------------------------------------------------
I'm nothing to do with ZM, ZMNinja or ZMES, I just use them.
Contribute funding to the devs here - https://opencollective.com/zoneminder
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Thanks. I will try that and report back. "Save JPEGs" was set to "Analysis Images Only (if available)".
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Just to add some more information, here are the settings for one of the cameras. There appears to be a limit, so will add more settings in another post.
mo-1-settings-general.JPG
mo-1-settings-general.JPG (61.32 KiB) Viewed 36785 times
mo-1-settings-source.JPG
mo-1-settings-source.JPG (38.26 KiB) Viewed 36785 times
mo-1-settings-storage.JPG
mo-1-settings-storage.JPG (32.24 KiB) Viewed 36785 times
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Settings for buffers:
mo-1-settings-buffers.JPG
mo-1-settings-buffers.JPG (30.56 KiB) Viewed 36785 times
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Magic919 wrote: Thu Jan 06, 2022 9:19 am Try disable saving JPEGs if you haven't already.
ZM got killed by OOM after 3 days (?) after disabling JPEGs. I will note that I was running with OOMD off (see oomd.conf in my first post).

Here are some "top"s over time. I only got 2 days worth. ZM was killed about 12 hours after the last image.

Day 1:
f35sec3-top-no-jpg.JPG
f35sec3-top-no-jpg.JPG (88.2 KiB) Viewed 36745 times
Day 2:
f35sec3-top-no-jpg-day-2.JPG
f35sec3-top-no-jpg-day-2.JPG (77.56 KiB) Viewed 36745 times
Day 2 (some hours after above):
f35sec3-top-no-jpg-day-2b.JPG
f35sec3-top-no-jpg-day-2b.JPG (66.24 KiB) Viewed 36745 times
Last edited by chue on Sat Jan 08, 2022 3:32 pm, edited 2 times in total.
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Also, for some reason I could no longer see any live views (e.g. montage or camera view). I need to investigate if disabling JPEGs caused this.

I will add that if ZM crashes after 3 days, I can work around this with nightly reboots. Still probably needs some investigation by the devs at some point though.
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Turning off Save JPGs does disable live view, which is a dealbreaker. I don't know what to do at this point, perhaps I'll try a different distribution perhaps CentOS or Ubuntu. Or perhaps install an older version of ZoneMinder (1.34).
Magic919
Posts: 865
Joined: Wed Sep 18, 2013 6:56 am

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by Magic919 »

My live views are not affected, so for me it's a useful way to cut memory use.
-----------------------------------------------------------------------
I'm nothing to do with ZM, ZMNinja or ZMES, I just use them.
Contribute funding to the devs here - https://opencollective.com/zoneminder
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

I installed an older version of ZM (1.34.26-1) on Fedora 34. This was installed without on disk swap, so with only compressed swap in RAM.

This older version of ZM was unavailable on Fedora 35, otherwise I would have gone with the newer Fedora.

The memory usage has been low and stable over time - memory usage does not grow. I also noticed that on this version of ZM, when something moves on camera, the live views stay responsive. The newer version live views would freeze for 20 or more seconds. So for now this is what I will go with.

Here is a "top" after a few days:
f34sec4-day3.JPG
f34sec4-day3.JPG (75.66 KiB) Viewed 36552 times
chue
Posts: 13
Joined: Thu Dec 30, 2021 10:19 pm

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Post by chue »

Memory usage still quite stable after 5+ days
f34sec4-day5.JPG
f34sec4-day5.JPG (79.49 KiB) Viewed 36451 times
Post Reply