Page 1 of 2

Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Thu Dec 30, 2021 10:39 pm
by chue
Hello, I recently installed Zoneminder on Fedora. I was having some issues with Zoneminder being killed by the system OOMD and OOM processes. I changed my settings for both and so far all seems ok. I'm hoping that what I post here can help someone. Or perhaps the powers that be can suggest better settings than I have used.

Install details:

Fedora 34 (server edition, very minimal install)
Zoneminder 1.36.12
Number of cameras: 2
fcgiwrap daemon procs: 6
Zoneminder Machine RAM: 8 GB

Files added/updated:

/etc/systemd/oomd.conf

Code: Select all

[OOM]
SwapUsedLimit=99%
DefaultMemoryPressureLimit=100%
DefaultMemoryPressureDurationSec=30s
/etc/sysctl.d/00-reserve-ratio.conf

Code: Select all

vm.lowmem_reserve_ratio = 128 128 32

I tried to update the OOMD configuration (oomd.conf) so that OOMD would not kill any processes. This seemed to work, but then to my surprise the old OOM process kicked in and would kill my Zoneminder processes.

To prevent OOM from doing this, I followed the advice from this link: https://blog.kamens.us/2016/11/10/linux ... of-memory/

The above link suggests changing the defaults from "256 256 32" to "128 128 32", which is what I did in the file 00-reserve-ratio.conf.

As I said, so far Zoneminder seems to be running well. I'll post an update here if that changes.

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Thu Dec 30, 2021 10:45 pm
by chue
Forgot to mention Zoneminder is running under nginx, and not Apache. It's probably not relevant, but thought I'd include this.

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Fri Dec 31, 2021 12:06 am
by chue
Spoke too soon... Zoneminder has been killed. I will keep on playing with the values for the reserve ratio and see what happens.

Here's the service log, in case anyone is interested:

Code: Select all

Dec 30 16:53:03 zmstats[868]: INF [Stats Daemon exiting]
Dec 30 16:53:03 zmdc[805]: INF ['zmc -m 1' exited normally]
Dec 30 16:53:03 zmdc[805]: INF ['zmwatch.pl' exited normally]
Dec 30 16:53:03 zmdc[805]: INF ['zmstats.pl' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmfilter.pl --filter_id=1 --daemon' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmfilter.pl --filter_id=2 --daemon' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmc -m 2' exited normally]
Dec 30 16:53:04 zmdc[805]: INF [Server shutdown at 21/12/30 16:53:04]
Dec 30 16:53:04 systemd[1]: zoneminder.service: Failed with result 'oom-kill'.

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Mon Jan 03, 2022 10:36 pm
by chue
Adjusting the reserve ratio did not work. I kept halving the numbers until I reached the minimum. In other words, I tried these values for vm.lowmem_reserve_ratio:

Code: Select all

128 128 32
64 64 32
32 32 32
16 16 32
8 8 32
4 4 32
2 2 32
1 1 32
However, after each adjustment OOM always kicked in and killed ZM.

I then did a fresh install on Fedora 35 (not 34), but included 8 GB of old style disk swap space. By default Fedora 34 (?) and later do not include this. They include a compressed swap (zram) that exists in RAM only.

In this install I did not do anything to adjust OOMD or OOM.

In any case, Zoneminder seems much happier with the included on disk swap. It ran for 24 hours and wasn't killed, whereas without the disk swap ZM would be killed within 12 hours.

As an extra test, I removed the swap from the working install. Immediately, ZM was killed by OOMD.

For now, I'll add the swap back in and run with it. Hopefully this is solved.

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Wed Jan 05, 2022 6:30 pm
by chue
Not solved. ZM keeps getting killed. It looks like ZM is hanging onto too much memory. If there's anything else I can post here that can be of help, let me know. Here's a "top" of the system after running for a while:

f35sec3-top-w-virt-mem.JPG
f35sec3-top-w-virt-mem.JPG (97.86 KiB) Viewed 70502 times

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Thu Jan 06, 2022 9:19 am
by Magic919
Try disable saving JPEGs if you haven't already.

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Thu Jan 06, 2022 4:09 pm
by chue
Thanks. I will try that and report back. "Save JPEGs" was set to "Analysis Images Only (if available)".

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Thu Jan 06, 2022 4:35 pm
by chue
Just to add some more information, here are the settings for one of the cameras. There appears to be a limit, so will add more settings in another post.
mo-1-settings-general.JPG
mo-1-settings-general.JPG (61.32 KiB) Viewed 70471 times
mo-1-settings-source.JPG
mo-1-settings-source.JPG (38.26 KiB) Viewed 70471 times
mo-1-settings-storage.JPG
mo-1-settings-storage.JPG (32.24 KiB) Viewed 70471 times

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Thu Jan 06, 2022 4:35 pm
by chue
Settings for buffers:
mo-1-settings-buffers.JPG
mo-1-settings-buffers.JPG (30.56 KiB) Viewed 70471 times

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Sat Jan 08, 2022 3:26 pm
by chue
Magic919 wrote: Thu Jan 06, 2022 9:19 am Try disable saving JPEGs if you haven't already.
ZM got killed by OOM after 3 days (?) after disabling JPEGs. I will note that I was running with OOMD off (see oomd.conf in my first post).

Here are some "top"s over time. I only got 2 days worth. ZM was killed about 12 hours after the last image.

Day 1:
f35sec3-top-no-jpg.JPG
f35sec3-top-no-jpg.JPG (88.2 KiB) Viewed 70431 times
Day 2:
f35sec3-top-no-jpg-day-2.JPG
f35sec3-top-no-jpg-day-2.JPG (77.56 KiB) Viewed 70431 times
Day 2 (some hours after above):
f35sec3-top-no-jpg-day-2b.JPG
f35sec3-top-no-jpg-day-2b.JPG (66.24 KiB) Viewed 70431 times

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Sat Jan 08, 2022 3:30 pm
by chue
Also, for some reason I could no longer see any live views (e.g. montage or camera view). I need to investigate if disabling JPEGs caused this.

I will add that if ZM crashes after 3 days, I can work around this with nightly reboots. Still probably needs some investigation by the devs at some point though.

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Tue Jan 11, 2022 12:25 am
by chue
Turning off Save JPGs does disable live view, which is a dealbreaker. I don't know what to do at this point, perhaps I'll try a different distribution perhaps CentOS or Ubuntu. Or perhaps install an older version of ZoneMinder (1.34).

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Tue Jan 11, 2022 9:02 am
by Magic919
My live views are not affected, so for me it's a useful way to cut memory use.

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Tue Jan 18, 2022 5:18 pm
by chue
I installed an older version of ZM (1.34.26-1) on Fedora 34. This was installed without on disk swap, so with only compressed swap in RAM.

This older version of ZM was unavailable on Fedora 35, otherwise I would have gone with the newer Fedora.

The memory usage has been low and stable over time - memory usage does not grow. I also noticed that on this version of ZM, when something moves on camera, the live views stay responsive. The newer version live views would freeze for 20 or more seconds. So for now this is what I will go with.

Here is a "top" after a few days:
f34sec4-day3.JPG
f34sec4-day3.JPG (75.66 KiB) Viewed 70238 times

Re: Zoneminder processes being killed by OOM and OOMD on Fedora

Posted: Sat Jan 22, 2022 3:40 pm
by chue
Memory usage still quite stable after 5+ days
f34sec4-day5.JPG
f34sec4-day5.JPG (79.49 KiB) Viewed 70137 times