Memory creep when using NAS storage, can't find the issue

[knotworking]

Baffled currently, I've been trying to troubleshoot this for days. My system:

Quad-core i7, 8th gen
32gb DDR4
Debian 11
ZM 1.36.32 running in container (I've tried multiple images and ZM versions, doesn't change the issue)
2 hikvision cameras

If I use my local NAS (freeBSD-based on dual-core i3 7th gen, 40GB memory) for storage @ 1080p (on either camera), the memory usage climbs and climbs until the system becomes unusable. If I switch to any other lower resolution (currently using 1280x960p) the issue is not present, but higher resolutions burn through the memory even faster than 1080. If I use the 256gb SSD for storage, there is no issue.

I've tested the network connection between the two using iperf3, there's no packet loss or other issue I can see on the NAS (system load average is <1). Network is 1Gbps and I can transfer multi-gigabyte files to/fro the NAS and my PC in under a seconds, I've ruled out it being a NAS issue. Cameras are recording h264+ (I've tried plain h264 with I-frame intervals at 25, no difference) with no decoding or encoding. If I switch cameras to Monitor, there is no memory issue. When I switching debugging on @ level 9, I don't see any warnings/errors. I've played with the Image Buffer, no real difference between 3-5. Max Image Buffer is 0 and Stream Replay Buffer is also set to 0. Setting the frame rate to 8 (I prefer 10) slows the creep, but it still maxes out eventually.

I'm thinking it's a container/network/packet issue of some type (duh), but I'm not seeing any lost packets or obvious errors. I am wondering if a straight install of ZM would encounter the same errors, but that's a time-suck I'm really hoping to avoid. I can live with 1280x960p, but it will bug me continually. Anybody have any tips on what I might be missing? Some of the tools I've installed to try and debug: atop, iotop, ethtool, nload, tcptrack....

[dougmccrary]

That's why it's called "The memory Remains"

Are you using any hardware decoding?
That seems to improve but not eliminate the problem.

[knotworking]

Ha, yeah, I did see a pun in the release name

I had that checkbox unchecked since I was using h.264 passthrough (Decoding Enabled had me thinking I'd only be putting more load on the RAM), but I checked it for the cameras and the memory issue seems to have completely subsided. Both cameras running @ 1080p 10fps is only taking up ~8GB of RAM and load is ~2 for the quad-core. All the dang things I tweaked and tested, can't believe I didn't trying checking that box.

Thanks Doug!!

[iconnor]

Slow disks, or slow db, or just really huge resolutions all can cause ZM to fall behind.It's a problem I am working on it, but but at the end of the day, remember that your NAS is 1000 times slower than a local disk in terms of latency. The best thing to do might be to write to a local disk, and move the events to the NAS in the background.

[JasonBorn]

With whitch settings can i delete the nas storage in this setup?

[burger]

Are you using any hardware decoding?
That seems to improve but not eliminate the problem.

I thought if you are hardware decoding, you should be using more memory.I don't understand. But if it resolves the issue, then whatever works.

I'm getting occasional short events, that look something like picture attached. I'm going to try enabling hw decoding
as recommended above to see if that resolves it. Notice that some of these are kB, and the time/date is wrong. That's something (probably memory) breaking. I'll have to investigate. I also tried increasing the image buffer to 50 from 20. It seems to give more warnings when it's set to 50, though it's not affecting the recordings. Maybe that isn't required. I have plenty of RAM/CPU though, so it shouldn't hurt.

A weird thing: the logs option is missing on the top bar. Though I can go to it by typing log in the url. However, it only shows results
for zmtelemetry, even when set to debug for the weblog. I am using devuan, but I hope that's not the cause.

Later short events that aren't kB are from me restarting zm, those can be ignored.