Memory creep when using NAS storage, can't find the issue
Posted: Tue Feb 14, 2023 6:41 pm
Baffled currently, I've been trying to troubleshoot this for days. My system:
Quad-core i7, 8th gen
32gb DDR4
Debian 11
ZM 1.36.32 running in container (I've tried multiple images and ZM versions, doesn't change the issue)
2 hikvision cameras
If I use my local NAS (freeBSD-based on dual-core i3 7th gen, 40GB memory) for storage @ 1080p (on either camera), the memory usage climbs and climbs until the system becomes unusable. If I switch to any other lower resolution (currently using 1280x960p) the issue is not present, but higher resolutions burn through the memory even faster than 1080. If I use the 256gb SSD for storage, there is no issue.
I've tested the network connection between the two using iperf3, there's no packet loss or other issue I can see on the NAS (system load average is <1). Network is 1Gbps and I can transfer multi-gigabyte files to/fro the NAS and my PC in under a seconds, I've ruled out it being a NAS issue. Cameras are recording h264+ (I've tried plain h264 with I-frame intervals at 25, no difference) with no decoding or encoding. If I switch cameras to Monitor, there is no memory issue. When I switching debugging on @ level 9, I don't see any warnings/errors. I've played with the Image Buffer, no real difference between 3-5. Max Image Buffer is 0 and Stream Replay Buffer is also set to 0. Setting the frame rate to 8 (I prefer 10) slows the creep, but it still maxes out eventually.
I'm thinking it's a container/network/packet issue of some type (duh), but I'm not seeing any lost packets or obvious errors. I am wondering if a straight install of ZM would encounter the same errors, but that's a time-suck I'm really hoping to avoid. I can live with 1280x960p, but it will bug me continually. Anybody have any tips on what I might be missing? Some of the tools I've installed to try and debug: atop, iotop, ethtool, nload, tcptrack....
Quad-core i7, 8th gen
32gb DDR4
Debian 11
ZM 1.36.32 running in container (I've tried multiple images and ZM versions, doesn't change the issue)
2 hikvision cameras
If I use my local NAS (freeBSD-based on dual-core i3 7th gen, 40GB memory) for storage @ 1080p (on either camera), the memory usage climbs and climbs until the system becomes unusable. If I switch to any other lower resolution (currently using 1280x960p) the issue is not present, but higher resolutions burn through the memory even faster than 1080. If I use the 256gb SSD for storage, there is no issue.
I've tested the network connection between the two using iperf3, there's no packet loss or other issue I can see on the NAS (system load average is <1). Network is 1Gbps and I can transfer multi-gigabyte files to/fro the NAS and my PC in under a seconds, I've ruled out it being a NAS issue. Cameras are recording h264+ (I've tried plain h264 with I-frame intervals at 25, no difference) with no decoding or encoding. If I switch cameras to Monitor, there is no memory issue. When I switching debugging on @ level 9, I don't see any warnings/errors. I've played with the Image Buffer, no real difference between 3-5. Max Image Buffer is 0 and Stream Replay Buffer is also set to 0. Setting the frame rate to 8 (I prefer 10) slows the creep, but it still maxes out eventually.
I'm thinking it's a container/network/packet issue of some type (duh), but I'm not seeing any lost packets or obvious errors. I am wondering if a straight install of ZM would encounter the same errors, but that's a time-suck I'm really hoping to avoid. I can live with 1280x960p, but it will bug me continually. Anybody have any tips on what I might be missing? Some of the tools I've installed to try and debug: atop, iotop, ethtool, nload, tcptrack....