Persistent dropped frames using MODECT using 1.34.26

Forum for questions and support relating to the 1.34.x releases only.
Post Reply
texastexas
Posts: 12
Joined: Wed Oct 06, 2021 4:06 am

Persistent dropped frames using MODECT using 1.34.26

Post by texastexas »

I'm running my first ever install of ZM and I think it's great. Being a newbie, I may be missing something. I'm using an Amcrest IP8 camera in 4k at 15FPS. It's the only camera on the network.

When I get an alarm, the timeline is missing about 2-3 seconds of frames. If I encode with h265 it's about 3 seconds and with h264 about 2 seconds. I've tried playing with the buffers, and that gets me more frames before and after events, but I can't seem to recover or find the missing 2-3 seconds.

I don't know all of the settings one can adjust in ZM, so any help is welcome.
User avatar
Andyrh
Posts: 238
Joined: Sat Oct 28, 2017 3:55 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by Andyrh »

This may not be good news, but I saw a similar problem when I was running on a gen 1 i3 and 2 cameras, I tried many things and could not solve it on that HW. It would drop frames around the time of the motion detection. I suspect it had to do with the buffer dump to disk and processing the new frames and updating the DB and stuff I do not know about all at the same time.
I ended up upgrading to a gen 4 i7 and the problem went away, even with 6 cameras.

You might try a lower frame rate. Many here will recommend 5FPS, I like 10FPS.

What CPU are you using?
Andy
o||||o

Ubuntu 22.04
ZM 1.36.33
E5-1650-v4 Xeon
16 GB RAM
6 cameras -> 54 FPS modect
texastexas
Posts: 12
Joined: Wed Oct 06, 2021 4:06 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by texastexas »

It's a VM on ESXi 6.7. I have a fair bit of horsepower on the VM. 16 CPU sockets and 4GB or RAM reserved. The hardware is two E5-2690 @ 2.90GHz and 64GB on the host (I can allocate more to ZM if helpful).

It seems to be a straight buffer interaction with the DB. When I record constantly, there are zero dropped frames. Doesn't that imply a bug and not a CPU issue? I've tried buffer settings, and more buffer seems to mean bigger losses. With:

Image Buffer Size (frames) 50
Warmup Frames 0
Pre Event Image Count 30
Post Event Image Count 10
Stream Replay Image Buffer 0
Alarm Frame Count 1

I get a 3.5 second drop out!
User avatar
Andyrh
Posts: 238
Joined: Sat Oct 28, 2017 3:55 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by Andyrh »

Is the host busy? Try reducing the vCPU count. ESXi will wait for 16 cores to be available before allowing the VM to execute. With 1 camera I would look at 2 cores and work up from there. And yes, it is counter-intuitive.
Andy
o||||o

Ubuntu 22.04
ZM 1.36.33
E5-1650-v4 Xeon
16 GB RAM
6 cameras -> 54 FPS modect
texastexas
Posts: 12
Joined: Wed Oct 06, 2021 4:06 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by texastexas »

I cut the vCPUs to 4 and still have a 2 second drop. I'd think it's buffer to DB problem because steady recording has no drops. I'm guessing, but when the alarm is triggered, there's some branch in the program that fails to pull in the entire buffer.
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Persistent dropped frames using MODECT using 1.34.26

Post by iconnor »

4k video takes too much cpu.You might try 1.36 series where we have been trying to improve things. Lots of people still have trouble with super high res cameras though.
texastexas
Posts: 12
Joined: Wed Oct 06, 2021 4:06 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by texastexas »

I'll try it tonight.

I tried a BlueIris demo and there's no problem with lost frames. I would think Linux is lighter weight and faster than a Win program. I'd like to think the dropped frame issue for ZM is fixable if BI can do it with the same camera in the same conditions.
texastexas
Posts: 12
Joined: Wed Oct 06, 2021 4:06 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by texastexas »

I'm now running 1.36.8.

I'm hopeful.

Now my events have the alarmed frames without drops but I get a different weird thing happening. For a single trigger event (me walking in a room, turning around and walking out) ZM cuts the event into two equal 20 second events. I set my minimum event time to 20 seconds, so it sort of makes sense, but for a trigger like the one I described, shouldn't it be one event? I'm running BI on the same camera, and it logs it as one event.

I'm hopeful, now that all the alarm frames are present, there's just a small adjustment I can make to turn it into a single event.

Thanks
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Persistent dropped frames using MODECT using 1.34.26

Post by iconnor »

Set post event frames to something larger than it is.

So that it keeps recording the event even if it isn't detecting motion.
texastexas
Posts: 12
Joined: Wed Oct 06, 2021 4:06 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by texastexas »

After a large amount of experimentation : clean installs of 1.34, 1.36 and different hardware with each, I found what I believe if the issue.

When I use systemd to mount my NAS, I lose 1-3 seconds of frame capture after each alarm no matter what version of ZM I use. When I store locally, no loss of frames. One might say this is latency built into the reaching of the NAS, but that doesn't seem true. The pre alarm prames are there, they should be the ones lost if latency was the issue. I don't know ZM programming well enough to know if this can be fixed, but it seems an issue.

For background, both my NAS and my ZM ubuntu server are both on the same ESXi server and communicate through vmx0 network connections (high speed). If anyone knows how to resolve this please let me know. I'd like to have my NAS save the ZM output.

Thanks
User avatar
iconnor
Posts: 2862
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Persistent dropped frames using MODECT using 1.34.26

Post by iconnor »

When you say you are dropping frames... you mean they aren't in the db or in the mp4 right?

And you are encoding to h265?!

Here's the thing... writing an mp4, the ffmpeg libraries and encoders buffer a ton of frames. So when it comes time to finalize the file, it can take a lot of time. Like 10 seconds, during which we are not doing motion detection or anything else. You exacerbate the problem by saving to a NAS with latency that is large orders of magnitude slower than a local disk.

In 1.36 I moved finalising the mp4 to it's own thread so that we can back to motion detection etc. Seems to work. I tried to do the same in 1.32/1.34 but the x264 libraries are not thread safe so the results were interesting.

My advice is to store locally and move events to the NAS in the background using a filter.
texastexas
Posts: 12
Joined: Wed Oct 06, 2021 4:06 am

Re: Persistent dropped frames using MODECT using 1.34.26

Post by texastexas »

I'm getting h265 from the camera. Happy to pass it on to the drive if that helps.

My set up is: 4K 30FPS camera to ZM on ESXi to NAS on same ESXi server (transfer should be pretty good with vmx0 ethernet).

OK. I'll give local storage a try. Is there a chance the NAS would work if the buffer were big enough? Are most folks transferring files with a script or backup SW? I'm pretty new to Linux, but I would think a script is better.
Post Reply