1.34.22 (and .20 and .21) MODECT woes

Forum for questions and support relating to the 1.34.x releases only.
sideband
Posts: 27
Joined: Sat Oct 24, 2020 2:19 pm

1.34.22 (and .20 and .21) MODECT woes

Post by sideband »

I'm having a peculiar issue that's only cropped up in the last few releases (and since I changed a couple of cameras)...

I'm about 80% certain this issue is PEBKAC... I just don't know where to look to fix it.

What's happening is this: When an event is triggered, recording starts, just as it should, however, during the "red line" part of playback (and all downloads of events, too....) where the "trigger" is, there are no images saved except the first one in the "red line" zone.

Now, I'm fairly certain I've got enough CPU, RAM, and Network Bandwidth to handle recording events on this machine (SuperMicro X10DRi, with two E5-2620V3s and 128G DDR4-2133, and all cam I/O is done on a separate gigabit ethernet port that doesn't have ANY other traffic on it).

The system is running Ubuntu 20.04.1 with (just updated) ZoneMinder 1.34.22.

I've got 3 @ 5MP cams and 4 @ 2MP cams, all PoE and going through a gigabit switch (Cisco 4948, if that matters to anyone).

I've got plenty of SHM (only 58% utilized, on average) and the system load sits somewhere around 7.2 most of the time.

I'm 99% sure it's not hardware causing the issue.

I've got my buffers set up thus:

On the 5MP cams:
Image Buffer Size (frames)
420
Warmup Frames
30
Pre Event Image Count
30
Post Event Image Count
60
Stream Replay Image Buffer
240
Alarm Frame Count
1

On the 2MP (1080) cams:
Image Buffer Size (frames)
120
Warmup Frames
30
Pre Event Image Count
30
Post Event Image Count
60
Stream Replay Image Buffer
120
Alarm Frame Count
1

All cams are at 15fps, though I've tried dropping them to as slow as 5fps (in the cameras themselves, the frame rates in ZM are left blank) and had no change in behavior.

the 5MP and the 2MP cams both do the same thing... which means (to me... I could be wrong) whatever is causing this is resolution-independent.

I've been fighting this for quite some time, and I'd really like to get it solved.

Any and All advice would be greatly appreciated.

Thanks in advance.

-TP

Copypasta of system specs:
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz

# free
total used free shared buff/cache available
Mem: 131912704 18183328 9047408 29449700 104681968 83216428
Swap: 33554428 2601216 30953212

# cat /proc/meminfo
MemTotal: 131912704 kB
MemFree: 8008068 kB
MemAvailable: 82406764 kB
Buffers: 2423048 kB
Cached: 95856780 kB
SwapCached: 156080 kB
Active: 29522128 kB
Inactive: 85272848 kB

# df -h
...
tmpfs 48G 28G 21G 59% /dev/shm
...
User avatar
Bluemax
Posts: 121
Joined: Wed Jun 12, 2019 5:15 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by Bluemax »

Alarm frames have to be enabled explicitly on ffmpeg/passthrough.
sideband
Posts: 27
Joined: Sat Oct 24, 2020 2:19 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by sideband »

Bluemax wrote: Sat Oct 24, 2020 3:39 pm Alarm frames have to be enabled explicitly on ffmpeg/passthrough.
Howso? Every part of the video plays, except the time during the trigger. ZM is doing the alarm detection, not the cameras, so I'm not certain how ffmpeg/passthrough would have an effect. Educate me?
User avatar
Bluemax
Posts: 121
Joined: Wed Jun 12, 2019 5:15 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by Bluemax »

sideband wrote: Sat Oct 24, 2020 2:52 pm there are no images saved
You probably don't mean separate jpg alarm frames on passthrough mode then i guess.

What are your 'source' and 'storage' settings?
sideband
Posts: 27
Joined: Sat Oct 24, 2020 2:19 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by sideband »

Bluemax wrote: Sat Oct 24, 2020 5:44 pm What are your 'source' and 'storage' settings?
Source:

Source Path
rtsp://someusername:noththatgullible@10.42.2.27:554/stream0
Method?)
TCP
Options (?)
DecoderHWAccelName (?)
cuda
DecoderHWAccelDevice (?)
Target colorspace
32 bit color
Capture Resolution (pixels)
2592

1944

2592x1944 5MP
Preserve Aspect Ratio
Orientation
Normal
Deinterlacing
Disabled

Storage:

Storage Area
Exports
Save JPEGs
Frames + Analysis images (if available)
Video Writer
X264 Encode
Optional Encoder Parameters (?)
# Lines beginning with # are a comment
# For changing quality, use the crf option
# 1 is best, 51 is worst quality
#crf=23
Whether to store the audio stream when saving an event.


As you can see... nothing unusual. Yes, I've tried removing the cuda HW accelerator, as well, no difference.

Again.. the videos play on replay, up until the alarm starts in the video (the red line), then only the first image in the alarm is displayed until the alarm ends. I have timestamps turned on on the cameras as well as ZM, and I can watch the ZM timestamp count up in seconds while the image's timestamp stays the same.. until the alarm ends, then it picks up from that time frame and continues. NO .jpgs are saved in /exports/cams during the alarm, but the before and after alarm jpgs are saved.

Perplexing, isn't it?
User avatar
Bluemax
Posts: 121
Joined: Wed Jun 12, 2019 5:15 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by Bluemax »

sideband wrote: Sat Oct 24, 2020 6:36 pm Save JPEGs
Frames + Analysis images (if available)
This is usually only used when tweaking zones or debugging. For regular usage this should be rather off.
sideband wrote: Sat Oct 24, 2020 6:36 pm Video Writer
X264 Encode
Try 'H264 camera passthrough' if possible.

I rather suspect bad stream encoding. If x264 uses h/w acceleration (gpu) try to turn it off. GPUs might produce exotic streams.
It also doesn't mean the stream is corrupt. Maybe only the decoders have difficulties. You could try to remultiplex the MP4 file (ffmpeg -i input.mp4 -c copy output.mp4) to see if it fixes the stream to achieve more informations.
sideband
Posts: 27
Joined: Sat Oct 24, 2020 2:19 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by sideband »

Bluemax wrote: Sat Oct 24, 2020 7:22 pm I rather suspect bad stream encoding. If x264 uses h/w acceleration (gpu) try to turn it off. GPUs might produce exotic streams.
It also doesn't mean the stream is corrupt. Maybe only the decoders have difficulties. You could try to remultiplex the MP4 file (ffmpeg -i input.mp4 -c copy output.mp4) to see if it fixes the stream to achieve more informations.
I'll try your suggestions, but, again, I'm not sure why that would be. All video that isn't in the "red" trigger zone plays smoothly and without issue. it's only the video in the "red" trigger area that disappears.
Also, the lead-in and lead-out before and after the red "trigger" zone in every video is good, but the red "trigger" zone has no associated .jpg files, either, so it's impossible to see what has happened in that 4+ seconds... just what the trigger itself was.
User avatar
Bluemax
Posts: 121
Joined: Wed Jun 12, 2019 5:15 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by Bluemax »

What have you set in 'General' tab? Did you check the logs?
Maybe backup your config/database and restart ZM from scratch for testing. My 1.34.22 doesn't behave like this. I cannot give you specific help but that's what i would try.
sideband
Posts: 27
Joined: Sat Oct 24, 2020 2:19 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by sideband »

EDIT: NO, I haven't found the culprit.

Well, I've found the culprit.

It appears that recent versions of ZM don't like the cameras sending video in Variable Bit Rate. When I switched the cams to Constant Bit Rate, ZM started writing the alarm .jpgs and showing video through the actual trigger again.

Seems inefficient, but at least it's a workaround that's easily implemented.


EDIT: I'm wrong... It wasn't VBR/CBR causing the issue...

The alarmed frames aren't being encoded by ZM for some reason. Logs don't show why.

If I set for H264 passthrough on storage, the whole video writes (and why shouldn't it? there's nothing to re-encode) but ZM is unable to put a timestamp on the events. If I set for X264 encode (with .jpeg on or off - doesn't matter) the video gets encoded, EXCEPT for the alarmed frames.. Everything before and after is encoded properly.

The problem persists.
sideband
Posts: 27
Joined: Sat Oct 24, 2020 2:19 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by sideband »

I wonder if @iconnor has seen this yet? He knows this software better than anyone else...
User avatar
Bluemax
Posts: 121
Joined: Wed Jun 12, 2019 5:15 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by Bluemax »

sideband wrote: Sun Oct 25, 2020 12:09 pm If I set for H264 passthrough on storage, the whole video writes
The red areas are ok?

What are your settings in 'general' tab?
sideband
Posts: 27
Joined: Sat Oct 24, 2020 2:19 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by sideband »

Bluemax wrote: Sun Oct 25, 2020 4:33 pm What are your settings in 'general' tab?

Name
CAR-SE
Notes
Server
None
Source Type
Ffmpeg
Function
Modect
Enabled
Linked Monitors (?)
Select Some Options
Groups
Select Some Options
Analysis FPS
Maximum FPS (?)
Alarm Maximum FPS (?)
Reference Image Blend %ge
6
Alarm Reference Image Blend %ge
6
Triggers None available

Nothing unusual.
User avatar
Bluemax
Posts: 121
Joined: Wed Jun 12, 2019 5:15 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by Bluemax »

So, to sum it up:
- H264 passthrough is ok.
- H264 Encode is not.
- Red areas have black video but the ZM timestamps work properly.
- Affected versions are (at least) 20, 21, 22.

I will test the 'h264 encode' mode on my system.

The 'h264 encode' mode is usually just needed for h265 cameras. You pay a high price just for timestamps especially when your camera provides them already.
User avatar
iconnor
Posts: 2881
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by iconnor »

I've taken a quick look at the code and nothing obvious jumped out at me. I will switch to encode and do some testing. I would expect if it isn't encoding the alarm frames, then it would also not be saving the jpegs (if jpeg saving is turned on).

So far my testing is showing me alarm frames in the encoded mp4 so I'm not sure what's going on.
User avatar
Bluemax
Posts: 121
Joined: Wed Jun 12, 2019 5:15 pm

Re: 1.34.22 (and .20 and .21) MODECT woes

Post by Bluemax »

Bluemax wrote: Sun Oct 25, 2020 11:55 pm I will test the 'h264 encode' mode on my system.
I cannot test 'h264 encode' (i haven't compiled it in) but exclusive JPG saving does work here as well.

Just noticed that your 'Analysis FPS' field is empty. Maybe this is related. Otherwise, as stated before, i'd go for a factory/database reset. You can always backup and restore your current db.
Post Reply