Excessive RAM usage using "RECORD" Mode

[tinymouse]

Hi all -
I have a Ubuntu 20.04 running Zoneminder 1.36.32 with 9 cams of various manufactures (WYZE/TAPO/SV3C). The Majority of the cams are MODECT, one in MOCORD, and two in "RECORD" mode. All are working fine. However, the two set in "RECORD" mode (Wyze v3 / Tapo C310) are consuming and holding onto RAM. Most noticeably is the C310. This one seems to consume and keep RAM over the course of several days. I have had this one grow to over 16gb. I have some screen shots from TOP showing the RAM usage (Monitor #1) . Monitor #8 is the WYZE v3 in record mode too, but doesn't grow as much; but it does grow.

Here are the settings for the C310: (the Wyze v3 is the same only it's 1920x1080)
Tapo C310 20fps 2304x1296; save JPEGS: Disabled; Camera Passthrough; options crf=18,reorder_queue_size=150
Image Buffer size=4; Max Image Buff Size=325; Warmup=0; Pre/Post=25; Section Length=600

Is there something I have missed in the setup?

Here is output from TOP

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4223 www-data 20 0 3211912 2.4g 75804 S 17.0 7.8 1:49.46 /usr/bin/zmc -m 1 (Record)
4464 www-data 20 0 2195668 1.7g 61072 S 13.0 5.5 1:20.65 /usr/bin/zmc -m 8 (Record)
4410 www-data 20 0 1507816 1.1g 61732 S 27.0 3.4 2:54.88 /usr/bin/zmc -m 7 (Modect)
4374 www-data 20 0 1231880 895396 61772 S 40.0 2.7 3:44.76 /usr/bin/zmc -m 4 (Modect)
4510 www-data 20 0 1132300 809572 86140 S 14.3 2.5 1:34.01 /usr/bin/zmc -m 9 (Nodect - lnk to id#10)
4742 www-data 20 0 966908 693676 60720 S 28.7 2.1 2:44.61 /usr/bin/zmc -m 15 (Modect)
4338 www-data 20 0 822464 550548 59964 S 28.0 1.7 2:49.40 /usr/bin/zmc -m 3 (Modect)
4288 www-data 20 0 770020 509620 60916 S 24.3 1.6 2:08.28 /usr/bin/zmc -m 2 (Modect)
4601 www-data 20 0 873836 494532 34796 S 4.7 1.5 0:30.59 /usr/bin/zmc -m 10 (Mocord)

Next Day:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4223 www-data 20 0 6837572 5.8g 66492 S 12.5 18.5 231:43.39 /usr/bin/zmc -m 1
4464 www-data 20 0 5639756 4.8g 51640 S 12.5 15.3 202:50.70 /usr/bin/zmc -m 8
4742 www-data 20 0 2886760 2.3g 51812 S 25.0 7.5 370:51.27 /usr/bin/zmc -m 15
4410 www-data 20 0 2165632 1.7g 51996 S 25.0 5.4 355:48.61 /usr/bin/zmc -m 7

Two Days later:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4223 www-data 20 0 9852476 8.6g 60200 S 14.6 27.4 462:44.70 /usr/bin/zmc -m 1
4464 www-data 20 0 5370052 4.5g 45876 S 11.6 14.4 408:38.51 /usr/bin/zmc -m 8
4410 www-data 20 0 2100752 1.6g 46284 S 20.6 5.1 714:51.03 /usr/bin/zmc -m 7

Three Days Later:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4223 www-data 20 0 13.1g 12.1g 60120 S 21.0 38.7 720:51.37 /usr/bin/zmc -m 1
4464 www-data 20 0 3214900 2.5g 45712 S 15.3 8.1 634:14.98 /usr/bin/zmc -m 8
4374 www-data 20 0 2490636 2.0g 46052 S 40.0 6.3 1491:18 /usr/bin/zmc -m 4

Thanks--

[dougmccrary]

Is your storage keeping up?
Maybe try cutting fps to 10 or even 5 - see what if any diff it makes.

[tinymouse]

Unfortunately the C310 is fixed at 20fps. Yes, storage is keeping up.

[dougmccrary]

Not sure if you’re doing analysis, but you can limit analysis fps near the bottom of that page.

[tinymouse]

I have set that; but no luck after about 5 days now it's up to 10gb

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1532477 www-data 20 0 10.4g 9.5g 53916 S 6.2 30.4 826:17.41 /usr/bin/zmc -m 1
1566991 www-data 20 0 5778104 4.9g 39352 S 12.5 15.7 626:38.80 /usr/bin/zmc -m 8

[dougmccrary]

Geeze.
Unless that's using UDP you can get rid of the reorder que. Also crf is meaningless with passthrough.
Would you post your zm settings pages?
I don't see it in specs, but is there a lower res stream you can tap to see if it affects it at all?

[tinymouse]

If I remove the reorder_queue_size, I get these in the LOG:

Date/Time,Component,Server,PID,Level,Message,File,Line
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"pkt.dts(0) must be",zm_videostore.cpp,1409
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"non increasing dts, fixing. our dts -4410 stream 0 last_dts 0. reorder_queue_size=0",zm_videostore.cpp,1398
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"pkt.dts(0) must be",zm_videostore.cpp,1409
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"non increasing dts, fixing. our dts -8820 stream 0 last_dts 0. reorder_queue_size=0",zm_videostore.cpp,1398
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"pkt.dts(0) must be",zm_videostore.cpp,1409
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"non increasing dts, fixing. our dts -18000 stream 0 last_dts 0. reorder_queue_size=0",zm_videostore.cpp,1398
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"pkt.dts(0) must be",zm_videostore.cpp,1409
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"non increasing dts, fixing. our dts -22500 stream 0 last_dts 0. reorder_queue_size=0",zm_videostore.cpp,1398
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"pkt.dts(0) must be",zm_videostore.cpp,1409
"Fri 01/06/2023 10:15:12",zmc_m1,,4583,WAR,"non increasing dts, fixing. our dts -26910 stream 0 last_dts 0. reorder_queue_size=0",zm_videostore.cpp,1398

Which zm settings pages are you referring to?

[iconnor]

The reorder_queue_size when used in source options is for buffering packets when using UDP. When used in encoder parameters it is used to buffer out of order packets when writing out in passthrough mode.

[dougmccrary]

Which zm settings pages are you referring to?

Sorry, I mean the pages for this camera, like source, general, storage ...

Also, if an option, is the camera set to vbr or cbr? (variable/constant) bit rate?

[tinymouse]

Here are the Cam Settings:
General Tab
Name: s01
Server: None
Source Type: Ffmpge
Function: Record
Analysis Enabled: UNCHECKED
Reference Image Blend %ge 12.5% (Outdoor)
Alarm Referenct Image Blend %ge 12.5%

Source Tab
Source Path rtsp://user:pw@192.168.x.x:554/stream1
Method TCP
Target colorspace 32 bit color
Capture Resolution 2304 1296 2304x1296
Orientation Normal
Deinterlacing Disabled

Storage Tab
Storage Area ZMStorage
Save JPEGs Disabled
Video Writer Camera Passthrough
Optionsa Encoder Parms: reorder_ques_size=150
store audio stream: CHECKED

Buffers
Image BUffer Size: 4
Max Image BUffer size: 325
Warmup Frames: 0
Pre Event Image Count: 25
Post Event Image Count: 25
Stream Replay Image Buffer: 0
Alarm Frame Count: 1

Estimaged Ram Use Min: 330.375MB Max: 4GB

Misc Tab mostly defaults; only:
Seciont Length: 600
Min Section Length: 16

[dougmccrary]

So there's no setting on the camera for cbr/vbr? If it's not cbr, suggest you make it so.

[iconnor]

So your max image buffer size is 325, which as the estimated ram use indicates could be up to 4gb. Reduce your keyframe interval. MaxImageBuffer should be roughly 2*keyframe interval, plus maybe a little more to handle slow IO/DB. Then you can also reduce your reorder_queue_size, again to roughly 2*keyframe interval.

[tinymouse]

This cam is a Tapo C310. I don't see anywhere to set the keyframe interval in the cam. It does appear to be vbr @20fps during the daylight and @15fps night time. Also vbr/cbr can not be set within the cam.

[robzio]

I just have to. Reducing FPS causes RAM to fill up slower, but it will fill up anyway. Is there any solution for this?

[tinymouse]

I read another post with similar problems and they discovered that having "Whether to store the audio stream when saving an event" (Found on the storage tab) checked caused their problem. So I looked at my cam and sure enough I too had that option selected. I have since unchecked that option and now this cam works much better; no more massive growth. The RAM has only grown to a max of 7gb, but usually hangs out at about 4gb-5gb.