Ffmpeg and smearing issues

Forum for questions and support relating to the 1.30.x releases only.
johnnytaco
Posts: 26
Joined: Sun Feb 26, 2017 3:16 pm

Ffmpeg and smearing issues

Post by johnnytaco »

Hi -

First off - I **LOVE** ZoneMinder. I've really enjoyed installing, configuring, tweaking, and playing with it over the last month. It's been a great home project for me. It gets me back in Linux which is always fun... I plan to contribute somehow, if possible. I'm not much of a dev but I have been spreading the word to friends who are looking at security cam systems in their homes.

I have Reolink cameras and (at least) 2 of them give me smearing issues after a few days of running. Once smearing starts for a camera, it doesn't stop until I restart ZoneMinder or the camera itself. Then it takes at least a day or more for it to happen again. It seems that changing from FFMPEG (RTP/RTSP) to Remote is a fix (2 days in, at least), but I've read all over the forums not to use Remote as it will likely go away. Also, I get random "unexpected channel errors" in the logs for my Remote systems.

Notes:
ZoneMinder 1.30.0 (distro, not compiled, used the read-me/install doc and followed to the T)
/dev/shm size fine (using <50%)
CPU load under 2, often under 1 in a 4 core system
other disk space is fine
clean CentOS/minimal build, yum updated everything (including ffmpeg) (nothing else running to speak of)
TCP (not UDP) for FFMPEG
cameras are all hardwired, cat6, 1Gb POE switch
cameras do NOT allow keyframe changes so I can't adjust those
VLC and direct URL camera viewing are fine during the smearing issue w/ ZM
once smearing starts, it doesn't stop on its own and thus records motion events
all cams are MODECT with minimal zone sizes

I really don't want to get rid of my Reolink cameras - I have 5 of them and really like them (other than this issue). Does anyone have any ideas on what else I can look at, tweak, or do to fix? I suspect it's an FFMPEG bug or something between that and ZM, but I don't know how to prove it. Also the fact that it just happens on 2 of my cameras is really weird (and suspect). But since Remote seems to avoid (or hide) the issue, I'm further stumped.

Thank you.
jt
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Ffmpeg and smearing issues

Post by bbunge »

You did well with providing info! Looks like you have done your homework to fix the issue.

Are you using 32 bit color?
Did you try to lower the resolution?
Remote should stay around in the package, I hope, as there are a lot of older cameras in use that do not work with ffmpeg...

As for the CentOS, There is a post at the top of the list about some changes to ffmpeg in CentOS. I did a test install last week of CentOS and everything went well. Of course, I did not test it with a ffmpeg camera.
johnnytaco
Posts: 26
Joined: Sun Feb 26, 2017 3:16 pm

Re: Ffmpeg and smearing issues

Post by johnnytaco »

Ha! I tried to answer all questions... missed a few though!

32 bit color - yes

Resolution - yes, dropped it quite a bit. The camera does 2560x1440 but I have it at 1080p. Actually that makes me wonder if it's EXACTLY 1920x1080. I'll check. :)

Also I have dropped the frame rate down to 10. Would prefer not to go lower than that if possible.

I'll chase the CentOS/ffmpeg a bit and see what I can find. I updated with yum though so I'd assume I have the latest and greatest outside of even CentOS.

One other thing I haven't done is a good job at looking at the logs when this happens. I have been troubleshooting with camera settings, restarting, etc. I just added an ffmpeg monitor back in so I'll see if I can reproduce again and catch the logs.

Curious - why did you include remote if you aren't pushing its use? I thought base on my research that it is fairly new. My guess is that if ffmpeg works and it's not something you have to maintain, use that and not spend the dev cycles on remote? If that's the case .. why bother including it at all?

Thanks for your help! And thanks even more for all your work on ZM.

jt
Last edited by johnnytaco on Sun Mar 19, 2017 4:27 am, edited 1 time in total.
johnnytaco
Posts: 26
Joined: Sun Feb 26, 2017 3:16 pm

Re: Ffmpeg and smearing issues

Post by johnnytaco »

As luck would have it, I have the issue going on right now, for a 3rd camera (one I hadn't moved to remote). The logs are interesting:

Code: Select all

Mar 18 19:51:02 zoneminder-vm zma_m1[2766]: INF [Back Door: 192000 - Analysing at 10.10 fps]
Mar 18 19:51:31 zoneminder-vm zmwatch[2733]: INF [Restarting capture daemon for Back Door, time since last capture 7 seconds (1489884691-1489884684)]
Mar 18 19:51:31 zoneminder-vm zmdc[2410]: INF ['zmc -m 1' sending stop to pid 2625 at 17/03/18 19:51:31]
Mar 18 19:51:31 zoneminder-vm zmc_m1[2625]: INF [Got signal 15 (Terminated), exiting]
Mar 18 19:51:37 zoneminder-vm zmdc[2410]: WAR ['zmc -m 1' has not stopped at 17/03/18 19:51:37. Sending KILL to pid 2625]
Mar 18 19:51:37 zoneminder-vm zmdc[2410]: INF ['zmc -m 1' crashed, signal 8]
Mar 18 19:51:37 zoneminder-vm zmdc[2410]: INF [Starting pending process, zmc -m 1]
Mar 18 19:51:37 zoneminder-vm zmdc[2410]: INF ['zmc -m 1' starting at 17/03/18 19:51:37, pid = 8419]
Mar 18 19:51:37 zoneminder-vm zmdc[8419]: INF ['zmc -m 1' started at 17/03/18 19:51:37]
Mar 18 19:51:38 zoneminder-vm zmc_m1[8419]: INF [Starting Capture version 1.30.0]
Mar 18 19:51:38 zoneminder-vm zmc_m1[8419]: INF [Priming capture from rtsp://[camera_ip]/h264Preview_01_main]
Mar 18 19:51:38 zoneminder-vm zmc_m1[8419]: WAR [Unable to parse ffmpeg option 0 '', expecting key=value]
Mar 18 19:51:41 zoneminder-vm zma_m1[2766]: INF [Back Door: 192239 - Gone into prealarm state]
Mar 18 19:51:42 zoneminder-vm zma_m1[2766]: INF [ExtendAlarm count: 0, ExtendAlarm frames: 0]
Mar 18 19:51:42 zoneminder-vm zma_m1[2766]: INF [Back Door: 192255 - Gone into prealarm state]
Mar 18 19:51:43 zoneminder-vm zma_m1[2766]: INF [Back Door: 192264 - Gone into alarm state]
Mar 18 19:51:43 zoneminder-vm zma_m1[2766]: INF [Back Door: 192264 - Opening new event 7560, alarm start]
Mar 18 19:51:46 zoneminder-vm zma_m1[2766]: INF [Back Door: 192293 - Gone into alert state]
Mar 18 19:51:48 zoneminder-vm zma_m1[2766]: INF [Back Door: 192318 - Left alarm state (7560) - 88(38) images]
Mar 18 19:51:48 zoneminder-vm zma_m1[2766]: INF [Back Door: 192318 - Closing event 7560, alarm end]
That time since last capture of 7 seconds is odd... Doesn't look like the camera stopped though.

Ring any bells ?

jt
Last edited by johnnytaco on Sun Mar 19, 2017 4:46 am, edited 1 time in total.
johnnytaco
Posts: 26
Joined: Sun Feb 26, 2017 3:16 pm

Re: Ffmpeg and smearing issues

Post by johnnytaco »

Did some more trial and error. Killed the zmc process manually and the smearing / artifacts continued. Then killed the zma process, same thing. Then I restarted zoneminder - the issue still persisted. :(

Rebooted the camera... boom. Problem gone. I have to assume it's a problem with the camera, but that doesn't explain to me why Remote doesn't seem to show the issue. Is it doing anything crazy different than ffmpeg rtp/rstp that you know of?

I've just set all my Reolink cameras to reboot every day... Maybe there's a memory leak or something. Argh...

jt
johnnytaco
Posts: 26
Joined: Sun Feb 26, 2017 3:16 pm

Re: Ffmpeg and smearing issues

Post by johnnytaco »

One more thought. I'm using ffmpeg 2.8.11 but there's a 3.2.4 version out there. Huge delta there, yet 2.8.11 is still a recent version. I've searched the forums here but don't see a lot of talk about different versions of ffmpeg pros/cons/requirements etc. Are there any benefits to getting a 3.x ver? Looks like I'm going to have to compile it, if so.

Sorry for the spam... stopping for now! :)

jt
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: Ffmpeg and smearing issues

Post by rockedge »

there has been some success resolving issues by upgrading FFMPEG to 3+ it has been reported.
pgrunwald
Posts: 71
Joined: Mon Mar 06, 2017 9:26 pm

Re: Ffmpeg and smearing issues

Post by pgrunwald »

JT - I have seen the same symptoms as you. I have a Reolink 411 and 410 setup. My cameras are running with both the low modect (640x360x7fps) and high stream. My high stream is at nodect 2304x1296 at 24-bit set at 20fps from the camera. I could not tell any difference between 24-bit and 32-bit performance or behavior. As far as I can tell, this is related to the CPU instruction set. I have E5620 CPUs with SSE 4.2. CPU extensions are turned on.

At times, I have only seen the smearing on the low stream and it required a camera reboot to clear.

I'm running Debian 9 with 1.30.0. ffmpeg updated in Stretch a couple of days ago:
ffmpeg -v
ffmpeg version 3.2.4-1 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-6) 20170205
configuration: --prefix=/usr --extra-version=1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil 55. 34.101 / 55. 34.101
libavcodec 57. 64.101 / 57. 64.101
libavformat 57. 56.101 / 57. 56.101
libavdevice 57. 1.100 / 57. 1.100
libavfilter 6. 65.100 / 6. 65.100
libavresample 3. 1. 0 / 3. 1. 0
libswscale 4. 2.100 / 4. 2.100
libswresample 2. 3.100 / 2. 3.100
libpostproc 54. 1.100 / 54. 1.100

I added a camera yesterday and have not seen this in a few days BTW.
Please let me know if I can help with troubleshooting.
Last edited by pgrunwald on Sun Mar 19, 2017 6:30 pm, edited 1 time in total.
pgrunwald
Posts: 71
Joined: Mon Mar 06, 2017 9:26 pm

Re: Ffmpeg and smearing issues

Post by pgrunwald »

thatguy wrote:Set rtsp_transport to tcp and upgrade to ffmpeg 3. If you can increase analyzation duration and probesize, you should.

are those Zoneminder options, ffmpeg options, or ffmpeg compile switches?

edit - I RTFM: http://www.ffmpeg-archive.org/analyzedu ... 58160.html
ffmpeg version 3.2.4-1 Copyright (c) 2000-2017 the FFmpeg developers
.
.
.

-probesize <int64> .D...... set probing size (from 32 to I64_MAX) (default 5e+06)
-formatprobesize <int> .D...... number of bytes to probe file format (from 0 to 2.14748e+09) (default 1.04858e+06)
-analyzeduration <int64> .D...... specify how many microseconds are analyzed to probe the input (from 0 to I64_MAX) (default 0)
-fpsprobesize <int> .D...... number of frames used to probe fps (from -1 to 2.14748e+09) (default -1)


Suggestions?
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Ffmpeg and smearing issues

Post by bbunge »

Johnny,
You asked about remote. It has been around a long while. In my memory of Zoneminder as long as local which seems almost forever. I seldom have problems with remote but have a lot of old cameras including an Axis that may be 20 years old and bangs away at 1(one) FPS using a 352x288 jpg. The developers have hinted that no new work is being done on remote as most new cameras need something more robust.

All my Ubuntu machines use ffmpeg 2.8.11 and work well.

bb
johnnytaco
Posts: 26
Joined: Sun Feb 26, 2017 3:16 pm

Re: Ffmpeg and smearing issues

Post by johnnytaco »

Pgrunwald - I grabbed a static binary of ffmpeg 3.2.4 but it was something like 4 (or 40??) megs compared to 212K for 2.8.11 (CentOS /yum repo) so I trashed it. I'll try compiling or looking more for another binary for centos. I may play with those ffmpeg options, or at least read about them. I have 4 Reolink 422's and 1 410. I was going to get 2 more 422's but am weary of them. The 410 seems to show smearing the most often, but the 422's do too on occasion. It's weird that once it starts, it doesn't stop until I reboot them. I also have update firmware on them... no change.

bbunge - yeah I must have misread something. I though I saw 1.28 was the first time remote was around, then 1.30 i read that we shouldn't use it. Makes more sense now. ;)

jt
SteveGilvarry
Posts: 494
Joined: Sun Jun 29, 2014 1:12 pm
Location: Melbourne, AU

Re: Ffmpeg and smearing issues

Post by SteveGilvarry »

FYI rtsp_transport is set through the Remote Method option of FFmpeg monitor, should be set to RTP/RTSP, which is currently the third option. You will also see people refer to ?tcp on the end of the URL this is the old method to set the Option.
Someone was lazy in GUI when adding ffmpeg monitor, the pull request below renames this option and sets the default to TCP.
https://github.com/ZoneMinder/ZoneMinde ... 1765/files.
Production Zoneminder 1.37.x (Living dangerously)
Random Selection of Cameras (Dahua and Hikvision)
johnnytaco
Posts: 26
Joined: Sun Feb 26, 2017 3:16 pm

Re: Ffmpeg and smearing issues

Post by johnnytaco »

Upgraded to ffmpeg 3.2.4 and made a few options changes with probesize and analyzeduration while having the issue with no noticeable effect. I have both a Remote monitor and an Ffmpeg monitor going at the same time and the Remote monitor has absolutely no spearing while the Ffmpeg one has constant smearing. I'm at a loss...

Don't know what else to do, so I'll just stick with Remote for now.

Thanks for everyone's input.

jt
pgrunwald
Posts: 71
Joined: Mon Mar 06, 2017 9:26 pm

Re: Ffmpeg and smearing issues

Post by pgrunwald »

I played the same game last night as well. My RLC-410 started freaking out in late afternoon. I thought probesize was making a difference but nothing changed until the camera went to black and white after dark. The Reolink app and high stream was clear the entire time. Reboots of the camera did not change anything. I also went through frame rate (7,10, 15), bit rate (64, 128, 160, 256, 384). and profile (high, main, base) to no effect.

Question? My camera was facing West at sundown so there was a great deal of contrast in the image. Are any of your cameras that are having problems facing the sun or bright lights?
SteveGilvarry
Posts: 494
Joined: Sun Jun 29, 2014 1:12 pm
Location: Melbourne, AU

Re: Ffmpeg and smearing issues

Post by SteveGilvarry »

Pgrunwald's issue appears maybe the camera, test that by connecting something like vlc or ffmpeg -i url command to record the stream.
JT,
Reason why you don't see it on remote is the code to retrieve the packets in remote is all zm code for RTSP, then uses ffmpeg from decoding onwards. The issue I have seen always seems to be related to RTSP UDP buffering in ffmpeg code. The buffer gets full and you only get part of the frame, the smear is ffmpeg stopping decoding on a particular line and that last line of pixels just dribbles down the rest of the frame. By changing the camera resolution you can change where it occurs and also different conditions vary the size of the frames so it moves around. There is a fifo buffer option that I need to play with in ffmpeg that may actually solve this, but the below option has worked for most people.
You didn't mention changing the "remote method" option in the ffmpeg camera to RTP/RTSP, did you do that?
Production Zoneminder 1.37.x (Living dangerously)
Random Selection of Cameras (Dahua and Hikvision)
Locked