HW Optimisation?

[zmaximz]

Hello! It's great that ZM now supports HW acceleration h264...

But what about:
" Post by iconnor » Wed Aug 22, 2018 10:07 pm
zmc always decodes the video, no matter what. Maybe some future version won't, but right now, it always decodes. You are decoding an 8MP stream... without hardware acceleration..."

It's too expensive to use old ZM on large system (about 100 cams) because of high CPU usage... too many decoding... it's electricity... expensive servers...
Is it still decoding all video?

[iconnor]

Yes it is.

[fontiano]

Hi,

I'd like to enable cuda support on ZM 1.34.0 - Debian 10.

This is the output of "ffmpeg -hwaccels" provided by ZM package:

root@videosorveglianza:~# /usr/bin/ffmpeg -hwaccels
ffmpeg version 4.1.4-1~deb10u1 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 8 (Debian 8.3.0-6)
---
Hardware acceleration methods:
vdpau
vaapi
drm

and this is the output of recompiled ffmpeg binary with cuda support:

root@videosorveglianza:~# /usr/bin/ffmpeg -hwaccels
ffmpeg version N-96640-gb4f300f8ea Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 8 (Debian 8.3.0-6)
---
Hardware acceleration methods:
vdpau
cuda
vaapi
cuvid

In the box "DecoderHWAccelName" I writed "cuda", started browsing event frames but I did not see improvements and looking at ffmpeg process i see this:

ps -auxww

www-data 24776 0.0 0.0 2388 768 ? S 22:30 0:00 sh -c ffmpeg -ss 34.30 -i /var/cache/zoneminder/events/1/2020-02-04/50027/50027-video.mp4 -frames:v 1 /var/cache/zoneminder/events/1/2020-02-04/50027/00020-capture.jpg

It seems that ffmpeg is launched without "-hwaccel cuda" option.

Here the performance comparison between ffmpeg with and without hwaccel option:

sudo -u www-data ffmpeg -ss 34.26 -i /var/cache/zoneminder/events/1/2020-02-04/50027/50027-video.mp4 -frames:v 1 /var/cache/zoneminder/events/1/2020-02-04/50027/00019-capture.jpg

result: frame= 1 fps=0.4 q=7.1 Lsize=N/A time=00:00:00.10 bitrate=N/A speed=0.0384x

sudo -u www-data ffmpeg -ss 34.26 -hwaccel cuda -i /var/cache/zoneminder/events/1/2020-02-04/50027/50027-video.mp4 -frames:v 1 /var/cache/zoneminder/events/1/2020-02-04/50027/00019-capture.jpg

result: frame= 1 fps=1.0 q=7.1 Lsize=N/A time=00:00:00.10 bitrate=N/A speed=0.0962x

Is there a way to "inject" -hwaccel cuda parameter when ZM launches ffmpeg ?

[DALDON]

Yes it is.

Thank a lot for you answer. If I understand correct, zoneminder will be still intesive use of cpu in regardless h.264 acceleration enabled or not. Is it try?

[iconnor]

You can wait for 1.34.3, where the code will be updated to use the Options->Images->PATH_FFMPEG setting, at which point you could add -hwaccel cuda to that setting.

In the meantime, you can edit /usr/share/zoneminder/www/views/image.php line 203 to add whatever you want.

The place you SHOULD be seeing an improvement in cpu use is the cpu use of the zmc processes (assuming that they are using your recompiled libraries). In order to tell for sure, you will need to consult the output of nvidia-smi or turn on debug logging and view your zmc logs.

[DALDON]

You can wait for 1.34.3, where the code will be updated to use the Options->Images->PATH_FFMPEG setting, at which point you could add -hwaccel cuda to that setting.

In the meantime, you can edit /usr/share/zoneminder/www/views/image.php line 203 to add whatever you want.

The place you SHOULD be seeing an improvement in cpu use is the cpu use of the zmc processes (assuming that they are using your recompiled libraries). In order to tell for sure, you will need to consult the output of nvidia-smi or turn on debug logging and view your zmc logs.

Thank you!

[lbm]

Im curious, would you please come back with some numbers, with CPU utilization reduction, when done ?

[fontiano]

Im curious, would you please come back with some numbers, with CPU utilization reduction, when done ?

Thanks to iconnor now ffmpeg uses cuda when extracts jpgs from saved videos.
Unfortunately it seems that zmc processes don't use cuda acceleration:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.44 Driver Version: 440.44 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1650 On | 00000000:01:00.0 On | N/A |
| 56% 24C P8 8W / 75W | 80MiB / 3911MiB | 3% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 905 G /usr/lib/xorg/Xorg 46MiB |
| 0 1519 G /usr/bin/gnome-shell 31MiB |
+-----------------------------------------------------------------------------+

[fontiano]

After I installed ffmpeg libraries in the right way now zmc processes are using cuda acceleration but CPU utilization of "zma" process is increased for monitor number 7:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59 Driver Version: 440.59 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1650 On | 00000000:01:00.0 On | N/A |
| 56% 33C P0 20W / 75W | 1416MiB / 3911MiB | 8% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 914 G /usr/lib/xorg/Xorg 46MiB |
| 0 1520 G /usr/bin/gnome-shell 31MiB |
| 0 1595 C /usr/bin/zmc 249MiB |
| 0 1617 C /usr/bin/zmc 117MiB |
| 0 1627 C /usr/bin/zmc 99MiB |
| 0 1653 C /usr/bin/zmc 99MiB |
| 0 1679 C /usr/bin/zmc 237MiB |
| 0 1845 C ffmpeg 297MiB |
| 0 21664 C /usr/bin/zmc 225MiB |
+-----------------------------------------------------------------------------+


this is with cuda support:
root@videosorveglianza:~# ps auxww | grep "m 7"
www-data 21664 18.4 15.4 8778540 3791912 ? Sl 19:39 0:24 /usr/bin/zmc -m 7
www-data 21819 30.8 15.8 4099168 3905776 ? S 19:40 0:39 /usr/bin/zma -m 7

and this is without cuda support:
root@videosorveglianza:~# ps auxww | grep "m 7"
www-data 16109 22.0 14.9 3871704 3664784 ? S 19:36 0:12 /usr/bin/zmc -m 7
www-data 16615 22.0 15.7 4051288 3858068 ? S 19:36 0:10 /usr/bin/zma -m 7

root@videosorveglianza:~# ffmpeg
ffmpeg version 4.1.5 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 8 (Debian 8.3.0-6)

It's a Dahua Ipcamera running @ 2688x1520 ffmpeg passtrhough

CPU usage for other monitors is lower than before.

[lbm]

I've would have expected much better numbers... Like CPU usage, to drop to almost zero. But maybe im wrong, in the way it works, so this would not happen at all.

[iconnor]

Yeah the numbers are disappointing. I actually get around 7% reduction in cpu use for zmc without any noticeable increase in zma. Others have very different results.

I can't really think of any reason for zma to use more cpu.

[lbm]

I have no experience in GPU offload at all, but I would asume, that it would take almost all of the CPU usage. But I could be wrong, so even though its 7-10%, its still not that much IMHO.

If you have A LOT of cameras, it would of course give you something.

[iconnor]

The problem is that we have to copy the image to the gpu ram (and anything else that the underlaying layers do) and then get the gpu to do the decode, and then copy the raw image data back (which is fairly large) and then translate it to the format that we want RGBA generally... so we add extra processing just in order to do the decode on the gpu. So ymmv.

It will be more useful for encoding. Which may come soon.

[zmaximz]

After I installed ffmpeg libraries in the right way...

Hello, can you (or anyone) please to tell what is this right way? I'm playing with it by several days.
I have installed 1.34.6 with 19.10 Ubuntu, Installed nvidia driver, removed old ffmpeg and... where to get proper ffmpeg? for cuvid cuda support? Does I need to install nvidia-cuda-toolkit?
As I can understand vdpau hw acceleration is supported by my graphic card and is present in list of the default ffmpeg installation, but looks like no any impact if it's in the options of ZM

[fontiano]

Sorry for the delay.

I first tried to install all packages manually but I had a lot of troubles. So i found a fast and simple solution reading on some debian support forums but I did not remeber all the steps and unfortunately I haven't marked them on the clipboard.

I try to remember.

My GPU is GTX1650 and SO Debian 10

I modified preference file like this:

cat /etc/apt/preferences.d/preferences
#Package: *
#Pin: release a=stable
#Pin-Priority: 1001

Package: *
Pin: release a=unstable
Pin-Priority: 10

Added this repo on apt sources: "deb http://debian.fastweb.it/debian unstable main contrib non-free"

apt-get update
apt-get install nvidia-detect

init 6

then I installed nvidia-cuda-toolkit ( downloaded the binary installation file from nvidia website. the name of the file is like this "cuda_10.2.89_440.33.01_linux.run" and it's "huge" .. about 2.5 Gb

after this I lanched nvidia-smi to check if everything was ok:
this is first output's line:

NVIDIA-SMI 440.59 Driver Version: 440.59 CUDA Version: 10.2

After this I installed ffmpeg with cuda support with all libraries by adding this repo on sources:
deb https://www.deb-multimedia.org buster main non-free
apt-get update
apt-get install ffmpeg

Remember to remove "multimedia repo" after you installed ffmpeg because if you don't modify priorities on apt config files your zoneminder installation will be downgraded to 1.32.x at next "apt-get dist-upgrade"

Surely I forgot something, such as some dependencies to install. Before doing this backup all your data and/or configuration files.