Zoneminder Event Server object detection GPU is running at 100%

Discussion topics related to mobile applications and ZoneMinder Event Server (including machine learning)
Post Reply
JasonBorn
Posts: 37
Joined: Fri Jan 08, 2021 5:31 pm

Zoneminder Event Server object detection GPU is running at 100%

Post by JasonBorn »

Maybe someone can help me here. Thought on reddit would be more readers.

Hi there,

i'm running zoneminder 1.36.32 + event server on a Dell R730XD on Proxmox in a LXC with a Tesla P40.

CPU is loaded with 120GB RAM and 25 Cores CPU.

I got 13 cams. All of them are integrated in zm with the main (nodect mode, 4k) and the substream (modect mode, 640x480...).

Unfortunately the database is growing from time to time. So i increased mysql to 700.

I think the main problem is that my 24GB Tesla P40 is running full. See the Host nvidia-smi.

I also got sereval log errors. Such this:


Error running model: Timeout waiting for pyzm_uid33_gpu_lock portalock for 100 seconds

--------------------------------------------------------------------------------------

Timeout waiting for pyzm_uid33_gpu_lock portalock for 100 seconds

--------------------------------------------------------------------------------------

Error loading same model variation for face:Error while calling cudaMallocHost(&data, new_size*sizeof(float))

in file /tmp/pip-install-bg4asj5v/dlib_ffc5739eb1204a7e9def39b29673b390/dlib/cuda/gpu_data.cpp:211. code: 2, reason: out of memory


Not all hooks are failing. So it is working. But not all...

Can you please help me with this problem? Would it be helpful when i create a additional LXC with mlapi?

Thank you


Host:

Code: Select all

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.12    Driver Version: 525.60.13    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P40           On   | 00000000:82:00.0 Off |                  Off |
| N/A   87C    P0    67W / 250W |  21224MiB / 24576MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   1464885      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1465149      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1465544      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1465775      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1466093      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1466095      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1466355      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1466356      C   /usr/bin/python3                  192MiB |
|    0   N/A  N/A   1466972      C   /usr/bin/python3                  192MiB |
....
LXC:

Code: Select all

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.13    Driver Version: 525.60.13    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P40           On   | 00000000:82:00.0 Off |                  Off |
| N/A   88C    P0    74W / 250W |  21362MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
Post Reply