Face Training Issues

[jkno]

Sorry this is a double post. I posted this in the 1.34 forums and was advised to post this in the mobile section instead.

I have been unable to get face training to work correctly on Ubuntu 20.04. I have tried creating the system twice now and still have the same error with training faces. If I enable face detection in objectconfig.ini I see the same errors everytime a hook is run. Everything else for detection works fine using the hooks. I have been unable to find a solution here or on google. Any help with narrowing down the cause of this would be greatly appreciated. Thank you,

I am running Cuda 11.1 with CuDNN-11.1-linux-x64-v8.0.4.30
OpenCV 4.4.0

Console Output

$ sudo -u www-data /var/lib/zmeventnotification/bin/zm_train_faces.py

CONSOLE:Face Recognition library load time took: 0.004 milliseconds
CONSOLE:secret filename: /etc/zm/secrets.ini
CONSOLE:Secret token found in config: !ZM_PORTAL
CONSOLE:Secret token found in config: !ZM_API_PORTAL
CONSOLE:Secret token found in config: !ZM_USER
CONSOLE:Secret token found in config: !ZM_PASSWORD
CONSOLE:Secret token found in config: !ML_USER
CONSOLE:Secret token found in config: !ML_PASSWORD
CONSOLE:Secret token found in config: !PLATEREC_ALPR_KEY
CONSOLE:allowing self-signed certs to work...
CONSOLE:Ignoring monitor specific settings, as you did not provide a monitor id
CONSOLE:key [config] is '/var/lib/zmeventnotification/models/yolov3/yolov3.cfg' after substitution
CONSOLE:key [weights] is '/var/lib/zmeventnotification/models/yolov3/yolov3.weig hts' after substitution
CONSOLE:key [labels] is '/var/lib/zmeventnotification/models/yolov3/coco.names' after substitution
CONSOLE:key [tiny_config] is '/var/lib/zmeventnotification/models/tinyyolo/yolov 3-tiny.cfg' after substitution
CONSOLE:key [tiny_weights] is '/var/lib/zmeventnotification/models/tinyyolo/yolo v3-tiny.weights' after substitution
CONSOLE:key [tiny_labels] is '/var/lib/zmeventnotification/models/tinyyolo/yolov 3-tiny.txt' after substitution
CONSOLE:key [known_images_path] is '/var/lib/zmeventnotification/known_faces' af ter substitution
CONSOLE:key [unknown_images_path] is '/var/lib/zmeventnotification/unknown_faces ' after substitution
CONSOLE:josh_k is a directory. Processing all images inside it
CONSOLE:loading face from josh_k/1.jpg
CONSOLE:Error initializing face recognition: Error while calling cudaOccupancyMa xPotentialBlockSize(&num_blocks,&num_threads,K) in file /tmp/pip-install-scqomot r/dlib/dlib/cuda/cuda_utils.h:186. code: 98, reason: invalid device function
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/zmes_hook_helpers/face_train.py", line 77, in train
face_encodings = face_recognition.face_encodings(
File "/usr/local/lib/python3.8/dist-packages/face_recognition/api.py", line 21 4, in face_encodings
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landma rk_set, num_jitters)) for raw_landmark_set in raw_landmarks]
File "/usr/local/lib/python3.8/dist-packages/face_recognition/api.py", line 21 4, in <listcomp>
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landma rk_set, num_jitters)) for raw_landmark_set in raw_landmarks]
RuntimeError: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks ,&num_threads,K) in file /tmp/pip-install-scqomotr/dlib/dlib/cuda/cuda_utils.h:1 86. code: 98, reason: invalid device function

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/lib/zmeventnotification/bin/zm_train_faces.py", line 26, in <module >
train.train()
File "/usr/local/lib/python3.8/dist-packages/zmes_hook_helpers/face_train.py", line 109, in train
raise ValueError(
ValueError: Error opening known faces directory. Is the path correct?


nvidia-smi:

Mon Oct 12 14:11:35 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.05 Driver Version: 455.23.05 CUDA Version: 11.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GT 710 On | 00000000:03:00.0 N/A | N/A |
| 50% 46C P0 N/A / N/A | 93MiB / 2002MiB | N/A Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

$ ls -l /var/lib/zmeventnotification/known_faces
total 4
drwxrwxr-x 2 www-data www-data 4096 Oct 11 14:46 josh_k

$ ls -l /var/lib/zmeventnotification/known_faces/josh_k
total 36
-rw-rw-r-- 1 www-data www-data 35437 Oct 11 14:43 1.jpg


zmesdectect logs:

2020-10-12 14:17:18 zmesdetect_m2 102072 ERR Error initializing face recognition: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K) in file /tmp/pip-install-scqomotr/dlib/dlib/cuda/cuda_utils.h:164. code: 98, reason: invalid device function face_train.py 108
2020-10-12 14:17:17 zmesdetect_m2 102072 INF Importing local classes for Yolo/Face zm_detect.py 210
2020-10-12 14:17:17 zmesdetect_m2 102072 INF ---------| hook version: 5.15.6, ES version: 5.15 , OpenCV version: 4.4.0|------------ zm_detect.py 181
2020-10-12 14:16:54 zmesdetect_m2 101935 ERR Error initializing face recognition: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K) in file /tmp/pip-install-scqomotr/dlib/dlib/cuda/cuda_utils.h:164. code: 98, reason: invalid device function face_train.py 108
2020-10-12 14:16:52 zmesdetect_m2 101935 INF Importing local classes for Yolo/Face zm_detect.py 210
2020-10-12 14:16:52 zmesdetect_m2 101935 INF ---------| hook version: 5.15.6, ES version: 5.15 , OpenCV version: 4.4.0|------------ zm_detect.py 181


Thank you.

[asker]

Try disabling CUDA in the config and see if it works. If it does, then you have a software driver/version/Dlib mismatch.

[jkno]

Thanks for your help.

How do I disable Cuda in the config? under Yolov4(and tried Yolov3) i have tried CPU and GPU both modes detection hooks run fine just not face_detect. I dont see any option for face recognition in the config. I checked the compatibility matrix and the drivers and cuda are listed as compatible.

I tried another system with Cuda 11.0, and CuDNN 8.0.4.30 for Cuda 11.0 and still the same error pops up. Running OpenCV 4.5.0-dev

[asker]

facedetection is dlib. If you compiled it from source, there is an option to turn it off. It is documented on their site. My apologies, you can't do this via config - that is only for opencv.

[jkno]

Great thanks for the dlib information. I was able to narrow the issue down at least. I will attempt to fix it at a later time when I have more time to devote to it - I will update this post if I find the solution for others in the future. I have tried CUDA 10.2 as well with the same results. Have yet to try a different nvidia driver than what comes with Ubuntu 20.04. I have found a lot of contradicting information on the cuda capabilities of the GT710 card...

Removed old installation:

Code: Select all

sudo python3 uninstall dlib

Downloaded latest release from github:
https://github.com/davisking/dlib/releases


Installed DLIB with:

Code: Select all

sudo python3 setup.py install --no DLIB_USE_CUDA

Verified no cuda installation:

Code: Select all

python3
import dlib
dlib.DLIB_USE_CUDA

Has to return FALSE.

Code: Select all

sudo -u www-data /var/lib/zmeventnotification/bin/zm_train_faces.py

and face detection works

Thanks again for your help. I was really struggling to figure out where the issue could lie on this one.