Sorry this is a double post. I posted this in the 1.34 forums and was advised to post this in the mobile section instead.
I have been unable to get face training to work correctly on Ubuntu 20.04. I have tried creating the system twice now and still have the same error with training faces. If I enable face detection in objectconfig.ini I see the same errors everytime a hook is run. Everything else for detection works fine using the hooks. I have been unable to find a solution here or on google. Any help with narrowing down the cause of this would be greatly appreciated. Thank you,
I am running Cuda 11.1 with CuDNN-11.1-linux-x64-v8.0.4.30
OpenCV 4.4.0
Console Output
$ sudo -u www-data /var/lib/zmeventnotification/bin/zm_train_faces.py
CONSOLE:Face Recognition library load time took: 0.004 milliseconds
CONSOLE:secret filename: /etc/zm/secrets.ini
CONSOLE:Secret token found in config: !ZM_PORTAL
CONSOLE:Secret token found in config: !ZM_API_PORTAL
CONSOLE:Secret token found in config: !ZM_USER
CONSOLE:Secret token found in config: !ZM_PASSWORD
CONSOLE:Secret token found in config: !ML_USER
CONSOLE:Secret token found in config: !ML_PASSWORD
CONSOLE:Secret token found in config: !PLATEREC_ALPR_KEY
CONSOLE:allowing self-signed certs to work...
CONSOLE:Ignoring monitor specific settings, as you did not provide a monitor id
CONSOLE:key [config] is '/var/lib/zmeventnotification/models/yolov3/yolov3.cfg' after substitution
CONSOLE:key [weights] is '/var/lib/zmeventnotification/models/yolov3/yolov3.weig hts' after substitution
CONSOLE:key [labels] is '/var/lib/zmeventnotification/models/yolov3/coco.names' after substitution
CONSOLE:key [tiny_config] is '/var/lib/zmeventnotification/models/tinyyolo/yolov 3-tiny.cfg' after substitution
CONSOLE:key [tiny_weights] is '/var/lib/zmeventnotification/models/tinyyolo/yolo v3-tiny.weights' after substitution
CONSOLE:key [tiny_labels] is '/var/lib/zmeventnotification/models/tinyyolo/yolov 3-tiny.txt' after substitution
CONSOLE:key [known_images_path] is '/var/lib/zmeventnotification/known_faces' af ter substitution
CONSOLE:key [unknown_images_path] is '/var/lib/zmeventnotification/unknown_faces ' after substitution
CONSOLE:josh_k is a directory. Processing all images inside it
CONSOLE:loading face from josh_k/1.jpg
CONSOLE:Error initializing face recognition: Error while calling cudaOccupancyMa xPotentialBlockSize(&num_blocks,&num_threads,K) in file /tmp/pip-install-scqomot r/dlib/dlib/cuda/cuda_utils.h:186. code: 98, reason: invalid device function
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/zmes_hook_helpers/face_train.py", line 77, in train
face_encodings = face_recognition.face_encodings(
File "/usr/local/lib/python3.8/dist-packages/face_recognition/api.py", line 21 4, in face_encodings
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landma rk_set, num_jitters)) for raw_landmark_set in raw_landmarks]
File "/usr/local/lib/python3.8/dist-packages/face_recognition/api.py", line 21 4, in <listcomp>
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landma rk_set, num_jitters)) for raw_landmark_set in raw_landmarks]
RuntimeError: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks ,&num_threads,K) in file /tmp/pip-install-scqomotr/dlib/dlib/cuda/cuda_utils.h:1 86. code: 98, reason: invalid device function
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/zmeventnotification/bin/zm_train_faces.py", line 26, in <module >
train.train()
File "/usr/local/lib/python3.8/dist-packages/zmes_hook_helpers/face_train.py", line 109, in train
raise ValueError(
ValueError: Error opening known faces directory. Is the path correct?
nvidia-smi:
Mon Oct 12 14:11:35 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.05 Driver Version: 455.23.05 CUDA Version: 11.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GT 710 On | 00000000:03:00.0 N/A | N/A |
| 50% 46C P0 N/A / N/A | 93MiB / 2002MiB | N/A Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
$ ls -l /var/lib/zmeventnotification/known_faces
total 4
drwxrwxr-x 2 www-data www-data 4096 Oct 11 14:46 josh_k
$ ls -l /var/lib/zmeventnotification/known_faces/josh_k
total 36
-rw-rw-r-- 1 www-data www-data 35437 Oct 11 14:43 1.jpg
zmesdectect logs:
2020-10-12 14:17:18 zmesdetect_m2 102072 ERR Error initializing face recognition: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K) in file /tmp/pip-install-scqomotr/dlib/dlib/cuda/cuda_utils.h:164. code: 98, reason: invalid device function face_train.py 108
2020-10-12 14:17:17 zmesdetect_m2 102072 INF Importing local classes for Yolo/Face zm_detect.py 210
2020-10-12 14:17:17 zmesdetect_m2 102072 INF ---------| hook version: 5.15.6, ES version: 5.15 , OpenCV version: 4.4.0|------------ zm_detect.py 181
2020-10-12 14:16:54 zmesdetect_m2 101935 ERR Error initializing face recognition: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K) in file /tmp/pip-install-scqomotr/dlib/dlib/cuda/cuda_utils.h:164. code: 98, reason: invalid device function face_train.py 108
2020-10-12 14:16:52 zmesdetect_m2 101935 INF Importing local classes for Yolo/Face zm_detect.py 210
2020-10-12 14:16:52 zmesdetect_m2 101935 INF ---------| hook version: 5.15.6, ES version: 5.15 , OpenCV version: 4.4.0|------------ zm_detect.py 181
Thank you.
Face Training Issues
Re: Face Training Issues
Try disabling CUDA in the config and see if it works. If it does, then you have a software driver/version/Dlib mismatch.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.
Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Re: Face Training Issues
Thanks for your help.
How do I disable Cuda in the config? under Yolov4(and tried Yolov3) i have tried CPU and GPU both modes detection hooks run fine just not face_detect. I dont see any option for face recognition in the config. I checked the compatibility matrix and the drivers and cuda are listed as compatible.
I tried another system with Cuda 11.0, and CuDNN 8.0.4.30 for Cuda 11.0 and still the same error pops up. Running OpenCV 4.5.0-dev
How do I disable Cuda in the config? under Yolov4(and tried Yolov3) i have tried CPU and GPU both modes detection hooks run fine just not face_detect. I dont see any option for face recognition in the config. I checked the compatibility matrix and the drivers and cuda are listed as compatible.
I tried another system with Cuda 11.0, and CuDNN 8.0.4.30 for Cuda 11.0 and still the same error pops up. Running OpenCV 4.5.0-dev
Re: Face Training Issues
facedetection is dlib. If you compiled it from source, there is an option to turn it off. It is documented on their site. My apologies, you can't do this via config - that is only for opencv.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.
Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
Re: Face Training Issues
Great thanks for the dlib information. I was able to narrow the issue down at least. I will attempt to fix it at a later time when I have more time to devote to it - I will update this post if I find the solution for others in the future. I have tried CUDA 10.2 as well with the same results. Have yet to try a different nvidia driver than what comes with Ubuntu 20.04. I have found a lot of contradicting information on the cuda capabilities of the GT710 card...
Removed old installation:
Downloaded latest release from github:
https://github.com/davisking/dlib/releases
Installed DLIB with:
Verified no cuda installation:
Has to return FALSE.
and face detection works
Thanks again for your help. I was really struggling to figure out where the issue could lie on this one.
Removed old installation:
Code: Select all
sudo python3 uninstall dlib
https://github.com/davisking/dlib/releases
Installed DLIB with:
Code: Select all
sudo python3 setup.py install --no DLIB_USE_CUDA
Verified no cuda installation:
Code: Select all
python3
import dlib
dlib.DLIB_USE_CUDA
Code: Select all
sudo -u www-data /var/lib/zmeventnotification/bin/zm_train_faces.py
Thanks again for your help. I was really struggling to figure out where the issue could lie on this one.