Face training seg fault, OpenCV works

[calmor15014]

Hoping someone else might have had this problem and found a way around it.

• ZM1.34.15
• Ubuntu 18.04 in Virtual Machine on Proxmox, upgraded from 16.04
• MySQL server on remote machine
• OpenCV 4.2.0 built from source and seems to run normally
• 8 monitors - 4 Hikvision cameras, MJPEG stream and h.264 stream (currently not using h.264 monitors)
• Hardware is a 24-core AMD server (six cores exposed to ZM VM)
  • There is no GPU acceleration on the hardware or VM and CUDA was intentionally left out of the build of OpenCV

ZMEventNotification installed and object detection functioning normally. Events being sent to ZMNinja as expected, car and person detection works.

I wanted to enable face detection, so I added some face images to the appropriate folder and ran the recommended command:

Code: Select all

sudo -u www-data /var/lib/zmeventnotification/bin/zm_train_faces.py

It reads the config file then tries to load a picture and fails with a segmentation fault. No other info.

I dug into that python code and into face_train.py and found that it was failing on execution of the face_detection() step. Same thing - segfault, no other info.

I uninstalled face_detection, uninstalled dlib, ensured the blas was detected, reinstalled face_detection. I tried running zm_train_faces as root and even just running face_detection from the python interpreter and then from the command line directly. I uninstalled face_detection and installed the previous version. I limited CPUs to 1. It segfaults every time. I tried different training images. cv2.imread is successful, but face_detection() will not complete regardless of what I try.

If I just allow zmeventnotification to try to detect faces, the logs don't give any additional information, just that the model hadn't been trained and it is going to try, then does not succeed (as expected since the command line execution doesn't work either) and quits processing.

Any thoughts?

[asker]

1. If you have faces.dat in known_faces, delete it
2. Turn on debug logs
3. Post them

[calmor15014]

Didn't expect to hear from the man immediately... thanks!

1. If you have faces.dat in known_faces, delete it

It never generated one, so nothing to delete. Folder and files all owned by www-data, but even root wouldn't generate it.

2. Turn on debug logs

Debug logs are on per the documentation recommended settings

3. Post them

Code: Select all

05/25/20 11:52:46 zmesdetect_m5[5713] INF zm_detect.py:177 [---------| hook version: 5.13.3, ES version: 5.13 , OpenCV version: 4.2.0|------------]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 utils.py:265 [secret filename: /etc/zm/secrets.ini]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:238 [Secret token found in config: !ZM_PORTAL]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:238 [Secret token found in config: !ZM_API_PORTAL]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:238 [Secret token found in config: !ZM_USER]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:238 [Secret token found in config: !ZM_PASSWORD]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:238 [Secret token found in config: !PLATEREC_ALPR_KEY]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 utils.py:288 [allowing self-signed certs to work...]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [config] is '/var/lib/zmeventnotification/models/yolov3/yolov3.cfg' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [weights] is '/var/lib/zmeventnotification/models/yolov3/yolov3.weights' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [labels] is '/var/lib/zmeventnotification/models/yolov3/yolov3_classes.txt' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [tiny_config] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.cfg' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [tiny_weights] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.weights' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [tiny_labels] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.txt' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [known_images_path] is '/var/lib/zmeventnotification/known_faces' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:343 [key [unknown_images_path] is '/var/lib/zmeventnotification/unknown_faces' after substitution]
05/25/20 11:52:46 zmesdetect_m5[5713] INF zm_detect.py:206 [Importing local classes for Yolo/Face]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 utils.py:143 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405627&fid=alarm&username=username&password=*****]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 utils.py:161 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405627&fid=snapshot&username=username&password=*****]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 zm_detect.py:263 [No polygon area specfied, so adding a full image polygon:[{'name': 'full_image', 'value': [(0, 0), (640, 0), (640, 480), (0, 480)]}]]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 zm_detect.py:266 [resizing to 1200 before analysis...]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 utils.py:37 [resized polygons x=1.0/y=1.0: [{'name': 'full_image', 'value': [(0, 0), (640, 0), (640, 480), (0, 480)]}]]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 zm_detect.py:287 [User ALPR if vehicle found: False]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 zm_detect.py:369 [Using model: yolo with /var/lib/zmeventnotification/images/405627-alarm.jpg]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 yolo.py:48 [|---------- YOLO (input image: 640w*480h, resized to: 416w*416h) ----------|]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-1 yolo.py:59 [Initializing Yolo]
05/25/20 11:52:46 zmesdetect_m5[5713] DBG-2 yolo.py:61 [config:/var/lib/zmeventnotification/models/yolov3/yolov3.cfg, weights:/var/lib/zmeventnotification/models/yolov3/yolov3.weights]
05/25/20 11:52:49 zmesdetect_m5[5713] DBG-1 yolo.py:83 [Not using CUDA backend]
05/25/20 11:52:49 zmesdetect_m5[5713] DBG-1 yolo.py:88 [YOLO initialization (loading model from disk) took: 183.559 milliseconds]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-1 yolo.py:103 [YOLO detection took: 11.73 milliseconds]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-1 yolo.py:136 [YOLO NMS filtering took: 4.489 milliseconds]
05/25/20 11:52:52 zmesdetect_m5[5713] INF yolo.py:162 [object:person at [7, 296, 263, 480] has a acceptable confidence:0.9700866341590881 compared to min confidence of: 0.3, adding]
05/25/20 11:52:52 zmesdetect_m5[5713] INF yolo.py:162 [object:laptop at [201, 382, 375, 474] has a acceptable confidence:0.8613473176956177 compared to min confidence of: 0.3, adding]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-2 image_manip.py:269 [intersection: polygon in process=[(7, 296), (263, 296), (263, 480), (7, 480)]]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-2 image_manip.py:276 [full_image intersects object:person[[(7, 296), (263, 296), (263, 480), (7, 480)]]]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-2 image_manip.py:269 [intersection: polygon in process=[(201, 382), (375, 382), (375, 474), (201, 474)]]
05/25/20 11:52:52 zmesdetect_m5[5713] INF image_manip.py:283 [discarding "laptop" as it does not match your filters]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-1 image_manip.py:287 [full_image intersects object:laptop[[(201, 382), (375, 382), (375, 474), (201, 474)]] but does NOT match your detect_pattern filter of (person|car|motorbike|truck|dog|cat|bird)]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-2 zm_detect.py:598 [ALPR not in use, no need for look aheads in processing]
05/25/20 11:52:52 zmesdetect_m5[5713] INF zm_detect.py:615 [labels found: ['person']]
05/25/20 11:52:52 zmesdetect_m5[5713] DBG-2 zm_detect.py:618 [match found in /var/lib/zmeventnotification/images/405627-alarm.jpg, breaking file loop...]
05/25/20 11:52:57 zmesdetect_m5[5713] DBG-1 face_train.py:20 [Face Recognition library load time took: 0.009 milliseconds]
05/25/20 11:52:57 zmesdetect_m5[5713] DBG-1 face.py:24 [Initializing face recognition with model:cnn upsample:1, jitters:1]
05/25/20 11:52:57 zmesdetect_m5[5713] DBG-1 face.py:53 [trained file not found, reading from images and doing training...]
05/25/20 11:52:57 zmesdetect_m5[5713] DBG-1 face.py:56 [If you are using a GPU and run out of memory, do the training using zm_train_faces.py. In this case, other models like yolo may already take up a lot of GPU memory]
05/25/20 11:52:57 zmesdetect_m5[5713] DBG-1 face_train.py:53 [person_name is a directory. Processing all images inside it]
05/25/20 11:52:57 zmesdetect_m5[5713] DBG-1 face_train.py:58 [loading face from  person_name/3.jpeg]
05/25/20 11:53:01 zmesdetect_m5[5789] INF zm_detect.py:177 [---------| hook version: 5.13.3, ES version: 5.13 , OpenCV version: 4.2.0|------------]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 utils.py:265 [secret filename: /etc/zm/secrets.ini]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:238 [Secret token found in config: !ZM_PORTAL]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:238 [Secret token found in config: !ZM_API_PORTAL]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:238 [Secret token found in config: !ZM_USER]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:238 [Secret token found in config: !ZM_PASSWORD]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:238 [Secret token found in config: !PLATEREC_ALPR_KEY]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 utils.py:288 [allowing self-signed certs to work...]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [config] is '/var/lib/zmeventnotification/models/yolov3/yolov3.cfg' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [weights] is '/var/lib/zmeventnotification/models/yolov3/yolov3.weights' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [labels] is '/var/lib/zmeventnotification/models/yolov3/yolov3_classes.txt' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [tiny_config] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.cfg' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [tiny_weights] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.weights' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [tiny_labels] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.txt' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [known_images_path] is '/var/lib/zmeventnotification/known_faces' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:343 [key [unknown_images_path] is '/var/lib/zmeventnotification/unknown_faces' after substitution]
05/25/20 11:53:01 zmesdetect_m5[5789] INF zm_detect.py:206 [Importing local classes for Yolo/Face]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 utils.py:143 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405628&fid=alarm&username=username&password=*****]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 utils.py:161 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405628&fid=snapshot&username=username&password=*****]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 zm_detect.py:263 [No polygon area specfied, so adding a full image polygon:[{'name': 'full_image', 'value': [(0, 0), (640, 0), (640, 480), (0, 480)]}]]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 zm_detect.py:266 [resizing to 1200 before analysis...]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 utils.py:37 [resized polygons x=1.0/y=1.0: [{'name': 'full_image', 'value': [(0, 0), (640, 0), (640, 480), (0, 480)]}]]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 zm_detect.py:287 [User ALPR if vehicle found: False]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 zm_detect.py:369 [Using model: yolo with /var/lib/zmeventnotification/images/405628-alarm.jpg]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 yolo.py:48 [|---------- YOLO (input image: 640w*480h, resized to: 416w*416h) ----------|]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-1 yolo.py:59 [Initializing Yolo]
05/25/20 11:53:01 zmesdetect_m5[5789] DBG-2 yolo.py:61 [config:/var/lib/zmeventnotification/models/yolov3/yolov3.cfg, weights:/var/lib/zmeventnotification/models/yolov3/yolov3.weights]
05/25/20 11:53:03 zmesdetect_m5[5789] DBG-1 yolo.py:83 [Not using CUDA backend]
05/25/20 11:53:03 zmesdetect_m5[5789] DBG-1 yolo.py:88 [YOLO initialization (loading model from disk) took: 687.538 milliseconds]
05/25/20 11:53:06 zmesdetect_m5[5789] DBG-1 yolo.py:103 [YOLO detection took: 992.804 milliseconds]
05/25/20 11:53:07 zmesdetect_m5[5789] DBG-1 yolo.py:136 [YOLO NMS filtering took: 2.414 milliseconds]
05/25/20 11:53:07 zmesdetect_m5[5789] INF yolo.py:162 [object:person at [388, 80, 440, 242] has a acceptable confidence:0.38470223546028137 compared to min confidence of: 0.3, adding]
05/25/20 11:53:07 zmesdetect_m5[5789] DBG-2 image_manip.py:269 [intersection: polygon in process=[(388, 80), (440, 80), (440, 242), (388, 242)]]
05/25/20 11:53:07 zmesdetect_m5[5789] DBG-2 image_manip.py:276 [full_image intersects object:person[[(388, 80), (440, 80), (440, 242), (388, 242)]]]
05/25/20 11:53:07 zmesdetect_m5[5789] DBG-2 zm_detect.py:598 [ALPR not in use, no need for look aheads in processing]
05/25/20 11:53:07 zmesdetect_m5[5789] INF zm_detect.py:615 [labels found: ['person']]
05/25/20 11:53:07 zmesdetect_m5[5789] DBG-2 zm_detect.py:618 [match found in /var/lib/zmeventnotification/images/405628-alarm.jpg, breaking file loop...]
05/25/20 11:53:11 zmesdetect_m5[5789] DBG-1 face_train.py:20 [Face Recognition library load time took: 0.009 milliseconds]
05/25/20 11:53:11 zmesdetect_m5[5789] DBG-1 face.py:24 [Initializing face recognition with model:cnn upsample:1, jitters:1]
05/25/20 11:53:11 zmesdetect_m5[5789] DBG-1 face.py:53 [trained file not found, reading from images and doing training...]
05/25/20 11:53:11 zmesdetect_m5[5789] DBG-1 face.py:56 [If you are using a GPU and run out of memory, do the training using zm_train_faces.py. In this case, other models like yolo may already take up a lot of GPU memory]
05/25/20 11:53:11 zmesdetect_m5[5789] DBG-1 face_train.py:53 [person_name is a directory. Processing all images inside it]
05/25/20 11:53:11 zmesdetect_m5[5789] DBG-1 face_train.py:58 [loading face from  person_name/3.jpeg]
05/25/20 12:46:34 zmesdetect_m5[10840] INF zm_detect.py:177 [---------| hook version: 5.13.3, ES version: 5.13 , OpenCV version: 4.2.0|------------]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:265 [secret filename: /etc/zm/secrets.ini]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:238 [Secret token found in config: !ZM_PORTAL]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:238 [Secret token found in config: !ZM_API_PORTAL]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:238 [Secret token found in config: !ZM_USER]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:238 [Secret token found in config: !ZM_PASSWORD]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:238 [Secret token found in config: !PLATEREC_ALPR_KEY]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:288 [allowing self-signed certs to work...]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:307 [[monitor-5] overrides key:import_zm_zones with value:yes]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:307 [[monitor-5] overrides key:detect_pattern with value:(person)]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:307 [[monitor-5] overrides key:models with value:yolo]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:307 [[monitor-5] overrides key:frame_id with value:bestmatch]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:307 [[monitor-5] overrides key:match_past_detections with value:no]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:60 [Getting ZM zones using https://***hostname***/zm/api/zones/forMonitor/5.json?user=xxx&pass=yyy]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:94 [importing zoneminder polygon: DoorPad [1,275 82,232 250,223 544,291 639,479 0,479]]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:94 [importing zoneminder polygon: EntryLight [6,241 19,230 28,255 14,264]]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:94 [importing zoneminder polygon: Window1 [592,33 609,33 595,177 558,312 547,282]]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:94 [importing zoneminder polygon: GarageSideEntry [406,242 402,54 531,52 526,150 504,282 391,256]]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:94 [importing zoneminder polygon: DrivewayEntry [140,111 293,80 310,235 246,222 84,230 118,191 182,170]]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:94 [importing zoneminder polygon: Window2 [45,116 138,111 180,169 115,190 83,231]]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [config] is '/var/lib/zmeventnotification/models/yolov3/yolov3.cfg' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [weights] is '/var/lib/zmeventnotification/models/yolov3/yolov3.weights' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [labels] is '/var/lib/zmeventnotification/models/yolov3/yolov3_classes.txt' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [tiny_config] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.cfg' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [tiny_weights] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.weights' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [tiny_labels] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.txt' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [known_images_path] is '/var/lib/zmeventnotification/known_faces' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:343 [key [unknown_images_path] is '/var/lib/zmeventnotification/unknown_faces' after substitution]
05/25/20 12:46:34 zmesdetect_m5[10840] INF zm_detect.py:206 [Importing local classes for Yolo/Face]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:143 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405631&fid=alarm&username=username&password=*****]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 utils.py:161 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405631&fid=snapshot&username=username&password=*****]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 zm_detect.py:266 [resizing to 1200 before analysis...]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 utils.py:37 [resized polygons x=1.0/y=1.0: [{'name': 'DoorPad', 'value': [(1, 275), (82, 232), (250, 223), (544, 291), (639, 479), (0, 479)]}, {'name': 'EntryLight', 'value': [(6, 241), (19, 230), (28, 255), (14, 264)]}, {'name': 'Window1', 'value': [(592, 33), (609, 33), (595, 177), (558, 312), (547, 282)]}, {'name': 'GarageSideEntry', 'value': [(406, 242), (402, 54), (531, 52), (526, 150), (504, 282), (391, 256)]}, {'name': 'DrivewayEntry', 'value': [(140, 111), (293, 80), (310, 235), (246, 222), (84, 230), (118, 191), (182, 170)]}, {'name': 'Window2', 'value': [(45, 116), (138, 111), (180, 169), (115, 190), (83, 231)]}]]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 zm_detect.py:287 [User ALPR if vehicle found: False]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 zm_detect.py:369 [Using model: yolo with /var/lib/zmeventnotification/images/405631-alarm.jpg]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 yolo.py:48 [|---------- YOLO (input image: 640w*480h, resized to: 416w*416h) ----------|]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 yolo.py:59 [Initializing Yolo]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-2 yolo.py:61 [config:/var/lib/zmeventnotification/models/yolov3/yolov3.cfg, weights:/var/lib/zmeventnotification/models/yolov3/yolov3.weights]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 yolo.py:83 [Not using CUDA backend]
05/25/20 12:46:34 zmesdetect_m5[10840] DBG-1 yolo.py:88 [YOLO initialization (loading model from disk) took: 434.54 milliseconds]
05/25/20 12:46:38 zmesdetect_m5[10840] DBG-1 yolo.py:103 [YOLO detection took: 186.499 milliseconds]
05/25/20 12:46:38 zmesdetect_m5[10840] DBG-1 yolo.py:136 [YOLO NMS filtering took: 2.549 milliseconds]
05/25/20 12:46:38 zmesdetect_m5[10840] INF yolo.py:162 [object:person at [10, 278, 268, 478] has a acceptable confidence:0.9766791462898254 compared to min confidence of: 0.3, adding]
05/25/20 12:46:38 zmesdetect_m5[10840] INF yolo.py:162 [object:laptop at [199, 385, 373, 475] has a acceptable confidence:0.9535558819770813 compared to min confidence of: 0.3, adding]
05/25/20 12:46:38 zmesdetect_m5[10840] INF yolo.py:171 [rejecting object:chair at [304, 392, 528, 476] because its confidence is :0.23611921072006226 compared to min confidence of: 0.3]
05/25/20 12:46:38 zmesdetect_m5[10840] DBG-2 image_manip.py:269 [intersection: polygon in process=[(10, 278), (268, 278), (268, 478), (10, 478)]]
05/25/20 12:46:38 zmesdetect_m5[10840] DBG-2 image_manip.py:276 [DoorPad intersects object:person[[(10, 278), (268, 278), (268, 478), (10, 478)]]]
05/25/20 12:46:38 zmesdetect_m5[10840] DBG-2 image_manip.py:269 [intersection: polygon in process=[(199, 385), (373, 385), (373, 475), (199, 475)]]
05/25/20 12:46:38 zmesdetect_m5[10840] INF image_manip.py:283 [discarding "laptop" as it does not match your filters]
05/25/20 12:46:38 zmesdetect_m5[10840] DBG-1 image_manip.py:287 [DoorPad intersects object:laptop[[(199, 385), (373, 385), (373, 475), (199, 475)]] but does NOT match your detect_pattern filter of (person)]
05/25/20 12:46:38 zmesdetect_m5[10840] DBG-2 zm_detect.py:598 [ALPR not in use, no need for look aheads in processing]
05/25/20 12:46:39 zmesdetect_m5[10840] INF zm_detect.py:615 [labels found: ['person']]
05/25/20 12:46:39 zmesdetect_m5[10840] DBG-2 zm_detect.py:618 [match found in /var/lib/zmeventnotification/images/405631-alarm.jpg, breaking file loop...]
05/25/20 12:46:39 zmesdetect_m5[10840] DBG-1 zm_detect.py:700 [Writing detected image to /var/cache/zoneminder/events/5/2020-05-25/405631/objdetect.jpg]
05/25/20 12:46:39 zmesdetect_m5[10840] DBG-1 zm_detect.py:704 [Writing JSON output to /var/cache/zoneminder/events/5/2020-05-25/405631/objects.json]
05/25/20 12:46:39 zmesdetect_m5[10840] INF zm_detect.py:757 [Prediction string:[a] detected:person:98% ]
05/25/20 12:46:39 zmesdetect_m5[10840] DBG-1 zm_detect.py:760 [Prediction string JSON:[{"type": "object", "label": "person", "box": [10, 278, 268, 478], "confidence": "97.67%"}]]
05/25/20 14:37:17 zmesdetect_m5[23318] INF zm_detect.py:177 [---------| hook version: 5.13.3, ES version: 5.13 , OpenCV version: 4.2.0|------------]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:265 [secret filename: /etc/zm/secrets.ini]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:238 [Secret token found in config: !ZM_PORTAL]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:238 [Secret token found in config: !ZM_API_PORTAL]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:238 [Secret token found in config: !ZM_USER]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:238 [Secret token found in config: !ZM_PASSWORD]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:238 [Secret token found in config: !PLATEREC_ALPR_KEY]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:288 [allowing self-signed certs to work...]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:307 [[monitor-5] overrides key:import_zm_zones with value:yes]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:307 [[monitor-5] overrides key:detect_pattern with value:(person)]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:307 [[monitor-5] overrides key:models with value:yolo]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:307 [[monitor-5] overrides key:frame_id with value:bestmatch]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:307 [[monitor-5] overrides key:match_past_detections with value:no]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:60 [Getting ZM zones using https://***hostname***/zm/api/zones/forMonitor/5.json?user=xxx&pass=yyy]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:94 [importing zoneminder polygon: DoorPad [1,275 82,232 250,223 544,291 639,479 0,479]]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:94 [importing zoneminder polygon: EntryLight [6,241 19,230 28,255 14,264]]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:94 [importing zoneminder polygon: Window1 [592,33 609,33 595,177 558,312 547,282]]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:94 [importing zoneminder polygon: GarageSideEntry [406,242 402,54 531,52 526,150 504,282 391,256]]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:94 [importing zoneminder polygon: DrivewayEntry [140,111 293,80 310,235 246,222 84,230 118,191 182,170]]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:94 [importing zoneminder polygon: Window2 [45,116 138,111 180,169 115,190 83,231]]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [config] is '/var/lib/zmeventnotification/models/yolov3/yolov3.cfg' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [weights] is '/var/lib/zmeventnotification/models/yolov3/yolov3.weights' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [labels] is '/var/lib/zmeventnotification/models/yolov3/yolov3_classes.txt' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [tiny_config] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.cfg' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [tiny_weights] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.weights' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [tiny_labels] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.txt' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [known_images_path] is '/var/lib/zmeventnotification/known_faces' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:343 [key [unknown_images_path] is '/var/lib/zmeventnotification/unknown_faces' after substitution]
05/25/20 14:37:17 zmesdetect_m5[23318] INF zm_detect.py:206 [Importing local classes for Yolo/Face]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:143 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405637&fid=alarm&username=username&password=*****]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 utils.py:161 [Trying to download https://***hostname***/zm/index.php?view=image&eid=405637&fid=snapshot&username=username&password=*****]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 zm_detect.py:266 [resizing to 1200 before analysis...]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 utils.py:37 [resized polygons x=1.0/y=1.0: [{'name': 'DoorPad', 'value': [(1, 275), (82, 232), (250, 223), (544, 291), (639, 479), (0, 479)]}, {'name': 'EntryLight', 'value': [(6, 241), (19, 230), (28, 255), (14, 264)]}, {'name': 'Window1', 'value': [(592, 33), (609, 33), (595, 177), (558, 312), (547, 282)]}, {'name': 'GarageSideEntry', 'value': [(406, 242), (402, 54), (531, 52), (526, 150), (504, 282), (391, 256)]}, {'name': 'DrivewayEntry', 'value': [(140, 111), (293, 80), (310, 235), (246, 222), (84, 230), (118, 191), (182, 170)]}, {'name': 'Window2', 'value': [(45, 116), (138, 111), (180, 169), (115, 190), (83, 231)]}]]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 zm_detect.py:287 [User ALPR if vehicle found: False]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 zm_detect.py:369 [Using model: yolo with /var/lib/zmeventnotification/images/405637-alarm.jpg]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 yolo.py:48 [|---------- YOLO (input image: 640w*480h, resized to: 416w*416h) ----------|]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 yolo.py:59 [Initializing Yolo]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-2 yolo.py:61 [config:/var/lib/zmeventnotification/models/yolov3/yolov3.cfg, weights:/var/lib/zmeventnotification/models/yolov3/yolov3.weights]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 yolo.py:83 [Not using CUDA backend]
05/25/20 14:37:17 zmesdetect_m5[23318] DBG-1 yolo.py:88 [YOLO initialization (loading model from disk) took: 369.957 milliseconds]
05/25/20 14:37:21 zmesdetect_m5[23318] DBG-1 yolo.py:103 [YOLO detection took: 81.65 milliseconds]
05/25/20 14:37:21 zmesdetect_m5[23318] DBG-1 yolo.py:136 [YOLO NMS filtering took: 2.503 milliseconds]
05/25/20 14:37:21 zmesdetect_m5[23318] DBG-2 zm_detect.py:598 [ALPR not in use, no need for look aheads in processing]
05/25/20 14:37:21 zmesdetect_m5[23318] DBG-1 zm_detect.py:627 [No match found in /var/lib/zmeventnotification/images/405637-alarm.jpg using model:yolo]
05/25/20 14:37:21 zmesdetect_m5[23318] DBG-1 zm_detect.py:369 [Using model: yolo with /var/lib/zmeventnotification/images/405637-snapshot.jpg]
05/25/20 14:37:21 zmesdetect_m5[23318] DBG-1 yolo.py:48 [|---------- YOLO (input image: 640w*480h, resized to: 416w*416h) ----------|]
05/25/20 14:37:23 zmesdetect_m5[23318] DBG-1 yolo.py:103 [YOLO detection took: 246.303 milliseconds]
05/25/20 14:37:24 zmesdetect_m5[23318] DBG-1 yolo.py:136 [YOLO NMS filtering took: 2.901 milliseconds]
05/25/20 14:37:24 zmesdetect_m5[23318] DBG-2 zm_detect.py:598 [ALPR not in use, no need for look aheads in processing]
05/25/20 14:37:24 zmesdetect_m5[23318] DBG-1 zm_detect.py:627 [No match found in /var/lib/zmeventnotification/images/405637-snapshot.jpg using model:yolo]
05/25/20 14:37:24 zmesdetect_m5[23318] INF zm_detect.py:640 [No patterns found using any models in all files]

I shut off face detection after the first two failures grabbed in this log, so that at least the object detection would work until I got this sorted out, which is why additional logs didn't show the same failures. It just fails silently.

The only thing zmeventnotification showed in the frontend log was:

Code: Select all

|----> FORK:FrontDoor (5), eid:405627 Not sending event end alarm, as we did not send a start alarm for this, or start hook processing failed

I tried running the examples for the face_recognition project in a Python3 shell. I'm able to do the facial features recognition example successfully on one of my training images, and successfully exported a photo with lines around the features. However, the recognition example segfaults with no error message at all at face_recognition.face_encodings() regardless of the photo I use.

When I took your face_train.py and executed it line-by-line in the shell, it segfaulted on the face_locations() command.

I'm thinking I might have to build dlib from scratch?

[asker]

Hmm possible.

1. You posted debug logs for an event detection. Can you post full debug logs for zm_train_faces?
2. Also, can you keep a tab of memory when you do it and see if you are exhausting memory?
3. While not your environment, I found this thread on GH https://github.com/ageitgey/face_recognition/issues/294

[calmor15014]

Thanks again for the quick response, and of course for all of your work on this and ZMNinja. I bought that for my Android and my work iPhone, as well as my wife's iPhone.

Ultimately this isn't the most important thing ever, but was something I wanted to get up and running over the holiday weekend. The machine was on 16.04 and running a pretty old ZM so lots of updating. I'm happy that it can see objects, sends notifications that are useful (not just another bird or bee flying too close to the camera), and hey, it even thinks I'm 99.92% person. (yes I know that's not what it means but it's amusing anyway)

Unfortunately, I did see that GH issue and tried setting those environment variables, as well as running the command line executable with the cpu limit flag, restricting it to one CPU.

The only thing in zm_train_faces.log is a bunch of INF logs at utils.py 322 ignoring monitor specific settings as I did not provide a monitor (which should be normal for training), and an error I'd made in the objectconfig.ini file duplicating the face_model option in one of the monitors at one point. Doesn't seem like there are any DBG logs so I'll check my log settings and try it again.

Right now the hypervisor and top both report that the machine has over 4GB free memory so I'm hoping that's not it, but I'll run some more testing this week and see what I can tease out of it. top reports 660MiB actually used and 7.5GB buffer/cache of 12GB.

Because of all the upgrading, it might be worth just installing all from scratch. Is it worth moving to Ubuntu 20.04 LTS, or do you have a preferred distro/installation method?

[asker]

How big are the images? What happens if you reduce the size for training?

[calmor15014]

It tries 3.jpeg first for some reason. It's 378x550, 26KiB.

When I delete 3, it tries 1.jpeg first. It's 153x214, 4.8KiB.

Segfaults either way. I'd also tried hog first then went to cnn, both segfaulted. running help(cv2) in the Python shell lists dnn as a module.

[calmor15014]

Also interesting, if I run the command line program face_detection on 1.jpeg, it returns:

1.jpeg,77,112,181,8

If I run the program face_recognition, which expects a folder of known faces and then a file, and run both against 1.jpeg, it segfaults too regardless of where 1.jpeg lives.

[calmor15014]

Took 1.jpeg down to 107x149, 3.0KiB.

running face_detection 6.jpeg returns:

6.jpeg,45,93,131,6

Still segfaults on both zm_train_faces.py and directly running sudo -u www-data face_recognition --cpus 1 . 6.jpeg

I'm not so familiar with this level of debugging but ran strace and see a few things to dig into... near the end starts popping up with:

stat("/usr/lib/python3/dist-packages/click/__init__.so" No such file or directory. Then it waits for a bit and crashes. But, click reports as installed. I'll look into that later.. have a thing to go to. Thanks again for kicking it around!

[asker]

In that case try removing the dlib install and compiling from source.

[calmor15014]

So, definitely going WAY down the rabbit hole here... but before I scrap the whole thing and build a new one from scratch, I'd like to know why it's choking. Much of this is probably useless for most but it might help someone in the future... who knows.

installed gdb and launched /usr/local/lib/python3.6/dist-packages/face_recognition/face_recognition_cli.py with the necessary arguments to check the known_faces folder. Wanted to get as close to the issue as i could to start out. Got the following:

Code: Select all

Thread 6 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe58cb700 (LWP 1106)]
0x00007ffff1738080 in () from /usr/lib/x86_64-linux-gnu/libopenblas.so.0

Looks like OpenBLAS *might* be the offending party here. It's detecting the processor correctly, though. The host and guest VM both present with the 'pni' CPU flag, which is the SSE3 flag, and the host processors are in fact Opteron 6172 chips. The Ubuntu 18.04 packaged version comes with that as one of the several kernel options compiled in.

For reference, I'm running a mostly up-to-date Proxmox as the hypervisor, which uses KVM for virtual machines.

Since you mention in the instructions that OpenBLAS is not mandatory, I might this week remove that and try to see if it will at least run, even if it's a bit slow, and work up from there. If it runs without OpenBLAS, I may try to recompile it from source.

[asker]

Since you mention in the instructions that OpenBLAS is not mandatory, I might this week remove that and try to see if it will at least run, even if it's a bit slow, and work up from there. If it runs without OpenBLAS, I may try to recompile it from source.

Ok. Do note that a segfault at a library may not always imply the problem is with that library. It's where the program _eventually_ crashes. So even if it doesn't work without openBLAS, it may be a good idea to purge the pre-compiled libs and do it from scratch.

[calmor15014]

Ok. Do note that a segfault at a library may not always imply the problem is with that library. It's where the program _eventually_ crashes. So even if it doesn't work without openBLAS, it may be a good idea to purge the pre-compiled libs and do it from scratch.

Thanks... as mentioned, diagnosing segfaults and digging into kernel and CPU architecture specifics are a lot further than I tend to venture normally so all new territory. I feared that might be the case, but I was hopeful because I saw a number of issues on the OpenBLAS GH about segfaults as well, especially people segfaulting on just including the OpenBLAS variant of numpy. Considering it has instruction set configurations for even subclasses of CPUs, I'm sure there's lots of room in there for bugs.

Took a break from work, removed face-recognition, face-recognition-models, and dlib via pip3, uninstalled the OpenBLAS files in apt-get, and re-installed dlib and then face-recognition with --no-cache-dir.

Went for broke and ran zm_train_faces.py instead of trying more direct routes. Success! There is now a faces.dat file. 5 files, 523ms processing time.

So... it would appear that something with OpenBLAS isn't happy with my architecture/setup... which means it's likely that even a clean install wasn't going to fix anything unless I changed something else.

After work, I'll re-enable face detection in objectconfig.ini, give it a run to see if it will process in the event processor. If so, next stop will be rebuilding OpenBLAS from source and trying again.

Thank you again for taking your time out to work this through!

[asker]

Sounds good. Glad you got things up and running!

[calmor15014]

Enabled the face detection in objectconfig.ini again and it picked me up first time, 100% confidence even with sub-optimal camera angle and pretty crappy Facebook screen captures as training images. I'm impressed!

For posterity's sake in case someone else runs up against this some day...:

Problem Summary
Installed zmeventnotification per instructions, built OpenCV from scratch. Object recognition worked but face detection crashed. /var/lib/zmeventnotification/bin/zm_train_faces.py would crash with "Segmentation Fault (core dumped)" and no other information.

ZM 1.34.15, Ubuntu 18.04 in KVM-QEMU on Proxmox 6.1-8 (Debian core). Opteron 6172 CPUs

Troubleshooting Detail
Command line face_detection ran, but face_recognition also segfaulted. gdb run of direct Python code for face_recognition_cli.py indicated segfault upon OpenBLAS library thread launch.

Resolution

• Removed dlib, face-recognition, and face-recognition-models via pip3
• Removed all OpenBLAS files via apt-get remove performed on the files installed in zmeventnotification instructions
• Installed dlib via pip3 with --no-cache-dir option, then installed face-recognition via pip3 with --no-cache-dir option

Assumed Issue
OpenBLAS optimizes linear algebra used in machine learning algorithms. Optimization of these routines is best achieved utilizing the quickest possible CPU instruction to perform a task, which changes per CPU architecture. Possible bug within OpenBLAS for the OPTERON_SSE3 instruction set compiled into stock Ubutnu 18.04 OpenBLAS packages, or OpenBLAS was not compiled with KVM support.