object detection using cuda 11.1 yolov4 "Illegal instruction" ??

Discussion topics related to mobile applications and ZoneMinder Event Server (including machine learning)
Post Reply
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

object detection using cuda 11.1 yolov4 "Illegal instruction" ??

Post by tsp84 »

Here is the output of running object detection against an event Ii have logging enabled and I dont know where to get more info about the "Illegal instruction".

Code: Select all

$ sudo -u www-data /var/lib/zmeventnotification/bin/zm_detect.py --config /etc/zm/objectconfig.ini  --eventid 198 --monitorid 2 --debug
01/25/21 23:53:26 zmesdetect_m2[3430] INF ZMLog.py:219 [Setting up signal handler for logs]

01/25/21 23:53:26 zmesdetect_m2[3430] INF zm_detect.py:261 [---------| pyzm version:0.3.32, hook version:6.1.8,  ES version:6.1.8 , OpenCV version:4.5.1|------------]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG1 utils.py:406 [secret filename: /etc/zm/secrets.ini]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_PORTAL]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_USER]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_PASSWORD]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_API_PORTAL]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ML_USER]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ML_PASSWORD]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !PLATEREC_ALPR_KEY]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG1 utils.py:441 [allowing self-signed certs to work...]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG4 utils.py:451 [Now checking for monitor overrides]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:482 [adding polygon: front [550,259 1454,230 1908,378 1919,1079 0,1067]]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG4 utils.py:514 [Finally, doing parameter substitution]

01/25/21 23:53:26 zmesdetect_m2[3430] INF zm_detect.py:286 [Importing local classes for Object/Face]

01/25/21 23:53:27 zmesdetect_m2[3430] INF zm_detect.py:309 [Connecting with ZM APIs]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 api.py:69 [API SSL certificate check has been disbled]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 api.py:171 [using username/password for login]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 api.py:200 [Using new token API]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 api.py:207 [Access token expires on:2021-01-26 01:53:27.497124 [7200s]]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 api.py:211 [Refresh token expires on:2021-01-26 23:53:27.498114 [86400s]]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 api.py:134 [No need to relogin as access token still has 119.99996715 minutes remaining]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG4 api.py:270 [make_request called with url=https://tsphomestead.duckdns.org/zm/api/host/gettimezone.json payload={} type=get query={'token': 'REDACTED'}]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 zm_detect.py:317 [using ml_sequence]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 zm_detect.py:329 [using stream_sequence]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:379 [Using manual locking as we are only using one model]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:49 [Media get SSL certificate check has been disbled]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:96 [Using URL 198 for stream]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:109 [We will only process frames: ['snapshot', 'alarm']]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:132 [No need to start streams, we are picking images from https://tsphomestead.duckdns.org/zm/index.php?view=image&eid=198]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 Media.py:245 [Reading https://tsphomestead.duckdns.org/zm/index.php?view=image&eid=198&fid=snapshot]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 api.py:134 [No need to relogin as access token still has 119.99950471666666 minutes remaining]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG4 api.py:270 [make_request called with url=https://tsphomestead.duckdns.org/zm/index.php?view=image&eid=198&fid=snapshot payload={} type=get query={'token': 'REDACTED}]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 detect_sequence.py:398 [perf: Starting for frame:snapshot]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 detect_sequence.py:411 [============ Frame: snapshot Running object model in sequence ==================]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:136 [load_models (just init, actual load happens at first detect): ['object']]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 detect_sequence.py:145 [Initializing model  type:object with options:{'object_config': '/var/lib/zmeventnotification/models/yolov4/yolov4.cfg', 'object_weights': '/var/lib/zmeventnotification/models/yolov4/yolov4.weights', 'object_labels': '/var/lib/zmeventnotification/models/yolov4/coco.names', 'object_min_confidence': 0.3, 'object_framework': 'opencv', 'object_processor': 'gpu', 'gpu_max_processes': 20, 'gpu_max_lock_wait': 100, 'cpu_max_processes': 9, 'cpu_max_lock_wait': 100, 'max_detection_size': '90%', 'disable_locks': 'no'}]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:38 [portalock: max:20, name:pyzm_uid33_gpu_lock, timeout:100]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:50 [Waiting for pyzm_uid33_gpu_lock portalock...]

filenames [PosixPath('/tmp/pyzm_uid33_gpu_lock.00.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.01.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.02.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.03.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.04.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.05.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.06.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.07.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.08.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.09.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.10.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.11.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.12.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.13.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.14.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.15.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.16.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.17.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.18.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.19.lock')]
trying lock [PosixPath('/tmp/pyzm_uid33_gpu_lock.00.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.01.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.02.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.03.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.04.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.05.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.06.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.07.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.08.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.09.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.10.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.11.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.12.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.13.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.14.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.15.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.16.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.17.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.18.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.19.lock')]
trying lock for /tmp/pyzm_uid33_gpu_lock.00.lock
locked /tmp/pyzm_uid33_gpu_lock.00.lock
01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:52 [Got pyzm_uid33_gpu_lock portalock]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:431 [object has a same_model_sequence strategy of first]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:442 [--------- Frame:snapshot Running variation: #1 -------------]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:47 [pyzm_uid33_gpu_lock portalock already acquired]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 yolo.py:81 [|--------- Loading Yolo model from disk ---------|]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 yolo.py:88 [perf: processor:gpu Yolo initialization (loading /var/lib/zmeventnotification/models/yolov4/yolov4.weights model from disk) took: 194.62 ms]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:104 [Setting CUDA backend for OpenCV]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 yolo.py:105 [If you did not set your CUDA_ARCH_BIN correctly during OpenCV compilation, you will get errors during detection related to invalid device/make_policy]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 yolo.py:143 [|---------- YOLO (input image: 800w*450h, model resize dimensions: 416w*416h) ----------|]

Illegal instruction
$ wtf????
Last edited by tsp84 on Tue Jan 26, 2021 5:04 pm, edited 3 times in total.
tsp84
Posts: 227
Joined: Thu Dec 24, 2020 4:04 am

Re: object detection using cuda 11.1 yolov4 "Illegal instruction" ??

Post by tsp84 »

I have the correct ARCH set for cuda. Its a 1660 ti so the compute capability is 7.5. opencv 4.5, cuda 11.1 and cudnn for cuda 11.1 (I had 11.2 installed but cudnn errors saying reported version mismatch). I am watching nvidia-smi while running the detection and I can see python uses roughly 500MB of ram on the GPU and suddenly dies. I'm very confused as it seems everything is doing what it's supposed to. I am reading up how to enable or view the logs for CUDA, cudnn and opencv to see if I can get a clue to what's going on.

Switched to yolov3 with same results.

I had this working with opencv 4.3 and yolov3 but wanted to try yolov4. Hope someone can shed some light on this issue. Thanks!
Post Reply