object detection using cuda 11.1 yolov4 "Illegal instruction" ??

Discussion topics related to mobile applications for ZoneMinder
Post Reply
tsp84
Posts: 68
Joined: Thu Dec 24, 2020 4:04 am

object detection using cuda 11.1 yolov4 "Illegal instruction" ??

Post by tsp84 » Tue Jan 26, 2021 6:58 am

Here is the output of running object detection against an event Ii have logging enabled and I dont know where to get more info about the "Illegal instruction".

Code: Select all

$ sudo -u www-data /var/lib/zmeventnotification/bin/zm_detect.py --config /etc/zm/objectconfig.ini  --eventid 198 --monitorid 2 --debug
01/25/21 23:53:26 zmesdetect_m2[3430] INF ZMLog.py:219 [Setting up signal handler for logs]

01/25/21 23:53:26 zmesdetect_m2[3430] INF zm_detect.py:261 [---------| pyzm version:0.3.32, hook version:6.1.8,  ES version:6.1.8 , OpenCV version:4.5.1|------------]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG1 utils.py:406 [secret filename: /etc/zm/secrets.ini]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_PORTAL]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_USER]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_PASSWORD]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ZM_API_PORTAL]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ML_USER]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !ML_PASSWORD]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:379 [Secret token found in config: !PLATEREC_ALPR_KEY]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG1 utils.py:441 [allowing self-signed certs to work...]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG4 utils.py:451 [Now checking for monitor overrides]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG2 utils.py:482 [adding polygon: front [550,259 1454,230 1908,378 1919,1079 0,1067]]

01/25/21 23:53:26 zmesdetect_m2[3430] DBG4 utils.py:514 [Finally, doing parameter substitution]

01/25/21 23:53:26 zmesdetect_m2[3430] INF zm_detect.py:286 [Importing local classes for Object/Face]

01/25/21 23:53:27 zmesdetect_m2[3430] INF zm_detect.py:309 [Connecting with ZM APIs]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 api.py:69 [API SSL certificate check has been disbled]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 api.py:171 [using username/password for login]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 api.py:200 [Using new token API]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 api.py:207 [Access token expires on:2021-01-26 01:53:27.497124 [7200s]]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 api.py:211 [Refresh token expires on:2021-01-26 23:53:27.498114 [86400s]]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 api.py:134 [No need to relogin as access token still has 119.99996715 minutes remaining]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG4 api.py:270 [make_request called with url=https://tsphomestead.duckdns.org/zm/api/host/gettimezone.json payload={} type=get query={'token': 'REDACTED'}]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 zm_detect.py:317 [using ml_sequence]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 zm_detect.py:329 [using stream_sequence]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:379 [Using manual locking as we are only using one model]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:49 [Media get SSL certificate check has been disbled]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:96 [Using URL 198 for stream]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:109 [We will only process frames: ['snapshot', 'alarm']]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 Media.py:132 [No need to start streams, we are picking images from https://tsphomestead.duckdns.org/zm/index.php?view=image&eid=198]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 Media.py:245 [Reading https://tsphomestead.duckdns.org/zm/index.php?view=image&eid=198&fid=snapshot]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 api.py:134 [No need to relogin as access token still has 119.99950471666666 minutes remaining]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG4 api.py:270 [make_request called with url=https://tsphomestead.duckdns.org/zm/index.php?view=image&eid=198&fid=snapshot payload={} type=get query={'token': 'REDACTED}]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 detect_sequence.py:398 [perf: Starting for frame:snapshot]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 detect_sequence.py:411 [============ Frame: snapshot Running object model in sequence ==================]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:136 [load_models (just init, actual load happens at first detect): ['object']]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 detect_sequence.py:145 [Initializing model  type:object with options:{'object_config': '/var/lib/zmeventnotification/models/yolov4/yolov4.cfg', 'object_weights': '/var/lib/zmeventnotification/models/yolov4/yolov4.weights', 'object_labels': '/var/lib/zmeventnotification/models/yolov4/coco.names', 'object_min_confidence': 0.3, 'object_framework': 'opencv', 'object_processor': 'gpu', 'gpu_max_processes': 20, 'gpu_max_lock_wait': 100, 'cpu_max_processes': 9, 'cpu_max_lock_wait': 100, 'max_detection_size': '90%', 'disable_locks': 'no'}]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:38 [portalock: max:20, name:pyzm_uid33_gpu_lock, timeout:100]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:50 [Waiting for pyzm_uid33_gpu_lock portalock...]

filenames [PosixPath('/tmp/pyzm_uid33_gpu_lock.00.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.01.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.02.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.03.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.04.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.05.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.06.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.07.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.08.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.09.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.10.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.11.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.12.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.13.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.14.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.15.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.16.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.17.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.18.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.19.lock')]
trying lock [PosixPath('/tmp/pyzm_uid33_gpu_lock.00.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.01.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.02.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.03.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.04.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.05.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.06.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.07.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.08.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.09.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.10.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.11.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.12.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.13.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.14.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.15.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.16.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.17.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.18.lock'), PosixPath('/tmp/pyzm_uid33_gpu_lock.19.lock')]
trying lock for /tmp/pyzm_uid33_gpu_lock.00.lock
locked /tmp/pyzm_uid33_gpu_lock.00.lock
01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:52 [Got pyzm_uid33_gpu_lock portalock]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:431 [object has a same_model_sequence strategy of first]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 detect_sequence.py:442 [--------- Frame:snapshot Running variation: #1 -------------]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:47 [pyzm_uid33_gpu_lock portalock already acquired]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 yolo.py:81 [|--------- Loading Yolo model from disk ---------|]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 yolo.py:88 [perf: processor:gpu Yolo initialization (loading /var/lib/zmeventnotification/models/yolov4/yolov4.weights model from disk) took: 194.62 ms]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG2 yolo.py:104 [Setting CUDA backend for OpenCV]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG3 yolo.py:105 [If you did not set your CUDA_ARCH_BIN correctly during OpenCV compilation, you will get errors during detection related to invalid device/make_policy]

01/25/21 23:53:27 zmesdetect_m2[3430] DBG1 yolo.py:143 [|---------- YOLO (input image: 800w*450h, model resize dimensions: 416w*416h) ----------|]

Illegal instruction
$ wtf????
Last edited by tsp84 on Tue Jan 26, 2021 5:04 pm, edited 3 times in total.

tsp84
Posts: 68
Joined: Thu Dec 24, 2020 4:04 am

Re: object detection using cuda 11.1 yolov4 "Illegal instruction" ??

Post by tsp84 » Tue Jan 26, 2021 7:16 am

I have the correct ARCH set for cuda. Its a 1660 ti so the compute capability is 7.5. opencv 4.5, cuda 11.1 and cudnn for cuda 11.1 (I had 11.2 installed but cudnn errors saying reported version mismatch). I am watching nvidia-smi while running the detection and I can see python uses roughly 500MB of ram on the GPU and suddenly dies. I'm very confused as it seems everything is doing what it's supposed to. I am reading up how to enable or view the logs for CUDA, cudnn and opencv to see if I can get a clue to what's going on.

Switched to yolov3 with same results.

I had this working with opencv 4.3 and yolov3 but wanted to try yolov4. Hope someone can shed some light on this issue. Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests