Failed to load delegate from libedgetpu.so.1

[fiatguy85]

I'm having an issue with a M2 PCI Coral EdgeTPU and I believe it's due to multiple cameras requesting the interface at the same time. My setup involves two cameras on the front of the house, so they are sometimes triggered by an event passing in front of the house all the way from one side to the other.

Currently, one camera is delayed slightly more than the other and the error is occurring on the second camera almost all of the time. I was also able to recreate this behavior by also running two manual requests concurrently and an example of the failure "Failed to load delegate from libedgetpu.so.1" is shown below. This error does not occur at all in multiple frames for the first instance request and occurs multiple times in the second requested instance. Running a single isolated request is successful and does not create the issue.

I currently have tpu_max_processes=1 in objectconfig.ini and I have also tried 2, but the error occurs either way. If there's a setting I'm missing that can correct this or any additional information that would be helpful, please let me know.

Code: Select all

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG1 detect_sequence.py:654 [perf: Starting for frame:81]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG1 detect_sequence.py:664 [Sequence of detection types to execute: ['object', 'face']]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG1 detect_sequence.py:669 [============ Frame: 81 Running object detection type in sequence ==================]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG3 detect_sequence.py:689 [object has a same_model_sequence strategy of first]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG1 detect_sequence.py:701 [--------- Frame:81 Running variation: #1 -------------]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG2 coral_edgetpu.py:51 [Waiting for pyzm_uid33_tpu_lock portalock...]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG2 coral_edgetpu.py:53 [Got pyzm_uid33_tpu_lock portalock]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG1 coral_edgetpu.py:89 [|--------- Loading "TPU object detection" model from disk -------------|]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG2 coral_edgetpu.py:68 [Released portalock pyzm_uid33_tpu_lock]

07/25/23 06:15:38 zmesdetect_m2[1961308] ERR detect_sequence.py:713 [Error running model: Failed to load delegate from libedgetpu.so.1
]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG2 detect_sequence.py:714 [Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/lib/python3.10/site-packages/tflite_runtime/interpreter.py", line 119, in __init__
    raise ValueError(capture.message)
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pyzm/ml/detect_sequence.py", line 710, in detect_stream
    _b,_l,_c,_m = m.detect(image=frame)
  File "/usr/lib/python3.10/site-packages/pyzm/ml/object.py", line 71, in detect
    b,l,c,_model_names = self.model.detect(image)
  File "/usr/lib/python3.10/site-packages/pyzm/ml/coral_edgetpu.py", line 161, in detect
    self.load_model()
  File "/usr/lib/python3.10/site-packages/pyzm/ml/coral_edgetpu.py", line 94, in load_model
    self.model = make_interpreter(self.options.get('object_weights'))
  File "/usr/lib/python3.10/site-packages/pycoral/utils/edgetpu.py", line 87, in make_interpreter
    delegates = [load_edgetpu_delegate({'device': device} if device else {})]
  File "/usr/lib/python3.10/site-packages/pycoral/utils/edgetpu.py", line 52, in load_edgetpu_delegate
    return tflite.load_delegate(_EDGETPU_SHARED_LIB, options or {})
  File "/usr/lib/python3.10/site-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1

]

07/25/23 06:15:38 zmesdetect_m2[1961308] DBG1 detect_sequence.py:701 [--------- Frame:81 Running variation: #2 -------------]

[fiatguy85]

I tried digging into this a little bit, but still no success so far. The issue appears to be that the second analysis process called is not being properly informed of the TPU device being locked by the first process.

I tried specifying the device used in line 94 in pyzm/ml/coral_edgetpu.py, to use the same device every time, but the result was the same with the 2nd process activated failing, with the same error message.

Code: Select all

self.model = make_interpreter(self.options.get('object_weights'), device=':0')

Pycoral API documentation where I got the idea: https://coral.ai/docs/edgetpu/multiple- ... ycoral-api

I also tried specifying "disable_locks=yes" in objectconfig.ini and that also made no difference.

[Magic919]

I've never experienced, but I see Asker (PliablePixels) made some comments here - https://github.com/tensorflow/tensorflow/issues/32743

[fiatguy85]

Are you running multiple camera, which are calling the TPU concurrently?

I did find that thread already, and I have double checked the permissions on the TPU device and that's not the issue. It runs fine manually for a single analysis process, just not a 2nd one concurrently.

I did also try reloading the device and it doesn't make a difference. I'm running a PCI M2 coral and not a USB one.

[Magic919]

I have two cameras that can potentially do that. However, I run USB and use MLAPI, so not the same setup you have.