[BUG] Occasional test suite deadlock in `test_parallel_tasks` or `test_multiple_async_done_multiple_streams`

[gmarkall]

Locally the test suite has just deadlocked for me, ending in:

test_add_callback (numba.cuda.tests.cudadrv.test_streams.TestCudaStream.test_add_callback) ... ok
test_add_callback_with_default_arg (numba.cuda.tests.cudadrv.test_streams.TestCudaStream.test_add_callback_with_default_arg) ... ok
test_async_done (numba.cuda.tests.cudadrv.test_streams.TestCudaStream.test_async_done) ... ok
test_cancelled_future (numba.cuda.tests.cudadrv.test_streams.TestCudaStream.test_cancelled_future) ... ok
test_multiple_async_done (numba.cuda.tests.cudadrv.test_streams.TestCudaStream.test_multiple_async_done) ... ok
test_multiple_async_done_multiple_streams (numba.cuda.tests.cudadrv.test_streams.TestCudaStream.test_multiple_async_done_multiple_streams) ... ok
test_parallel_tasks (numba.cuda.tests.cudadrv.test_streams.TestCudaStream.test_parallel_tasks) ... ^C^C

Hitting ctrl-C to get a traceback has no effect.

A CI run for a previous test also exhibited similar behaviour (will add more details when I can retrieve them).

[gmarkall]

info threads:

(gdb) info threads
  Id   Target Id                                            Frame 
* 1    Thread 0x7efaec9d8740 (LWP 146484) "python"          0x00007ef9ef64f83e in ?? () from /lib/x86_64-linux-gnu/libcuda.so
  2    Thread 0x7ef9ea1ff6c0 (LWP 146529) "cuda00001800007" 0x00007efaec71b4cd in __GI___poll (fds=0x64682dac4b80, nfds=3, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
  3    Thread 0x7ef9dc3ff6c0 (LWP 147272) "python"          0x00007efaec698d71 in __futex_abstimed_wait_common64 (private=32505, 
    cancel=true, abstime=0x7ef9dc3fee40, op=393, expected=0, futex_word=0x64682e124db0) at ./nptl/futex-internal.c:57
  4    Thread 0x7efa233ff6c0 (LWP 147273) "cuda-EvtHandlr"  0x00007efaec71b4cd in __GI___poll (fds=0x7ef9bc000c20, nfds=11, 
    timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
  5    Thread 0x7efac5bff6c0 (LWP 147275) "python"          0x00007efaec698d71 in __futex_abstimed_wait_common64 (private=32506, 
    cancel=true, abstime=0x7efac5bfe750, op=137, expected=0, futex_word=0x646819d3d194 <_PyRuntime+436>) at ./nptl/futex-internal.c:57

[gmarkall]

Traceback for thread 1:

#0  0x00007ef9ef43f831 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#1  0x00007ef9eff4f28d in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#2  0x00007ef9ef3da6b8 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#3  0x00007ef9ef529ac7 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#4  0x00007ef9d9e15713 in ?? ()
   from /home/gmarkall/miniforge3/envs/numbadev/lib/python3.11/site-packages/rmm/pylibrmm/../../../../libcudart.so.12
#5  0x00007ef9d9e58218 in cudaFree ()
   from /home/gmarkall/miniforge3/envs/numbadev/lib/python3.11/site-packages/rmm/pylibrmm/../../../../libcudart.so.12
#6  0x00007ef9fa9ba1bc in ?? ()
   from /home/gmarkall/miniforge3/envs/numbadev/lib/python3.11/site-packages/rmm/pylibrmm/device_buffer.cpython-311-x86_64-linux-gnu.so
#7  0x00007ef9fa9bd1a0 in ?? ()
   from /home/gmarkall/miniforge3/envs/numbadev/lib/python3.11/site-packages/rmm/pylibrmm/device_buffer.cpython-311-x86_64-linux-gnu.so
#8  0x00006468199e5cf0 in Py_DECREF (op=<rmm.pylibrmm.device_buffer.DeviceBuffer at remote 0x7efae6ab6800>)
    at /usr/local/src/conda/python-3.11.9/Include/object.h:538
#9  delitem_common (old_value=<rmm.pylibrmm.device_buffer.DeviceBuffer at remote 0x7efae6ab6800>, ix=<optimized out>, 
    hash=139610774568960, mp=0x7efae6a7ce90) at /usr/local/src/conda/python-3.11.9/Objects/dictobject.c:1986
#10 _PyDict_DelItem_KnownHash (hash=139610774568960, key=139610774568960, op=<UniqueDict at remote 0x7efae6a7ce90>)
    at /usr/local/src/conda/python-3.11.9/Objects/dictobject.c:2028
#11 PyDict_DelItem (op=<UniqueDict at remote 0x7efae6a7ce90>, key=139610774568960)
    at /usr/local/src/conda/python-3.11.9/Objects/dictobject.c:2003
#12 0x0000646819af1699 in wrap_delitem (self=<UniqueDict at remote 0x7efae6a7ce90>, args=(139610774568960,), 
    wrapped=0x6468199c8d90 <dict_ass_sub>) at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:6874
#13 0x0000646819a60f33 in wrapperdescr_raw_call (kwds=0x0, args=(139610774568960,), self=<UniqueDict at remote 0x7efae6a7ce90>, 
    descr=0x7efaec9112b0) at /usr/local/src/conda/python-3.11.9/Objects/descrobject.c:536
#14 wrapperdescr_call (descr=0x7efaec9112b0, args=(139610774568960,), kwds=0x0)
    at /usr/local/src/conda/python-3.11.9/Objects/descrobject.c:573
#15 0x00006468199c98b3 in _PyObject_MakeTpCall (tstate=0x646819d65998 <_PyRuntime+166328>, 
    callable=<wrapper_descriptor at remote 0x7efaec9112b0>, args=<optimized out>, nargs=2, keywords=0x0)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:214
#16 0x0000646819a43b41 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ffdf696a210, 
    callable=<wrapper_descriptor at remote 0x7efaec9112b0>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:90
#17 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffdf696a210, 
    callable=<wrapper_descriptor at remote 0x7efaec9112b0>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:77
#18 vectorcall_unbound (nargs=<optimized out>, args=0x7ffdf696a210, func=<optimized out>, unbound=<optimized out>, 
    tstate=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:1641
#19 vectorcall_method (name=name@entry='__delitem__', args=args@entry=0x7ffdf696a210, nargs=nargs@entry=2)
    at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:1672
#20 0x0000646819a4e41d in slot_mp_ass_subscript (self=<optimized out>, key=<optimized out>, value=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:7407
#21 0x00006468199dbdee in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae8f48, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:2376
#22 0x00006468199fa981 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae8f48, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#23 _PyEval_Vector (kwnames=<optimized out>, argcount=0, args=0x646819d4b610 <_PyRuntime+58928>, locals=0x0, func=<optimized out>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#24 _PyFunction_Vectorcall (func=<optimized out>, stack=0x646819d4b610 <_PyRuntime+58928>, nargsf=<optimized out>, 
    kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#25 0x00006468199db6e4 in do_call_core (use_tracing=<optimized out>, kwdict={}, callargs=(), 
    func=<function at remote 0x7efae7bd4540>, tstate=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:7349
#26 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae8ec0, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:5376
#27 0x00006468199fa981 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae8ec0, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#28 _PyEval_Vector (kwnames=<optimized out>, argcount=2, args=0x7ffdf696a670, locals=0x0, func=<optimized out>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#29 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7ffdf696a670, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#30 0x00006468199ce3a3 in _PyObject_FastCallDictTstate (tstate=0x646819d65998 <_PyRuntime+166328>, 
    callable=<function at remote 0x7efaebedec00>, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:141
#31 0x0000646819a02399 in _PyObject_Call_Prepend (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, 
    callable=callable@entry=<function at remote 0x7efaebedec00>, obj=obj@entry=<finalize at remote 0x7efae6a17260>, 
    args=args@entry=(<weakref.ReferenceType at remote 0x7efae6bb40e0>,), kwargs=kwargs@entry=0x0)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:482
#32 0x0000646819ad4108 in slot_tp_call (self=<finalize at remote 0x7efae6a17260>, 
    args=(<weakref.ReferenceType at remote 0x7efae6bb40e0>,), kwds=0x0)
    at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:7624
#33 0x00006468199c98b3 in _PyObject_MakeTpCall (tstate=0x646819d65998 <_PyRuntime+166328>, 
    callable=<finalize at remote 0x7efae6a17260>, args=<optimized out>, nargs=1, keywords=0x0)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:214
#34 0x00006468199ffb18 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7ffdf696a788, 
    callable=<finalize at remote 0x7efae6a17260>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:90
#35 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7ffdf696a788, 
--Type <RET> for more, q to quit, c to continue without paging--
    callable=<finalize at remote 0x7efae6a17260>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:77
#36 PyObject_CallOneArg (func=func@entry=<finalize at remote 0x7efae6a17260>, arg=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:376
#37 0x0000646819aa649f in handle_callback (ref=<optimized out>, callback=<finalize at remote 0x7efae6a17260>)
    at /usr/local/src/conda/python-3.11.9/Objects/weakrefobject.c:928
#38 0x0000646819a21c4b in PyObject_ClearWeakRefs (
    object=<MemoryPointer(context=<Context(device=<weakref.ProxyType at remote 0x7ef9ee1e39c0>, handle=<c_void_p at remote 0x7efae6af1010>, allocations=<UniqueDict at remote 0x7efae6a7cfb0>, deallocations=<_PendingDeallocs(_cons=<collections.deque at remote 0x7efae6b6a200>, _disable_count=0, _size=0, memory_capacity=<_SizeNotSet at remote 0x7ef9fab87400>) at remote 0x7efae699e950>, memory_manager=<RMMNumbaManager(context=<...>, allocations=<UniqueDict at remote 0x7efae6a7ce90>, deallocations=<_PendingDeallocs(_cons=<collections.deque at remote 0x7efae6b6b3d0>, _disable_count=0, _size=0, memory_capacity=<...>) at remote 0x7efae6824510>) at remote 0x7efae699f7d0>, modules=<UniqueDict at remote 0x7efae6a7cf50>, extras={}) at remote 0x7efae696c310>, device_pointer=<c_ulong at remote 0x7efae6a479b0>, size=64, _cuda_memsize_=64, is_managed=True, refct=1, handle=<...>, _owner=None, _finalizer=<finalize at remote 0x7efae6a17260>) at remote 0x7efae6ab7d50>) at /usr/local/src/conda/python-3.11.9/Objects/weakrefobject.c:974
#39 subtype_dealloc (
    self=<MemoryPointer(context=<Context(device=<weakref.ProxyType at remote 0x7ef9ee1e39c0>, handle=<c_void_p at remote 0x7efae6af1010>, allocations=<UniqueDict at remote 0x7efae6a7cfb0>, deallocations=<_PendingDeallocs(_cons=<collections.deque at remote 0x7efae6b6a200>, _disable_count=0, _size=0, memory_capacity=<_SizeNotSet at remote 0x7ef9fab87400>) at remote 0x7efae699e950>, memory_manager=<RMMNumbaManager(context=<...>, allocations=<UniqueDict at remote 0x7efae6a7ce90>, deallocations=<_PendingDeallocs(_cons=<collections.deque at remote 0x7efae6b6b3d0>, _disable_count=0, _size=0, memory_capacity=<...>) at remote 0x7efae6824510>) at remote 0x7efae699f7d0>, modules=<UniqueDict at remote 0x7efae6a7cf50>, extras={}) at remote 0x7efae696c310>, device_pointer=<c_ulong at remote 0x7efae6a479b0>, size=64, _cuda_memsize_=64, is_managed=True, refct=1, handle=<...>, _owner=None, _finalizer=<finalize at remote 0x7efae6a17260>) at remote 0x7efae6ab7d50>) at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:1389
#40 0x00006468199c84bc in Py_DECREF (op=<optimized out>) at /usr/local/src/conda/python-3.11.9/Include/object.h:538
#41 Py_XDECREF (op=<optimized out>) at /usr/local/src/conda/python-3.11.9/Include/object.h:602
#42 dict_dealloc (mp=0x7efae699f0c0) at /usr/local/src/conda/python-3.11.9/Objects/dictobject.c:2371
#43 0x0000646819a21dc6 in Py_DECREF (op=<optimized out>) at /usr/local/src/conda/python-3.11.9/Include/object.h:538
#44 subtype_dealloc (
    self=<DeviceNDArray(ndim=1, _dummy=<Array(dims=<unknown at remote 0x7efae6ab7ab0>, strides=<type at remote 0x64682db813a0>, itemsize=<unknown at remote 0x2>, size=0, extent=64) at remote 0x7efae6ab4790>, shape=(139610774568960,), strides=(<weakref.ReferenceType at remote 0x7efae6bb40e0>,), dtype=<numpy.dtypes.Float64DType at remote 0x7efaeb9c4ee0>, size=8, alloc_size=64, gpu_data=<MemoryPointer(context=<Context(device=<weakref.ProxyType at remote 0x7ef9ee1e39c0>, handle=<c_void_p at remote 0x7efae6af1010>, allocations=<UniqueDict at remote 0x7efae6a7cfb0>, deallocations=<_PendingDeallocs(_cons=<collections.deque at remote 0x7efae6b6a200>, _disable_count=0, _size=0, memory_capacity=<_SizeNotSet at remote 0x7ef9fab87400>) at remote 0x7efae699e950>, memory_manager=<RMMNumbaManager(context=<...>, allocations=<UniqueDict at remote 0x7efae6a7ce90>, deallocations=<_PendingDeallocs(_cons=<collections.deque at remote 0x7efae6b6b3d0>, _disable_count=0, _size=0, memory_capacity=<...>) at remote 0x7efae6824510>) at remote 0x7...(truncated))
    at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:1429
#45 0x00006468199e5917 in _Py_Dealloc (op=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/object.c:2390
#46 Py_DECREF (op=<optimized out>) at /usr/local/src/conda/python-3.11.9/Include/object.h:538
#47 Py_XDECREF (op=<optimized out>) at /usr/local/src/conda/python-3.11.9/Include/object.h:602
#48 _PyFrame_Clear (frame=0x7efae71b8990) at /usr/local/src/conda/python-3.11.9/Python/frame.c:141
#49 0x0000646819aa2d5b in gen_send_ex2 (gen=0x7efae71b8940, arg=None, presult=0x7ffdf696a9b8, exc=<optimized out>, 
    closing=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/genobject.c:272
#50 0x00007efaa2e1e8b6 in task_step_impl (exc=0x0, task=0x7efae7b67d00)
    at /usr/local/src/conda/python-3.11.9/Modules/_asynciomodule.c:2693
#51 task_step (task=0x7efae7b67d00, exc=<optimized out>) at /usr/local/src/conda/python-3.11.9/Modules/_asynciomodule.c:2993
#52 0x00006468199e93fa in cfunction_vectorcall_O (
    func=<built-in method task_wakeup of _asyncio.Task object at remote 0x7efae7b67d00>, args=0x7efae72affa0, nargsf=<optimized out>, 
    kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Include/cpython/methodobject.h:52
#53 0x0000646819982fb5 in _PyObject_VectorcallTstate (tstate=0x646819d65998 <_PyRuntime+166328>, 
    callable=<built-in method task_wakeup of _asyncio.Task object at remote 0x7efae7b67d00>, args=<optimized out>, 
    nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:92
#54 0x0000646819985065 in context_run (self=0x7efae6973840, args=0x7efae72aff98, nargs=2, kwnames=0x0)
    at /usr/local/src/conda/python-3.11.9/Python/context.c:673
#55 0x00006468199e3fbf in cfunction_vectorcall_FASTCALL_KEYWORDS (
    func=<built-in method run of _contextvars.Context object at remote 0x7efae6973840>, args=0x7efae72aff98, nargsf=<optimized out>, 
    kwnames=0x0) at /usr/local/src/conda/python-3.11.9/Include/cpython/methodobject.h:52
#56 0x00006468199df8d4 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, 
    callargs=(<built-in method task_wakeup of _asyncio.Task object at remote 0x7efae7b67d00>, <_asyncio.Future at remote 0x7efae6a47ac0>), func=<built-in method run of _contextvars.Context object at remote 0x7efae6973840>, tstate=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Python/ceval.c:7321
#57 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae8a60, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:5376
#58 0x0000646819a18fc2 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae8a60, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#59 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7ffdf696acd0, locals=0x0, func=0x7ef9faed39c0, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#60 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7ffdf696acd0, 
    func=<function at remote 0x7ef9faed39c0>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#61 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7ffdf696acd0, 
    callable=<function at remote 0x7ef9faed39c0>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:92
#62 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/classobject.c:89
#63 0x00006468199db6e4 in do_call_core (use_tracing=<optimized out>, kwdict={}, 
    callargs=(<ParallelTestResult(failfast=True, failures=[], errors=[], testsRun=630, skipped=[(<TestNvvmDriver(_testMethodName='test_nvvm_ir_verify_fail', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2--Type <RET> for more, q to quit, c to continue without paging--
ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqual', <type at remote 0x646819d22f60>: 'assertMultiLineEqual'}) at remote 0x7ef9d8801950>, "Bad triple doesn't fail verify on CUDA >= 12.5"), (<TestContextAPI(_testMethodName='test_context_memory', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqua...(truncated), 
    func=<method at remote 0x7efae6905a00>, tstate=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:7349
#64 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae89d8, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:5376
#65 0x00006468199fa981 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae89d8, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#66 _PyEval_Vector (kwnames=<optimized out>, argcount=2, args=0x7ffdf696af90, locals=0x0, func=<optimized out>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#67 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7ffdf696af90, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#68 0x00006468199ce3a3 in _PyObject_FastCallDictTstate (tstate=0x646819d65998 <_PyRuntime+166328>, 
    callable=<function at remote 0x7ef9faed3ba0>, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:141
#69 0x0000646819a02399 in _PyObject_Call_Prepend (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, 
    callable=callable@entry=<function at remote 0x7ef9faed3ba0>, 
    obj=obj@entry=<TestCudaStream(_testMethodName='test_parallel_tasks', _outcome=<_Outcome(expecting_failure=False, result=<ParallelTestResult(failfast=True, failures=[], errors=[], testsRun=630, skipped=[(<TestNvvmDriver(_testMethodName='test_nvvm_ir_verify_fail', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqual', <type at remote 0x646819d22f60>: 'assertMultiLineEqual'}) at remote 0x7ef9d8801950>, "Bad triple doesn't fail verify on CUDA >= 12.5"), (<TestContextAPI(_testMethodName='test_context_memory', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleE...(truncated), 
    args=args@entry=(<ParallelTestResult(failfast=True, failures=[], errors=[], testsRun=630, skipped=[(<TestNvvmDriver(_testMethodName='test_nvvm_ir_verify_fail', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqual', <type at remote 0x646819d22f60>: 'assertMultiLineEqual'}) at remote 0x7ef9d8801950>, "Bad triple doesn't fail verify on CUDA >= 12.5"), (<TestContextAPI(_testMethodName='test_context_memory', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqua...(truncated), 
    kwargs=kwargs@entry=0x0) at /usr/local/src/conda/python-3.11.9/Objects/call.c:482
#70 0x0000646819ad4108 in slot_tp_call (
    self=<TestCudaStream(_testMethodName='test_parallel_tasks', _outcome=<_Outcome(expecting_failure=False, result=<ParallelTestResult(failfast=True, failures=[], errors=[], testsRun=630, skipped=[(<TestNvvmDriver(_testMethodName='test_nvvm_ir_verify_fail', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqual', <type at remote 0x646819d22f60>: 'assertMultiLineEqual'}) at remote 0x7ef9d8801950>, "Bad triple doesn't fail verify on CUDA >= 12.5"), (<TestContextAPI(_testMethodName='test_context_memory', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleE...(truncated), 
    args=(<ParallelTestResult(failfast=True, failures=[], errors=[], testsRun=630, skipped=[(<TestNvvmDriver(_testMethodName='test_nvvm_ir_verify_fail', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqual', <type at remote 0x646819d22f60>: 'assertMultiLineEqual'}) at remote 0x7ef9d8801950>, "Bad triple doesn't fail verify on CUDA >= 12.5"), (<TestContextAPI(_testMethodName='test_context_memory', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqua...(truncated), kwds=0x0)
    at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:7624
#71 0x00006468199c98b3 in _PyObject_MakeTpCall (tstate=0x646819d65998 <_PyRuntime+166328>, 
    callable=<TestCudaStream(_testMethodName='test_parallel_tasks', _outcome=<_Outcome(expecting_failure=False, result=<ParallelTestResult(failfast=True, failures=[], errors=[], testsRun=630, skipped=[(<TestNvvmDriver(_testMethodName='test_nvvm_ir_verify_fail', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleEqual', <type at remote 0x646819d26ae0>: 'assertSetEqual', <type at remote 0x646819d26940>: 'assertSetEqual', <type at remote 0x646819d22f60>: 'assertMultiLineEqual'}) at remote 0x7ef9d8801950>, "Bad triple doesn't fail verify on CUDA >= 12.5"), (<TestContextAPI(_testMethodName='test_context_memory', _outcome=None, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x646819d2ba80>: 'assertDictEqual', <type at remote 0x646819d2c640>: 'assertListEqual', <type at remote 0x646819d257c0>: 'assertTupleE...(truncated), args=<optimized out>, 
    nargs=1, keywords=0x0) at /usr/local/src/conda/python-3.11.9/Objects/call.c:214
#72 0x00006468199d73b6 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae8428, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:4769
#73 0x0000646819a198d4 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae8428, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#74 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7efaebdaaf30, locals=0x0, func=0x7ef9fad2de40, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#75 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7efaebdaaf30, 
    func=<function at remote 0x7ef9fad2de40>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#76 _PyObject_VectorcallTstate (tstate=0x646819d65998 <_PyRuntime+166328>, callable=<function at remote 0x7ef9fad2de40>, 
    args=0x7efaebdaaf30, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:92
#77 0x0000646819a190be in method_vectorcall (method=<optimized out>, args=0x7efaebdaaf38, nargsf=<optimized out>, 
    kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/classobject.c:59
--Type <RET> for more, q to quit, c to continue without paging--
#78 0x0000646819a046b0 in _PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, 
    callable=<method at remote 0x7ef9fae7a400>, func=0x646819a18d90 <method_vectorcall>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:257
#79 _PyObject_Call (kwargs=<optimized out>, args=<optimized out>, callable=<method at remote 0x7ef9fae7a400>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:328
#80 PyObject_Call (callable=<method at remote 0x7ef9fae7a400>, args=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:355
#81 0x00006468199db6e4 in do_call_core (use_tracing=<optimized out>, 
    kwdict={'argv': ['/home/gmarkall/numbadev/numba/numba/runtests.py', 'numba.cuda.tests', '-vf', '-m'], 'module': None, 'defaultTest': 'numba.tests', 'testRunner': None, 'exit': False, 'verbosity': 1, 'testLoader': <TestLoader(errors=[], _loading_packages=set(), _top_level_dir='/home/gmarkall/numbadev/numba-cuda/numba_cuda') at remote 0x7efaebda9b10>}, callargs=(), 
    func=<method at remote 0x7ef9fae7a400>, tstate=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:7349
#82 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae8390, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:5376
#83 0x00006468199fa981 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae8390, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#84 _PyEval_Vector (kwnames=<optimized out>, argcount=1, args=0x7ef9fae5fb48, locals=0x0, func=<optimized out>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#85 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7ef9fae5fb48, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#86 0x0000646819a01e03 in _PyObject_FastCallDictTstate (
    kwargs={'argv': ['/home/gmarkall/numbadev/numba/numba/runtests.py', 'numba.cuda.tests', '-vf', '-m'], 'module': None, 'defaultTest': 'numba.tests', 'topleveldir': None, 'testRunner': None, 'exit': False, 'verbosity': 1, 'nomultiproc': False}, 
    nargsf=<optimized out>, args=0x7ffdf696b590, callable=<function at remote 0x7ef9fad65c60>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:152
#87 _PyObject_Call_Prepend (
    kwargs={'argv': ['/home/gmarkall/numbadev/numba/numba/runtests.py', 'numba.cuda.tests', '-vf', '-m'], 'module': None, 'defaultTest': 'numba.tests', 'topleveldir': None, 'testRunner': None, 'exit': False, 'verbosity': 1, 'nomultiproc': False}, args=<optimized out>, 
    obj=<optimized out>, callable=<function at remote 0x7ef9fad65c60>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:482
#88 slot_tp_init (self=<optimized out>, args=<optimized out>, 
    kwds={'argv': ['/home/gmarkall/numbadev/numba/numba/runtests.py', 'numba.cuda.tests', '-vf', '-m'], 'module': None, 'defaultTest': 'numba.tests', 'topleveldir': None, 'testRunner': None, 'exit': False, 'verbosity': 1, 'nomultiproc': False})
    at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:7855
#89 0x00006468199c9893 in type_call (
    kwds={'argv': ['/home/gmarkall/numbadev/numba/numba/runtests.py', 'numba.cuda.tests', '-vf', '-m'], 'module': None, 'defaultTest': 'numba.tests', 'topleveldir': None, 'testRunner': None, 'exit': False, 'verbosity': 1, 'nomultiproc': False}, args=(), 
    type=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:1103
#90 _PyObject_MakeTpCall (tstate=0x646819d65998 <_PyRuntime+166328>, callable=<type at remote 0x64682d9fed50>, args=<optimized out>, 
    nargs=<optimized out>, 
    keywords=('argv', 'module', 'defaultTest', 'topleveldir', 'testRunner', 'exit', 'verbosity', 'nomultiproc'))
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:214
#91 0x00006468199d73b6 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae82b0, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:4769
#92 0x00006468199fa981 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae82b0, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#93 _PyEval_Vector (kwnames=<optimized out>, argcount=1, args=0x7efa19957fd8, locals=0x0, func=<optimized out>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#94 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7efa19957fd8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#95 0x0000646819a046b0 in _PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, 
    callable=<function at remote 0x7ef9fae84d60>, func=0x6468199fa800 <_PyFunction_Vectorcall>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:257
#96 _PyObject_Call (kwargs=<optimized out>, args=<optimized out>, callable=<function at remote 0x7ef9fae84d60>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:328
#97 PyObject_Call (callable=<function at remote 0x7ef9fae84d60>, args=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:355
#98 0x00006468199db6e4 in do_call_core (use_tracing=<optimized out>, kwdict={'defaultTest': 'numba.tests'}, 
    callargs=(['/home/gmarkall/numbadev/numba/numba/runtests.py', 'numba.cuda.tests', '-vf', '-m'],), 
    func=<function at remote 0x7ef9fae84d60>, tstate=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:7349
#99 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae81b8, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:5376
#100 0x0000646819a8ea8d in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae81b8, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#101 _PyEval_Vector (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, func=func@entry=0x7efaec9a5f80, 
    locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba.runtests', path='/home/gmarkall/numbadev/numba/numba/runtests.py') at remote 0x7efaebda9a90>, '__spec__': <ModuleSpec(name='numba.runtests', loader=<...>, origin='/home/gmarkall/numbadev/numba/numba/runtests.py', loader_state=None, submodule_search_locations=None, _uninitialized_submodules=[], _set_fileattr=True, _cached='/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc') at remote 0x7ef9fae79ed0>, '__annotations__': {}, '__builtins__': <module at remote 0x7efaec960a90>, '__file__': '/home/gmarkall/numbadev/numba/numba/runtests.py', '__cached__': '/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc', '_main': <function at remote 0x7ef9fad66c00>, 'sys': <module at remote 0x7efaec952ca0>, 'freeze_support': <method at remote 0x7ef9fad5cf00>}, 
    args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0)
    at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#102 0x0000646819a8e11f in PyEval_EvalCode (co=co@entry=<code at remote 0x7ef9faf32e20>, 
    globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba.runtests', path='/home/gmarkall/numbadev/numba/numba/runtests.py') at remote 0x7efaebda9a90>, '__spec__': <ModuleSpec(name='numba.runtests', loader=<...>, origin='/home/gmarkall/numbadev/numba/numba/runtests.py', loader_state=None, submodule_search_locations=None, _uni--Type <RET> for more, q to quit, c to continue without paging--
nitialized_submodules=[], _set_fileattr=True, _cached='/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc') at remote 0x7ef9fae79ed0>, '__annotations__': {}, '__builtins__': <module at remote 0x7efaec960a90>, '__file__': '/home/gmarkall/numbadev/numba/numba/runtests.py', '__cached__': '/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc', '_main': <function at remote 0x7ef9fad66c00>, 'sys': <module at remote 0x7efaec952ca0>, 'freeze_support': <method at remote 0x7ef9fad5cf00>}, 
    locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba.runtests', path='/home/gmarkall/numbadev/numba/numba/runtests.py') at remote 0x7efaebda9a90>, '__spec__': <ModuleSpec(name='numba.runtests', loader=<...>, origin='/home/gmarkall/numbadev/numba/numba/runtests.py', loader_state=None, submodule_search_locations=None, _uninitialized_submodules=[], _set_fileattr=True, _cached='/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc') at remote 0x7ef9fae79ed0>, '__annotations__': {}, '__builtins__': <module at remote 0x7efaec960a90>, '__file__': '/home/gmarkall/numbadev/numba/numba/runtests.py', '__cached__': '/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc', '_main': <function at remote 0x7ef9fad66c00>, 'sys': <module at remote 0x7efaec952ca0>, 'freeze_support': <method at remote 0x7ef9fad5cf00>})
    at /usr/local/src/conda/python-3.11.9/Python/ceval.c:1148
#103 0x0000646819aa56ee in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, 
    locals={'__name__': '__main__', '__doc__': None, '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba.runtests', path='/home/gmarkall/numbadev/numba/numba/runtests.py') at remote 0x7efaebda9a90>, '__spec__': <ModuleSpec(name='numba.runtests', loader=<...>, origin='/home/gmarkall/numbadev/numba/numba/runtests.py', loader_state=None, submodule_search_locations=None, _uninitialized_submodules=[], _set_fileattr=True, _cached='/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc') at remote 0x7ef9fae79ed0>, '__annotations__': {}, '__builtins__': <module at remote 0x7efaec960a90>, '__file__': '/home/gmarkall/numbadev/numba/numba/runtests.py', '__cached__': '/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc', '_main': <function at remote 0x7ef9fad66c00>, 'sys': <module at remote 0x7efaec952ca0>, 'freeze_support': <method at remote 0x7ef9fad5cf00>}, 
    globals={'__name__': '__main__', '__doc__': None, '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba.runtests', path='/home/gmarkall/numbadev/numba/numba/runtests.py') at remote 0x7efaebda9a90>, '__spec__': <ModuleSpec(name='numba.runtests', loader=<...>, origin='/home/gmarkall/numbadev/numba/numba/runtests.py', loader_state=None, submodule_search_locations=None, _uninitialized_submodules=[], _set_fileattr=True, _cached='/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc') at remote 0x7ef9fae79ed0>, '__annotations__': {}, '__builtins__': <module at remote 0x7efaec960a90>, '__file__': '/home/gmarkall/numbadev/numba/numba/runtests.py', '__cached__': '/home/gmarkall/numbadev/numba/numba/__pycache__/runtests.cpython-311.pyc', '_main': <function at remote 0x7ef9fad66c00>, 'sys': <module at remote 0x7efaec952ca0>, 'freeze_support': <method at remote 0x7ef9fad5cf00>}, 
    source=<code at remote 0x7ef9faf32e20>) at /usr/local/src/conda/python-3.11.9/Python/bltinmodule.c:1077
#104 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Python/clinic/bltinmodule.c.h:465
#105 0x00006468199e3fbf in cfunction_vectorcall_FASTCALL_KEYWORDS (
    func=<built-in method exec of module object at remote 0x7efaec960a90>, args=0x7efaecae8180, nargsf=<optimized out>, kwnames=0x0)
    at /usr/local/src/conda/python-3.11.9/Include/cpython/methodobject.h:52
#106 0x00006468199e3eac in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
    callable=<built-in method exec of module object at remote 0x7efaec960a90>, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:92
#107 PyObject_Vectorcall (callable=<built-in method exec of module object at remote 0x7efaec960a90>, args=<optimized out>, 
    nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:299
#108 0x00006468199d73b6 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x646819d65998 <_PyRuntime+166328>, frame=<optimized out>, 
    frame@entry=0x7efaecae8020, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:4769
#109 0x00006468199fa981 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaecae8020, tstate=0x646819d65998 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#110 _PyEval_Vector (kwnames=<optimized out>, argcount=2, args=0x7efaebd8a918, locals=0x0, func=<optimized out>, 
    tstate=0x646819d65998 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#111 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7efaebd8a918, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#112 0x0000646819ab8158 in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1)
    at /usr/local/src/conda/python-3.11.9/Modules/main.c:300
#113 0x0000646819ab7ad9 in pymain_run_python (exitcode=0x7ffdf696be14) at /usr/local/src/conda/python-3.11.9/Modules/main.c:595
#114 Py_RunMain () at /usr/local/src/conda/python-3.11.9/Modules/main.c:680
#115 0x0000646819a7e027 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Modules/main.c:734
#116 0x00007efaec62a1ca in __libc_start_call_main (main=main@entry=0x646819a7df80 <main>, argc=argc@entry=6, 
    argv=argv@entry=0x7ffdf696c078) at ../sysdeps/nptl/libc_start_call_main.h:58
#117 0x00007efaec62a28b in __libc_start_main_impl (main=0x646819a7df80 <main>, argc=6, argv=0x7ffdf696c078, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdf696c068) at ../csu/libc-start.c:360
#118 0x0000646819a7decd in _start ()

Traceback for thread 2:

#0  0x00007efaec71b4cd in __GI___poll (fds=0x64682dac4b80, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ef9ef508517 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#2  0x00007ef9ef5cb17f in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#3  0x00007ef9ef4f8b23 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#4  0x00007efaec69caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#5  0x00007efaec729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Traceback for thread 3:

#0  0x00007efaec698d71 in __futex_abstimed_wait_common64 (private=32505, cancel=true, abstime=0x7ef9dc3fee40, op=393, expected=0, 
    futex_word=0x64682e124db0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=32505, abstime=0x7ef9dc3fee40, clockid=32505, expected=0, 
    futex_word=0x64682e124db0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x64682e124db0, expected=expected@entry=0, 
    clockid=clockid@entry=0, abstime=abstime@entry=0x7ef9dc3fee40, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007efaec69bc8e in __pthread_cond_wait_common (abstime=0x7ef9dc3fee40, clockid=0, mutex=0x64682e15b978, cond=0x64682e124d88)
    at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x64682e124d88, mutex=0x64682e15b978, abstime=0x7ef9dc3fee40) at ./nptl/pthread_cond_wait.c:652
#5  0x00007ef9ef473eea in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#6  0x00007ef9ef4f8b23 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#7  0x00007efaec69caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#8  0x00007efaec729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Traceback for thread 4:

#0  0x00007efaec71b4cd in __GI___poll (fds=0x7ef9bc000c20, nfds=11, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ef9ef508517 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#2  0x00007ef9ef5cb17f in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#3  0x00007ef9ef4f8b23 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#4  0x00007efaec69caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#5  0x00007efaec729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Traceback for thread 5:

#0  0x00007efaec698d71 in __futex_abstimed_wait_common64 (private=32506, cancel=true, abstime=0x7efac5bfe750, op=137, expected=0, 
    futex_word=0x646819d3d194 <_PyRuntime+436>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=32506, abstime=0x7efac5bfe750, clockid=0, expected=0, 
    futex_word=0x646819d3d194 <_PyRuntime+436>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x646819d3d194 <_PyRuntime+436>, expected=expected@entry=0, 
    clockid=clockid@entry=1, abstime=abstime@entry=0x7efac5bfe750, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007efaec69bc8e in __pthread_cond_wait_common (abstime=0x7efac5bfe750, clockid=1, mutex=0x646819d3d198 <_PyRuntime+440>, 
    cond=0x646819d3d168 <_PyRuntime+392>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x646819d3d168 <_PyRuntime+392>, mutex=mutex@entry=0x646819d3d198 <_PyRuntime+440>, 
    abstime=abstime@entry=0x7efac5bfe750) at ./nptl/pthread_cond_wait.c:652
#5  0x00006468199af768 in PyCOND_TIMEDWAIT (us=<optimized out>, mut=<optimized out>, cond=0x646819d3d168 <_PyRuntime+392>)
    at /usr/local/src/conda/python-3.11.9/Python/condvar.h:73
#6  take_gil (tstate=tstate@entry=0x7ef9d4ea4df0) at /usr/local/src/conda/python-3.11.9/Python/ceval_gil.h:231
#7  0x00006468199d40e2 in PyEval_RestoreThread (tstate=tstate@entry=0x7ef9d4ea4df0)
    at /usr/local/src/conda/python-3.11.9/Python/ceval.c:521
#8  0x00007efac4c43c3c in sock_call_ex (s=s@entry=0x7efae6ed7340, writing=writing@entry=1, 
    sock_func=sock_func@entry=0x7efac4c41462 <sock_send_impl>, data=data@entry=0x7efac5bfe838, connect=connect@entry=0, 
    err=err@entry=0x0, timeout=0) at /usr/local/src/conda/python-3.11.9/Modules/socketmodule.c:921
#9  0x00007efac4c43eaa in sock_call (data=0x7efac5bfe838, func=0x7efac4c41462 <sock_send_impl>, writing=1, s=0x7efae6ed7340)
    at /usr/local/src/conda/python-3.11.9/Modules/socketmodule.c:972
#10 sock_send (s=0x7efae6ed7340, args=<optimized out>) at /usr/local/src/conda/python-3.11.9/Modules/socketmodule.c:4206
#11 0x00006468199d11c8 in method_vectorcall_VARARGS (func=<method_descriptor at remote 0x7efa199e8450>, args=0x7efaec575258, 
    nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/descrobject.c:330
#12 0x00006468199e3eac in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
    callable=<method_descriptor at remote 0x7efa199e8450>, tstate=0x7ef9d4ea4df0)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:92
#13 PyObject_Vectorcall (callable=<method_descriptor at remote 0x7efa199e8450>, args=<optimized out>, nargsf=<optimized out>, 
    kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:299
#14 0x00006468199d73b6 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x7ef9d4ea4df0, frame=<optimized out>, 
    frame@entry=0x7efaec575020, throwflag=throwflag@entry=0) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:4769
#15 0x00006468199fa981 in _PyEval_EvalFrame (throwflag=0, frame=0x7efaec575020, tstate=0x7ef9d4ea4df0)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#16 _PyEval_Vector (kwnames=<optimized out>, argcount=3, args=0x7efac5bfeba0, locals=0x0, func=<optimized out>, tstate=0x7ef9d4ea4df0)
    at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#17 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7efac5bfeba0, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#18 0x00006468199e3eac in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
    callable=<function at remote 0x7ef9fab968e0>, tstate=0x7ef9d4ea4df0)
    at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_call.h:92
#19 PyObject_Vectorcall (callable=<function at remote 0x7ef9fab968e0>, args=<optimized out>, nargsf=<optimized out>, 
    kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:299
#20 0x00007efac4c9c619 in _CallPythonObject (mem=0x7efac5bfee00, restype=<optimized out>, setfunc=0x0, 
    callable=<function at remote 0x7ef9fab968e0>, converters=<optimized out>, flags=257, pArgs=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Modules/_ctypes/callbacks.c:231
#21 0x00007efac4c9c7bb in closure_fcn (cif=<optimized out>, resp=<optimized out>, args=<optimized out>, userdata=<optimized out>)
    at /usr/local/src/conda/python-3.11.9/Modules/_ctypes/callbacks.c:310
#22 0x00007efac4c88892 in ffi_closure_unix64_inner ()
   from /home/gmarkall/miniforge3/envs/numbadev/lib/python3.11/lib-dynload/../../libffi.so.8
#23 0x00007efac4c88be0 in ffi_closure_unix64 ()
   from /home/gmarkall/miniforge3/envs/numbadev/lib/python3.11/lib-dynload/../../libffi.so.8
#24 0x00007ef9ef507528 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#25 0x00007ef9ef4f8b23 in ?? () from /lib/x86_64-linux-gnu/libcuda.so
#26 0x00007efaec69caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#27 0x00007efaec729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

[wence-]

RMM doesn't drop the gil when calling into the C++ layer nearly as often as it should. I think perhaps this might fix things:

diff --git a/python/rmm/rmm/pylibrmm/device_buffer.pyx b/python/rmm/rmm/pylibrmm/device_buffer.pyx
index 0d6825ed..d02197e0 100644
--- a/python/rmm/rmm/pylibrmm/device_buffer.pyx
+++ b/python/rmm/rmm/pylibrmm/device_buffer.pyx
@@ -106,6 +106,10 @@ cdef class DeviceBuffer:
                 if stream.c_is_default():
                     stream.c_synchronize()
 
+    def __dealloc__(self):
+        with nogil:
+            self.c_obj.reset()
+
     def __len__(self):
         return self.size
 

And rebuilding rmm

[gmarkall]

I have noticed that this deadlock seems to occur quite consistently when using RMM and the CUDA bindings, so I should be able to test that shortly.

[gmarkall]

@wence- Your fix does seem to resolve the deadlock, at least for my testing leaving it running for 30 minutes repeatedly (previously it would deadlock within seconds of starting running tests over and over).

[kkraus14]

@wence- Your fix does seem to resolve the deadlock, at least for my testing leaving it running for 30 minutes repeatedly (previously it would deadlock within seconds of starting running tests over and over).

@gmarkall if the patch fixes the deadlock, then the root cause of the issue is likely that we're trying to call a Python function via a CUDA callback or a cuLaunchHostFunc or something like that. This is generally unsafe for us to be doing and we should fix that in Numba-CUDA, otherwise if anyone ever calls a CUDA API without releasing the GIL we run the risk of this same deadlock.

[kkraus14]

I think the tests that are deadlocking are related to our usage of callbacks for async functions:

numba-cuda/numba_cuda/numba/cuda/cudadrv/driver.py

Line 2440 in 20a2e3b

self.add_callback(callback, future)

These Stream callbacks are calling Python functions which require the GIL and will do so while holding the CUDA driver lock in a CUDA driver managed thread. Then if another thread, i.e. the main Python thread, is ever trying to acquire the CUDA driver lock without releasing the GIL, it leads to a potential deadlock, which I believe is what we're seeing here.

I think this is popping up now because of known issues where cuda.bindings doesn't currently release the GIL in all of its APIs. We plan to have this fixed for the next cuda.bindings release.

I'm not sure on a shorter term fix here other than to avoid using CUDA callbacks in numba-cuda.

[kkraus14]

Had an offline conversation with @gmarkall on this issue:

• The above tracebacks are from the test_parallel_tasks and only deadlock when running with RMM which we don't currently do in CI. This is why we haven't seen this issue before. We believe this deadlock is irrespective of cuda.bindings vs ctypes and is a lock ordering issue across threads with the CUDA driver lock and the GIL. @wence-'s patch above should potentially resolve this issue when using RMM, but in general if anyone is using a Python library that calls into CUDA without releasing the GIL, there's a potential deadlock because of how we're adding Python functions as CUDA Stream callbacks.
• In test_multiple_async_done_multiple_streams @gmarkall indicated that the traceback(s) showed cuda.bindings in a deadlock related to the GIL. cuda.bindings doesn't unconditionally release the GIL currently, but that will be changed in the next release of cuda.bindings which should resolve this issue.
• In general, as long as there's the GIL, there's potentially for lock ordering issues with the GIL and CUDA where I'll open a PR adding documentation to the Stream.add_callback and Stream.async_done methods indicating the possibility of deadlock if you're using a Python library that makes CUDA calls without releasing the GIL.