deadlock while using hazmat #2911
Description
Deadlock is reproducible while using tornado https and requests. One thread is trying to perform handshake, acquires CRYPTO_LOCK_RSA lock and is trying to call to Cryptography_rand_bytes which required GIL. Another thread is keeping GIL and trying to execute PySSL_dealloc, which required locking of CRYPTO_LOCK_RSA.
Here is a repro:
import asyncio
import cryptography.hazmat.backends.openssl.backend
import requests
import tornado.platform.asyncio
import tornado.web
import tornado.httpserver
# Will cause deadlock
cryptography.hazmat.backends.openssl.backend.activate_osrandom_engine()
# Will fix deadlock
#cryptography.hazmat.backends.openssl.backend.activate_builtin_random()
# Port for local server
_PORT = 10001
# local server
class Hello(tornado.web.RequestHandler):
def get(self):
self.write("hello")
application = tornado.web.Application([('/', Hello)])
http_server = tornado.httpserver.HTTPServer(application, ssl_options={
"certfile": "/tmp/cert.pem"
})
tornado.platform.asyncio.AsyncIOMainLoop().install()
http_server.listen(_PORT)
@asyncio.coroutine
def tester():
while True:
yield from asyncio.get_event_loop().run_in_executor(
None, lambda: requests.get(
'https://127.0.0.1:{}'.format(_PORT), verify=False))
# start stressing
for _ in range(11):
asyncio.ensure_future(tester())
tornado.ioloop.IOLoop.instance().start()Callstacks:
Thread #1
(gdb) bt
#0 0x00007f35b3828790 in sem_wait () from /lib64/libpthread.so.0
#1 0x000000000052d77e in PyThread_acquire_lock_timed (lock=0x113dcc0, microseconds=<optimized out>, intr_flag=0) at Python/thread_pthread.h:352
#2 0x00007f35aabb9d97 in CRYPTO_add_lock () from /lib64/libcrypto.so.10
#3 0x00007f35aac154d9 in RSA_free () from /lib64/libcrypto.so.10
#4 0x00007f35aac410db in EVP_PKEY_free_it () from /lib64/libcrypto.so.10
#5 0x00007f35aac41818 in EVP_PKEY_free () from /lib64/libcrypto.so.10
#6 0x00007f35aac50b60 in pubkey_cb () from /lib64/libcrypto.so.10
#7 0x00007f35aac5646a in asn1_item_combine_free () from /lib64/libcrypto.so.10
#8 0x00007f35aac566ff in ASN1_template_free () from /lib64/libcrypto.so.10
#9 0x00007f35aac5658a in asn1_item_combine_free () from /lib64/libcrypto.so.10
#10 0x00007f35aac566ff in ASN1_template_free () from /lib64/libcrypto.so.10
#11 0x00007f35aac5658a in asn1_item_combine_free () from /lib64/libcrypto.so.10
#12 0x00007f35aac56645 in ASN1_item_free () from /lib64/libcrypto.so.10
#13 0x00007f35aaf7ff48 in SSL_SESSION_free () from /lib64/libssl.so.10
#14 0x00007f35aaf7e2cd in SSL_free () from /lib64/libssl.so.10
#15 0x00007f35a8374a40 in PySSL_dealloc (self=0x7f35a4c6fb20) at /home/alyukhan/Python-3.4.4/Modules/_ssl.c:1481
#16 0x000000000045f44b in insertdict (mp=0x7f35a43c85c8, key=0x7f35a85fb810, hash=3968200442044068234, value=0x8496a0 <_Py_NoneStruct>) at Objects/dictobject.c:824
RSA_free call CRYPTO_add_lock on CRYPTO_LOCK_RSA
...
Thread #2
(gdb) bt
#0 0x00007f35b3826a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000004dafdf in PyCOND_TIMEDWAIT (cond=0x897340 <gil_cond>, mut=0x897300 <gil_mutex>, us=5000) at Python/condvar.h:103
#2 take_gil (tstate=tstate@entry=0xef74e0) at Python/ceval_gil.h:224
#3 0x00000000004dbc39 in PyEval_RestoreThread (tstate=0xef74e0) at Python/ceval.c:448
#4 0x00007f35a6aba6d5 in gil_ensure () at c/misc_thread_common.h:104
#5 cffi_call_python (externpy=0x7f35a631a160 <_cffi_externpy__Cryptography_rand_status>, args=0x7ffdc9dacae0 "\340\365i\001") at c/call_python.c:234
#6 0x00007f35a607afb4 in Cryptography_rand_status () at build/temp.linux-x86_64-3.4/_openssl.c:12097
#7 0x00007f35aac1ab9e in RSA_setup_blinding () from /lib64/libcrypto.so.10
#8 0x00007f35aac128cc in rsa_get_blinding () from /lib64/libcrypto.so.10
#9 0x00007f35aac13c3c in RSA_eay_private_encrypt () from /lib64/libcrypto.so.10
#10 0x00007f35aac1596e in RSA_sign () from /lib64/libcrypto.so.10
#11 0x00007f35aac1a7dd in pkey_rsa_sign () from /lib64/libcrypto.so.10
#12 0x00007f35aac40d7e in EVP_SignFinal () from /lib64/libcrypto.so.10
#13 0x00007f35aaf57933 in ssl3_send_server_key_exchange () from /lib64/libssl.so.10
#14 0x00007f35aaf5b5d2 in ssl3_accept () from /lib64/libssl.so.10
#15 0x00007f35aaf690c8 in ssl23_accept () from /lib64/libssl.so.10
#16 0x00007f35a837a0a9 in PySSL_SSLdo_handshake (self=0x7f35a43d1fb8) at /home/alyukhan/Python-3.4.4/Modules/_ssl.c:570
Thread #2 obtained CRYPTO_LOCK_RSA at crypto/rsa/rsa_eay.c in rsa_get_blinding and making RSA_setup_blinding call under lock