CurlAsyncHTTPClient hangs

[saltedsword]

It hangs when executing method fetch with CurlAsyncHTTPClient. I tried working on CentOS, Python 3.7 ~ 3.9, but the behavior was the same.

However, when i use SimpleAsyncHTTPClient, it works fine.

Software environment:

• OS: Ubuntu 20.04.4 LTS
• Python: 3.8.13 (miniconda)
• tornado: 6.2
• pycurl: 7.45.1
• curl:

curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Here is my test code:

import asyncio
from tornado.httpclient import AsyncHTTPClient
AsyncHTTPClient.configure('tornado.curl_httpclient.CurlAsyncHTTPClient', max_clients=10)

http_client = AsyncHTTPClient()
async def main():
  print('fetch...')
  res = await http_client.fetch('http://localhost')
  print('res', len(res.body))

asyncio.run(main())

I wonder if my code is wrong, since no one mentioned the same issue.

[bdarnell]

Your code looks fine to me. It's not quite the same behavior but I wonder if this could be related to #3131 (different versions of everything, though).

Try enabling asyncio's debug mode and setting the logging level to debug to see more about what's going on.

[alex-ip]

G'day - I came up against the same problem. It had me stumped, because I could run the sample code for the e web-spider example in https://buildmedia.readthedocs.org/media/pdf/tornado/latest/tornado.pdf with no issues.

What I found was that when I pre-instantiated the AsyncHTTPClient, the fetch failed, but when I instantiated it on the fly, it worked.

The modified version of @saltedsword's code that now works is as follows:

import asyncio
import logging
from tornado.httpclient import AsyncHTTPClient
AsyncHTTPClient.configure('tornado.curl_httpclient.CurlAsyncHTTPClient', max_clients=10)

LOG_LEVEL = logging.DEBUG
root_logger = logging.getLogger()
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s"))
root_logger.addHandler(handler)
root_logger.setLevel(LOG_LEVEL)

logger = logging.getLogger(__name__)
logger.setLevel(LOG_LEVEL)

tornado_logger = logging.getLogger('tornado.curl_httpclient')
tornado_logger.setLevel(LOG_LEVEL)

asyncio_logger = logging.getLogger("asyncio")
asyncio_logger.setLevel(LOG_LEVEL)

url = "http://www.tornadoweb.org/en/stable/"

http_client = AsyncHTTPClient()
async def main():
  print(f'fetching {url}')
  #res = await http_client.fetch(url)
  res = await AsyncHTTPClient().fetch(url)
  print('res', len(res.body))

asyncio.run(main(), debug=True)

It looks to me like the instantiation only works when there is an open, running IOloop, i.e. if it happens inside a running coroutine.