[BUG] Async (nested sync detected) error in UI when summarizing large/long text files

[Stego72]

Pre-check

• I have searched the existing issues and none cover this bug.

Description

Using Docker (Ollama-CUDA default settings)

A "RuntimeError: Detected nested async" is raised when trying to summarize large documents from the UI. The model is generating the output in the trace but it fail to appear in the UI - after a while the RuntimeError is raised.

It appears to be a consequence of a previous runtime error:

File "/home/worker/app/.venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
raise exc from None
File "/home/worker/app/.venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 189, in handle_async_request
await self._close_connections(closing)
File "/home/worker/app/.venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 305, in _close_connections
await connection.aclose()
File "/home/worker/app/.venv/lib/python3.11/site-packages/httpcore/_async/connection.py", line 171, in aclose
await self._connection.aclose()
File "/home/worker/app/.venv/lib/python3.11/site-packages/httpcore/_async/http11.py", line 265, in aclose
await self._network_stream.aclose()
File "/home/worker/app/.venv/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 55, in aclose
await self._stream.aclose()
File "/home/worker/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1198, in aclose
self._transport.write_eof()
File "uvloop/handles/stream.pyx", line 699, in uvloop.loop.UVStream.write_eof
File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x7fd41cdb1090>; the handler is closed

Steps to Reproduce

1. Ingest large PDF document (150 pages or more)
2. From the UI, prompt to summarize document
3. Observe response being generated in the trace
4. RuntimeError is raised after a while

Expected Behavior

Summary provided

Actual Behavior

RuntimeError

Environment

Win10, RTX4090, Docker install (vanilla Ollama-GPU) as per installation guide

Additional Information

No response

Version

No response

Setup Checklist

• Confirm that you have followed the installation instructions in the project’s documentation.
• Check that you are using the latest version of the project.
• Verify disk space availability for model storage and data processing.
• Ensure that you have the necessary permissions to run the project.

NVIDIA GPU Setup Checklist

• Check that the all CUDA dependencies are installed and are compatible with your GPU (refer to CUDA's documentation)
• Ensure an NVIDIA GPU is installed and recognized by the system (run nvidia-smi to verify).
• Ensure proper permissions are set for accessing GPU resources.
• Docker users - Verify that the NVIDIA Container Toolkit is configured correctly (e.g. run sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi)

[amida168]

I have the same problem.

[jaluma]

It is a problem related to the async implementation of Ollama in llama-index... Can you try to move to sync version until we have a solution?

[Nerdnub]

Changing async to false in settings.yaml doesn't resolve this issue for me.

[meng-hui]

I have noticed that stopping the summarize prompt in the UI also does not stop the model from generating outputs in the trace.

I'm using Ollama API