Seeking help: Real-Time Message Streaming Issue with GroupChatManager in AutoGen Framework

[kalensr]

Hello all,

I'm seeking some assistance and have posted a pretty thorough description in Reddit. AutoGenAI group. But I have not had any good suggestions yet.

This is my first post in this forum. So, if it is preferred, I can repost my problem here, as opposed to reference Reddit. Just let me know. I'd appreciate some help in this space. I've been fighting through this for a week now.

For my context, I'm using pyautogen.

Thanks in advance for any help that you can provide.

link: Detailed Description of my problem.

[kalensr]

My research has surfaced the following. Can this be confirmed? Are there any workarounds to this?

Based on the research and understanding of how AutoGen handles streaming and WebSockets,

it appears that the user_proxy.initiate_chat method is blocking because it is designed to complete the conversation before returning control.

This behavior occurs despite setting llm_config["stream"] = True, which usually enables incremental output. Here are some insights and potential solutions:
Insights on Streaming and Blocking Behavior

Streaming Configuration:
Setting llm_config["stream"] = True is intended to enable the model to stream its responses incrementally. However, this setting alone might not affect the blocking nature of the initiate_chat method if the method itself is not designed to handle asynchronous or streaming operations.

IOStream Usage:
The use of IOStream in AutoGen suggests that output is being managed through a specific interface, which might not be fully integrated with asynchronous WebSocket operations. This could lead to blocking if the initiate_chat method waits for complete messages before processing further.

WebSocket Integration:
WebSockets are inherently asynchronous, but if the underlying methods called within initiate_chat are synchronous, they will block until completion.

What's interesting is even tried to register all agents to agent.regster_reply() using the following:

for agent in agents:
        agent.register_reply(
            trigger=[ConversableAgent, None],  # Trigger for any ConversableAgent or None
            reply_func=utils.queue_messages,   # The function to call, adds message to the queue, in utils.py
            config=None                        # No additional config needed
        )

where utils.queue_messages is an asyncio Queue, with async write and read ops. And still user_proxy.initiate_chat, blocks. queue messages are successfully read and streamed to the client, but only after the conversation has completed.

Can anyone help me here?

[kalensr]

Update: Although I was not able to make this work with WebSockets, I have had success using StreamingResponse, an ascyncio message queue, and setting up all agents to register with an async reply function that writes to the queue. I'm now able to stream the intermittent agent chat messages during the agent conversation back to the client via FastAPI from the message queue, realtime.

This is my second time trying this implementation, but this time I had set 'stream':True, in the llm_config.

All is well here now.

[ekzhu]

If you are just started, and needs build front end integration. Consider take a look at the v0.4 preview. The libraries are natively async and you can create agent that uses the model client's create_stream method to get an async iterator for incremental output to be written to a web socket.

See the core library: https://microsoft.github.io/autogen/dev/user-guide/core-user-guide/index.html

The agent chat layer will have similar feature but the core is much more stable.

[jackgerrits]

Yes, in 0.2 user_proxy.initiate_chat is a blocking call and doesn't return until the conversation is resolved. Intermediate output is done via the IOStream interface as you pointed out. Enabling stream dictates whether results that make it to that interface are intermediate/streamed or in complete units.

I agree with @ekzhu, I think you'll be way better served by 0.4 with what you're trying to do

[sundar-sarvam]

Is there an example on how to use v0.4 with websockets?

[rysweet]

@ekzhu @jackgerrits