FastAPI autogen asynchronous session not working

[gvijqb]

Hey guys. I am trying to implement an asynchronous thread based workflow for handling multiple chat sessions in parallel. But my fast API stalls when a code is being executed or a function is being run in a thread by autogen groupchat. The API starts giving 504 gateway timeout.

How should I handle this?

[rysweet]

@ekzhu any thoughts?

[ekzhu]

I am sorry can you provide code, model, package version, Python version

[ekzhu]

Do you mean running code execution as a tool? You can certainly wrap the executor into a function and use it s a tool

[gvijqb]

Do you mean running code execution as a tool? You can certainly wrap the executor into a function and use it s a tool

As a tool it can be helpful but it won't enable the agent to monitor what's happening in the code execution. Monitoring is especially helpful in long running code execution that might take 5-6 hours.

[ekzhu]

In that case the code execution should be performed on a long-running container that emits events to be tracked by some agents, which then make decisions such as whether to cancel the operations or send further events to the code execution container.

Ideally, your code execution container should be hosted in a separate service. Sounds like a long-running data processing or machine learning training stuff to me.

[gvijqb]

I see. That's interesting. So you're suggesting that I have a separate docker container that has like a pub-sub architecture workflow for publishing messages as the code snippets are getting executed? And the container itself has a subscriber collecting any messages published for code execution by an agent tool call?

[ekzhu]

Yes. For long running tasks I think it should be separate to avoid failure bringing down the whole app