Feature request: Add Asynchronous Message Queue Support with Rate Limiting

[kaustavbecs]

Use case

Regular enterprise use cases require support for spiky workloads without violating rate limits from LLM Providers. The Multi Agent Orchestrator should be able to:

1. Store the incoming messages in a queue
2. Check the classifier to identify top LLMs for the request
3. Check the rate limits and current consumption
4. Request the appropriate LLM endpoint
5. Receive the response and send it back to the requestor via Websockets

Solution/User Experience

The Multi Agent Orchestrator should be able to:

1. Store the incoming messages in a queue
2. Check the classifier to identify top LLMs for the request
3. Check the rate limits and current consumption
4. Request the appropriate LLM endpoint
5. Receive the response and send it back to the requestor via Websockets

Alternative solutions

Frameworks such as LLamaIndex supports async mode - but that targeted towards async IO non blocking http calls only. For a comprehensive enterprise grade Multi Agent Orchestrator solution, we need a custom solution

[cornelcroi]

Hi @kaustavbecs, thank you for the proposition.

To have a async behaviour I think the best way is to handle this per agent (not at orchestrator level), because each agent is not necessary an LLM and because each agent could have different limits.
To implement this you could create a custom agent and handle the asynchronous communication with the LLM inside.
I could see this being implemented in a LambdaAgent for example and this lambda will have the asynchronous communication with the LLM by using queues or any other mechanism.
It would be interesting to create such agent to be part of built in agents.
Happy to review a solution before you can start the implementation if you want to contribute to this repostitory.