Testing AutoGen CompressibleAgent or AutoGen + MemGPT for Token Limit Issues

[yiranwu0]

Seeking Collaborators to Tackle Token Limit Issues in AutoGen

Are you encountering token count limit issues with AutoGen? We need your insights and experiences! Our goal is to refine solutions that effectively address these challenges, and your contribution could be pivotal.

Join Our Experimentation Effort

We are actively experimenting with different strategies to manage the token count overflow. To that end, we’ve developed some promising tools, and we invite you to test and provide feedback:

• CompressibleAgent: Replacing AssistantAgent with CompressibleAgent to enable compression when a pre-set token limit is reached. Dive into the example here: agentchat_compression.ipynb

• CompressibleGroupManager ([Core] Compression in GroupChat #497): Replacing GroupManager with CompressibleGroupManager so that the group manager can compress messages and broadcast to all agents in the group. Explore how it works: agentchat_groupchat_compression.ipynb

Pull from #497 to try out both of them!

• Exploring MemGPT + AutoGen: MemGPT is a great solution to this problem and the MemGPT team has developed an integration for AutoGen. We might want to integrate MemGPT in the future. Check out the MemGPT Agent: MemGPT+AutoGen.

We Want to Hear From You!

• Try out CompressibleAgent or CompressibleGroupManager? Share your outcomes!
• Are you a MemGPT user? How do you like the MemGPT agent?
• Are you facing token limit issues? What is your current solution and your needs?

[kfsone]

I'm fairly sure that the future is ollama + memgpt + autogen, and some of the bigger token models like vicuna 16k. I hope that whomever glues this all together does it with Powershell. I may have been a shell evangelist for 35 years-or-so but using shell to manage python runing lms on diff platforms is too much like using forth to manage perl to implement a gui for my old bones :) Powershell core is open source and multi-platform and powershell remoting is going to be transformative to empowering people working outside of code to operate lms, so long as someone doesn't come along and make it all platform specific... coughapplecough

[jguo1002]

Hi Kevin, thank you for making the CompressibleAgent. It's performing remarkably well. When used solely for compressing the current conversation without relying on an external database, I found it quite similar to the native MemGPT agent.

One suggestion: it would be great if there are various compression modes to choose from. The current default method operates at the message level, which may trigger compression every time I input a new message. The MemGPT approach summarizes at the conversation level, so the summarization is shorter.

The group chat of Autogen + MemGPT is a bit messy, particularly because not all agents are equipped with MemGPT's memory mechanism. They are confused about which agents have archival memory capabilities lol…

[yiranwu0]

Hello there, thank you for trying out the compressible agent!

I will think of more ways to put in more modes to handle token limit. If you checkout out Example 3 of https://github.com/microsoft/autogen/blob/groupcompress/notebook/agentchat_compression.ipynb, you can see there is an option to customize your own function to prune the messages, if the defaults are not desirable. If you find your own customized method very useful, it would be great if you can contribute or share it with us, so it could be added as a default option for others to use!

If the compression is triggered too often, you might want to raise the trigger count. With the COMPRESSION mode, you can lower the last_n_messages to compress more messages at a time (if that is desirable).

For GroupChat, the CompressibleGroupManager PR is still on-going and sorry that it is dragging through a long time. Use MemGPT in a GroupChat is definitely messy just as you said, it requires more thoughts to have a generalized memory for the whole group, and that will be an interesting topic to explore.