Is it possible to use external compilers in autogen?

[PrashantSaikia]

If I ask autogen to generate python code, and go back and forth until all the bugs in the code are corrected, it can do that. But is the same possible for other languages that are not "embedded" into autogen? Say I ask it to write a shell script to do something, it generates the code, but cannot go and run it as it requires an external compiler and running environment (terminal, etc). Is there a way to incorporate that, and go back and forth until all the bugs in the shell script are corrected?

[jackgerrits]

Currently both shell and python code are supported.

@ekzhu is working on adding support for new code executors, as well as the ability to build custom executors. So with this you could build an executor that capable of executing any other language.

#1405

[PrashantSaikia]

Is there a way I could make it execute an external executor though? I understand it can work with whatever executors are "embedded" in Autogen, but my task involves generating code in, say, Ruby or PHP which most likely you will never support. So, is there a way I could make it take the following steps:

1. generate code in language XYZ
2. save the code in a file called filename.xyz
3. start the compiler of XYZ language from terminal/cmd
4. run the compiler against the generated code with a shell script (containing my instructions on how to run the compiler), something like: "./run_compiler.sh filename.xyz"
5. if the code runs without any error, halt. If it produces a compiler error, copy the error from terminal/cmd, paste it as input to autogen, ask autogen to fix the bug and regenerate the code.
6. run steps 2 through 5 in a loop

If this is possible, that would be awesome!

[ekzhu]

@PrashantSaikia you might want to see if you can implement your own executor following the CodeExecutor interface that works for your language. References can be found in the autogen.coding module. If you want to wait for some one else to implement it, please join our Discord and comment on #agent-coding channel so you can be notified when a new code execution feature gets added.

[PrashantSaikia]

@ekzhu Thanks for shedding light on the CodeExecutor interface. I found this script, but I'm not sure how to use it. Would you be able to kindly show a minimal example? For example, say I want to run this line in CMD/terminal: ./run_compiler.sh filename.xyz, where filename.xyz is the code produced by the LLM. How do I do that with this script, and then do the back and forth process of fixing errors and regenerating LLM responses?

[ekzhu]

1. Place your run_compiler.sh file in a working directory you have created.
2. Start you user proxy agent (assuming you are doing a one-on-one chat with an assistant agent) with the following setting:

proxy = UserProxyAgent("user proxy", code_execution_config={"executor" : "commandline-local", "commandline-local": {"work_dir" : "your working directory path"}}, human_input_mode="ALWAYS") 

assistant = AssistantAgent("assistant", llm_config=..., system_message="<you need to tune your system message here, based on the default one but adding the instruction about how to use run_compiler.sh>")

3. Initiate chat

proxy.initiate_chat(assistant, message="The task you want to do")

[ekzhu]

Oh, I forgot to mention, the command line code executor currently only supports Python. It takes a little work to make it support compiled-time languages -- by adding some additional handling for other languages. You can use the existing handling for Python as example.

See: https://github.com/microsoft/autogen/blob/main/autogen/coding/local_commandline_code_executor.py#L141. You can add additional handling after the Python part.

So, my example above is assuming the compiled-time languages have been added. Would love to get some help on this.

[PrashantSaikia]

Hi @ekzhu, I had already tried this, and in fact autogen with human_input_mode="ALWAYS" was the first thing I tried when I started this task. But, to get it to do this task with human_input_mode="NEVER" is the goal. That is, full automation of this back-and-forth process.

Here is my workaround to this, for my specific use case, that I built yesterday. Not sure if it can be readily adapted to make it a part of autogen, but it does automate the process of taking the LLM generated code, runs it against the external language compiler, takes the output (error, etc) from the compiler, pastes it back into the LLM, takes the regenerated code and continues this process until no more compiler errors are encountered.

If you think this can be adapted to make it a part of autogen, please let me know and I'd be more than happy to contribute.

[jackgerrits]

The custom executor PR will allow for code blocks in any language to be processed in any way that you want including compilation and execution. So you could have a markdown code block in Ruby and have the executor execute that code. This PR is not far from being merged so keep an eye on this space. We're actively trying to improve the code execution functionality.

Today though, you could technically use shell/python to generate files containing the content in the other language and get it to generate the required commands to compile/run. However, you will need to spend quite some time crafting the prompt to get it to work reliably I would imagine.