Is that possible to read pandas dataframe from memory and analyze via Autogen?

[mraguth]

I have a pandas data frame in memory. However, I am not sure how I read the data frame via Autogen agents and analyze or build later steps. Is that possible? please help.

[sonichi]

Do you mean you have a df in a jupyter notebook environment?

[ekzhu]

I think you may find the ipython-embedded executor useful. It will be released in the next version -- you will need to update your assistant agent's system prompt with instruction on how to use the new executor.

proxy = UserProxyAgent("proxy", code_execution_config={"executor" : "ipython-embedded"})

There is an example system prompt in autogen.coding.embedded_ipython_code_executor.py you can use but you may want to tune it for your specific use case.

You need to install autogen with IPython dependencies pip install "pyautogen[ipython]".

If you have a working use case, please share it on our Discord #agent-coding channel, and submit a PR to update our example and/or gallery page.

[abhijithnair1]

@ekzhu Any update on when we can expect the next version? Can you help how I can install the library from source in the meantime? I couldn't find steps on the main docs. Would just cloning the repo, doing python setup.py install does the job?

Yes once I have a working example, will share it on the discord channel.

[sonichi]

v0.2.13 has been released to pypi.

[abhijithnair1]

@sonichi @ekzhu Thank you. This idea of stateful execution is really good and thank you for all the work related to this.

[ekzhu]

You are welcome @abhijithnair1. This is an open source effort, so your PR for further enhancement is appreciated.

[mraguth]

Yes

…

On Mon, Oct 16, 2023 at 3:45 PM Chi Wang ***@***.***> wrote: Do you mean you have a df in a jupyter notebook environment? — Reply to this email directly, view it on GitHub <#261 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5IT75WAGXOWOKZSOJUP2NDX7WMHHAVCNFSM6AAAAAA6CXUJHOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TEOJXGIYDO> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[sonichi]

You can use the IPythonUserProxyAgent in https://github.com/microsoft/autogen/blob/main/notebook/agentchat_auto_feedback_from_code_execution.ipynb
The last section to execute code in the notebook environment.

[mraguth]

How I call my vscode dataframe in the prompt? Sorry I am new and please help. My screen shot is below. (heere my dataset name is 'sp1'

[huangqingyi-code]

Has this problem been solved?

[ekzhu]

The code execution module currently doesn't support persisted session. So each time the code will be executed from a new Python session.

However, there is a way to do this using function calls, with our latest release.

df = Dataframe(...)

assistant = ConversableAgent(name="assistant", system_message=f"You are AI assistant for data analysis, here is a dataframe you can query using the function df_query. The dataframe has the following columns: {df.columns} and here is a preview: {df.head()}.")
user_proxy = UserProxyAgent(name="user proxy")

@assistant.register_for_llm(description="Function to query the dataframe")
@user_proxy.register_for_execution()
def df_query(query: Annotated[str, "The query string used for df.query()"]) -> Annotated[str, "Output"]:
  return str(df.query(query))

user_proxy.initiate_chat(assistant, "Hey please tell me what is the maximum value of ...")

[mraguth]

Thank you and I will try. Is that works with pyautogen 0.20?

[sonichi]

Found an interesting demo about this from discord: https://www.youtube.com/watch?v=iF2RguoqmrA

[olimoz]

The demo was done with this...
https://github.com/olimoz/AI_Teams_AutoGen/blob/main/JupyterNotebooksForAutoGen.ipynb

Code is fairly rough, a Sunday afternoon project. It creates an executor for Jupyter Notebooks along the same lines as the autogen example for Chess. The Jupyter notebook effectively becomes a chessboard where agents take turns to write code/markdown into notebook cells.

Bear in mind that it was a Sunday afternoon project. So, beware...

1. Uses a local python kernel, not Docker.
2. Needs a python environment which has all the potential packages already installed.

• It will use the same conda environment as the code which instantiates the python kernel, so that is the environment to get the packages installed to.

Good news is, it creates a stateful environment, objects persist from one notebook cell to the next.

When the team's code leads to errors then the team overwrite that notebook cell with their corrections. So, we don't pay to retain the errored code in our subsequent tokens to OpenAI, and the final product is then error free. Hence no code errors in the youtube video.

The notebook is created in the background and the cells execute quietly in the background. The demo on youtube was created by reading a finalised notebook written by the agent team to VSCode via pyautogui. That then executes live for the video.

We could tie all that together, making a single code base which writes into vscode in front of the users' eyes. It would be pretty, but safer and easier to write into the background. The video was intended to make people aware of what autogen can do, not intended to be functionality.

[sonichi]

Love the idea.
@ekzhu @qingyun-wu @afourney