Execution Observability with OpenTelemetry

[nirga]

Summary
OpenTelemetry is a CNCF standard for cloud observability. It instruments HTTP calls, DB calls, etc as traces. Together with OpenLLMetry, which provides instrumentations for LLM providers like OpenA, vector DBs like Chroma and LLamaIndex - it gives a great way to view and debug agent exceutions.

It's natively supported by all major observability platforms like Datadog, Honeycomb, New Relic and others, so if we install the SDK in the repo it allows users to see and debug executions of OpenDevin.

Motivation

Debug and see executions of OpenDevin, including LLM calls, vector DB calls, HTTP calls, and others.

Technical Design

We can offer an option to initialize an SDK, like the one provided by OpenLLMetry so if the user enables it they can view execution traces.

Alternatives to Consider

None

Additional context

[li-boxuan]

We already have #1944 that tracks costs. We could incrementally build more, and eventually, refactor them to be OpenTelemetry compatible.

[yufansong]

Just curious, is there any other popular open source solution like OpenLLMetry?

[nirga]

@li-boxuan happy to contribute this if you give me the green light :)

@yufansong OpenLLMetry is the most popular one and is fully compliant with OpenTelemetry standards (I know that, because I'm defining them these days :) )

[li-boxuan]

@nirga that would be awesome, please go ahead!

Update: sorry I didn’t realize it’s a commercial product. Would you mind sharing some plans first before you start any dev work?

[yufansong]

@nirga that would be awesome, please go ahead!

Update: sorry I didn’t realize it’s a commercial product. Would you mind sharing some plans first before you start any dev work?

I test it locally, it seems require me to register and get the api_key before using sdk. If in this case, I think maybe not suitable to intergate into opendevin. Or at least I need to discuss with other core maintainers.

Also, if we try to add it into opendevin, it will collect prompts and response from user, which is also a large change need to discuss with other maintainers.

[nirga]

Hey @li-boxuan and @yufansong! it's not a commercial product. OpenLLMetry is completely open source Apache 2.0 project. The data it outputs can be connected to 20 different platforms for visualization, some of them oss (like signoz or hyperdx) and some not. Our platform, traceloop, is one of these.

See this list of integrations - https://www.traceloop.com/docs/openllmetry/integrations/introduction

The error you got @yufansong is basically asking you to connect it to some destination to see the traces.

[nirga]

@li-boxuan what are your thoughts on this?

[yufansong]

Hey @li-boxuan and @yufansong! it's not a commercial product. OpenLLMetry is completely open source Apache 2.0 project. The data it outputs can be connected to 20 different platforms for visualization, some of them oss (like signoz or hyperdx) and some not. Our platform, traceloop, is one of these.

See this list of integrations - https://www.traceloop.com/docs/openllmetry/integrations/introduction

The error you got @yufansong is basically asking you to connect it to some destination to see the traces.

Ah, sorry for the late reply. My previous problem is, when I run Traceloop.init(disable_batch=True), it will throw the following error:

Error: Missing Traceloop API key, go to https://app.traceloop.com/settings/api-keys to create one
Set the TRACELOOP_API_KEY environment variable to the key

Warning: Traceloop not initialized, make sure you call Traceloop.init()

Do you mean actually I don't need to register and can directly following like this way to avoid this initialization error?

[yufansong]

Update: I also find these two tools for LLM Observability, but it seems both of them are commercial products and no open source version can use.
https://www.helicone.ai/
https://www.langchain.com/

[nirga]

@yufansong exactly! There are 15 different platforms you can connect to, some of them (HyperDX and SigNoz) are completely OSS

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for over 30 days with no activity.

[kripper]

I'm trying Phoenix.
With LiteLLM and Vertex, I have this issue:
Arize-ai/phoenix#5813