Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🗺️ [tracing] Template Iteration #1164

Closed
mikeldking opened this issue Aug 21, 2023 · 1 comment
Closed

🗺️ [tracing] Template Iteration #1164

mikeldking opened this issue Aug 21, 2023 · 1 comment
Labels

Comments

@mikeldking
Copy link
Contributor

Milestone 3 - Template Iteration
Templates and how an LLM is prompted can have a deep impact on the quality. Given it's large impact, there should be tools around prompt iteration and the quality of the responses. The first step is to capture as much of the data as possible - notably the prompts embedded in frameworks like LlamaIndex and LangChain as well as prompt serving layers like Prompt Layer.

As an AI engineer, I don't know what prompts are being used by the frameworks. I want to make adjustments to the baked-in templates to improve the generation.

A clear demarkation of prompt template and variables will enable a new class of analytics, notably statistics and drift of variable values, which could un-lock a new category of analytics.

It's critical to capture all parameters that impact the generation. While this is largely available to OpenAI models, we will have to better understand the ecosystem. We should at a minimum support OpenAI, Anthropic, Cohere, OpenAI (Azure), Llama 2.

@dosubot dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Jun 7, 2024
@dosubot dosubot bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 14, 2024
@dosubot dosubot bot removed the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Jun 14, 2024
@mikeldking mikeldking reopened this Jun 15, 2024
@mikeldking
Copy link
Contributor Author

Duplicate of #3435

@mikeldking mikeldking marked this as a duplicate of #3435 Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

No branches or pull requests

1 participant