Switch to langchain

[Yiannis128]

Consider switching to langchain. Need to consider the positives and negatives. This requires some replacement of the backend, specifically in the BaseChatInterface class.

Requirements

• Use LLM as base class for all LLM models (currently OpenAI models and Falcon).
• Integrate HumanMessage, SystemMessage, and AIMessage inside of BaseChatInterface.
• Convert code extraction used by optimize-code and fix-code to be OutputParsers. #62
• RetryWithErrorOutputParser can be used in fix-code. #63
• Use ChatPromptTemplates to set the instructions of the system message.
• Use summarization in LangChain for UserChat compress message stack. Handled in Reimplement Conversation Summarizer using LangChain for ChatInterface #57.
• [Research] See if we can use Chains in place of Signal (This one is probably not applicable use case to replace Signal with)
• Switch -r or --raw-output arguments to verbose level 2 for extra output, since LangChain uses multiple aggregators of services, the concept of 'raw' output doesn't really apply.
• Add templates to falcon to increase accuracy and improve direction of responses since it's not a chat model so need to turn it into one.

[krrishdholakia]

Hey @Yiannis128 cool repo - curious, why switch to Langchain? was it to support Falcon?

[Yiannis128]

Hey @Yiannis128 cool repo - curious, why switch to Langchain? was it to support Falcon?

Hi, short answer: yes. Slightly longer answer is that it will also allow for easily adding other types of LLMs such as Google PaLM that all have different APIs. Langchain makes the issue of designing, implementing, and testing a custom interface go away.

The transition is mostly complete, with only a slight set of features left to achieve feature parity with pre-langchain ESBMC-AI.

[krrishdholakia]

Curious - would this have helped - https://github.com/BerriAI/litellm?

[Yiannis128]

Curious - would this have helped - https://github.com/BerriAI/litellm?

Nice suggestion, I will have to check it out. The current implementation of LangChain is very generic (only uses API calls for completions), so, in theory, I could replace the LangChain support with LiteLLM.

I would have to see if:

• LiteLLM is developed and supported regularly—wouldn't want to switch to something that is not going to be updated in the future.
• Does it contain a good API with features, LangChain is OK for this, so switching would only occur if the features offered are better.
• Is it worth it? If it's more lightweight, then maybe...

[krrishdholakia]

Any specific features you're looking for / problem you're facing with current implementation? Happy to submit a PR to help out here 😊

[Yiannis128]

Hello, sorry for the late reply. I am looking for easy interop between the different APIs, along with the following:

• Current context length measurements
• Compatibility with the newest models (makes it easier to add them into ESBMC-AI).

Currently, langchain is doing fine, however, please keep me informed :)

[krrishdholakia]

Hey @Yiannis128,

No worries. I went through the code and here's what I understand:

• You currently support OpenAI + Huggingface Text Gen API (falcon + starcoder).
• For Huggingface Text Gen API you have some logic for translating I/O before/after making the call

How do you expect users to make the call to falcon-7b or starcoder? would they have to deploy it themselves via huggingface inference api before using esbmc? both falcon-7b and starcoder are available pretty easily (1-click deploy) on other providers - e..g Baseten which also offers free credit

[krrishdholakia]

https://app.baseten.co/explore

[Yiannis128]

Hey @Yiannis128,

No worries. I went through the code and here's what I understand:

• You currently support OpenAI + Huggingface Text Gen API (falcon + starcoder).
• For Huggingface Text Gen API you have some logic for translating I/O before/after making the call

Yeah that's right, as the hugging face API is more generic due to the diverse amount of models it supports.

How do you expect users to make the call to falcon-7b or starcoder? would they have to deploy it themselves via huggingface inference api before using esbmc? both falcon-7b and starcoder are available pretty easily (1-click deploy) on other providers - e..g Baseten which also offers free credit

No need to use one click deploy for some models. The ones built-in to ESBMC-AI make calls to hugging face servers, as they're hosted free of charge. The only thing users need to provide is an API key for hugging face, as stated in the documentation.

Larger models as well as private models need to be added through the config as custom AI models that are hosted elsewhere.