Which embedding model should you use?

Table of contents

• Problem ❗
• Solution 🧠
• External services 🛠️
• Run the code 👨🏻‍💻
• Video lecture 🎬
• Wanna learn more real-time ML? 💡

Problem

Text embeddings are vector representations of raw text that you compute using an embedding model

These vectors representations are then used for downstream tasks, like

• Classification → for example, to classify tweet sentiment as either positive or negative.

• Clustering → for example, to automatically group news into topics.

• Retrieval → for example, to find similar documents to a given query.

Retrieval (the “R” in RAG) is the task of finding the most relevant documents given an input query. This is one of the most popular usages of embeddings these days, and the one we focus on in this repository.

There are many embedding models, both open and proprietary, so the question is:

What embedding model is best for you problem? 🤔

In this repository you can find an evaluation script that helps you find the right embedding model for your use case.

Solution

To evaluate a model for retrieval using a particular dataset we will

1. Load the model and your dataset from HuggingFace, with

   • questions
   • contexts, and
   • correct answers

2. Embed the context into the Vector DB, in our case Qdrant.

3. For each question retrieve the top K relevant documents from the Vector DB, and

4. Compare the overlap in information between the retrieved documents and the correct answers. We will use ragas, an open-source framework for RAG evalution, to compute context precision and context recall

5. Finally log the results, so you know what worked best.

External services

You will need

• An OpenAI API key, because ragas will be making calls to GPT-3.5 Turbo to evaluate the context precision and recall.

• A Qdrant Vector DB with its corresponding URL and API key, which you can get for FREE by signing up here

Run the code

1. Create an .env file and paste your OPENAI_API_KEY, QDRANT_URL and QDRANT_API_KEY

   $ cp .env.example .env
   

2. Create the virtual environment with Python Poetry

   $ make install
   

3. Update the list of models and datasets you want to test in config.yml

4. Run the evaluations

   $ make run-evals
   

Video lecture

Wanna learn more Real World ML?

Join 15k builders to the Real-World ML Newsletter.

Every Saturday morning.

For FREE.

→ Subscribe TODAY 🤗