Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for Index-Based Retrieval for ColBERT #1593

Open
sam-hey opened this issue Dec 14, 2024 · 0 comments
Open

Add Support for Index-Based Retrieval for ColBERT #1593

sam-hey opened this issue Dec 14, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@sam-hey
Copy link
Contributor

sam-hey commented Dec 14, 2024

The current implementation of ColBERT only utilizes the MaxSim operator for retrieval and does not leverage an index. While this works for smaller datasets, it is not scalable for large datasets, as the resource consumption becomes impractical.

To enable efficient retrieval for large-scale datasets, it is essential to integrate an indexing mechanism. This will ensure that resource consumption remains manageable while maintaining retrieval performance.

PyLate plans to introduce the PLAID Index to address this limitation. This index is designed to optimize retrieval tasks for larger datasets. For details on the current progress, refer to the PyLate Issue.

@KennethEnevoldsen KennethEnevoldsen added the enhancement New feature or request label Dec 14, 2024
@KennethEnevoldsen KennethEnevoldsen changed the title Enhancement: Add Support for Index-Based Retrieval for ColBERT Add Support for Index-Based Retrieval for ColBERT Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants