New features:
The Query object now supports two new methods for easier evaluation of your retrieval pipeline:
query.get_runs()returns a dictionary of TREC-style runs for all the agents that retrieved documents for that query. (the mapping is agent_id -> query_id->document_id->retrieval_score).query.get_qrels()returns a TREC-style qrels dictionary with the judgement scores assigned by an Evaluator. The mapping is query_id->document_id->relevance).
You can explore how these two methods work in the new example notebook here that uses the ir-measures package.
Another addition (by @RodrJ106) is the addition of a new LLMProvider for Ollama! Now you can also run RAGElo on locally, without the need to call an external provider. Thanks!
A potential breaking change is that the retrieved_docs and the answers attributes of the Query object are now dictionaries instead of lists (mapping the document id or the agent name, respectively, to the actual object). This was done to better support future changes where RAGElo relies less on CSV files everywhere, but instead saves and serializes its internal state as a dictionary, until the user actually asks for an output as a CSV.
What's Changed
- Add missing f-string to warning. by @RodrJ106 in #38
- Add ollama as new llm provider by @RodrJ106 in #39
- Remove extra domain sentence by @din0s in #40
- Add get_qrels and get_runs for Queries by @ArthurCamara in #41
New Contributors
Full Changelog: 0.1.7...0.1.8
