Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to support MS MARCO #68

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Changes to support MS MARCO #68

wants to merge 6 commits into from

Conversation

ryan-clancy
Copy link
Member

  • Add the qrels file
  • Make the measures parameter for trec_eval configurable

@ryan-clancy ryan-clancy requested review from charW and lintool May 10, 2019 15:06
searcher.py Outdated Show resolved Hide resolved
@ryan-clancy
Copy link
Member Author

@lintool How should topics be handled for MS MARCO? Currently, topic is a required parameter for the search hook - should we extract the topics file (queries.dev.small.tsv ?) and include it in jig, or make topic an optional parameter (with some collection == "msmarco" check) so the image generates the topics (like here?)

@lintool
Copy link
Member

lintool commented May 11, 2019

msmarco formatting should define a generic topic type, and the topics themselves in the tsv format should be passed in from outside the jig, i.e., in the same we pass in TREC topic files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants