Using pre-tokenized queries / documents does not work at the moment

[mam10eks]

This commit adds some failing unit tests: 4a747d4

Should be simple to resolve this. We load the term-pipeline from the terrier index which we implemented at a time when the pre-tokenized feature was not yet available in PyTerrier, so we likely have a wrong pipeline in case pre-tokenized is specified.

[mam10eks]

cc @Parry-Parry, @heinrichreimer.

[mam10eks]

Alright, for pretokenized indexes, termpipelines= is in the index/data.properties file, and in this case ir_axioms uses a default term-pipeline that applies some normalization.

@heinrichreimer Do you have any preferences how we could solve this? E.g., so that it is usable but maybe still compatible with previous behaviour?

[Parry-Parry]

@heinrichreimer @mam10eks So I assume the default pipe is stopwords, porter stemmer, this is always included in data.properties should shouldn't be an issue in the default case

[mam10eks]

one possible suggestion could also be that we introduce a new PreTokenizedTerrierIndexContext that is a TerrierIndexContext and jst overrides the termpipeline property?

[janheinrichmerker]

I'd say it would be best to fix this in the PyTerrier backend here:

ir_axioms/ir_axioms/backend/pyterrier/__init__.py

Lines 229 to 234 in 4212946

	def terms(
	self,
	query_or_document: Union[Query, Document]
	) -> Sequence[str]:
	text = self.contents(query_or_document)
	return self._terms(text)

Is there a PyTerrier API to access the pre-tokenized terms given the document ID?