How to predict the Answer for Question answering task using FARMReader?

[karndeepsingh]

Hi,
I have fine tuned a bert model on my dataset and want to do prediction using the fin tuned model. Please, explain how I can use this fine tuned model to predict on my new set of context and its respective query without using any retriever just passing the context and queries to my trained model?

Thanks

[julian-risch]

Hi @karndeepsingh you can use the Pipeline concept of Haystack for that and create a pipeline that consists of a reader node. I suggest that you get familiar with the concept of pipelines, documents and maybe even document stores in Haystack with the help of our documentation and tutorials:
In short, you could do something like:

from haystack import Pipeline, Document
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")
p = Pipeline()
p.add_node(component=reader, name="Reader", inputs=["Query"])
res = p.run(
    query="Who is the father of Arya Stark?", documents=[Document(content="Your text here. Ned is the father of Arya.")]
)
print_answers(res, details="minimum")

[bogdankostic]

Hi @karndeepsingh, you can't set a filter to exclude answers below a certain threshold, you can only set the number of answers the Reader returns by setting the top_k value. However, you can process the returned list of answers yourself and exclude answers with scores below your threshold like this:

threshold = 0.9
answers_above_threshold = [answer for answer in res["answers"] if answer.score > threshold]

[karndeepsingh]

Hi @karndeepsingh, you can't set a filter to exclude answers below a certain threshold, you can only set the number of answers the Reader returns by setting the top_k value. However, you can process the returned list of answers yourself and exclude answers with scores below your threshold like this:

threshold = 0.9
answers_above_threshold = [answer for answer in res["answers"] if answer.score > threshold]

@bogdankostic Thanks for your answer.
I would also like to know following things please help me to understand:

1. How can we pass List of queries or questions and get answers for them in a single run? I don’t want to pass one by one query to the model and get output for it, instead in one shot I want to pass a list of queries to the model and get output for the same. How this can be achieved?
2. Once I have trained the model, how I can check the inferencing time taken to get answers from the model.

Your suggestions could help me to build a robust QA system.

Thanks

[bogdankostic]

Unfortunately, we do not provide yet the possibility to pass a list of queries to the Pipeline's run method. We plan to tackle this #1239.

We do not measure inference time inside haystack, however, you might use Python's standard library time to measure this yourself.

[karndeepsingh]

@bogdankostic Thanks for your reply.
How can we pass batch of Questions for batch of documents while doing inferencing?
As i have to deploy the trained QnA model on to the server and for faster processing I need to ask 20 questions for each batch of context lets say 5 context. How that can be done using haytstack?

[tstadel]

@karndeepsingh as @bogdankostic already mentioned currently you can only pass one question per pipeline.run. Retrieved documents are automatically processed in parallel for each question.