Why is Relevance score is always = 0?

[Vasiliy-b]

Despite the LLM providing the correct response, the documents listed in the informational panel always contain irrelevant retrievals, and their scores are always 0. Why is this happening, any guess?
*Using hybrid search

[Vasiliy-b]

Also, it seems that while using the 'complex' reasoning settings, the model is also using web search. This is not clear to me—when does it have access to web search and when does it not? Is this related to hybrid search? I thought hybrid search combines text and vector search.

Any hints on how this works? When I decompose the request, there is no sign of the model using tools like web search...

Chosen rewrite pipeline DecomposeQuestionPipeline( (llm): ChatOpenAI(api_key=sk-DmivWJqglpxu..., base_url=https://api.ope..., frequency_penalty=None, logit_bias=None, logprobs=None, max_retries=None, max_retries_=2, max_tokens=None, model=gpt-4o, n=1, organization=None, presence_penalty=None, stop=None, temperature=None, timeout=20, tool_choice=None, tools=None, top_logprobs=None, top_p=None)

[Vasiliy-b]

After adding cohere API key, relevance score is calculating propperly.

But, strange thing with web search are still there. Does model uses tools by default?

[Vasiliy-b]

Seems like when model fail to find relevant answer in docs it is going web by default.

[vap0rtranz]

But, strange thing with web search are still there. Does model uses tools by default?

Not all models support tools[], so even if Kotaemon requests a tool, the model could reject it.

Look for errors from Kotaemon app. I usually forget to switch to a model that supports tools[]. The UI does not tell you when the model fails tools[].

[vap0rtranz]

My relevance scores report non-zero when only using my local vectorstore.

Notice that my re-ranking score is the only 0.

It seems the settings that funnel into the relevance scores are scatter about the UI. For example, the checkbox for "Use re-ranking" under Settings->File Collection will attempt to call Cohere. I found that about because I saw an error in the app logs.