[BUG] Filter on Parent Doc fields inside Nested knn query fails for many Query types

[krishy91]

What is the bug?

When a document contains vectors in nested documents, and we perform a nested knn query with filters set on the parent documents fields, the filters can only be specific Query types (like TermQuery). If for example a Phrase Query is specified as a filter, the knn query fails to retrieve any results at all. There are several other Query types (like exists, range etc) which also fail.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create a simple index with nested vector objects & text fields on the parent document to apply filters over
2. Index a couple of example documents
3. Perform a nested neural search or nested knn search with filter in the neural search query /knn query set on the parent document field - (let the term be an exact match with one of the doucments in index)

{
    "query_string": {
        "query": "field_standard: \"Hello World\""
    }
}

5. You can see that no results are returned

What is the expected behavior?

Evene when filters specify PhraseQuery or range query etc. the filters should be applied & results should be returned if any.

What is your host/environment?

• OS: Windows
• Version OpenSearch 2.17
• Plugins - Neural Serach & knn plugin

Do you have any additional context?

On analysis, we found that @navneet1v added the functionality to support applying filters on parent documents here: #1356

The code uses the NestedHelper.mightMatchNestedDocs method determine whether to filter is applied on Parent doucment or nested document. Unfortunately, mightMatchNestedDocs method checks for speicifc Query types individually to see if they contain "field" & check if it is present in the parent or the nested doc. This list of Query types in not complete. Many commonly uses Query types which have "field" are missing like Phrase query, Range query etc.

https://github.com/opensearch-project/OpenSearch/blob/f1c98a4da0cf6583212eecc9ed8ebc3cd426a918/server/src/main/java/org/opensearch/index/search/NestedHelper.java#L65

[krishy91]

Although this issue might have to resolved directly on NestedHelper, I wanted to know the others opinion on this issue and how to go about it. It affects the knn search & hence the Neural Search (for nested documents) directly.

[jmazanec15]

@heemin32 could you take a look at this?

[brianjyee]

I am also finding that must_not does not work.

Create index

PUT /knn
{
	"settings": {
		"index": {
			"knn": true,
			"knn.algo_param.ef_search": 100
		}
	},
	"mappings": {
		"properties": {
			"nested_field": {
				"type": "nested",
				"properties": {
					"my_vector1": {
						"type": "knn_vector",
						"dimension": 3,
						"method": {
							"name": "hnsw",
							"space_type": "l2",
							"engine": "faiss",
							"parameters": {
								"ef_construction": 128,
								"m": 24
							}
						}
					}
				}
			}
		}
	}
}

Index documents

PUT /_bulk?refresh=true
{ "index": { "_index": "knn", "_id": "1" } }
{"nested_field":[{"my_vector1":[1,1,1]},{"my_vector1":[2,2,2]},{"my_vector1":[3,3,3]}], "parking": "false"}
{ "index": { "_index": "knn", "_id": "2" } }
{"nested_field":[{"my_vector1":[10,10,10]},{"my_vector1":[11,11,11]},{"my_vector1":[12,12,12]}], "parking": "true"}
{ "index": { "_index": "knn", "_id": "3" } }
{"nested_field":[{"my_vector1":[1,1,1], "parking": "false"},{"my_vector1":[2,2,2]},{"my_vector1":[3,3,3]}]}
{ "index": { "_index": "knn", "_id": "4" } }
{"nested_field":[{"my_vector1":[10,10,10], "parking": "true"},{"my_vector1":[11,11,11]},{"my_vector1":[12,12,12]}]}

Query using must_not

GET knn/_search
{
	"query": {
		"nested": {
			"path": "nested_field",
			"query": {
				"knn": {
					"nested_field.my_vector1": {
						"vector": [
							1,
							1,
							1
						],
						"k": 2,
						"filter": {
							"bool": {
								"must_not": [
									{
										"term": {
											"parking": "false"
										}
									}
								]
							}
						}
					}
				}
			}
		}
	}
}

Should exclude id 1 but it does not.

[kristib]

I opened this ticket #2339 that was closed as a duplicate of this ticket, however I wanted to note that I am seeing the issue with range filters on non-nested vector fields. Specifically, I am seeing no hits when using a knn query range filter for a date field.

[heemin32]

Hi @kristib. I think I closed the issue to quickly. Your issue might not be a duplication of this issue as your case is not with nested field. Let me reopen it. Thanks!

[navneet1v]

@buddharajusahil will look into this.

[buddharajusahil]

Please assign this issue to me

[buddharajusahil]

Hi @krishy91 do you mind including the exact commands used to reproduce this bug? Just want to make sure I am reproducing one to one.