[BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results #2339

kristib · 2024-12-16T21:36:49Z

What is the bug?
We have a cluster with about 60 million documents spread across weekly indexes. The weekly indexes are all added to a "documents" alias. Each weekly index has about 1 million documents, and each doc has a faiss hnsw knn_vector field with 256 dimensions.

We upgraded to OpenSearch 2.17 last week, and some of our k-NN queries no longer return hits. We have isolated the issue to k-NN queries (that are using efficient filtering) that have a range filter for a date field.

Here is the vector field (embeddings.OAI_TE3L_256) mapping:

"embeddings": {
          "properties": {
            "OAI_TE3L_256": {
              "type": "knn_vector",
              "dimension": 256,
              "method": {
                "engine": "faiss",
                "space_type": "l2",
                "name": "hnsw",
                "parameters": {
                  "m": 32
                }
              }
            }

Here is an example query that returns 0 hits:

POST documents/_search
{
  "from": 0,
  "query": {
    "knn": {
      "embeddings.OAI_TE3L_256": {
        "vector": [
          -0.05308306,
          -0.040473435,
          -0.020121736,
          -0.024970116,
          0.017065149,
          0.0007180346,
          -0.041048344,
          0.025315061,
          -0.0696787,
          -0.04986358,
          0.11812417,
          0.004086629,
          -0.121267,
          -0.03683236,
          -0.063508034,
          -0.10087697,
          -0.1159012,
          -0.032597214,
          0.020332536,
          -0.103329904,
          0.15353844,
          -0.05373462,
          -0.000034284803,
          -0.03058504,
          0.04346295,
          0.038748715,
          -0.088458985,
          0.03915115,
          0.066957474,
          0.14510646,
          0.029665189,
          -0.0040171607,
          -0.079260476,
          -0.022766307,
          -0.06937208,
          0.0005449637,
          0.1864231,
          -0.054577816,
          0.010750757,
          0.038729552,
          0.06327807,
          0.048292167,
          0.015273356,
          0.0040890244,
          -0.07489119,
          -0.009126645,
          0.033459574,
          -0.011258591,
          -0.023954447,
          0.009629688,
          -0.03322961,
          -0.03882537,
          -0.070905164,
          0.026349893,
          0.121267,
          -0.096431024,
          0.0071336348,
          0.13782431,
          -0.034226116,
          -0.10509296,
          0.021673985,
          -0.038729552,
          -0.016672296,
          0.061859965,
          -0.06864387,
          0.12824254,
          0.030086787,
          -0.05423287,
          0.004139329,
          -0.046644103,
          0.0077324957,
          0.08370642,
          -0.091985084,
          0.084856234,
          0.017975418,
          0.0033632044,
          0.011910152,
          -0.050630122,
          0.116131164,
          -0.08439631,
          -0.030201769,
          -0.06734075,
          0.104862995,
          0.04043511,
          0.056379188,
          0.0061515025,
          0.064006284,
          -0.23870128,
          0.03117911,
          -0.0060892208,
          0.03867206,
          -0.04162325,
          0.00463279,
          -0.11774091,
          0.0556893,
          -0.07496784,
          -0.0155608095,
          -0.0087960735,
          -0.0011372374,
          -0.0022026114,
          -0.03468604,
          -0.0071384255,
          -0.010175849,
          -0.08907222,
          0.08255661,
          -0.064159594,
          -0.06829892,
          -0.056417517,
          -0.043386295,
          -0.023494523,
          -0.022517182,
          0.00840322,
          -0.038767878,
          0.054616142,
          -0.08439631,
          0.06262651,
          -0.027633851,
          -0.03644909,
          0.039017007,
          -0.05047681,
          0.026714,
          -0.045954213,
          0.10011043,
          -0.03150489,
          -0.026503202,
          0.1031766,
          -0.05756733,
          0.029933479,
          0.060135245,
          -0.17323858,
          -0.04150827,
          0.043194663,
          -0.09367147,
          -0.033574555,
          0.0029607697,
          0.032482233,
          0.034283604,
          0.010607029,
          -0.07044524,
          -0.021942275,
          -0.080486946,
          0.036219127,
          0.021348204,
          -0.005686786,
          -0.074086316,
          -0.041776557,
          -0.033267938,
          -0.056915767,
          -0.07719082,
          0.07742078,
          -0.1773779,
          -0.058295544,
          -0.052431498,
          0.030182606,
          -0.013251601,
          -0.0072294525,
          0.0696787,
          -0.042849716,
          0.019163558,
          0.061131753,
          0.074507914,
          0.095511176,
          -0.039285295,
          0.100187086,
          0.03825046,
          -0.02579415,
          0.13245852,
          0.0035692127,
          -0.022919616,
          0.11375488,
          -0.031485725,
          -0.08133014,
          -0.051818263,
          -0.07059855,
          -0.00053478306,
          0.00737797,
          0.056340862,
          -0.11697436,
          0.071556725,
          0.0053514238,
          0.1167444,
          0.042888045,
          0.025123425,
          -0.0140852155,
          -0.020351699,
          0.0119963875,
          -0.0105207935,
          -0.031428237,
          0.012695857,
          -0.009773415,
          -0.031370744,
          -0.013558217,
          -0.03526095,
          -0.03656407,
          -0.0095913615,
          0.02100326,
          0.053351346,
          0.05825722,
          -0.006208993,
          -0.03665989,
          0.07941379,
          -0.09213839,
          -0.067762345,
          -0.058218893,
          0.07174836,
          0.09443802,
          0.09006873,
          -0.006343138,
          -0.0071911253,
          -0.12555964,
          0.025698332,
          0.014554723,
          0.07339643,
          0.0067072455,
          0.02454852,
          0.030144278,
          0.01432476,
          -0.025947459,
          0.0651561,
          0.039074495,
          0.021463186,
          -0.040090166,
          -0.025104262,
          -0.04346295,
          -0.08753914,
          -0.028400393,
          0.09505125,
          0.14012393,
          -0.055804282,
          0.013059965,
          -0.022306383,
          -0.0034662087,
          -0.0038973887,
          -0.120807074,
          -0.06281815,
          -0.047525626,
          0.009073945,
          0.04411451,
          -0.062281564,
          -0.055114396,
          -0.0583722,
          0.04032013,
          0.00021514088,
          -0.018780287,
          -0.019048577,
          -0.03284634,
          -0.06856721,
          0.069985315,
          0.056839116,
          -0.02127155,
          0.06140004,
          0.028745338,
          -0.0561109,
          0.033363756,
          -0.025621679,
          0.06872052
        ],
        "k": 1000,
        "filter": {
          "bool": {
            "filter": [
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "documentType.keyword": "RAW"
                      }
                    },
                    {
                      "term": {
                        "contentLanguage.keyword": "English"
                      }
                    },
                    {
                      "exists": {
                        "field": "attachment"
                      }
                    },
                    {
                      "match_phrase_prefix": {
                        "topic.country": "Australia"
                      }
                    },
                    {
                      "range": {
                        "dateReceived": {
                          "gte": "2024-12-12T12:30:47.000Z"
                        }
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      }
    }
  }
}

The filtered subset for this query should result in 968 docs, which is less than k, and I believe should result in an exact knn search through the efficient knn filtering algorithm, however it does not seem like we are seeing this behavior. Instead, we are just getting 0 hits.

Overall, we are seeing inconsistent query results. For example:

If we run this query on a single weekly index rather than the entire documents alias, we do get expected hits back.
If we remove the range portion of the knn query filter, we get expected hits back
If we move the range filter outside of the knn query filter (and transform the query into a knn post-filter instead), we do get hits back
If we use a knn filter range query for a non-date field (such as a float field), we get expected hits back

What is the expected behavior?
K-NN efficient filtering should work as described https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/ for knn range filters on a date field.

What is your host/environment?

Opensearch 2.17
k-NN plugin

The text was updated successfully, but these errors were encountered:

heemin32 · 2024-12-16T21:45:16Z

~~Duplication of #2222~~
This issue is happening without nested field

navneet1v · 2024-12-17T06:42:30Z

@kristib did you try reproducing on a smaller index with like 10-20 docs? We can try reproducing this with an example index.

Also when you say you upgraded the index, can you please tell from which version of Opensearch you upgraded from?

kristib · 2024-12-18T13:44:26Z

I will try to determine more factors for reproducing it. We are seeing the issue when we have an alias for at least 4 or so indexes that have around 1 million documents. I haven't been able to reproduce it on a small index. We upgraded from 2.13 to 2.17. Thanks!

navneet1v · 2024-12-18T21:30:08Z

@kristib one thing I would suggest you to try is, instead of sending the alias in the search if you pass the all the indices behind the alias as a comma separated values in the URL of search.
This will help us nail down the issue better if its happening due to k-NN queries or something else is going on.
Example:
index alias: my-index

indices: my-index1, my-index2

Alias based search

POST my-index/_search
// your query

Putting all indices instead of alias

POST my-index1,my-index2/_search
// your query

kristib · 2024-12-18T21:45:40Z

The issue seems weirdly related to querying using an alias or across indexes, along with the range knn filter. I just tried the comma separated list too and still see the issue.

To reproduce, I created an index with these settings and mappings:

PUT testindex1
{
  "settings": {
    "index": {
      "replication": {
        "type": "DOCUMENT"
      },
      "mapping": {
        "nested_objects": {
          "limit": "20000"
        },
        "total_fields": {
          "limit": "1200"
        }
      },
      "number_of_shards": "6",
      "knn": "true",
      "number_of_replicas": "2"
    }
  },
  "mappings": {
    "properties": {
      "embeddings": {
        "properties": {
          "OAI_TE3L_256": {
            "type": "knn_vector",
            "dimension": 256,
            "method": {
              "engine": "faiss",
              "space_type": "l2",
              "name": "hnsw",
              "parameters": {
                "m": 32
              }
            }
          }
        }
      },
      "guide": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "dateReceived": {
        "type": "date"
      },
      "documentType": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "contentLanguage": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "creator": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

I then populated that index with around 35 documents. Everything works fine with my knn query using this index.

Next, I created a second index with the same settings and mappings, testindex2. I populated it with 20 other documents. Everything works fine with my knn query using this index.

Then I create an alias and add testindex1 and testindex2 to the alias.

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "testindex1",
        "alias": "testindex_alias"
      }
    },
    {
      "add": {
        "index": "testindex2",
        "alias": "testindex_alias"
      }
    }
  ]
}

If I query the alias with my knn query, the hits in the response start to vary and return 0 sometimes, which isn't expected behavior.
Query:

GET testindex_alias/_search
{
  "from": 0,
  "query": {
    "knn": {
      "embeddings.OAI_TE3L_256": {
"vector": [
          -0.05308306,
          -0.040473435,
          -0.020121736,
          -0.024970116,
          0.017065149,
          0.0007180346,
          -0.041048344,
          0.025315061,
          -0.0696787,
          -0.04986358,
          0.11812417,
          0.004086629,
          -0.121267,
          -0.03683236,
          -0.063508034,
          -0.10087697,
          -0.1159012,
          -0.032597214,
          0.020332536,
          -0.103329904,
          0.15353844,
          -0.05373462,
          -0.000034284803,
          -0.03058504,
          0.04346295,
          0.038748715,
          -0.088458985,
          0.03915115,
          0.066957474,
          0.14510646,
          0.029665189,
          -0.0040171607,
          -0.079260476,
          -0.022766307,
          -0.06937208,
          0.0005449637,
          0.1864231,
          -0.054577816,
          0.010750757,
          0.038729552,
          0.06327807,
          0.048292167,
          0.015273356,
          0.0040890244,
          -0.07489119,
          -0.009126645,
          0.033459574,
          -0.011258591,
          -0.023954447,
          0.009629688,
          -0.03322961,
          -0.03882537,
          -0.070905164,
          0.026349893,
          0.121267,
          -0.096431024,
          0.0071336348,
          0.13782431,
          -0.034226116,
          -0.10509296,
          0.021673985,
          -0.038729552,
          -0.016672296,
          0.061859965,
          -0.06864387,
          0.12824254,
          0.030086787,
          -0.05423287,
          0.004139329,
          -0.046644103,
          0.0077324957,
          0.08370642,
          -0.091985084,
          0.084856234,
          0.017975418,
          0.0033632044,
          0.011910152,
          -0.050630122,
          0.116131164,
          -0.08439631,
          -0.030201769,
          -0.06734075,
          0.104862995,
          0.04043511,
          0.056379188,
          0.0061515025,
          0.064006284,
          -0.23870128,
          0.03117911,
          -0.0060892208,
          0.03867206,
          -0.04162325,
          0.00463279,
          -0.11774091,
          0.0556893,
          -0.07496784,
          -0.0155608095,
          -0.0087960735,
          -0.0011372374,
          -0.0022026114,
          -0.03468604,
          -0.0071384255,
          -0.010175849,
          -0.08907222,
          0.08255661,
          -0.064159594,
          -0.06829892,
          -0.056417517,
          -0.043386295,
          -0.023494523,
          -0.022517182,
          0.00840322,
          -0.038767878,
          0.054616142,
          -0.08439631,
          0.06262651,
          -0.027633851,
          -0.03644909,
          0.039017007,
          -0.05047681,
          0.026714,
          -0.045954213,
          0.10011043,
          -0.03150489,
          -0.026503202,
          0.1031766,
          -0.05756733,
          0.029933479,
          0.060135245,
          -0.17323858,
          -0.04150827,
          0.043194663,
          -0.09367147,
          -0.033574555,
          0.0029607697,
          0.032482233,
          0.034283604,
          0.010607029,
          -0.07044524,
          -0.021942275,
          -0.080486946,
          0.036219127,
          0.021348204,
          -0.005686786,
          -0.074086316,
          -0.041776557,
          -0.033267938,
          -0.056915767,
          -0.07719082,
          0.07742078,
          -0.1773779,
          -0.058295544,
          -0.052431498,
          0.030182606,
          -0.013251601,
          -0.0072294525,
          0.0696787,
          -0.042849716,
          0.019163558,
          0.061131753,
          0.074507914,
          0.095511176,
          -0.039285295,
          0.100187086,
          0.03825046,
          -0.02579415,
          0.13245852,
          0.0035692127,
          -0.022919616,
          0.11375488,
          -0.031485725,
          -0.08133014,
          -0.051818263,
          -0.07059855,
          -0.00053478306,
          0.00737797,
          0.056340862,
          -0.11697436,
          0.071556725,
          0.0053514238,
          0.1167444,
          0.042888045,
          0.025123425,
          -0.0140852155,
          -0.020351699,
          0.0119963875,
          -0.0105207935,
          -0.031428237,
          0.012695857,
          -0.009773415,
          -0.031370744,
          -0.013558217,
          -0.03526095,
          -0.03656407,
          -0.0095913615,
          0.02100326,
          0.053351346,
          0.05825722,
          -0.006208993,
          -0.03665989,
          0.07941379,
          -0.09213839,
          -0.067762345,
          -0.058218893,
          0.07174836,
          0.09443802,
          0.09006873,
          -0.006343138,
          -0.0071911253,
          -0.12555964,
          0.025698332,
          0.014554723,
          0.07339643,
          0.0067072455,
          0.02454852,
          0.030144278,
          0.01432476,
          -0.025947459,
          0.0651561,
          0.039074495,
          0.021463186,
          -0.040090166,
          -0.025104262,
          -0.04346295,
          -0.08753914,
          -0.028400393,
          0.09505125,
          0.14012393,
          -0.055804282,
          0.013059965,
          -0.022306383,
          -0.0034662087,
          -0.0038973887,
          -0.120807074,
          -0.06281815,
          -0.047525626,
          0.009073945,
          0.04411451,
          -0.062281564,
          -0.055114396,
          -0.0583722,
          0.04032013,
          0.00021514088,
          -0.018780287,
          -0.019048577,
          -0.03284634,
          -0.06856721,
          0.069985315,
          0.056839116,
          -0.02127155,
          0.06140004,
          0.028745338,
          -0.0561109,
          0.033363756,
          -0.025621679,
          0.06872052
        ],
        "k": 1000,
        "filter": {
          "bool": {
            "filter": [
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "contentLanguage.keyword": "Spanish"
                      }
                    },
                    {
                      "range": {
                        "dateReceived": {
                          "gte": "2024-12-09T02:10:20.840Z"
                        }
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      }
    }
  },
  "size": 10,
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ]
}

I tried testindex1,textindex2/_search and see the same unexpected results.

Also, if I remove the knn filter range portion of the query, the query results look as expected

navneet1v · 2024-12-18T22:00:14Z

Thanks @kristib for providing the reproducing steps. We will start looking into this. This could be an issue from Opensearch too, but we can say more once we complete the deep-dive(@buddharajusahil is doing the deep-dive).

buddharajusahil · 2024-12-19T20:52:23Z

Please assign this issue to me

buddharajusahil · 2024-12-20T00:03:39Z

Hi @kristib I tried recreating the issue on OpenSearch 2.17 but was unable to. I used the same index settings except with dimension 4. Here are the commands I used in sequence:

curl -X PUT "https://localhost:9200/testindex1?pretty" -H 'Content-Type: application/json' -d '{
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "mapping": {
          "nested_objects": {
            "limit": "20000"
          },
          "total_fields": {
            "limit": "1200"
          }
        },
        "number_of_shards": "6",
        "knn": true,
        "number_of_replicas": "2"
      }
    },
    "mappings": {
      "properties": {
        "embeddings": {
          "properties": {
            "OAI_TE3L_256": {
              "type": "knn_vector",
              "dimension": 4,
              "method": {
                "engine": "faiss",
                "space_type": "l2",
                "name": "hnsw",
                "parameters": {
                  "m": 32
                }
              }
            }
          }
        },
        "guide": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "dateReceived": {
          "type": "date"
        },
        "documentType": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "contentLanguage": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "creator": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
}'

curl -X POST "https://localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d'
{"index":{"_index":"testindex1"}}
{"guide":"Document 1", "dateReceived":"2023-06-01", "documentType":"Type A", "contentLanguage":"English", "creator":"User 1", "embeddings":{"OAI_TE3L_256":[0.1, 0.2, 0.3, 0.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 2", "dateReceived":"2023-06-02", "documentType":"Type B", "contentLanguage":"French", "creator":"User 2", "embeddings":{"OAI_TE3L_256":[0.2, 0.3, 0.4, 0.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 3", "dateReceived":"2023-06-03", "documentType":"Type C", "contentLanguage":"German", "creator":"User 3", "embeddings":{"OAI_TE3L_256":[0.3, 0.4, 0.5, 0.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 4", "dateReceived":"2023-06-04", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 4", "embeddings":{"OAI_TE3L_256":[0.4, 0.5, 0.6, 0.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 5", "dateReceived":"2023-06-05", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 5", "embeddings":{"OAI_TE3L_256":[0.5, 0.6, 0.7, 0.8]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 6", "dateReceived":"2023-06-06", "documentType":"Type C", "contentLanguage":"English", "creator":"User 6", "embeddings":{"OAI_TE3L_256":[0.6, 0.7, 0.8, 0.9]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 7", "dateReceived":"2023-06-07", "documentType":"Type A", "contentLanguage":"French", "creator":"User 7", "embeddings":{"OAI_TE3L_256":[0.7, 0.8, 0.9, 1.0]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 8", "dateReceived":"2023-06-08", "documentType":"Type B", "contentLanguage":"German", "creator":"User 8", "embeddings":{"OAI_TE3L_256":[0.8, 0.9, 1.0, 1.1]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 9", "dateReceived":"2023-06-09", "documentType":"Type C", "contentLanguage":"Spanish", "creator":"User 9", "embeddings":{"OAI_TE3L_256":[0.9, 1.0, 1.1, 1.2]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 10", "dateReceived":"2023-06-10", "documentType":"Type A", "contentLanguage":"Italian", "creator":"User 10", "embeddings":{"OAI_TE3L_256":[1.0, 1.1, 1.2, 1.3]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 11", "dateReceived":"2023-06-11", "documentType":"Type B", "contentLanguage":"English", "creator":"User 11", "embeddings":{"OAI_TE3L_256":[1.1, 1.2, 1.3, 1.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 12", "dateReceived":"2023-06-12", "documentType":"Type C", "contentLanguage":"French", "creator":"User 12", "embeddings":{"OAI_TE3L_256":[1.2, 1.3, 1.4, 1.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 13", "dateReceived":"2023-06-13", "documentType":"Type A", "contentLanguage":"German", "creator":"User 13", "embeddings":{"OAI_TE3L_256":[1.3, 1.4, 1.5, 1.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 14", "dateReceived":"2023-06-14", "documentType":"Type B", "contentLanguage":"Spanish", "creator":"User 14", "embeddings":{"OAI_TE3L_256":[1.4, 1.5, 1.6, 1.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 15", "dateReceived":"2023-06-15", "documentType":"Type C", "contentLanguage":"Italian", "creator":"User 15", "embeddings":{"OAI_TE3L_256":[1.5, 1.6, 1.7, 1.8]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 16", "dateReceived":"2023-06-16", "documentType":"Type A", "contentLanguage":"English", "creator":"User 16", "embeddings":{"OAI_TE3L_256":[1.6, 1.7, 1.8, 1.9]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 17", "dateReceived":"2023-06-17", "documentType":"Type B", "contentLanguage":"French", "creator":"User 17", "embeddings":{"OAI_TE3L_256":[1.7, 1.8, 1.9, 2.0]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 18", "dateReceived":"2023-06-18", "documentType":"Type C", "contentLanguage":"German", "creator":"User 18", "embeddings":{"OAI_TE3L_256":[1.8, 1.9, 2.0, 2.1]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 19", "dateReceived":"2023-06-19", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 19", "embeddings":{"OAI_TE3L_256":[1.9, 2.0, 2.1, 2.2]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 20", "dateReceived":"2023-06-20", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 20", "embeddings":{"OAI_TE3L_256":[2.0, 2.1, 2.2, 2.3]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 21", "dateReceived":"2023-06-21", "documentType":"Type C", "contentLanguage":"English", "creator":"User 21", "embeddings":{"OAI_TE3L_256":[2.1, 2.2, 2.3, 2.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 22", "dateReceived":"2023-06-22", "documentType":"Type A", "contentLanguage":"French", "creator":"User 22", "embeddings":{"OAI_TE3L_256":[2.2, 2.3, 2.4, 2.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 23", "dateReceived":"2023-06-23", "documentType":"Type B", "contentLanguage":"German", "creator":"User 23", "embeddings":{"OAI_TE3L_256":[2.3, 2.4, 2.5, 2.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 24", "dateReceived":"2023-06-24", "documentType":"Type C", "contentLanguage":"Spanish", "creator":"User 24", "embeddings":{"OAI_TE3L_256":[2.4, 2.5, 2.6, 2.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 25", "dateReceived":"2023-06-25", "documentType":"Type A", "contentLanguage":"Italian", "creator":"User 25", "embeddings":{"OAI_TE3L_256":[2.5, 2.6, 2.7, 2.8]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 26", "dateReceived":"2023-06-26", "documentType":"Type B", "contentLanguage":"English", "creator":"User 26", "embeddings":{"OAI_TE3L_256":[2.6, 2.7, 2.8, 2.9]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 27", "dateReceived":"2023-06-27", "documentType":"Type C", "contentLanguage":"French", "creator":"User 27", "embeddings":{"OAI_TE3L_256":[2.7, 2.8, 2.9, 3.0]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 28", "dateReceived":"2023-06-28", "documentType":"Type A", "contentLanguage":"German", "creator":"User 28", "embeddings":{"OAI_TE3L_256":[2.8, 2.9, 3.0, 3.1]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 29", "dateReceived":"2023-06-29", "documentType":"Type B", "contentLanguage":"Spanish", "creator":"User 29", "embeddings":{"OAI_TE3L_256":[2.9, 3.0, 3.1, 3.2]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 30", "dateReceived":"2023-06-30", "documentType":"Type C", "contentLanguage":"Italian", "creator":"User 30", "embeddings":{"OAI_TE3L_256":[3.0, 3.1, 3.2, 3.3]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 31", "dateReceived":"2023-07-01", "documentType":"Type A", "contentLanguage":"English", "creator":"User 31", "embeddings":{"OAI_TE3L_256":[3.1, 3.2, 3.3, 3.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 32", "dateReceived":"2023-07-02", "documentType":"Type B", "contentLanguage":"French", "creator":"User 32", "embeddings":{"OAI_TE3L_256":[3.2, 3.3, 3.4, 3.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 33", "dateReceived":"2023-07-03", "documentType":"Type C", "contentLanguage":"German", "creator":"User 33", "embeddings":{"OAI_TE3L_256":[3.3, 3.4, 3.5, 3.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 34", "dateReceived":"2023-07-04", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 34", "embeddings":{"OAI_TE3L_256":[3.4, 3.5, 3.6, 3.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 35", "dateReceived":"2023-07-05", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 35", "embeddings":{"OAI_TE3L_256":[3.5, 3.6, 3.7, 3.8]}}
' 


curl -X PUT "https://localhost:9200/testindex2?pretty" -H 'Content-Type: application/json' -d'{
"settings": {
    "index": {
    "replication": {
        "type": "DOCUMENT"
    },
    "mapping": {
        "nested_objects": {
        "limit": "20000"
        },
        "total_fields": {
        "limit": "1200"
        }
    },
    "number_of_shards": "6",
    "knn": "true",
    "number_of_replicas": "2"
    }
},
"mappings": {
    "properties": {
    "embeddings": {
        "properties": {
        "OAI_TE3L_256": {
            "type": "knn_vector",
            "dimension": 4,
            "method": {
            "engine": "faiss",
            "space_type": "l2",
            "name": "hnsw",
            "parameters": {
                "m": 32
            }
            }
        }
        }
    },
    "guide": {
        "type": "text",
        "fields": {
        "keyword": {
            "type": "keyword",
            "ignore_above": 256
        }
        }
    },
    "dateReceived": {
        "type": "date"
    },
    "documentType": {
        "type": "text",
        "fields": {
        "keyword": {
            "type": "keyword",
            "ignore_above": 256
        }
        }
    },
    "contentLanguage": {
        "type": "text",
        "fields": {
        "keyword": {
            "type": "keyword",
            "ignore_above": 256
        }
        }
    },
    "creator": {
        "type": "text",
        "fields": {
        "keyword": {
            "type": "keyword",
            "ignore_above": 256
        }
        }
    }
    }
}
}' 


curl -X POST "https://localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d'
{"index":{"_index":"testindex2"}}
{"guide":"Document 1", "dateReceived":"2023-06-01", "documentType":"Type A", "contentLanguage":"English", "creator":"User 1", "embeddings":{"OAI_TE3L_256":[0.1, 0.2, 0.3, 0.4]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 2", "dateReceived":"2023-06-02", "documentType":"Type B", "contentLanguage":"French", "creator":"User 2", "embeddings":{"OAI_TE3L_256":[0.2, 0.3, 0.4, 0.5]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 3", "dateReceived":"2023-06-03", "documentType":"Type C", "contentLanguage":"German", "creator":"User 3", "embeddings":{"OAI_TE3L_256":[0.3, 0.4, 0.5, 0.6]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 4", "dateReceived":"2023-06-04", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 4", "embeddings":{"OAI_TE3L_256":[0.4, 0.5, 0.6, 0.7]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 5", "dateReceived":"2023-06-05", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 5", "embeddings":{"OAI_TE3L_256":[0.5, 0.6, 0.7, 0.8]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 6", "dateReceived":"2023-06-06", "documentType":"Type C", "contentLanguage":"English", "creator":"User 6", "embeddings":{"OAI_TE3L_256":[0.6, 0.7, 0.8, 0.9]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 7", "dateReceived":"2023-06-07", "documentType":"Type A", "contentLanguage":"French", "creator":"User 7", "embeddings":{"OAI_TE3L_256":[0.7, 0.8, 0.9, 1.0]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 8", "dateReceived":"2023-06-08", "documentType":"Type B", "contentLanguage":"German", "creator":"User 8", "embeddings":{"OAI_TE3L_256":[0.8, 0.9, 1.0, 1.1]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 9", "dateReceived":"2023-06-09", "documentType":"Type C", "contentLanguage":"Spanish", "creator":"User 9", "embeddings":{"OAI_TE3L_256":[0.9, 1.0, 1.1, 1.2]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 10", "dateReceived":"2023-06-10", "documentType":"Type A", "contentLanguage":"Italian", "creator":"User 10", "embeddings":{"OAI_TE3L_256":[1.0, 1.1, 1.2, 1.3]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 11", "dateReceived":"2023-06-11", "documentType":"Type B", "contentLanguage":"English", "creator":"User 11", "embeddings":{"OAI_TE3L_256":[1.1, 1.2, 1.3, 1.4]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 12", "dateReceived":"2023-06-12", "documentType":"Type C", "contentLanguage":"French", "creator":"User 12", "embeddings":{"OAI_TE3L_256":[1.2, 1.3, 1.4, 1.5]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 13", "dateReceived":"2023-06-13", "documentType":"Type A", "contentLanguage":"German", "creator":"User 13", "embeddings":{"OAI_TE3L_256":[1.3, 1.4, 1.5, 1.6]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 14", "dateReceived":"2023-06-14", "documentType":"Type B", "contentLanguage":"Spanish", "creator":"User 14", "embeddings":{"OAI_TE3L_256":[1.4, 1.5, 1.6, 1.7]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 15", "dateReceived":"2023-06-15", "documentType":"Type C", "contentLanguage":"Italian", "creator":"User 15", "embeddings":{"OAI_TE3L_256":[1.5, 1.6, 1.7, 1.8]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 16", "dateReceived":"2023-06-16", "documentType":"Type A", "contentLanguage":"English", "creator":"User 16", "embeddings":{"OAI_TE3L_256":[1.6, 1.7, 1.8, 1.9]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 17", "dateReceived":"2023-06-17", "documentType":"Type B", "contentLanguage":"French", "creator":"User 17", "embeddings":{"OAI_TE3L_256":[1.7, 1.8, 1.9, 2.0]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 18", "dateReceived":"2023-06-18", "documentType":"Type C", "contentLanguage":"German", "creator":"User 18", "embeddings":{"OAI_TE3L_256":[1.8, 1.9, 2.0, 2.1]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 19", "dateReceived":"2023-06-19", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 19", "embeddings":{"OAI_TE3L_256":[1.9, 2.0, 2.1, 2.2]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 20", "dateReceived":"2023-06-20", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 20", "embeddings":{"OAI_TE3L_256":[2.0, 2.1, 2.2, 2.3]}}
' 

curl -X POST "https://localhost:9200/_aliases?pretty" -H 'Content-Type: application/json' -d'
{
"actions": [
    {
    "add": {
        "index": "testindex1",
        "alias": "testindex_alias"
    }
    },
    {
    "add": {
        "index": "testindex2",
        "alias": "testindex_alias"
    }
    }
]
}
'

curl -X GET "https://localhost:9200/testindex_alias/_search?pretty" -H 'Content-Type: application/json' -d'
{
"from": 0,
"query": {
    "knn": {
    "embeddings.OAI_TE3L_256": {
        "vector": [
        -0.05308306,
        -0.040473435,
        -0.020121736,
        -0.024970116
        ],
        "k": 1000,
        "filter": {
        "bool": {
            "filter": [
            {
                "bool": {
                "must": [
                    {
                    "term": {
                        "contentLanguage.keyword": "Spanish"
                    }
                    },
                    {
                    "range": {
                        "dateReceived": {
                        "gte": "2022-12-09T02:10:20.840Z"
                        }
                    }
                    }
                ]
                }
            }
            ]
        }
        }
    }
    }
},
"size": 10,
"sort": [
    {
    "_score": {
        "order": "desc"
    }
    }
]
}
'

I was able to get consistent search results using these commands. Do you have more context to your issue?

kristib · 2024-12-20T01:44:59Z

@buddharajusahil thanks for investigating! I tried your exact steps and could not reproduce with that set of data. However, I tried modifying the dates and I could reproduce it.

Here's a set of data where I see inconsistent results returned. Sometimes I even get results outside of the date range.

POST /_bulk?pretty
{"index":{"_index":"testindex1"}}
{"guide":"Document 1", "dateReceived":"2024-12-09T02:10:23.798Z", "documentType":"Type A", "contentLanguage":"English", "creator":"User 1", "embeddings":{"OAI_TE3L_256":[0.1, 0.2, 0.3, 0.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 2", "dateReceived":"2024-12-09T02:10:23.711Z", "documentType":"Type B", "contentLanguage":"French", "creator":"User 2", "embeddings":{"OAI_TE3L_256":[0.2, 0.3, 0.4, 0.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 3", "dateReceived":"2024-12-09T02:10:23.222Z", "documentType":"Type C", "contentLanguage":"German", "creator":"User 3", "embeddings":{"OAI_TE3L_256":[0.3, 0.4, 0.5, 0.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 4", "dateReceived":"2023-06-04T18:37:43.333Z", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 4", "embeddings":{"OAI_TE3L_256":[0.4, 0.5, 0.6, 0.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 5", "dateReceived":"2024-12-09T02:10:23.111Z", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 5", "embeddings":{"OAI_TE3L_256":[0.5, 0.6, 0.7, 0.8]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 6", "dateReceived":"2023-06-06T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"English", "creator":"User 6", "embeddings":{"OAI_TE3L_256":[0.6, 0.7, 0.8, 0.9]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 7", "dateReceived":"2023-06-07T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"French", "creator":"User 7", "embeddings":{"OAI_TE3L_256":[0.7, 0.8, 0.9, 1.0]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 8", "dateReceived":"2023-06-08T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"German", "creator":"User 8", "embeddings":{"OAI_TE3L_256":[0.8, 0.9, 1.0, 1.1]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 9", "dateReceived":"2023-02-09T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"Spanish", "creator":"User 9", "embeddings":{"OAI_TE3L_256":[0.9, 1.0, 1.1, 1.2]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 10", "dateReceived":"2023-06-10T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"Italian", "creator":"User 10", "embeddings":{"OAI_TE3L_256":[1.0, 1.1, 1.2, 1.3]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 11", "dateReceived":"2024-12-09T00:10:23.798Z", "documentType":"Type B", "contentLanguage":"English", "creator":"User 11", "embeddings":{"OAI_TE3L_256":[1.1, 1.2, 1.3, 1.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 12", "dateReceived":"2023-06-12T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"French", "creator":"User 12", "embeddings":{"OAI_TE3L_256":[1.2, 1.3, 1.4, 1.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 13", "dateReceived":"2023-06-13T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"German", "creator":"User 13", "embeddings":{"OAI_TE3L_256":[1.3, 1.4, 1.5, 1.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 14", "dateReceived":"2023-06-14T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"Spanish", "creator":"User 14", "embeddings":{"OAI_TE3L_256":[1.4, 1.5, 1.6, 1.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 15", "dateReceived":"2023-06-15T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"Italian", "creator":"User 15", "embeddings":{"OAI_TE3L_256":[1.5, 1.6, 1.7, 1.8]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 16", "dateReceived":"2023-01-16T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"English", "creator":"User 16", "embeddings":{"OAI_TE3L_256":[1.6, 1.7, 1.8, 1.9]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 17", "dateReceived":"2023-06-17T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"French", "creator":"User 17", "embeddings":{"OAI_TE3L_256":[1.7, 1.8, 1.9, 2.0]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 18", "dateReceived":"2023-06-30T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"German", "creator":"User 18", "embeddings":{"OAI_TE3L_256":[1.8, 1.9, 2.0, 2.1]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 19", "dateReceived":"2023-06-19T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 19", "embeddings":{"OAI_TE3L_256":[1.9, 2.0, 2.1, 2.2]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 20", "dateReceived":"2023-06-20T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 20", "embeddings":{"OAI_TE3L_256":[2.0, 2.1, 2.2, 2.3]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 21", "dateReceived":"2023-06-21T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"English", "creator":"User 21", "embeddings":{"OAI_TE3L_256":[2.1, 2.2, 2.3, 2.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 22", "dateReceived":"2023-06-22T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"French", "creator":"User 22", "embeddings":{"OAI_TE3L_256":[2.2, 2.3, 2.4, 2.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 23", "dateReceived":"2023-06-23T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"German", "creator":"User 23", "embeddings":{"OAI_TE3L_256":[2.3, 2.4, 2.5, 2.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 24", "dateReceived":"2023-06-24T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"Spanish", "creator":"User 24", "embeddings":{"OAI_TE3L_256":[2.4, 2.5, 2.6, 2.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 25", "dateReceived":"2023-06-25T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"Italian", "creator":"User 25", "embeddings":{"OAI_TE3L_256":[2.5, 2.6, 2.7, 2.8]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 26", "dateReceived":"2023-06-26T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"English", "creator":"User 26", "embeddings":{"OAI_TE3L_256":[2.6, 2.7, 2.8, 2.9]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 27", "dateReceived":"2023-06-27T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"French", "creator":"User 27", "embeddings":{"OAI_TE3L_256":[2.7, 2.8, 2.9, 3.0]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 28", "dateReceived":"2023-06-28T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"German", "creator":"User 28", "embeddings":{"OAI_TE3L_256":[2.8, 2.9, 3.0, 3.1]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 29", "dateReceived":"2023-06-29T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"Spanish", "creator":"User 29", "embeddings":{"OAI_TE3L_256":[2.9, 3.0, 3.1, 3.2]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 30", "dateReceived":"2023-06-30T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"Italian", "creator":"User 30", "embeddings":{"OAI_TE3L_256":[3.0, 3.1, 3.2, 3.3]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 31", "dateReceived":"2023-07-01T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"English", "creator":"User 31", "embeddings":{"OAI_TE3L_256":[3.1, 3.2, 3.3, 3.4]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 32", "dateReceived":"2023-07-02T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"French", "creator":"User 32", "embeddings":{"OAI_TE3L_256":[3.2, 3.3, 3.4, 3.5]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 33", "dateReceived":"2023-07-03T18:37:43.415Z", "documentType":"Type C", "contentLanguage":"German", "creator":"User 33", "embeddings":{"OAI_TE3L_256":[3.3, 3.4, 3.5, 3.6]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 34", "dateReceived":"2023-07-04T18:37:43.415Z", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 34", "embeddings":{"OAI_TE3L_256":[3.4, 3.5, 3.6, 3.7]}}
{"index":{"_index":"testindex1"}}
{"guide":"Document 35", "dateReceived":"2023-07-05T18:37:43.415Z", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 35", "embeddings":{"OAI_TE3L_256":[3.5, 3.6, 3.7, 3.8]}}
 

POST /_bulk?pretty
{"index":{"_index":"testindex2"}}
{"guide":"Document 1", "dateReceived":"2024-11-01T19:37:43.123Z", "documentType":"Type A", "contentLanguage":"English", "creator":"User 1", "embeddings":{"OAI_TE3L_256":[0.1, 0.2, 0.3, 0.4]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 2", "dateReceived":"2024-11-01T19:37:43.111", "documentType":"Type B", "contentLanguage":"French", "creator":"User 2", "embeddings":{"OAI_TE3L_256":[0.2, 0.3, 0.4, 0.5]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 3", "dateReceived":"2024-11-01T19:37:43.333Z", "documentType":"Type C", "contentLanguage":"German", "creator":"User 3", "embeddings":{"OAI_TE3L_256":[0.3, 0.4, 0.5, 0.6]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 4", "dateReceived":"2024-11-01T19:37:43.312Z", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 4", "embeddings":{"OAI_TE3L_256":[0.4, 0.5, 0.6, 0.7]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 5", "dateReceived":"2024-11-01T19:30:43.123Z", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 5", "embeddings":{"OAI_TE3L_256":[0.5, 0.6, 0.7, 0.8]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 6", "dateReceived":"2023-06-06T19:37:43.123Z", "documentType":"Type C", "contentLanguage":"English", "creator":"User 6", "embeddings":{"OAI_TE3L_256":[0.6, 0.7, 0.8, 0.9]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 7", "dateReceived":"2023-06-07T19:37:44.123Z", "documentType":"Type A", "contentLanguage":"French", "creator":"User 7", "embeddings":{"OAI_TE3L_256":[0.7, 0.8, 0.9, 1.0]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 8", "dateReceived":"2023-06-08T19:37:43.123Z", "documentType":"Type B", "contentLanguage":"German", "creator":"User 8", "embeddings":{"OAI_TE3L_256":[0.8, 0.9, 1.0, 1.1]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 9", "dateReceived":"2023-06-09T00:30:43.123Z", "documentType":"Type C", "contentLanguage":"Spanish", "creator":"User 9", "embeddings":{"OAI_TE3L_256":[0.9, 1.0, 1.1, 1.2]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 10", "dateReceived":"2023-06-10T19:37:43.123Z", "documentType":"Type A", "contentLanguage":"Italian", "creator":"User 10", "embeddings":{"OAI_TE3L_256":[1.0, 1.1, 1.2, 1.3]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 11", "dateReceived":"2023-06-11T19:37:43.123Z", "documentType":"Type B", "contentLanguage":"English", "creator":"User 11", "embeddings":{"OAI_TE3L_256":[1.1, 1.2, 1.3, 1.4]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 12", "dateReceived":"2023-06-12T19:37:43.123Z", "documentType":"Type C", "contentLanguage":"French", "creator":"User 12", "embeddings":{"OAI_TE3L_256":[1.2, 1.3, 1.4, 1.5]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 13", "dateReceived":"2023-06-13T19:37:43.123Z", "documentType":"Type A", "contentLanguage":"German", "creator":"User 13", "embeddings":{"OAI_TE3L_256":[1.3, 1.4, 1.5, 1.6]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 14", "dateReceived":"2023-08-14T19:37:43.123Z", "documentType":"Type B", "contentLanguage":"Spanish", "creator":"User 14", "embeddings":{"OAI_TE3L_256":[1.4, 1.5, 1.6, 1.7]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 15", "dateReceived":"2023-06-15T02:37:43.123Z", "documentType":"Type C", "contentLanguage":"Italian", "creator":"User 15", "embeddings":{"OAI_TE3L_256":[1.5, 1.6, 1.7, 1.8]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 16", "dateReceived":"2023-06-16T19:37:43.123Z", "documentType":"Type A", "contentLanguage":"English", "creator":"User 16", "embeddings":{"OAI_TE3L_256":[1.6, 1.7, 1.8, 1.9]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 17", "dateReceived":"2023-06-17T19:37:43.123Z", "documentType":"Type B", "contentLanguage":"French", "creator":"User 17", "embeddings":{"OAI_TE3L_256":[1.7, 1.8, 1.9, 2.0]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 18", "dateReceived":"2023-06-18T19:37:43.123Z", "documentType":"Type C", "contentLanguage":"German", "creator":"User 18", "embeddings":{"OAI_TE3L_256":[1.8, 1.9, 2.0, 2.1]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 19", "dateReceived":"2023-06-19T19:37:43.123Z", "documentType":"Type A", "contentLanguage":"Spanish", "creator":"User 19", "embeddings":{"OAI_TE3L_256":[1.9, 2.0, 2.1, 2.2]}}
{"index":{"_index":"testindex2"}}
{"guide":"Document 20", "dateReceived":"2023-06-20T19:37:43.123Z", "documentType":"Type B", "contentLanguage":"Italian", "creator":"User 20", "embeddings":{"OAI_TE3L_256":[2.0, 2.1, 2.2, 2.3]}}

Using this query:

GET /testindex_alias/_search?pretty
{
  "from": 0,
  "query": {
    "knn": {
      "embeddings.OAI_TE3L_256": {
        "vector": [
          -0.05308306,
          -0.040473435,
          -0.020121736,
          -0.024970116
        ],
        "k": 1000,
        "filter": {
          "bool": {
            "filter": [
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "contentLanguage.keyword": "Spanish"
                      }
                    },
                    {
                      "range": {
                        "dateReceived": {
                          "gte": "2023-06-30T19:37:43.123Z"
                        }
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      }
    }
  },
  "size": 10,
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ]
}

Is it related to the date format?

buddharajusahil · 2024-12-20T17:39:15Z

Hi @kristib. Thanks for the data! With your commands I was able to reproduce the issue, will look into it! Additionally, I seem to get inconsistent results even when querying the individual indexes, so maybe not an alias problem.

kristib · 2024-12-20T18:25:17Z

ok thanks! maybe it is just more noticeable across indexes. I'm experimenting with indexing different date formats to try to help isolate where it occurs

buddharajusahil · 2024-12-23T20:44:01Z

@kristib I have narrowed the problem down to this PR: #1874. Looking into it further.

kristib added bug Something isn't working untriaged labels Dec 16, 2024

heemin32 closed this as completed Dec 16, 2024

kristib mentioned this issue Dec 16, 2024

[BUG] Filter on Parent Doc fields inside Nested knn query fails for many Query types #2222

Open

heemin32 reopened this Dec 16, 2024

navneet1v removed the untriaged label Dec 17, 2024

navneet1v assigned buddharajusahil Dec 19, 2024

CorentinLimier mentioned this issue Dec 20, 2024

[BUG] Insufficient number of hits for nested knn queries with efficient filter #2347

Open

navneet1v added this to Vector Search RoadMap Dec 24, 2024

github-project-automation bot moved this to Backlog in Vector Search RoadMap Dec 24, 2024

navneet1v moved this from Backlog to 2.19.0 in Vector Search RoadMap Dec 24, 2024

navneet1v added the v2.19.0 label Dec 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results #2339

[BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results #2339

kristib commented Dec 16, 2024 •

edited

Loading

heemin32 commented Dec 16, 2024 •

edited

Loading

navneet1v commented Dec 17, 2024

kristib commented Dec 18, 2024

navneet1v commented Dec 18, 2024

kristib commented Dec 18, 2024 •

edited

Loading

navneet1v commented Dec 18, 2024 •

edited

Loading

buddharajusahil commented Dec 19, 2024

buddharajusahil commented Dec 20, 2024 •

edited

Loading

kristib commented Dec 20, 2024

buddharajusahil commented Dec 20, 2024 •

edited

Loading

kristib commented Dec 20, 2024

buddharajusahil commented Dec 23, 2024

[BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results #2339

[BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results #2339

Comments

kristib commented Dec 16, 2024 • edited Loading

heemin32 commented Dec 16, 2024 • edited Loading

navneet1v commented Dec 17, 2024

kristib commented Dec 18, 2024

navneet1v commented Dec 18, 2024

kristib commented Dec 18, 2024 • edited Loading

navneet1v commented Dec 18, 2024 • edited Loading

buddharajusahil commented Dec 19, 2024

buddharajusahil commented Dec 20, 2024 • edited Loading

kristib commented Dec 20, 2024

buddharajusahil commented Dec 20, 2024 • edited Loading

kristib commented Dec 20, 2024

buddharajusahil commented Dec 23, 2024

kristib commented Dec 16, 2024 •

edited

Loading

heemin32 commented Dec 16, 2024 •

edited

Loading

kristib commented Dec 18, 2024 •

edited

Loading

navneet1v commented Dec 18, 2024 •

edited

Loading

buddharajusahil commented Dec 20, 2024 •

edited

Loading

buddharajusahil commented Dec 20, 2024 •

edited

Loading