[Feature] Pagination in hybrid query #963

vibrantvarun · 2024-10-23T17:19:51Z

Description

This PR contains changes for enabling support for pagination in hybrid query.
The highlight of this PR are

Introduction of a new parameter "pagination_depth" to set a reference of hybrid query search results on which pagination can be applied.
Handling of single shard scenario where fetch phase can run before the normalization process.
Handling of from parameter conditions in Normalization processor.
Disabling scroll operation in hybrid query.

Related Issues

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

martin-gaievski · 2024-10-24T04:39:59Z

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

+        int paginationDepth;
+        HybridQuery hybridQuery;
+        Query query = searchContext.query();
+        if (query instanceof BooleanQuery) {


when such scenario can happen? we should not allow hybrid if it's not the top clause

I have added the comment on top why it is written like that. Basically in case of nested fields and alias filter, hybrid query gets wrapped under bool query.

simple case with nested fields is taken care of in QueryPhaseSearcher class. This logic doesn't belong here, unless this is a special scenario that we missed in that extract query method I mentioned above.

martin-gaievski · 2024-10-24T04:40:51Z

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

+        Query query = searchContext.query();
+        if (query instanceof BooleanQuery) {
+            BooleanQuery booleanQuery = (BooleanQuery) query;
+            hybridQuery = (HybridQuery) booleanQuery.clauses().get(0).getQuery();


this looks hacky, can we avoid this logic?

I have added the comment on top why it is written like that. Basically in case of nested fields and alias filter, hybrid query gets wrapped under bool query.

see my previous comment, this should be handled in here

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

martin-gaievski · 2024-11-05T01:34:41Z

.github/CODEOWNERS

@@ -1,2 +1,2 @@
 # This should match the owning team set up in https://github.com/orgs/opensearch-project/teams
-*   @heemin32 @navneet1v @VijayanB @vamshin @jmazanec15 @naveentatikonda @junqiu-lei @martin-gaievski @sean-zheng-amazon @model-collapse @zane-neo @ylwu-amzn @jngz-es @vibrantvarun @zhichao-aws @yuye-aws
+*   @heemin32 @navneet1v @VijayanB @vamshin @jmazanec15 @naveentatikonda @junqiu-lei @martin-gaievski @sean-zheng-amazon @model-collapse @zane-neo @vibrantvarun @zhichao-aws @yuye-aws @minalsha


this looks like artifacts of improper rebase, can you please rebase on main properly

martin-gaievski · 2024-11-05T01:36:14Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

@@ -48,16 +50,23 @@ public final class HybridQueryBuilder extends AbstractQueryBuilder<HybridQueryBu
    public static final String NAME = "hybrid";

    private static final ParseField QUERIES_FIELD = new ParseField("queries");
+    private static final ParseField DEPTH_FIELD = new ParseField("pagination_depth");


Suggested change

private static final ParseField DEPTH_FIELD = new ParseField("pagination_depth");

private static final ParseField PAGINATION_DEPTH_FIELD = new ParseField("pagination_depth");

martin-gaievski · 2024-11-05T01:42:00Z

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java

 public final class HybridQuery extends Query implements Iterable<Query> {

    private final List<Query> subQueries;
+    private Integer paginationDepth;


why this is not primitive int? Operating with wrapper class is potentially error prone when boxing/unboxing a null value.

martin-gaievski · 2024-11-05T01:48:37Z

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java

            querySearchResult.topDocs(updatedTopDocsAndMaxScore, querySearchResult.sortValueFormats());
        }
+
+        final int from = querySearchResults.get(0).from();


do we need final?

martin-gaievski · 2024-11-05T01:52:57Z

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java

            querySearchResult.topDocs(updatedTopDocsAndMaxScore, querySearchResult.sortValueFormats());
        }
+
+        final int from = querySearchResults.get(0).from();
+        if (from > 0 && from > totalScoreDocsCount) {


first check looks redundant, can't we rely only on from > totalScoreDocsCount?

martin-gaievski · 2024-11-05T02:13:48Z

src/test/java/org/opensearch/neuralsearch/query/HybridQueryIT.java

@@ -637,6 +639,7 @@ public void testWrappedQueryWithFilter_whenIndexAliasHasFilterAndIndexWithNested

            HybridQueryBuilder hybridQueryBuilder = new HybridQueryBuilder();
            hybridQueryBuilder.add(QueryBuilders.existsQuery(TEST_TEXT_FIELD_NAME_1));
+            // hybridQueryBuilder.paginationDepth(10);


is this intentional?

martin-gaievski · 2024-11-05T02:17:57Z

src/test/java/org/opensearch/neuralsearch/query/HybridQueryIT.java

    @SneakyThrows
-    public void testHybridQuery_whenFromIsSetInSearchRequest_thenFail() {
+    public void testPaginationDepth_whenSubqueriesCountIsGreaterThanFive_thenFail() {


What are we testing here? check for max sub-query limit is part of the HybridQueryBuilderTests.

martin-gaievski · 2024-11-05T02:23:37Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

@@ -97,6 +109,9 @@ protected void doXContent(XContentBuilder builder, Params params) throws IOExcep
            queryBuilder.toXContent(builder, params);
        }
        builder.endArray();
+        if (isClusterOnOrAfterMinReqVersionForPaginationInHybridQuery()) {
+            builder.field(DEPTH_FIELD.getPreferredName(), paginationDepth == null ? DEFAULT_PAGINATION_DEPTH : paginationDepth);


when we're doing today's logic? from this statement we'll either get user provided value or default which is 10. Today's logic is equivalent to PD equals to 1, I'm not exactly sure, but definitely not 10.

martin-gaievski · 2024-11-05T02:27:06Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

+        if (paginationDepth != null
+            && (paginationDepth < LOWER_BOUND_OF_PAGINATION_DEPTH || paginationDepth > UPPER_BOUND_OF_PAGINATION_DEPTH)) {
+            throw new IllegalArgumentException(
+                String.format(Locale.ROOT, "Pagination depth should lie in the range of 1-1000. Received: %s", paginationDepth)


please came up with better error message. The range is static here, but that's different in the logic where you have variables for lower and upper bounds, and we can drop the user-provided value as that's potentially unsafe, actual value should be obvious to user from the search request

martin-gaievski · 2024-11-05T02:29:53Z

src/test/java/org/opensearch/neuralsearch/query/HybridQueryIT.java

+    }
+
+    @SneakyThrows
+    public void testPaginationOnSingleShard_whenConcurrentSearchEnabled_thenSuccessful() {


Looks like you're basically put all tests for pagination to integ test. Can this be moved to unit tests, so we have only minimal increase in new integ tests. If that's not possible please consider moving them to a new class, this one because way overloaded, about 1000 lines.

Signed-off-by: Varun Jain <[email protected]>

Signed-off-by: Varun Jain <[email protected]> Signed-off-by: Martin Gaievski <[email protected]>

Signed-off-by: Martin Gaievski <[email protected]>

vibrantvarun changed the title ~~Pagination in hybrid query~~ [Feature] Pagination in hybrid query Oct 23, 2024

martin-gaievski reviewed Oct 24, 2024

View reviewed changes

vibrantvarun marked this pull request as ready for review November 4, 2024 03:08

vibrantvarun requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, sean-zheng-amazon, model-collapse, zane-neo, zhichao-aws and yuye-aws as code owners November 4, 2024 03:08

martin-gaievski reviewed Nov 5, 2024

View reviewed changes

martin-gaievski mentioned this pull request Dec 12, 2024

[DOC] Adding pagination in Hybrid query opensearch-project/documentation-website#8952

Open

4 tasks

martin-gaievski changed the base branch from Pagination_in_hybridQuery to feature/pagination_in_hybrid_query December 19, 2024 20:04

martin-gaievski requested a review from minalsha as a code owner December 19, 2024 20:04

vibrantvarun added 7 commits December 19, 2024 12:08

POC

bc73a50

Signed-off-by: Varun Jain <[email protected]>

Adding validation

7b20824

Signed-off-by: Varun Jain <[email protected]>

adding validation in depth

7dd0841

Signed-off-by: Varun Jain <[email protected]>

Integ Test and unit test

e0f8f4c

Signed-off-by: Varun Jain <[email protected]>

Update changelog

a8bf87b

Signed-off-by: Varun Jain <[email protected]>

Add IT and Unit test for subqueries count in hybrid clause

323b8c5

Signed-off-by: Varun Jain <[email protected]>

refactoring code

9e12976

Signed-off-by: Varun Jain <[email protected]>

martin-gaievski force-pushed the Pagination_in_hybrid_query branch from 3a9344f to 4580337 Compare December 19, 2024 21:41

Fixing Integ Tests

dc9020c

Signed-off-by: Varun Jain <[email protected]> Signed-off-by: Martin Gaievski <[email protected]>

martin-gaievski force-pushed the Pagination_in_hybrid_query branch from 4580337 to dc9020c Compare December 19, 2024 21:43

Use max_window_size as max for pagination depth

c3cacd1

Signed-off-by: Martin Gaievski <[email protected]>

vibrantvarun closed this Dec 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Pagination in hybrid query #963

[Feature] Pagination in hybrid query #963

vibrantvarun commented Oct 23, 2024 •

edited by martin-gaievski

Loading

martin-gaievski Oct 24, 2024

vibrantvarun Nov 4, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Oct 24, 2024

vibrantvarun Nov 4, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

	private static final ParseField DEPTH_FIELD = new ParseField("pagination_depth");
	private static final ParseField PAGINATION_DEPTH_FIELD = new ParseField("pagination_depth");

[Feature] Pagination in hybrid query #963

[Feature] Pagination in hybrid query #963

Conversation

vibrantvarun commented Oct 23, 2024 • edited by martin-gaievski Loading

Description

Related Issues

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vibrantvarun commented Oct 23, 2024 •

edited by martin-gaievski

Loading