-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DRAFT] nested spike discussion #193
Conversation
Codecov Report
@@ Coverage Diff @@
## draft-nested-spike #193 +/- ##
========================================================
- Coverage 98.36% 96.26% -2.11%
- Complexity 3643 3644 +1
========================================================
Files 343 348 +5
Lines 9017 9227 +210
Branches 585 605 +20
========================================================
+ Hits 8870 8882 +12
- Misses 142 334 +192
- Partials 5 11 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Can you please add some example queries? I'm not entirely sure how the nested function behaves and how it's meant to be used. |
@@ -149,6 +149,8 @@ public enum BuiltinFunctionName { | |||
STDDEV_POP(FunctionName.of("stddev_pop")), | |||
// take top documents from aggregation bucket. | |||
TAKE(FunctionName.of("take")), | |||
// TODO: nested aggregation function |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if we wanted to remove the TODO?
// TODO: nested aggregation function | |
//nested aggregation function |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, this should remain as it is not implemented yet.
@@ -53,6 +53,7 @@ public class TestsConstants { | |||
public final static String TEST_INDEX_BEER = TEST_INDEX + "_beer"; | |||
public final static String TEST_INDEX_NULL_MISSING = TEST_INDEX + "_null_missing"; | |||
public final static String TEST_INDEX_CALCS = TEST_INDEX + "_calcs"; | |||
public final static String TEST_INDEX_MULTI_NESTED= TEST_INDEX + "_multi_nested"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
public final static String TEST_INDEX_MULTI_NESTED= TEST_INDEX + "_multi_nested"; | |
public final static String TEST_INDEX_MULTI_NESTED = TEST_INDEX + "_multi_nested"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will be fixed in a non draft PR as there are many checkstyle failures with this PoC.
; | ||
|
||
nestedFunction | ||
: NESTED LR_BRACKET nestedField RR_BRACKET |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this support the following query?
SELECT nested(message.info) FROM multi_nested WHERE nested('message', message.info = 'a' OR message.info = 'a')
Specifically the second part
nested('message', message.info = 'a' OR message.info = 'a')
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does not. Nested in this PoC is only supported in SELECT. Nested in WHERE will be implemented in follow up tasks.
87d836a
to
e5fdf25
Compare
Sql Plugin integ tests require add files to keystore to add prometheus datasource. So adding a custom integ script which can be consumed by build repo while triggering integ tests during every release. Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: forestmvey <[email protected]>
Signed-off-by: Guian Gumpac <[email protected]>
Signed-off-by: forestmvey <[email protected]>
Signed-off-by: Guian Gumpac <[email protected]>
e5fdf25
to
da8a674
Compare
Description
This proposal will outline a suggestion for the scope and implementation of the nested function in the V2 engine.
Requirements
nested
in the new engine needs to support all functionality ofnested
in the legacy engine. This includes:nested
inSELECT
,WHERE
,GROUP BY
. example queries:SELECT nested(message.info) FROM multi_nested
SELECT nested(message.*) FROM multi_nested
SELECT nested(message.info) FROM multi_nested WHERE nested('message', message.info = 'a' OR message.info = 'a')
SELECT * FROM opensearch-sql_test_index_nested_type WHERE nested(message.info)='b';
SELECT nested(message.info) FROM multi_nested GROUP BY nested(message.info)
SELECT nested(message.info) FROM multi_nested GROUP BY message.info
alias of the result should be the argument of the
nested
functionnested
in function calls likeSELECT nested(message.info) FROM multi_nested WHERE wildcardquery(nested(message.info), 'z*')
multilevel nested field queries like
SELECT nested(message.author.address.street) FROM opensearch-sql_test_index_multi_nested
multiple nested functions in one query which is a limitation on elasticsearch like
SELECT nested(message.author.address.street), nested(message.info), nested(message.dayOfWeek) FROM opensearch-sql_test_index_multi_nested
nested
accepts strings as arguments for identifiers (This still needs to be investigated if it should not be supported). Example queries are:SELECT nested("field.subield") FROM test;
SELECT nested("field"."subield") FROM test;
SELECT nested('field.subield') FROM test;
SELECT nested('field'.'subield') FROM test;
Limitations
If we decide to not support strings as argument, then this will become a breaking change for 3.x
Implementation
The new SQL engine implementation of nested function will be similar to the implementation of nested function in legacy. The logical plan builder implementation will be the same. Legacy’s logical plan builder is a part of NestedFieldProjection while the new engine will contain it in OpenSearchRequestBuilder. This is currently implemented in this PoC for
nested
in only theSELECT
clause. Implementation fornested
inWHERE
andGROUP BY
may be different.Proof of Concept
The PoC does not satisfy all requirements for nested in
SELECT
. It only supports an end to end implementation ofSELECT nested(message.info) FROM opensearch-sql_test_index_nested_type;
. Further testing and support for more cases will be added once we are in agreement of the PoC.SQL queries to support for each stage of implementation
Schedule
nested
function working inSELECT
,WHERE
, andGROUP BY
Issues Resolved
opensearch-project#1111
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.