Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support json index #36750

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

feat: support json index #36750

wants to merge 3 commits into from

Conversation

sunby
Copy link
Contributor

@sunby sunby commented Oct 10, 2024

This PR adds json index support for json and dynamic fields. Now you can only do unary query like 'a["b"] > 1' using this index. We will support more filter type later.

basic usage:

collection.create_index("json_field", {"index_type": "INVERTED",
    "params": {"json_cast_type": DataType.STRING, "json_path":
'json_field["a"]["b"]'}})

There are some limits to use this index:

  1. If a record does not have the json path you specify, it will be ignored and there will not be an error.
  2. If a value of the json path fails to be cast to the type you specify, it will be ignored and there will not be an error.
  3. A specific json path can have only one json index.
  4. If you try to create more than one json indexes for one json field, sdk(pymilvus<=2.4.7) may return immediately because of internal implementation. This will be fixed in a later version.

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sunby
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Oct 10, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Oct 10, 2024
Copy link
Contributor

mergify bot commented Oct 10, 2024

@sunby Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

This PR adds json index support for json and dynamic fields. Now you can only do unary query like 'a["b"] > 1' using this index. We will support more filter type later.

basic usage:
```
collection.create_index("json_field", {"index_type": "INVERTED",
    "params": {"json_cast_type": DataType.STRING, "json_path":
'json_field["a"]["b"]'}})
```

There are some limits to use this index:
1. If a record does not have the json path you specify, it will be ignored and there will not be an error.
2. If a value of the json path fails to be cast to the type you specify,  it will be ignored and there will not be an error.
3. A specific json path can have only one json index.
4. If you try to create more than one json indexes for one json field, sdk(pymilvus<=2.4.7) may return immediately because of internal implementation. This will be fixed in a later version.

Signed-off-by: sunby <[email protected]>
Copy link
Contributor

mergify bot commented Oct 10, 2024

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

Copy link
Contributor

mergify bot commented Oct 10, 2024

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: sunby <[email protected]>
Copy link
Contributor

mergify bot commented Oct 12, 2024

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: sunby <[email protected]>
@sre-ci-robot sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Oct 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/internal-api dco-passed DCO check passed. do-not-merge/missing-related-issue kind/feature Issues related to feature request from users size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants