-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce memory copy in zstd compression #7681
Reduce memory copy in zstd compression #7681
Conversation
Gradle Check (Jenkins) Run Completed with:
|
...custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/ZstdCompressionMode.java
Show resolved
Hide resolved
@luyuncheng could you please fix DCO check https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin |
Curious what the impact on this change is, any benchmarks or ballpark numbers? cc: @backslasht @mulugetam who were doing some numbers |
Signed-off-by: luyuncheng <[email protected]>
a1b7871
to
1edf16f
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #7681 +/- ##
============================================
- Coverage 70.89% 70.67% -0.22%
+ Complexity 56242 56130 -112
============================================
Files 4682 4682
Lines 266102 266100 -2
Branches 39070 39070
============================================
- Hits 188658 188076 -582
- Misses 61478 62130 +652
+ Partials 15966 15894 -72
|
Signed-off-by: luyuncheng <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
@dblock looks like a legit fix (of not doing unneeded array copy), wdyt? |
Gradle Check (Jenkins) Run Completed with:
|
While this one looks straight forward but its always a good practise to understand the optimisation and check if there are regressions if any with performance |
@luyuncheng do you have any numbers? |
@dblock i think this optimize just reduce memory copy, mini benchmark can not show the exactly performance. AND it certainly not introduce any performance regression. i do some mini benchmark with rally: a little promote
|
Looks innocent enough. I merged it. Thanks @luyuncheng! |
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7681-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a0a8a181551c536228fc96bb0f27030b892ae51a
# Push it to GitHub
git push --set-upstream origin backport/backport-7681-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x Then, create a pull request where the |
@luyuncheng mind please creating a manual backport to |
* 1. reduce memory copy in zstd compression Signed-off-by: luyuncheng <[email protected]> * Add Change log Signed-off-by: luyuncheng <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> (cherry picked from commit a0a8a18)
Backported |
Thanks !!! |
* 1. reduce memory copy in zstd compression Signed-off-by: luyuncheng <[email protected]> * Add Change log Signed-off-by: luyuncheng <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> (cherry picked from commit a0a8a18) Co-authored-by: luyuncheng <[email protected]>
* 1. reduce memory copy in zstd compression Signed-off-by: luyuncheng <[email protected]> * Add Change log Signed-off-by: luyuncheng <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]>
…nsearch-project#7865) * 1. reduce memory copy in zstd compression Signed-off-by: luyuncheng <[email protected]> * Add Change log Signed-off-by: luyuncheng <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> (cherry picked from commit a0a8a18) Co-authored-by: luyuncheng <[email protected]>
* 1. reduce memory copy in zstd compression Signed-off-by: luyuncheng <[email protected]> * Add Change log Signed-off-by: luyuncheng <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>
As i see #3577 there is a zstd compression mode.
i think in some scenarios, like init compress buffer, we can save memory copy in
ArrayUtil.grow
instead of using ArrayUtil.growNoCopy