-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] After enabling zstd codec memory usage increased drastically #9312
Comments
@mgodwan yes, that might be it. Is this planned to be relased soon as part of 2.9.x version or will be only in 2.10+? |
🤦🏻♂️ Oye. So I'm kind of confused here. Zstd codec was sandboxed for almost three months. Was this never used while sandboxed such that this regression could be benchmarked and tested before promoting as a LTS feature? Do we need to now add an extra bake layer in the form of FEATURE_FLAG sandbox promoted features to prevent this from happening in the future? |
Hmmm.... why not switch to direct memory and keep the pressure off the heap?
|
FTR introducing a core feature that relies on native libraries is SUPER trappy. I didn't realize the native dependencies when I reviewed the first experimental PR. Our portability could easily be burned by incompatibilities in glibc (we ran into this on kNN w/ FAISS not building on M1 macbooks). We need to seriously think about moving all zstd out as optional plugins. Perhaps we even cut a patch release that deprecates this w/ a |
What is the solution? To pull the future? |
It looks like the |
What about current users, will demoting the feature break the cluster unless they enable the experimental feature? Maybe provide upgrade instructions |
@andrross should we close this? I believe the issue itself is fixed in 2.10 |
The fix for this will be available in the 2.10 release |
With opensearch-project/opensearch-build#3971 this won't be the case for users of the official distribution. |
Describe the bug
After enabling
zstd
codec memory usage increased drastically.To Reproduce
zstd
codeczstd
memory usage increases drasticallyExpected behavior
Memory usage doesn't increase that drastically
Screenshots
Here are memory graphs for 2 nodes from our cluster, it's clearly visible memory usage after enabling
zstd
codec and new indexes created (on 8/10). The memory usage is from the OS metrics.Host/Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: