Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add experimental support for zstd compression. #3577

Merged
merged 32 commits into from
Apr 4, 2023
Merged

Add experimental support for zstd compression. #3577

merged 32 commits into from
Apr 4, 2023

Commits on Jul 7, 2022

  1. Add experimental support for zstd and lz4 (native) compression. 

    Add experimental support for zstd (with and without dictionary support) and lz4 (native) compressions as discussed in: opensearch-project#3354.

Users would be able to set the index.codec setting with the values "lz4_native" (for lz4 native), "zstd" (for zstd with dictionary), and "zstd_no_dict" (for zstd without a dictionary).

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    fea4847 View commit details
    Browse the repository at this point in the history

  2. Fix license issues, add tests for zstd, zstd_no_dict, and lz4_native … 

    …compressions.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    8aac766 View commit details
    Browse the repository at this point in the history

  3. Fix DCO and and issues with CodecTests.java. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    33715b2 View commit details
    Browse the repository at this point in the history

  4. Fix forbidden api violation error for lz4-java. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    df93e0a View commit details
    Browse the repository at this point in the history

  5. Fix license headers. Remove and fix unnecessary fields. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    b0cf47e View commit details
    Browse the repository at this point in the history

  6. Fix magic numbers. Use more restrictive access modifiers. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    975cfd7 View commit details
    Browse the repository at this point in the history

  7. Use protected access modifier for Zstd and LZ4 compression mode classes. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    bdb9287 View commit details
    Browse the repository at this point in the history

  8. Allow negative compression levels for zstd. Use more restrictive acce… 

    …ss modifiers.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    1d6bbbb View commit details
    Browse the repository at this point in the history

  9. Use a more restrictive permission for loading zstd-jni and lz4-java l… 

    …ibraries.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    c6b39fa View commit details
    Browse the repository at this point in the history

  10. Rename a file (follow a consistent version naming convention). 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    1050738 View commit details
    Browse the repository at this point in the history

  11. Refactor and create a new custom-codecs sandbox module. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    cbfba87 View commit details
    Browse the repository at this point in the history

  12. Remove blank lines. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    3390e08 View commit details
    Browse the repository at this point in the history

  13. Restore Lucene92CustomCodec to extend from FilterCodec. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    a168c72 View commit details
    Browse the repository at this point in the history

  14. Make use of the compressionLevel argument. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    1491da7 View commit details
    Browse the repository at this point in the history

  15. Make Lucene92CustomCodec abstract and use a package-private access mo… 

    …difier.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    f74f011 View commit details
    Browse the repository at this point in the history

  16. Fix missing JavaDoc issues. Remove unused field in PerFieldMappingPos… 

    …tingFormatCodec.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    bdbe409 View commit details
    Browse the repository at this point in the history

  17. Fix lint errors. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    8120dae View commit details
    Browse the repository at this point in the history

  18. Fix the description for the custom-codecs plugin. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    28bd099 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2022

  1. Fix wildcard import and improve documentation. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Jul 8, 2022
    Configuration menu
    Copy the full SHA
    51b825b View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2023

  1. Access control exception fixed. Removed lz4-java support for now. 

    - PRs were made to zstd-jni and lz4-java to use AccessController.doPrivileged.
- The zstd-jni PR is merged since version 1.5.4-1.
- The lz4-java support temporarily removed until the PR gets merged.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    6757536 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2023

  1. Merge branch 'opensearch-project:main' into add-zstd-lz4-support

    @mulugetam
    mulugetam authored Feb 22, 2023
    Configuration menu
    Copy the full SHA
    9423abe View commit details
    Browse the repository at this point in the history

  2. Upgrade plugin to use Lucene95Codec. Rename files accordingly. 

    - Upgrade plugin to use Lucene95Codec. Rename files accordingly.
- Fix lint issue with plugin-security.
- Remove thridPartyAudit that was there for supporting lz4-java.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Feb 22, 2023
    Configuration menu
    Copy the full SHA
    45cd603 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2023

  1. Merge branch 'opensearch-project:main' into add-zstd-lz4-support

    @mulugetam
    mulugetam authored Mar 17, 2023
    Configuration menu
    Copy the full SHA
    c22d11d View commit details
    Browse the repository at this point in the history

  2. Add test cases for compression/decompression. Other minor changes. 

      - add test cases for compression/decompression.
  - rename package.
  - add a CHANGELOG entry.
  - add more checks for signed integer arithmetic.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    f39d093 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. Merge branch 'main' into add-zstd-lz4-support

    @dblock
    dblock authored Mar 21, 2023
    Configuration menu
    Copy the full SHA
    acf186f View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2023

  1. Remove ES grant in plugin-security.policy. Fix minor javadoc issue. 

      - Remove ES grant in plugin-security.policy file.
  - Replace @link and @see to fix javadoc error.

Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    ac1bd95 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'add-zstd-lz4-support' of https://github.com/mulugetam/O… 

    …penSearch into add-zstd-lz4-support
    @mulugetam
    mulugetam committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    b5ab68c View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2023

  1. Merge branch 'main' into add-zstd-lz4-support

    @dblock
    dblock authored Mar 27, 2023
    Configuration menu
    Copy the full SHA
    4d1e1a5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add-zstd-lz4-support

    @dblock
    dblock authored Mar 27, 2023
    Configuration menu
    Copy the full SHA
    634dfc0 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. Upgrade jettison version to 1.5.4. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    b48119d View commit details
    Browse the repository at this point in the history

  2. Update SHA for jettison 1.5.4. 

    Signed-off-by: Mulugeta Mammo <[email protected]>
    @mulugetam
    mulugetam committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    bc59912 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2023

  1. Merge branch 'main' of https://github.com/mulugetam/OpenSearch into a… 

    …dd-zstd-lz4-support
    @mulugetam
    mulugetam committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    06443ea View commit details
    Browse the repository at this point in the history