Add native Lz4, Snappy, and Zstd

[dain]

Add support for native Lz4, Snappy, and Zstd using the java.lang.foreign APIs

[wendigo]

That's awesome!

[wendigo]

I think that you could add Enable-Native-Access: ALL-UNNAMED to the MANIFEST.MF entries (https://docs.oracle.com/en/java/javase/22/core/restricted-methods.html)

[wendigo]

@dain we should apply for ARM runners so we can add a coverage here as well

[wendigo]

Right now some of the tests are not passing on ARM (due to the lack of the libgplcompression for aarch64)

[wendigo]

Some benchmarks:

jmh-result.json
benchmarks.log

[dain]

@wendigo

I think that you could add Enable-Native-Access: ALL-UNNAMED to the MANIFEST.MF entries (https://docs.oracle.com/en/java/javase/22/core/restricted-methods.html)

Ya, I think we should make this a proper module, not that most people use module capable systems.... follow up work

@dain we should apply for ARM runners so we can add a coverage here as well

Looks like the beta is open now https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/

Some benchmarks:

For benchmarks, you'll want to look at these algorithms:

• airlift_lz4
• airlift_lz4_native
• airlift_lzo
• airlift_snappy
• airlift_snappy_native
• airlift_zstd
• airlift_zstd_native

Then in the DataSet you'll want to narrow down to one of the collections, or it will take forever.

[wendigo]

Overall LGTM

[wendigo]

should we? or should this be configurable?

[dain]

Since we include our own libraries, loading this value from the library seems redundant.

[dnault]

Would you consider building a multi-release JAR, so us miserable folks stuck on Java 1.8 can still use the pure Java bits of this library?

[dain]

The 2.x branch (master) uses the new java.lang.foreign APIs that are only present in Java 22+. The release-0.x branch contains the code that works with Java 1.8.

[wendigo]

  compress    airlift_lz4             calgary/book2                  333,498   375.5MB/s ±    29.4MB/s ( 7.82%) (N = 3, α = 99.9%)
  compress    airlift_lz4_native      calgary/book2                  333,498   461.9MB/s ±    47.4MB/s (10.27%) (N = 3, α = 99.9%)
  compress    airlift_snappy          calgary/book2                  334,111   357.4MB/s ±    34.5MB/s ( 9.64%) (N = 3, α = 99.9%)
  compress    airlift_snappy_native   calgary/book2                  334,941   529.0MB/s ±   139.7MB/s (26.41%) (N = 3, α = 99.9%)
  compress    airlift_zstd            calgary/book2                  205,814   149.4MB/s ±    49.8MB/s (33.30%) (N = 3, α = 99.9%)
  compress    airlift_zstd_native     calgary/book2                  203,941   236.8MB/s ±    63.9MB/s (26.98%) (N = 3, α = 99.9%)
  decompress  airlift_lz4             calgary/book2                  333,498  2713.4MB/s ±   616.6MB/s (22.73%) (N = 3, α = 99.9%)
  decompress  airlift_lz4_native      calgary/book2                  333,498  3553.0MB/s ±   959.0MB/s (26.99%) (N = 3, α = 99.9%)
  decompress  airlift_snappy          calgary/book2                  334,111   735.0MB/s ±    26.7MB/s ( 3.64%) (N = 3, α = 99.9%)
  decompress  airlift_snappy_native   calgary/book2                  334,941  2225.0MB/s ±   105.1MB/s ( 4.72%) (N = 3, α = 99.9%)
  decompress  airlift_zstd            calgary/book2                  205,814   817.0MB/s ±    16.6MB/s ( 2.04%) (N = 3, α = 99.9%)
  decompress  airlift_zstd_native     calgary/book2                  203,941  1115.3MB/s ±   169.5MB/s (15.19%) (N = 3, α = 99.9%)

[wendigo]

LGTM

[wendigo]

Can you move the code to io.airlift.compressor package before merging? So it doesn't clash with the old aircompressor. That way we will be able to add a bridge that glues two APIs together.

[electrum]

you can specify the JAR-file manifest attribute Enable-Native-Access: ALL-UNNAMED in an executable JAR to enable warning-free use by all code on the class path

I think the key wording here is "an executable JAR". So this only works for the application's JAR being run with java -jar foo.jar. Otherwise, if any JAR could set this, it wouldn't protect the application (which is their goal with all of this).

[electrum]

Typo in commit message Add MessageSegment support ... should be MemorySegment

[electrum]

I actually like this idea. We could convert the OS names to macos and linux, which would make the directories cleaner:

String name = System.getProperty("os.name");
name = switch (name) {
    case "Linux" -> "linux";
    case "Mac OS X" -> "macos";
    default -> throw new LinkageError("Unsupported OS platform: " + name);
}

This will require changing the previous commit that downloads the native libraries.