Merge pull request #287 from t20100/update-libs

Updated embedded libraries:  c-blosc2 v2.12.0, zfp v1.0.1

doc/information.rst

@@ -71,12 +71,12 @@ HDF5 compression filters and compression libraries sources were obtained from:
   (commit `d48f960 <https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/d48f96064cb6e229ede4bf5e5c0e1935cf691036>`_)
   using LZ4.
 * `bitshuffle plugin <https://github.com/kiyo-masui/bitshuffle>`_ (v0.5.1) using LZ4 and ZStd.
-* bzip2 plugin (from `PyTables <https://github.com/PyTables/PyTables/>`_ v3.8.0)
+* bzip2 plugin (from `PyTables <https://github.com/PyTables/PyTables/>`_ v3.9.2)
   using `BZip2 <https://sourceware.org/git/bzip2.git>`_ (v1.0.8).
 * `hdf5-blosc plugin <https://github.com/Blosc/hdf5-blosc>`_ (v1.0.0)
   using `c-blosc <https://github.com/Blosc/c-blosc>`_ (v1.21.5), LZ4, Snappy, ZLib and ZStd.
-* hdf5-blosc2 plugin (from `PyTables <https://github.com/PyTables/PyTables/>`_ v3.9.2.dev0, commit `3ba4e78 <https://github.com/PyTables/PyTables/tree/3ba4e78336f21f5e32ddf49ced4560f610de70dd/hdf5-blosc2/src>`_)
-  using `c-blosc2 <https://github.com/Blosc/c-blosc2>`_ (v2.11.1), LZ4, ZLib and ZStd.
+* hdf5-blosc2 plugin (from `PyTables <https://github.com/PyTables/PyTables/>`_ v3.9.2)
+  using `c-blosc2 <https://github.com/Blosc/c-blosc2>`_ (v2.12.0), LZ4, ZLib and ZStd.
 * `FCIDECOMP plugin <ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_Test-Data/FCI_Decompression_Software_V1.0.2>`_ (v1.0.2)
   using `CharLS <https://github.com/team-charls/charls>`_
   (1.x branch, commit `25160a4 <https://github.com/team-charls/charls/tree/25160a42fb62e71e4b0ce081f5cb3f8bb73938b5>`_).
@@ -87,15 +87,15 @@ HDF5 compression filters and compression libraries sources were obtained from:
   (commit `4bbe9df7e4bcb <https://github.com/szcompressor/SZ3/commit/4bbe9df7e4bcb6ae6339fcb3033100da07fe7434>`_)
   using `SZ3 <https://github.com/szcompressor/SZ3>`_ and ZStd.
 * `HDF5-ZFP plugin <https://github.com/LLNL/H5Z-ZFP>`_ (v1.1.1)
-  using `zfp <https://github.com/LLNL/zfp>`_ (v1.0.0).
+  using `zfp <https://github.com/LLNL/zfp>`_ (v1.0.1).
 * `HDF5Plugin-Zstandard <https://github.com/aparamon/HDF5Plugin-Zstandard>`_
   (commit `d5afdb5 <https://github.com/aparamon/HDF5Plugin-Zstandard/tree/d5afdb5f04116d5c2d1a869dc9c7c0c72832b143>`_) using ZStd.
 
 Sources of compression libraries shared accross multiple filters were obtained from:
 
-* `LZ4 v1.9.4 <https://github.com/Blosc/c-blosc2/tree/v2.10.2/internal-complibs/lz4-1.9.4>`_
+* `LZ4 v1.9.4 <https://github.com/Blosc/c-blosc2/tree/v2.12.0/internal-complibs/lz4-1.9.4>`_
 * `Snappy v1.1.10 <https://github.com/google/snappy>`_
-* `ZStd v1.5.5 <https://github.com/Blosc/c-blosc2/tree/v2.10.2/internal-complibs/zstd-1.5.5>`_
+* `ZStd v1.5.5 <https://github.com/Blosc/c-blosc2/tree/v2.12.0/internal-complibs/zstd-1.5.5>`_
 * `ZLib v1.2.13 <https://github.com/Blosc/c-blosc/tree/v1.21.5/internal-complibs/zlib-1.2.13>`_
 
 When compiled with Intel IPP, the LZ4 compression library is replaced with `LZ4 v1.9.3 <https://github.com/lz4/lz4/releases/tag/v1.9.3>`_ patched with a patch from Intel IPP 2021.7.0.

src/c-blosc2/.github/workflows/fuzz.yml

@@ -18,7 +18,7 @@ jobs:
         dry-run: false
 
     - name: Upload Crash
-      uses: actions/upload-artifact@v3
+      uses: actions/upload-artifact@v4
       if: failure()
       with:
         name: artifacts

src/c-blosc2/ANNOUNCE.md

@@ -1,12 +1,12 @@
-# Announcing C-Blosc2 2.11.1
+# Announcing C-Blosc2 2.12.0
 A fast, compressed and persistent binary data store library for C.
 
 ## What is new?
 
-This is a maintenance release for fixing the ALTIVEC header.
-Only affects to IBM POWER builds.
-
-Also, some other fixes and improvements are in.
+Now the `grok` codec is available globally and will be loaded dynamically. See more
+info about the codec in our blog post: https://www.blosc.org/posts/blosc2-grok-release/
+Furthermore, a new function has been added to get the unidimensional chunk indexes
+needed to get the slice of a Blosc2 container.
 
 For more info, please see the release notes in:
 

src/c-blosc2/CMakeLists.txt

@@ -285,7 +285,10 @@ if(CMAKE_SYSTEM_PROCESSOR STREQUAL i386 OR
         else()
             set(COMPILER_SUPPORT_AVX2 FALSE)
         endif()
-        if(CMAKE_C_COMPILER_VERSION VERSION_GREATER 4.9 OR CMAKE_C_COMPILER_VERSION VERSION_EQUAL 4.9)
+        # GCC 10.3.2 (the version in manylinux_2014) seems to have issues supporting dynamic dispatching
+        # of AVX512.  GCC 11.4 is the first minimal version that works well here.
+        # That means that Linux wheels will have AVX512 disabled, but that's life.
+        if(CMAKE_C_COMPILER_VERSION VERSION_GREATER 11.4 OR CMAKE_C_COMPILER_VERSION VERSION_EQUAL 11.4)
             set(COMPILER_SUPPORT_AVX512 TRUE)
         else()
             set(COMPILER_SUPPORT_AVX512 FALSE)
@@ -297,7 +300,9 @@ if(CMAKE_SYSTEM_PROCESSOR STREQUAL i386 OR
         else()
             set(COMPILER_SUPPORT_AVX2 FALSE)
         endif()
-        if(CMAKE_C_COMPILER_VERSION VERSION_GREATER 10.0 OR CMAKE_C_COMPILER_VERSION VERSION_EQUAL 10.0)
+        # Clang 13 is the minimum version that we know that works with AVX512 dynamic dispatch.
+        # Perhaps lesser versions work too, better to err on the safe side.
+        if(CMAKE_C_COMPILER_VERSION VERSION_GREATER 13.0 OR CMAKE_C_COMPILER_VERSION VERSION_EQUAL 13.0)
             set(COMPILER_SUPPORT_AVX512 TRUE)
         else()
             set(COMPILER_SUPPORT_AVX512 FALSE)

src/c-blosc2/README.rst

@@ -250,4 +250,4 @@ See `THANKS document <https://github.com/Blosc/c-blosc2/blob/main/THANKS.rst>`_.
 
 ----
 
-**Enjoy data!**
+-- The Blosc Development Team.  **We make compression better.**

src/c-blosc2/RELEASE_NOTES.md

@@ -1,6 +1,45 @@
 Release notes for C-Blosc2
 ==========================
 
+Changes from 2.11.3 to 2.12.0
+=============================
+
+* New `blosc2_get_slice_nchunks` function for getting the unidimensional chunk indexes of a Blosc2 container slice.
+
+* Globally registered new codec `grok`. This will be loaded dynamically.
+
+
+Changes from 2.11.2 to 2.11.3
+=============================
+
+* Frames accept now typesizes that are larger than 255 (and up to 2**31).
+  See https://github.com/PyTables/PyTables/issues/1086.  Thanks to
+  @chris-allan for the report.
+
+* AVX512 runtime dispatching has been fixed (basically disabled) for GCC
+  versions <= 10.
+
+* Use typedef for blosc_timestamp_t.  Thanks to Magnus Ulimoen.
+
+
+Changes from 2.11.1 to 2.11.2
+=============================
+
+* Added support for ARMv7l platforms (Raspberry Pi).  The NEON version
+  of the bitshuffle filter was not compiling there, and besides it offered
+  no performance advantage over the generic bitshuffle version (it is 2x to
+  3x slower actually). So bitshuffle-neon.c has been disabled by default in
+  all ARM platforms.
+
+* Also, unaligned access has been disabled in all ARM non-64bits platforms.
+  It turned out that, at least the armv7l CPU in Raspberry Pi 4, had issues
+  because `__ARM_FEATURE_UNALIGNED` C macro was asserted in the compiler
+  (both gcc and clang), but it actually made binaries to raise a "Bus error".
+
+* Thanks to Ben Nuttall for providing a Raspberry Pi for tracking down these
+  issues.
+
+
 Changes from 2.11.0 to 2.11.1
 =============================
 

src/c-blosc2/blosc/CMakeLists.txt

@@ -280,7 +280,9 @@ if(NOT CMAKE_SYSTEM_PROCESSOR STREQUAL arm64)
 endif()
 if(COMPILER_SUPPORT_NEON)
     message(STATUS "Adding run-time support for NEON")
-    list(APPEND SOURCES blosc/shuffle-neon.c blosc/bitshuffle-neon.c)
+    # bitshuffle-neon.c does not offer better speed than generic on arm64 (Mac M1).
+    # Besides, it does not compile on raspberry pi (armv7l), so disable it.
+    list(APPEND SOURCES blosc/shuffle-neon.c)  # blosc/bitshuffle-neon.c)
 endif()
 if(COMPILER_SUPPORT_ALTIVEC)
     message(STATUS "Adding run-time support for ALTIVEC")

src/c-blosc2/blosc/b2nd-private.h

@@ -0,0 +1,40 @@
+/*********************************************************************
+  Blosc - Blocked Shuffling and Compression Library
+
+  Copyright (c) 2021  The Blosc Development Team <[email protected]>
+  https://blosc.org
+  License: BSD 3-Clause (see LICENSE.txt)
+
+  See LICENSE.txt for details about copyright and rights to use.
+**********************************************************************/
+
+
+#ifndef BLOSC_B2ND_PRIVATE_H
+#define BLOSC_B2ND_PRIVATE_H
+
+#include "b2nd.h"
+
+#include <stdbool.h>
+#include <stdlib.h>
+#include <stdint.h>
+
+/*********************************************************************
+
+  Functions meant to be used internally.
+
+*********************************************************************/
+
+/**
+ * @brief Get the chunk indexes needed to get the slice.
+ *
+ * @param array The b2nd array.
+ * @param start The coordinates where the slice will begin.
+ * @param stop The coordinates where the slice will end.
+ * @param chunks_idx The pointer to the buffer where the indexes of the chunks will be written.
+ *
+ * @return The number of chunks needed to get the slice. If some problem is
+ * detected, a negative code is returned instead.
+ */
+int b2nd_get_slice_nchunks(b2nd_array_t *array, const int64_t *start, const int64_t *stop, int64_t **chunks_idx);
+
+#endif /* BLOSC_B2ND_PRIVATE_H */

src/c-blosc2/blosc/b2nd.c

@@ -926,6 +926,92 @@ int b2nd_get_slice(b2nd_context_t *ctx, b2nd_array_t **array, const b2nd_array_t
 }
 
 
+int b2nd_get_slice_nchunks(b2nd_array_t *array, const int64_t *start, const int64_t *stop, int64_t **chunks_idx) {
+  BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER);
+  BLOSC_ERROR_NULL(start, BLOSC2_ERROR_NULL_POINTER);
+  BLOSC_ERROR_NULL(stop, BLOSC2_ERROR_NULL_POINTER);
+
+  int8_t ndim = array->ndim;
+
+  // 0-dim case
+  if (ndim == 0) {
+    *chunks_idx = malloc(1 * sizeof(int64_t));
+    *chunks_idx[0] = 0;
+    return 1;
+  }
+
+  int64_t chunks_in_array[B2ND_MAX_DIM] = {0};
+  for (int i = 0; i < ndim; ++i) {
+    chunks_in_array[i] = array->extshape[i] / array->chunkshape[i];
+  }
+
+  int64_t chunks_in_array_strides[B2ND_MAX_DIM];
+  chunks_in_array_strides[ndim - 1] = 1;
+  for (int i = ndim - 2; i >= 0; --i) {
+    chunks_in_array_strides[i] = chunks_in_array_strides[i + 1] * chunks_in_array[i + 1];
+  }
+
+  // Compute the number of chunks to update
+  int64_t update_start[B2ND_MAX_DIM];
+  int64_t update_shape[B2ND_MAX_DIM];
+
+  int64_t update_nchunks = 1;
+  for (int i = 0; i < ndim; ++i) {
+    int64_t pos = 0;
+    while (pos <= start[i]) {
+      pos += array->chunkshape[i];
+    }
+    update_start[i] = pos / array->chunkshape[i] - 1;
+    while (pos < stop[i]) {
+      pos += array->chunkshape[i];
+    }
+    update_shape[i] = pos / array->chunkshape[i] - update_start[i];
+    update_nchunks *= update_shape[i];
+  }
+
+  int nchunks = 0;
+  // Initially we do not know the number of chunks that will be affected
+  *chunks_idx = malloc(array->sc->nchunks * sizeof(int64_t));
+  int64_t *ptr = *chunks_idx;
+  for (int update_nchunk = 0; update_nchunk < update_nchunks; ++update_nchunk) {
+    int64_t nchunk_ndim[B2ND_MAX_DIM] = {0};
+    blosc2_unidim_to_multidim(ndim, update_shape, update_nchunk, nchunk_ndim);
+    for (int i = 0; i < ndim; ++i) {
+      nchunk_ndim[i] += update_start[i];
+    }
+    int64_t nchunk;
+    blosc2_multidim_to_unidim(nchunk_ndim, ndim, chunks_in_array_strides, &nchunk);
+
+    // Check if the chunk is inside the slice domain
+    int64_t chunk_start[B2ND_MAX_DIM] = {0};
+    int64_t chunk_stop[B2ND_MAX_DIM] = {0};
+    for (int i = 0; i < ndim; ++i) {
+      chunk_start[i] = nchunk_ndim[i] * array->chunkshape[i];
+      chunk_stop[i] = chunk_start[i] + array->chunkshape[i];
+      if (chunk_stop[i] > array->shape[i]) {
+        chunk_stop[i] = array->shape[i];
+      }
+    }
+    bool chunk_empty = false;
+    for (int i = 0; i < ndim; ++i) {
+      chunk_empty |= (chunk_stop[i] <= start[i] || chunk_start[i] >= stop[i]);
+    }
+    if (chunk_empty) {
+      continue;
+    }
+
+    ptr[nchunks] = nchunk;
+    nchunks++;
+  }
+
+  if (nchunks < array->sc->nchunks) {
+    *chunks_idx = realloc(ptr, nchunks * sizeof(int64_t));
+  }
+
+  return nchunks;
+}
+
+
 int b2nd_squeeze(b2nd_array_t *array) {
   BLOSC_ERROR_NULL(array, BLOSC2_ERROR_NULL_POINTER);
 

src/c-blosc2/blosc/blosc2.c

@@ -13,6 +13,8 @@
 #include "blosc-private.h"
 #include "../plugins/codecs/zfp/blosc2-zfp.h"
 #include "frame.h"
+#include "b2nd-private.h"
+#include "schunk-private.h"
 
 #if defined(USING_CMAKE)
   #include "config.h"
@@ -4656,3 +4658,25 @@ void blosc2_multidim_to_unidim(const int64_t *index, int8_t ndim, const int64_t
     *i += index[j] * strides[j];
   }
 }
+
+int blosc2_get_slice_nchunks(blosc2_schunk* schunk, int64_t *start, int64_t *stop, int64_t **chunks_idx) {
+  BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_NULL_POINTER);
+  if (blosc2_meta_exists(schunk, "b2nd") < 0) {
+    // Try with a caterva metalayer; we are meant to be backward compatible with it
+    if (blosc2_meta_exists(schunk, "caterva") < 0) {
+      return schunk_get_slice_nchunks(schunk, *start, *stop, chunks_idx);
+    }
+  }
+
+  b2nd_array_t *array;
+  int rc = b2nd_from_schunk(schunk, &array);
+  if (rc < 0) {
+    BLOSC_TRACE_ERROR("Could not get b2nd array from schunk.");
+    return rc;
+  }
+  rc = b2nd_get_slice_nchunks(array, start, stop, chunks_idx);
+  array->sc = NULL; // Free only array struct
+  b2nd_free(array);
+
+  return rc;
+}

src/c-blosc2/blosc/frame.c

@@ -428,8 +428,8 @@ int get_header_info(blosc2_frame_s *frame, int32_t *header_len, int64_t *frame_l
   }
   if (typesize != NULL) {
     from_big(typesize, framep + FRAME_TYPESIZE, sizeof(*typesize));
-    if (*typesize <= 0 || *typesize > BLOSC_MAX_TYPESIZE) {
-      BLOSC_TRACE_ERROR("`typesize` is zero or greater than max allowed.");
+    if (*typesize <= 0) {
+      BLOSC_TRACE_ERROR("`typesize` cannot be zero or negative.");
       return BLOSC2_ERROR_INVALID_HEADER;
     }
   }

src/c-blosc2/blosc/schunk-private.h

@@ -0,0 +1,40 @@
+/*********************************************************************
+  Blosc - Blocked Shuffling and Compression Library
+
+  Copyright (c) 2021  The Blosc Development Team <[email protected]>
+  https://blosc.org
+  License: BSD 3-Clause (see LICENSE.txt)
+
+  See LICENSE.txt for details about copyright and rights to use.
+**********************************************************************/
+
+
+#ifndef BLOSC_SCHUNK_PRIVATE_H
+#define BLOSC_SCHUNK_PRIVATE_H
+
+#include "b2nd.h"
+
+#include <stdbool.h>
+#include <stdlib.h>
+#include <stdint.h>
+
+/*********************************************************************
+
+  Functions meant to be used internally.
+
+*********************************************************************/
+
+/**
+ * @brief Get the chunk indexes needed to get the slice.
+ *
+ * @param schunk The super-chunk.
+ * @param start Index (0-based) where the slice begins.
+ * @param stop The first index (0-based) that is not in the selected slice.
+ * @param chunks_idx The pointer to the buffer where the indexes will be written.
+ *
+ *
+ * @return The number of chunks needed to get the slice. If some problem is
+ * detected, a negative code is returned instead.
+ */
+int schunk_get_slice_nchunks(blosc2_schunk *schunk, int64_t start, int64_t stop, int64_t **chunks_idx);
+#endif /* BLOSC_SCHUNK_PRIVATE_H */

src/c-blosc2/blosc/schunk.c

@@ -1379,6 +1379,27 @@ int blosc2_schunk_set_slice_buffer(blosc2_schunk *schunk, int64_t start, int64_t
 }
 
 
+int schunk_get_slice_nchunks(blosc2_schunk *schunk, int64_t start, int64_t stop, int64_t **chunks_idx) {
+  BLOSC_ERROR_NULL(schunk, BLOSC2_ERROR_NULL_POINTER);
+
+  int64_t byte_start = start * schunk->typesize;
+  int64_t byte_stop = stop * schunk->typesize;
+  int64_t nchunk_start = byte_start / schunk->chunksize;
+  int64_t nchunk_stop = byte_stop / schunk->chunksize;
+  if (byte_stop % schunk->chunksize != 0) {
+    nchunk_stop++;
+  }
+  int64_t nchunk = nchunk_start;
+  int nchunks = (int)(nchunk_stop - nchunk_start);
+  *chunks_idx = malloc(nchunks * sizeof(int64_t));
+  int64_t *ptr = *chunks_idx;
+  for (int64_t i = 0; i < nchunks; ++i) {
+    ptr[i] = nchunk;
+    nchunk++;
+  }
+  return nchunks;
+}
+
 /* Reorder the chunk offsets of an existing super-chunk. */
 int blosc2_schunk_reorder_offsets(blosc2_schunk *schunk, int64_t *offsets_order) {
   // Check that the offsets order are correct