Compression Methods in support

[Jahway]

Hi everyone,

I'm currently working with blosc2.Codec and noticed that the codec list contains the following compression methods:

• BLOSCLZ
• LZ4
• LZ4HC
• ZLIB
• ZSTD
• NDLZ
• ZFP_ACC
• ZFP_PREC
• ZFP_RATE
• OPENHTJ2K
• GROK

However, when I call blosc2.compressor_list(), only some of these are returned:

• BLOSCLZ
• LZ4
• LZ4HC
• ZLIB
• ZSTD

It seems like some of the other compression methods (e.g., NDLZ, ZFP, OPENHTJ2K, GROK) are not available by default.

Does this mean I need to install additional libraries or dependencies to enable these other compression methods? If yes, is there any guidance or documentation on how to install and configure these additional codecs?

Thanks!

[martaiborra]

Hi, thank you for opening this issue since it may be possible that other users have encountered the same problem.

You were not fully wrong. We could say that Blosc2 (both, C-Blosc2 and Python-Blosc2) has 3 "types" of codecs. The first ones are the ones that are included natively in the library, their id goes from 0 to blosc2.DEFINED_CODECS_STOP (which is 31). These are the ones listed with:

blosc2.compressor_list()
[<Codec.BLOSCLZ: 0>,
 <Codec.LZ4: 1>,
 <Codec.LZ4HC: 2>,
 <Codec.ZLIB: 4>,
 <Codec.ZSTD: 5>]

The second level, are the ones which are registered as Blosc2 global codecs, they are considered plugins. Their id goes from 32 to blosc2.GLOBAL_REGISTERED_CODECS_STOP (which is 159). This can be listed with the same function:

blosc2.compressor_list(plugins=True)
[<Codec.BLOSCLZ: 0>,
 <Codec.LZ4: 1>,
 <Codec.LZ4HC: 2>,
 <Codec.ZLIB: 4>,
 <Codec.ZSTD: 5>,
 <Codec.NDLZ: 32>,
 <Codec.ZFP_ACC: 33>,
 <Codec.ZFP_PREC: 34>,
 <Codec.ZFP_RATE: 35>,
 <Codec.OPENHTJ2K: 36>,
 <Codec.GROK: 37>]

Some of them may already be included in the library if you install the default version (and not the lite), but some others will need to be installed manually. We will be adding in the api reference which plugins need to be installed (for now, you can check the version in main). From the list above, the ones which need to be installed are Codec.OPENHTJ2K and Codec.GROK. They can be installed with pip install blosc2-openhtj2k and pip install blosc2-grok respectively. For more info you can check their pypi.org page or their repositories.

The third level, are the ones defined by the user. Their id goes from 160 to blosc2.USER_REGISTERED_CODECS_STOP (which is 255). These codecs will not appear in the list returned by blosc2.compressor_list. If you want to list them, you can print the registry of user-defined codecs: blosc2.ucodecs_registry. You can check this by printing the variable before and after the codec is registered in the ucodecs.py example.

This explanation also works when speaking about filters. Hope it was useful.