SDP Release 23.11

[adwk67]

Stackable Data Platform (SDP) Release 23.11 is now publicly available!

Highlights

This release introduces two further elements of platform stability - pod disruption budgets and graceful shutdown specifications - as well as a new management tool and updated product version support. See our blog for more details.

New / extended platform features

The following new major platform features were added:

PodDisruptionBudgets

Kubernetes has mechanisms to ensure minimal planned downtime. Our product operators deploy so-called PodDisruptionBudget (PDB) resources alongside the products. For every role that you specify (e.g. HDFS namenodes or Trino workers) a PDB is created. This will determine the extent to which roles for a given application may be inactive at any given time. See the documentation for more details.

Graceful shutdown

Graceful shutdown refers to the managed, controlled shutdown of service instances in the manner intended by the software authors. Typically, an instance will receive a signal indicating the intent for the server to shut down, and it will initiate a controlled shutdown. Our operators configure a sensible amount of time Pods are granted to properly shut down without disrupting the availability of the product. See the documentation for more details.

Signed SDP product images

As of this release all Stackable product images are signed (the signing of operator images was delivered in SDP 23.7). Please see this tutorial for more information.

Airflow KubernetesExecutor

Airflow clusters can now be configured to use Kubernetes executors, whereby pods are spun up for job tasks and terminated when complete, thus offering an alternative way to use resources without the need for job queuing.

Overridable Java security settings

For JVM-based products (i.e. Druid, HBase, HDFS, Hive, Kafka, NiFi, Spark, Trino and ZooKeeper) it is now possible to provide custom security settings that override the default values. This allows the user to control things such as DNS lookup caches.

Stackable Cockpit and stackablectl

This release includes a very early preview version of Stackable Cockpit, a browser-based management tool which interacts with the Stackable data platform to display e.g. deployed stacklets and their status. Our command line tool has been re-worked to use the same backbone as Stackable Cockpit: you can find out about the recent enhancements by visiting the online documentation.

Listener operator

The listener-operator was introduced in release 23.1 and the associated ServiceType field in 23.4. In this release we introduce configurable ListenerClass presets that map to the service types appropriate for different environments. This is discussed in more detail in the documentation.

New Versions

The following new product versions are now supported:

• Apache Airflow: 2.6.3, 2.7.2
• Apache Druid: 27.0.0
• Apache HBase: 2.4.17
• Apache HDFS: 3.2.4, 3.3.6
• Apache Kafka: 2.8.2, 3.4.1, 3.5.1
• Apache NiFi: 1.23.2
• OpenPolicyAgent: 0.57.0
• Apache Spark: 3.4.1, 3.5.0
• Apache Superset: 2.1.1, 3.0.1
• Trino: 428
• Apache ZooKeeper: 3.8.3

New product-specific features

Additionally, there are some other individual product features that are noteworthy:

• HBase: support for Hadoop native compression
• HBase: add operator tools to product image
• HDFS: support for FUSE
• Hive: update postgresql driver to support SCRAM authentication
• Spark: all product images contain pyspark
• Trino: support for the new OPA authorizer
• Vector: upgrade to 0.33.0

Learning Stackable

Further details on our release and how to upgrade can be found in our release notes as well as in the change logs of the individual operators.