Publish startup optimized variants of the Zeebe image

[npepinpe]

Is your feature request related to a problem? Please describe.

In most cases, starting a Zeebe container is not very lightweight. A ZeebeContainer instance typically takes 10 seconds to be ready (and a ZeebeBrokerContainer is quite similar), and a ZeebeGatewayContainer takes about 4 seconds.

If you're running many tests which rely on re-configuring the container (since neither application supports hot-reloading), then you have to start a container per test, sometimes multiple. This quickly adds up once your code base has hundreds of these, resulting in lengthy build times spent on auxiliary things (i.e. not your tests).

Of course, the ideal would be a way to isolate tests in Zeebe, and reuse the containers. But this is currently not really feasible and isn't on the roadmap for that project.

Describe the solution you'd like

One option would be to provide, as part of zeebe-test-container, startup optimized variants of Zeebe. They could be provided via ghcr.io, and there would be one per container type - something like:

• ghcr.io/zeebe-test-container/zeebe-container:<ZEEBE VERSION>
• ghcr.io/zeebe-test-container/zeebe-broker-container:<ZEEBE VERSION>
• ghcr.io/zeebe-test-container/zeebe-gateway-container:<ZEEBE VERSION>

These would be essentially the same as the base Zeebe image for that version, but would add settings such as disabling compiler optimization for short-lived JVMs (e.g. -XX:+TieredCompilation -XX:TieredStopAtLevel=1) and pre-generating/loading application class data sharing (so-called AppCDS - see https://docs.oracle.com/en/java/javase/14/vm/class-data-sharing.html for more).

Generating the AppCDS is what provides the biggest advantage. In a quick prototype, I saw startup time of a ZeebeContainer go from 10s to 4s, which is more than 2x as fast. However, it makes the image substantially bigger, and requires some minor changes to the way the class path definition.

Describe alternatives you've considered

Building a native application would be another option, using GraalVM. This is another option, but it may have a higher maintenance overhead - any modification to Zeebe may require more extensive modification of the GraalVM build settings here.

Additional context

I prototyped this, and came up with the following notes:

• Generating the CDS for the application fails if the config folder is in the classpath. It can be removed safely during the Zeebe build, but then the log4j2 configuration file has to be explicitly set to config/log4j2.xml. I'm not sure how we would do this though from the base image. We'd have to find a way to remove something, or overload, the class path.
• The Dockerfile has to be updated to run the generation once. This can be done in a separate build stage, and copied over in the final stage to reduce the size somewhat (though it's still rather sizable).

Sample Dockerfile:

# Generate CDS for faster startup times
FROM camunda/zeebe:1.2.2 as appCds

COPY generate-cds.sh /tmp/generate-cds.sh
RUN /tmp/generate-cds.sh /usr/local/zeebe/bin/broker /usr/local/zeebe/bin/broker.jsa

# Building application image
FROM camunda/zeebe:1.2.2 as app

ENV JAVA_OPTS="-Xshare:auto -XX:SharedArchiveFile=/usr/local/zeebe/bin/zeebe.jsa -Dlog4j2.configurationFile=/usr/local/zeebe/config/log4j2.xml"
COPY --from=appCds --chown=:1000:0 /usr/local/zeebe/bin/zeebe.jsa /usr/local/zeebe/bin/broker.jsa

Sample generate-cds.sh script:

#!/bin/bash -xeu
set -oxe pipefail

APP="${1}"
ARCHIVE="${2}"

if [[ ! -f "$APP" ]]; then
  echo "Expected to find an application at $APP, but there is no such file"
  exit 1
fi

# generate the actual class data archive
JAVA_OPTS="-Xshare:auto -XX:ArchiveClassesAtExit=${ARCHIVE}" exec "$APP" &

# wait until application is ready and then stop it
curl --connect-timeout 30 -f -s -o /dev/null \
  --retry-connrefused --retry-max-time 20 --retry 10 --retry-delay 1 \
  "http://localhost:9600/ready"
echo "Application is ready, shutting it down..."
kill "$(jobs -p)"
wait

echo "Generated CDS for application ${APP} at ${ARCHIVE}"

[npepinpe]

Tried this recently again, and it didn't provide much of a speed up in terms of start up :(