From ed3323b140adb63b87caa4470b88d0b4cf0031c2 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Mon, 12 Feb 2024 13:25:16 -0800 Subject: [PATCH 1/7] Stopgap message in release notes. --- CHANGES.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index bda5937a93d0..2fcb980aaa8f 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -72,6 +72,10 @@ * X behavior was changed ([#X](https://github.com/apache/beam/issues/X)). * Arrow version was bumped to 15.0.0 from 5.0.0 ([#30181](https://github.com/apache/beam/pull/30181)). +* Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes). + * The recommendation is to confinue to use the container for 2.53.0 as a base for now eg. `apache/beam_go_sdk:2.53.0`. + * The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on. + * If this affects you, please file an issue and contact the beam dev list. ## Deprecations From cb5f1ba047d3c9fda867a802a6be6922a32b0255 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Tue, 13 Feb 2024 08:14:49 -0800 Subject: [PATCH 2/7] Recommend 2.54.0 not 2.53.0. --- CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index 2fcb980aaa8f..1eab0cb48021 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -73,7 +73,7 @@ * X behavior was changed ([#X](https://github.com/apache/beam/issues/X)). * Arrow version was bumped to 15.0.0 from 5.0.0 ([#30181](https://github.com/apache/beam/pull/30181)). * Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes). - * The recommendation is to confinue to use the container for 2.53.0 as a base for now eg. `apache/beam_go_sdk:2.53.0`. + * The recommendation is to confinue to use the container for 2.54.0 as a base for now eg. `apache/beam_go_sdk:2.54.0`. * The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on. * If this affects you, please file an issue and contact the beam dev list. From c6bd5318ca5225127d0524662eaf68d127992e69 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Tue, 13 Feb 2024 10:04:28 -0800 Subject: [PATCH 3/7] Add from scratch instructions. --- CHANGES.md | 3 +- .../en/documentation/runtime/environments.md | 40 ++++++++++++++++++- 2 files changed, 40 insertions(+), 3 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 1eab0cb48021..607f1a1185fd 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -73,9 +73,8 @@ * X behavior was changed ([#X](https://github.com/apache/beam/issues/X)). * Arrow version was bumped to 15.0.0 from 5.0.0 ([#30181](https://github.com/apache/beam/pull/30181)). * Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes). - * The recommendation is to confinue to use the container for 2.54.0 as a base for now eg. `apache/beam_go_sdk:2.54.0`. * The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on. - * If this affects you, please file an issue and contact the beam dev list. + * See https://beam.apache.org/documentation/runtime/environments/#from-scratch-go for instructions on building and using a custom container. ## Deprecations diff --git a/website/www/site/content/en/documentation/runtime/environments.md b/website/www/site/content/en/documentation/runtime/environments.md index c860816e300d..1d43f9577f72 100644 --- a/website/www/site/content/en/documentation/runtime/environments.md +++ b/website/www/site/content/en/documentation/runtime/environments.md @@ -29,7 +29,7 @@ You may want to customize container images for many reasons, including: * Launching third-party software in the worker environment * Further customizing the execution environment - This guide describes how to create and use customized containers for the Beam SDK. + This guide describes how to create and use customized containers for the Beam SDKs. ### Prerequisites @@ -218,6 +218,44 @@ Beam offers a way to provide your own custom container image. The easiest way to docker push "${IMAGE_NAME}:${TAG}" ``` +#### Building a compatible container image from scratch (Go) {#from-scratch-go} + +From the 2.55.0 release, the Beam Go SDK has moved to using [distroless images](https://github.com/GoogleContainerTools/distroless) as a base. +These images have a reduced security attack surface by not including common tools and utilities. +This may cause difficulties customizing the image with using one of the above approaches. +As a fallback, it's possible to build a custom image from scratch, by building a matching boot loader, and setting +that as the container's entry point. + +If modifying a container image as above doesn't work, it's also possible to build one from scratch. +For example, if it's preferable to use alpine as the container OS your multi-stage docker file might +look like. + +``` +FROM golang:1.22-alpine AS build_base + +# Set the Current Working Directory inside the container +WORKDIR /tmp/beam + +# Build the Beam Go bootloader, to the local directory, matching your Beam version +# Similar go targets exist for other SDK languages. +RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/container@v2.53.0 + +# Set the real base image. +FROM alpine:3.9 +RUN apk add ca-certificates + +# Copy the boot loader `container` to the image. +COPY --from=build_base /tmp/beam/container /opt/apache/beam/boot + +# Set the container to use the newly built boot loader. +ENTRYPOINT ["/opt/apache/beam/boot"] +``` + +Build and push the new image as when [modifying an existing base image](#modify-existing-base-image) above. + +>**NOTE**: Java and Python require additional dependencies, such as their runtimes, and SDK packages for +> a valid container image. The bootloader isn't sufficient for creating a custom container for these SDKs. + ## Running pipelines with custom container images {#running-pipelines} The common method for providing a container image requires using the From 2735965583ae2d4d057da1f3afbfb77c93c65a2c Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Tue, 13 Feb 2024 10:11:41 -0800 Subject: [PATCH 4/7] remove specific go version --- .../site/content/en/documentation/runtime/environments.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/website/www/site/content/en/documentation/runtime/environments.md b/website/www/site/content/en/documentation/runtime/environments.md index 1d43f9577f72..4f2b81cf7574 100644 --- a/website/www/site/content/en/documentation/runtime/environments.md +++ b/website/www/site/content/en/documentation/runtime/environments.md @@ -231,12 +231,12 @@ For example, if it's preferable to use alpine as the container OS your multi-sta look like. ``` -FROM golang:1.22-alpine AS build_base +FROM golang:latest-alpine AS build_base # Set the Current Working Directory inside the container WORKDIR /tmp/beam -# Build the Beam Go bootloader, to the local directory, matching your Beam version +# Build the Beam Go bootloader, to the local directory, matching your Beam version. # Similar go targets exist for other SDK languages. RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/container@v2.53.0 @@ -244,6 +244,7 @@ RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/container@v2.53.0 FROM alpine:3.9 RUN apk add ca-certificates +# The following are required for the container to operate correctly. # Copy the boot loader `container` to the image. COPY --from=build_base /tmp/beam/container /opt/apache/beam/boot From 50ba4345795bdcf756cb03d6ea1122a96b823259 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Tue, 13 Feb 2024 10:32:11 -0800 Subject: [PATCH 5/7] ws --- .../www/site/content/en/documentation/runtime/environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/www/site/content/en/documentation/runtime/environments.md b/website/www/site/content/en/documentation/runtime/environments.md index 4f2b81cf7574..36d81eb2e60f 100644 --- a/website/www/site/content/en/documentation/runtime/environments.md +++ b/website/www/site/content/en/documentation/runtime/environments.md @@ -241,7 +241,7 @@ WORKDIR /tmp/beam RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/container@v2.53.0 # Set the real base image. -FROM alpine:3.9 +FROM alpine:3.9 RUN apk add ca-certificates # The following are required for the container to operate correctly. From 381fd628a5cad485e09030653ae6d2651afc28c6 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Tue, 13 Feb 2024 10:35:57 -0800 Subject: [PATCH 6/7] redundant line --- .../www/site/content/en/documentation/runtime/environments.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/www/site/content/en/documentation/runtime/environments.md b/website/www/site/content/en/documentation/runtime/environments.md index 36d81eb2e60f..07592f78c0c2 100644 --- a/website/www/site/content/en/documentation/runtime/environments.md +++ b/website/www/site/content/en/documentation/runtime/environments.md @@ -226,9 +226,8 @@ This may cause difficulties customizing the image with using one of the above ap As a fallback, it's possible to build a custom image from scratch, by building a matching boot loader, and setting that as the container's entry point. -If modifying a container image as above doesn't work, it's also possible to build one from scratch. For example, if it's preferable to use alpine as the container OS your multi-stage docker file might -look like. +look like the following: ``` FROM golang:latest-alpine AS build_base From 4b5f3c5c89ec43028404277bf7f5a439382467b4 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Tue, 13 Feb 2024 10:57:45 -0800 Subject: [PATCH 7/7] rm more ws --- .../www/site/content/en/documentation/runtime/environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/www/site/content/en/documentation/runtime/environments.md b/website/www/site/content/en/documentation/runtime/environments.md index 07592f78c0c2..d9a42db29e24 100644 --- a/website/www/site/content/en/documentation/runtime/environments.md +++ b/website/www/site/content/en/documentation/runtime/environments.md @@ -227,7 +227,7 @@ As a fallback, it's possible to build a custom image from scratch, by building a that as the container's entry point. For example, if it's preferable to use alpine as the container OS your multi-stage docker file might -look like the following: +look like the following: ``` FROM golang:latest-alpine AS build_base