Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add instructions for Go custom container builds, link from 2.55.0 release notes. #30290

Merged
merged 7 commits into from
Feb 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@

* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).
* Arrow version was bumped to 15.0.0 from 5.0.0 ([#30181](https://github.com/apache/beam/pull/30181)).
* Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes).
* The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on.
* See https://beam.apache.org/documentation/runtime/environments/#from-scratch-go for instructions on building and using a custom container.

## Deprecations

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ You may want to customize container images for many reasons, including:
* Launching third-party software in the worker environment
* Further customizing the execution environment

This guide describes how to create and use customized containers for the Beam SDK.
This guide describes how to create and use customized containers for the Beam SDKs.

### Prerequisites

Expand Down Expand Up @@ -218,6 +218,44 @@ Beam offers a way to provide your own custom container image. The easiest way to
docker push "${IMAGE_NAME}:${TAG}"
```

#### Building a compatible container image from scratch (Go) {#from-scratch-go}

From the 2.55.0 release, the Beam Go SDK has moved to using [distroless images](https://github.com/GoogleContainerTools/distroless) as a base.
These images have a reduced security attack surface by not including common tools and utilities.
This may cause difficulties customizing the image with using one of the above approaches.
As a fallback, it's possible to build a custom image from scratch, by building a matching boot loader, and setting
that as the container's entry point.

For example, if it's preferable to use alpine as the container OS your multi-stage docker file might
look like the following:

```
FROM golang:latest-alpine AS build_base

# Set the Current Working Directory inside the container
WORKDIR /tmp/beam

# Build the Beam Go bootloader, to the local directory, matching your Beam version.
# Similar go targets exist for other SDK languages.
RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/[email protected]

# Set the real base image.
FROM alpine:3.9
RUN apk add ca-certificates

# The following are required for the container to operate correctly.
# Copy the boot loader `container` to the image.
COPY --from=build_base /tmp/beam/container /opt/apache/beam/boot

# Set the container to use the newly built boot loader.
ENTRYPOINT ["/opt/apache/beam/boot"]
```

Build and push the new image as when [modifying an existing base image](#modify-existing-base-image) above.

>**NOTE**: Java and Python require additional dependencies, such as their runtimes, and SDK packages for
> a valid container image. The bootloader isn't sufficient for creating a custom container for these SDKs.

## Running pipelines with custom container images {#running-pipelines}

The common method for providing a container image requires using the
Expand Down
Loading