[TW-83031] TeamCity Agent containers with Podman

configs/linux/Agent/Ubuntu/Ubuntu-sudo.Dockerfile

@@ -29,7 +29,7 @@ RUN apt-get update && \
     apt-get install -y --no-install-recommends sudo && \
     # https://github.com/goodwithtech/dockle/blob/master/CHECKPOINT.md#dkl-di-0005
     apt-get clean && rm -rf /var/lib/apt/lists/* && \
-    echo 'buildagent ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \  
+    echo 'buildagent ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \
     rm -rf /var/lib/apt/lists/*
 
 USER buildagent

custom/README.md

@@ -1,9 +1,21 @@
 # Custom TeamCity Agent Images
-
 The folder includes Dockerfiles that you can utilize to create custom TeamCity Agent images.
 
-# 1. .NET SDK
+# Content
+<!-- TOC -->
+* [1. .NET SDK](#1-net-sdk)
+    * [1.1. Building Images](#11-building-images)
+* [1.2 .NET End of Support Dates](#12-net-end-of-support-dates)
+* [2. Podman](#2-podman)
+    * [2.1 Building Images](#21-building-images)
+    * [2.2 Execution](#22-execution)
+        * [2.2.1 Rootless Podman in Docker (no '--privileged')](#221-rootless-podman-in-docker-no---privileged)
+        * [2.2.2 Rootful Podman in Docker ('--privileged')](#222-rootful-podman-in-docker---privileged)
+    * [2.3 Podman - troubleshooting](#23-podman---troubleshooting)
+        * [2.3.1 Inability to execute images with rootful Podman](#231-inability-to-execute-images-with-rootful-podman)
+<!-- TOC -->
 
+# 1. .NET SDK
 | OS      | Arch    | .NET SDK                                                                | Dockerfile                                                 | `dotnetSdkVersion` | `dotnetSdkChecksum`                                                                                                                |
 |---------|---------|-------------------------------------------------------------------------|------------------------------------------------------------|--------------------|------------------------------------------------------------------------------------------------------------------------------------|
 | Linux   | `AMD64` | [.NET Core 3.1](https://dotnet.microsoft.com/en-us/download/dotnet/3.1) | [link](linux/agent/amd/custom.dotnet.sdk.amd.Dockerfile)   | `3.1.426`          | `6c3f9541557feb5d5b93f5c10b28264878948e8540f2b8bb7fb966c32bd38191e6b310dcb5f87a4a8f7c67a7046fa932cde3cce9dc8341c1365ae6c9fcc481ec` |
@@ -17,14 +29,13 @@ The folder includes Dockerfiles that you can utilize to create custom TeamCity A
 | Windows | `AMD64` | [.NET 7.0](https://dotnet.microsoft.com/en-us/download/dotnet/7.0)      | [link](windows/agent/custom.dotnet.sdk.win.amd.Dockerfile) | `7.0.401`          | `02a4ecc05d0b9dfa0c9e32f8a3d288f329e7338b2430fcbc1276ae356f9d8e14920f91382f3f141842bf1e6e6cd331e532b301edc71c26de9d9e5ad2371afbe0` |
 
 
-The .NET SDK version bundled within TeamCity Docker Images is aligned with [Microsoft's Long Term Support (LTS) release](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) 
-at the moment of a TeamCity release. Since it is sometimes necessary to use STS (which can be newer than LTS) or 
+The .NET SDK version bundled within TeamCity Docker Images is aligned with [Microsoft's Long Term Support (LTS) release](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core)
+at the moment of a TeamCity release. Since it is sometimes necessary to use STS (which can be newer than LTS) or
 older versions, we provide examples of building images with custom .NET SDK versions inside.
 
 The folder contains Dockerfiles that simplify this process, allowing you to easily replace any .NET SDK version within the image with a pre-defined one. These Dockerfiles can also be used as templates for installing any custom .NET version.
 
 ## 1.1. Building Images
-
 The table above references multiple versions of .NET framework. To build a custom image, specify the required SDK version (`dotnetSdkVersion`) and a checksum for it (`dotnetSdkChecksum`):
 ```
 docker build \
@@ -64,11 +75,91 @@ docker run teamcity-agent:windows-custom-dotnet-7 dotnet --version
 ```
 
 # 1.2 .NET End of Support Dates
-
 In the [.NET and .NET Core Support Policy](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) article, Microsoft states the following end of support dates for .NET:
 * **.NET Core 3.1** -  December 13th, 2022;
 * **.NET 5.0** -  May 10th, 2022;
 * **.NET 6** (LTS) - November 12, 2024;
 * **.NET 7.0** (STS) - May 14, 2024;
 
 We strongly encourage replacing your current .NET versions to newer ones if the support for your current version is nearing its end.
+
+# 2. Podman
+This section provides instructions for building and executing TeamCity Docker Images with Podman in both rootless and rootful modes.
+
+Please, note that the latest version of Podman for Ubuntu 20.04 is `Podman 3.4.2`, as indicated by the [libcontainers](https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/amd64/).
+
+To use Podman as a default container engine in TeamCity, set `teamcity.container.wrapper.use.podman=true`.
+
+## 2.1 Building Images
+Rootless:
+```
+$ docker build \
+    --build-arg teamCityAgentImage=jetbrains/teamcity-agent:2023.05.4 \
+    -f linux/agent/podman.Dockerfile \
+    -t jebrains/teamcity-agent:2023.05.4-podman .
+```
+
+Rootful (based on `sudo` image):
+```
+$ docker build \
+--build-arg teamCityAgentImage=jetbrains/teamcity-agent:2023.05.4-linux-sudo\
+-f linux/agent/podman.Dockerfile \
+-t jebrains/teamcity-agent:2023.05.4-podman-sudo .
+```
+Please ensure the OS/Arch of Docker image matches the expected host (see [2.3.1 Inability to execute images with rootful Podman](#231-inability-to-execute-images-with-rootful-podman)).
+
+## 2.2 Execution
+### 2.2.1 Rootless Podman in Docker (no '--privileged')
+Running Podman-in-Docker in Rootless mode involves extending the capabilities
+of the container and its `buildserver` user.
+
+Capabilities:
+* `sys_admin` - root access for Podman in order to mount required file systems;
+* `mknod` - creates `/dev` devices, such as `fuse-overlayfs`;
+
+Security options:
+* `unconfined`, `disable` - responsible for disabling of SElinux for container file mount permissions;
+
+Storage options:
+* `--device=/dev/fuse` - use [FUSE](https://www.kernel.org/doc/html/next/filesystems/fuse.html) for Podman container storage;
+```
+$ docker run --cap-add=sys_admin \
+                --cap-add mknod \
+                --device=/dev/fuse \
+                --security-opt seccomp=unconfined \
+                --security-opt label=disable \
+                -e SERVER_URL="<server url>" \
+                -v <agent conf>:/data/teamcity_agent/conf \
+                jebrains/teamcity-agent:2023.05.4-podman  \
+                podman run ubi8-minimal echo hello
+```
+
+### 2.2.2 Rootful Podman in Docker ('--privileged')
+Rootful Podman can be launched from non-sudo images using the `--privileged` flag.
+```
+$ docker run -itd --privileged \
+    -u 0 \
+    -e SERVER_URL="<server url>" \
+    -v <agent conf>:/data/teamcity_agent/conf \
+    jebrains/teamcity-agent:2023.05.4-podman-sudo \
+    podman run ubi8-minimal echo hello
+```
+
+## 2.3 Podman - troubleshooting
+### 2.3.1 Inability to execute images with rootful Podman
+**Problem**: If _rootful Podman-in-Docker_ runs on a platform where the host platform does not match the detected one,
+container execution fails due to overlayFS issues, affecting _CRUN_ and container storage.
+
+```
+docker run --privileged -u 0 docker.io/jebrains/teamcity-agent:2023.05.4-sudo-with-podman-sudo podman run ubi8-minimal echo hello
+...
+Error: writing blob: adding layer with blob "sha256:395bceae1ad3587036e94ca53ad1a297204f1ffa8f3af10c5a96c3c13b8aec8d": Error processing tar file(exit status 1): Error while loading /: Permission denied
+
+ Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
+...
+11:18:52   Error: writing blob: adding layer with blob "sha256:f992cb38fce665360a4d07f6f78db864a1f6e20a7ad304219f7f81d7fe608d97": Error processing tar file(exit status 1): Error while loading /: Permission denied
+...
+Failed to re-execute libcrun via memory file descriptor
+```
+
+**Solution**: Build TeamCity Agent Image with _Podman_ using Agent image whose OS/Arch matches the target host (`arm64` / `amd64`).

custom/linux/agent/configs/podman/rootful.containers.conf

@@ -0,0 +1,12 @@
+[containers]
+cgroupns="host"
+cgroups="disabled"
+ipcns="host"
+utsns="host"
+netns="host"
+userns="host"
+log_driver = "k8s-file"
+[engine]
+cgroup_manager = "cgroupfs"
+runtime="crun"
+events_logger="file"

custom/linux/agent/configs/podman/rootless.containers.conf

@@ -0,0 +1,5 @@
+[containers]
+volumes = [
+	"/proc:/proc",
+]
+default_sysctls = []

custom/linux/agent/podman.Dockerfile

@@ -0,0 +1,76 @@
+#
+# Dockerfile for Linux-based images with Podman Container Runtime.
+# Version: latest stable version for Ubuntu 20.04 is taken - 3.4.2. ...
+# ... See: https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/amd64/)
+# Docstrings annotation with "[Rootless]" prefix are required for rootless execution only and can be ...
+# ... omitted if only rootful execution is required.
+#
+ARG teamCityAgentImage
+
+FROM ${teamCityAgentImage}
+USER root
+
+# Install Podman 3.* - latest stable release
+RUN mkdir -p /etc/apt/keyrings && \
+    curl -fsSL "https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_$(lsb_release -rs)/Release.key" \
+      | gpg --dearmor \
+      | tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null && \
+    echo \
+      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg]\
+                https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_$(lsb_release -rs)/ /" \
+              | tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null && \
+    apt-get update -qq && \
+    apt install conmon containernetworking-plugins && \
+    apt-get -qq -y install podman;
+
+# Giving access to additional groups as per https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#enable-unprivileged-ping
+RUN echo buildagent:10000:5000 > /etc/subuid; \
+    echo buildagent:10000:5000 > /etc/subgid; \
+    usermod -aG docker buildagent;
+
+# [Rootless] Enable unprivileged ping as per https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#enable-unprivileged-ping
+RUN sysctl -w "net.ipv4.ping_group_range=0 2000000"
+
+# Create directories required for Podman
+RUN mkdir -p /var/lib/shared/overlay-images  \
+    /var/lib/shared/overlay-layers  \
+    /var/lib/shared/vfs-images  \
+    /var/lib/shared/vfs-layers  \
+    /home/buildagent/.local/share/containers; \
+    touch /var/lib/shared/overlay-images/images.lock;  \
+    touch /var/lib/shared/overlay-layers/layers.lock;  \
+    touch /var/lib/shared/vfs-images/images.lock;  \
+    touch /var/lib/shared/vfs-layers/layers.lock;
+
+# Add configuration files to use configure overlayFS / FUSE properly
+COPY linux/agent/configs/podman/rootful.containers.conf /etc/containers/containers.conf
+COPY linux/agent/configs/podman/rootless.containers.conf /home/buildagent/.config/containers/containers.conf
+
+# Update access policy for configuration files (containers.conf, storage.conf), update storage configuration ...
+# ... to enable FUSE storage.
+RUN chmod 644 /etc/containers/containers.conf;  \
+    sed -i -e 's|^#mount_program|mount_program|g'  \
+    -e '/additionalimage.*/a "/var/lib/shared",'  \
+    -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g'  \
+    /etc/containers/storage.conf
+
+# Adjust container storage ownership to have the ability to execute containers via `buildagent`
+RUN chown -R buildagent:buildagent /home/buildagent/ \
+    /home/buildagent/.local/share/containers/storage \
+    /home/buildagent/.config/containers/containers.conf; \
+    chmod -R 755 /home/buildagent /home/buildagent/.config/containers/containers.conf;
+
+# [Rootless] Enable user namespace to prevent "cannot clone: Invalid argument" @ podman.
+RUN echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf
+
+USER buildagent
+
+#
+# Creation of volumes for Podman containers. Please, note that Docker sets "root:root" ownership by default.
+#
+# -- Rootful containers
+VOLUME /var/lib/containers
+# -- Rootless containers.
+VOLUME /home/buildagent/.local/share/containers
+
+ENV _CONTAINERS_USERNS_CONFIGURED=""

dockerhub/teamcity-agent/README.md

@@ -40,7 +40,7 @@ docker run -e SERVER_URL="<url to TeamCity server>"
     jetbrains/teamcity-agent
 ```
 &nbsp;
-where `<url to TeamCity server>` is the full URL for TeamCity server, accessible by the agent. Note that "localhost" will generally not work as it will refer to the "localhost" inside the container.  
+where `<url to TeamCity server>` is the full URL for TeamCity server, accessible by the agent. Note that "localhost" will not generally not work as it will refer to the "localhost" inside the container.  
 `<path to agent config folder>` is the host machine directory to serve as the TeamCity agent config directory. We recommend providing this binding in order to persist the agent configuration, e.g. authorization on the server. Note that you should map a different folder for every new agent you create.
 
 Since version 2020.1, TeamCity agent Docker images __run under a non-root user__. Refer to our [upgrade notes](https://www.jetbrains.com/help/teamcity/upgrade-notes.html#UpgradeNotes-AgentDockerimagesrunundernon-rootuser) for information on possible affected use cases.
@@ -121,7 +121,7 @@ docker run -e SERVER_URL="<url to TeamCity server>"  \
     -v <path to agent config folder>:/data/teamcity_agent/conf \
     -v docker_volumes:/var/lib/docker \
     --privileged -e DOCKER_IN_DOCKER=start \    
-    jetbrains/teamcity-agent:2023.11-linux-sudo
+    jetbrains/teamcity-agent:2021.1.1-linux-sudo
 ```
 
 The option `-v docker_volumes:/var/lib/docker` is related to the case when the `aufs` filesystem is used and when a build agent is started from a Windows machine ([related issue](https://youtrack.jetbrains.com/issue/TW-52939)).   
@@ -130,14 +130,12 @@ If you want to start several build agents, you need to specify different volumes
 ### Windows Containers Limitations
 
 The details on the known problems in Windows containers are available in the [TeamCity documentation](https://www.jetbrains.com/help/teamcity/known-issues.html#KnownIssues-WindowsDockerContainers).
-
+ 
 ## Customization
 
-**Leveraging existing Dockerfiles**. Please, refer to [custom TeamCity Agent Images for more information](https://github.com/JetBrains/teamcity-docker-images/tree/master/custom).
-
-**Manually**. To customise the Agent image manually, please follow the procedure below.
+You can customize the image via the usual Docker procedure:
 
-1. Create a container.
+1. Run the image
 ```
 docker run -e SERVER_URL="<url to TeamCity server>"  \ 
     -v <path to agent config folder>:/data/teamcity_agent/conf  \
@@ -148,14 +146,13 @@ docker run -e SERVER_URL="<url to TeamCity server>"  \
 ```
 docker exec -it my-customized-agent bash
 ```
-3. Please make any required adjustments as needed
+3. Change whatever you need
 
 4. Exit and [create a new image](https://docs.docker.com/engine/reference/commandline/commit/) from the container:
 ```
 docker commit my-customized-agent <the registry where you what to store the image>
 ```
 
-
 ## License
 
 The image is available under the [TeamCity license](https://www.jetbrains.com/teamcity/buy/license.html).