From 4f8d39d1a6734c1499777d9ebd0fa8b15ce2803d Mon Sep 17 00:00:00 2001 From: Lachlan Donald Date: Wed, 30 Jan 2019 16:32:54 +1100 Subject: [PATCH 1/4] Add a warning about plugins and the agent in docker --- pages/agent/v3/docker.md.erb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/pages/agent/v3/docker.md.erb b/pages/agent/v3/docker.md.erb index 91794d0cb2..a22bf87b77 100644 --- a/pages/agent/v3/docker.md.erb +++ b/pages/agent/v3/docker.md.erb @@ -13,6 +13,12 @@ You can run the Buildkite Agent inside a Docker container using the [`buildkite/ <%= render_agent_setup :docker, nil, "v3" %> +## 🚨 Incompatibilities with plugins + +Whilst the agnt runs well in docker, there are presently some difficulties when combining your agent in a docker container and using many of our [plugins](https://buildkite.com/plugins). These can largely be mitigated by ensuring that the `builds-path` from the host is bind mounted into your agent container with the same path as on the host and having the buildkite-agent installed on the host so that it can be mounted into containers. + +See https://github.com/buildkite-plugins/docker-buildkite-plugin/issues/61 for more context. + ## Running on startup By default Docker [starts containers on restart](https://docs.docker.com/articles/host_integration/) so there's no need for upstart or similar. Simply start your container with `-d` to daemonize it and it will be restarted on system boot with the same environment variables and arguments. From b636f4262f3fc695b8545f3c69f8947960297a63 Mon Sep 17 00:00:00 2001 From: Harriet Lawrence Date: Wed, 30 Jan 2019 14:16:49 +0100 Subject: [PATCH 2/4] Minor words fixes, put url behind a link --- pages/agent/v3/docker.md.erb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pages/agent/v3/docker.md.erb b/pages/agent/v3/docker.md.erb index a22bf87b77..e550fdbdde 100644 --- a/pages/agent/v3/docker.md.erb +++ b/pages/agent/v3/docker.md.erb @@ -15,9 +15,9 @@ You can run the Buildkite Agent inside a Docker container using the [`buildkite/ ## 🚨 Incompatibilities with plugins -Whilst the agnt runs well in docker, there are presently some difficulties when combining your agent in a docker container and using many of our [plugins](https://buildkite.com/plugins). These can largely be mitigated by ensuring that the `builds-path` from the host is bind mounted into your agent container with the same path as on the host and having the buildkite-agent installed on the host so that it can be mounted into containers. +While the agent runs well in Docker, there are presently some difficulties when running an agent in a Docker container and using many of our [plugins](https://buildkite.com/plugins). These issues can largely be mitigated by ensuring that the `builds-path` from the host is bind mounted into your agent container with the same path as on the host, and having the `buildkite-agent` installed on the host so that it can be mounted into containers. -See https://github.com/buildkite-plugins/docker-buildkite-plugin/issues/61 for more context. +See [docker-buildkite-plugin GitHub issue #61](https://github.com/buildkite-plugins/docker-buildkite-plugin/issues/61) for more context. ## Running on startup From 24090485af7382c1ec24d7e737a4de2ad7d6f15c Mon Sep 17 00:00:00 2001 From: Lachlan Donald Date: Mon, 4 Feb 2019 11:15:30 +1100 Subject: [PATCH 3/4] Make caveat more generally about docker access --- pages/agent/v3/docker.md.erb | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/pages/agent/v3/docker.md.erb b/pages/agent/v3/docker.md.erb index e550fdbdde..295da1fdb1 100644 --- a/pages/agent/v3/docker.md.erb +++ b/pages/agent/v3/docker.md.erb @@ -13,11 +13,9 @@ You can run the Buildkite Agent inside a Docker container using the [`buildkite/ <%= render_agent_setup :docker, nil, "v3" %> -## 🚨 Incompatibilities with plugins +## 🚨 Caveats for builds that need Docker access -While the agent runs well in Docker, there are presently some difficulties when running an agent in a Docker container and using many of our [plugins](https://buildkite.com/plugins). These issues can largely be mitigated by ensuring that the `builds-path` from the host is bind mounted into your agent container with the same path as on the host, and having the `buildkite-agent` installed on the host so that it can be mounted into containers. - -See [docker-buildkite-plugin GitHub issue #61](https://github.com/buildkite-plugins/docker-buildkite-plugin/issues/61) for more context. +While the agent runs well in Docker, there are some challenges with giving the builds that run on that agent access to a docker socket, as volume mounts are relative to the host vs the paths in the agent container. This can cause compatibility issues with some of [plugins](https://buildkite.com/plugins). See [Invoking Docker from within a build](#invoking-docker-from-within-a-build) for more details. ## Running on startup @@ -115,15 +113,24 @@ Other options for configuring Git and SSH include: ## Invoking Docker from within a build -To invoke Docker from within builds you'll need to mount the Docker socket into the container: +To invoke Docker from within builds you can mount the host docker socket into the container. This can be tricky to get right, as paths that are mounted in containers created via the socket are relative to the host, not the container that the agent runs in, which can be very confusing. + +An example might be some code that calls `docker run --volume $PWD:/code ...`. The `$PWD` will resolve inside your agent container to something like `/buildkite/builds...`, which is the path inside the agent container. The docker daemon will try and mount that path from the host (where it doesn't exist) and docker will mount an empty directory without error to your container. + +The workaround to this is to mirror anything that you might want to volume mount into containers on the host system and in your agent container. Most usecases (especially [plugins](https://buildkite.com/plugins)) are the builds directories and `/usr/local/bin/buildkite-agent`. + +For example: ```bash -docker run -it \ - -v /var/run/docker.sock:/var/run/docker.sock \ - buildkite/agent:3 +docker run \ + -v /var/lib/buildkite/builds:/var/lib/buildkite/builds \ + -e "BUILDKITE_BUILDS_PATH=/var/lib/buildkite/builds" \ + -v "/usr/local/bin/buildkite-agent:/usr/local/bin/buildkite-agent" \ + -v "/var/run/docker.sock:/var/run/docker.sock" \ + buildkite/agent:3 start ``` -Note that this gives builds the same access to the host system as docker has, which is generally root. +Note that providing builds with a docker socket gives them access to whatever the docker daemon has access to on the host system. Typically this is `root`, which means builds have full root system access. This can be mitigated somewhat with [user namespace remapping](https://docs.docker.com/engine/security/userns-remap/), but caution should still be excercised. ## Entrypoint customizations From 1df5088cb3cdb6d193a7731283d6cb4a78f832b0 Mon Sep 17 00:00:00 2001 From: Harriet Lawrence Date: Mon, 4 Feb 2019 16:20:39 +0100 Subject: [PATCH 4/4] Words tweaks --- pages/agent/v3/docker.md.erb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pages/agent/v3/docker.md.erb b/pages/agent/v3/docker.md.erb index 295da1fdb1..f65ebc2674 100644 --- a/pages/agent/v3/docker.md.erb +++ b/pages/agent/v3/docker.md.erb @@ -15,7 +15,7 @@ You can run the Buildkite Agent inside a Docker container using the [`buildkite/ ## 🚨 Caveats for builds that need Docker access -While the agent runs well in Docker, there are some challenges with giving the builds that run on that agent access to a docker socket, as volume mounts are relative to the host vs the paths in the agent container. This can cause compatibility issues with some of [plugins](https://buildkite.com/plugins). See [Invoking Docker from within a build](#invoking-docker-from-within-a-build) for more details. +While the agent runs well in Docker, there are some challenges with giving that agent's builds access to a docker socket, as volume mounts are relative to the host rather than the paths in the agent container. This can cause compatibility issues with some [plugins](https://buildkite.com/plugins). See [Invoking Docker from within a build](#invoking-docker-from-within-a-build) for more details. ## Running on startup @@ -113,11 +113,11 @@ Other options for configuring Git and SSH include: ## Invoking Docker from within a build -To invoke Docker from within builds you can mount the host docker socket into the container. This can be tricky to get right, as paths that are mounted in containers created via the socket are relative to the host, not the container that the agent runs in, which can be very confusing. +To invoke Docker from within builds you can mount the host docker socket into the container. This can be tricky to get right, as paths that are mounted in containers created via the socket are relative to the host, not the container that the agent runs in. -An example might be some code that calls `docker run --volume $PWD:/code ...`. The `$PWD` will resolve inside your agent container to something like `/buildkite/builds...`, which is the path inside the agent container. The docker daemon will try and mount that path from the host (where it doesn't exist) and docker will mount an empty directory without error to your container. +For example, we have some code that calls `docker run --volume $PWD:/code ...`. The `$PWD` will resolve inside your agent container to something like `/buildkite/builds...`, which is the path inside the agent container. The docker daemon will try and mount that path from the host (where it doesn't exist) and docker will mount an empty directory without error to your container. -The workaround to this is to mirror anything that you might want to volume mount into containers on the host system and in your agent container. Most usecases (especially [plugins](https://buildkite.com/plugins)) are the builds directories and `/usr/local/bin/buildkite-agent`. +The workaround for this is to mirror anything that you might want to volume mount into containers on the host system and in your agent container. Most usecases (especially [plugins](https://buildkite.com/plugins)) are the builds directories and `/usr/local/bin/buildkite-agent`. For example: