multi-runners

🌈🌈🌈 Multi self-hosted GitHub action runners on single host! 🌈🌈🌈

Introduction

This application is designed for controlling multi self-hosted GitHub Action runners on single host, when Actions Runner Controller (ARC) is not feasible in your engineering environment. This application has following advantages:

• Single Linux host required.
• Single Bash script.
• Lightweight wrapper of GitHub official self-hosted runner.
• Both github.com and GitHub Enterprise are support.
• Either organization or repository or GitHub Cloud Enterprise level runners are supported.

Usage

mr.bash - https://github.com/vbem/multi-runners

Environment variables:
  MR_GITHUB_BASEURL=https://github.com
  MR_GITHUB_API_BASEURL=https://api.github.com
  MR_RELEASE_URL=<latest on github.com/actions/runner/releases>
  MR_USER_BASE=<default in /etc/default/useradd>
  MR_GITHUB_PAT=***

Sub-commands:
  add       Add one self-hosted runner on this host
            e.g. ./mr.bash add --org ORG --repo REPO --labels cloud:ali,region:cn-shanghai
            e.g. ./mr.bash add --org ORG --count 3
  del       Delete one self-hosted runner on this host
            e.g. ./mr.bash del --user runner-1
            e.g. ./mr.bash del --org ORG --count 3
  list      List all runners on this host
            e.g. ./mr.bash list
  download  Download GitHub Actions Runner release tar to /tmp/
            Detect latest on github.com/actions/runner/releases if MR_RELEASE_URL empty
            e.g. ./mr.bash download
  pat2token Get runner registration token from GitHub PAT (MR_GITHUB_PAT)
            e.g. ./mr.bash pat2token --org SOME_OWNER --repo SOME_REPO

Options:
  --enterprise  GitHub Cloud Enterprise name, optional
  --org         GitHub organization name
  --repo        GitHub repository name, registration on organization-level if empty
  --user        Linux local username of runner
  --labels      Extra labels for the runner
  --group       Runner group for the runner
  --token       Runner registration token, takes precedence over MR_GITHUB_PAT
  --dotenv      The lines to set in runner's '.env' files
  --count       The number to add or del, optional, defaults to 1 for add and all for del
  -h --help     Show this help.

Download this application

This application requires to be run under a Linux user with non-password sudo permission (e.g., %runners ALL=(ALL) NOPASSWD:ALL). It's also fine to run this application by root:

git clone https://github.com/vbem/multi-runners.git
cd multi-runners
./mr.bash --help

Setup PAT

This application requires a GitHub personal access token with smallest permissions and shortest expiration time. Only add/del/pat2token sub-commands need this PAT. You can remove it on GitHub after multi-runners' setup.

PAT types	Repository level runners	Organization level runners
Fine-grained PAT (recommended)	Referring to repository API, the administration:write permission is required.	Referring to organization policy & organization API, the organization_self_hosted_runners:write permission is required.
Classic PAT	Referring to repository API, need the repo scope	Refer to organization API, need the admin:org scope; if the repository is private, repo scope is also required.

During runtime, you can set your PAT in environment variable MR_GITHUB_PAT. To simplify subsequent execution, you can define any environment variable in .env file. For example,

# .env file under the directory of this application
MR_GITHUB_PAT='github_pat_***********'
ALL_PROXY=socks5h://localhost

You can run following command to check whether or not your PAT can generate GitHub Actions runners' registration-token:

./mr.bash pat2token --org <ORG-NAME> --repo <REPO-NAME>

Download the latest version of GitHub Actions package

If environment variable MR_RELEASE_URL is empty, this application will download the latest version of GitHub Actions runners tar package to local directory /tmp/ during runtime.

./mr.bash download

If your Linux host is internet bandwidth limited, you can also manually upload it from laptop to /tmp/<tar.gz file name>, and set the MR_RELEASE_URL env in .env file, e.g. /tmp/actions-runner-linux-x64-2.345.6.tar.gz.

GitHub Enterprise Server editions

GitHub Enterprise Server editions usually have different server and API URL prefixes comparing with github.com, you can set them in environment variables MR_GITHUB_BASEURL and MR_GITHUB_API_BASEURL.

GitHub Enterprise Cloud level registration

For GitHub Enterprise Cloud level registration, you can specify the --enterprise option to set the GitHub Enterprise Cloud name.

Setup multi-runners on single host

To setup multi-runners, you can simplify run following command multi times:

# 1 runner for repository `<ORG-NAME-1>/<REPO-NAME-1>`
./mr.bash add --org <ORG-NAME-1> --repo <REPO-NAME-1>

# 2 runners for repository `<ORG-NAME-1>/<REPO-NAME-2>`
./mr.bash add --org <ORG-NAME-1> --repo <REPO-NAME-2> --count 2

# 3 runners for organization `<ORG-NAME-2>`
./mr.bash add --org <ORG-NAME-2> --count 3

This application will create one Linux local user for one runner via useradd command. The Base Directory of these users is read from HOME setting in your /etc/default/useradd file by default (typically /home). You can also set it in environment variable MR_USER_BASE to override system-wide default.

List all runners on current host

This application also integrated status check of runners.

./mr.bash list

Which outputs,

runner-0 537M running https://github.com/<ORG-NAME-1>/<REPO-NAME-1>
runner-1 537M running https://github.com/<ORG-NAME-1>/<REPO-NAME-2>
runner-2 537M running https://github.com/<ORG-NAME-1>/<REPO-NAME-2>
runner-3 537M running https://github.com/<ORG-NAME-2>
runner-4 537M running https://github.com/<ORG-NAME-2>
runner-5 537M running https://github.com/<ORG-NAME-2>

Delete an existing runner

# delete an existing runner by its local Linux username.
./mr.bash del --user <runner-?>

# delete all runners for specific repository
./mr.bash del --org <ORG-NAME-1> --repo <REPO-NAME-2>

# delete multi runners by `--count` options.
./mr.bash del --org <ORG-NAME-2> --count 2

Specify runner in workflow file

In jobs.<job_id>.runs-on, target runners can be based on the labels as follows via GitHub context:

# For organization level self-hosted runners
runs-on: [self-hosted, '${{ github.repository_owner }}']

# For repository level self-hosted runners
runs-on: [self-hosted, '${{ github.repository }}']

Set environment variables into runners process

As described in GitHub official document, there's an approach to inject environment variables into runners process via the .env file before configuring or starting the self-hosted runners. This can be achieved via the --dotenv option, for example:

./mr.bash add --org <ORG> --repo <REPO> --dotenv 'TZ=Asia/Shanghai' --dotenv 'PATH=\$PATH:/mybin'

Then the following lines will be added to .env file located in self-hosted runner's directory before its configuring and starting:

TZ=Asia/Shanghai
PATH=$PATH:/mybin

Case Study - Deploy multi runners on single host which can not access GitHub directly

A multi-national corporation adopted GitHub as its centralized engineering efficiency platform. But in a country branch, according to some network blockade/bandwidth/QoS reasons, neither GitHub-hosted runners can access endpoints in this country stably, nor virtual machines in this country can access GitHub liberally.

In such a bad situation, we still need to setup reliable self-hosted runners in this country. What should we do? 🤣

A cost-conscious solution can be described as following architecture:

Endpoints <-------- VM-Runners ----> Firewall ----> VM-Proxy ----> GitHub
 \                          /                          |    \
  --------------------------                           |     ----> Other endpoints
     Branch office network                        Remote Proxy

A host VM-Runners is required for self-hosted runners, which is placed in this country and:

• Can access endpoints of this country branch
• Can NOT access GitHub directly or stably

A tiny specification host VM-Proxy is required as Remote Proxy, which is deployed in a place that:

• Can access GitHub directly and stably
• Can be accessed by VM-Runners directly and stably

Meanwhile, outbound traffics from VM-Runners MUST be routed by predefined rules:

• Requests to GitHub endpoints and non-local endpoints should be forward to the Remote Proxy on VM-Proxy
• Requests to local endpoints should be handled directly

Let's implement this solution. 🧐

On VM-Proxy, we can setup a Remote Proxy that's not easy to be blocked by the firewall, such as SS, TJ, XR, etc. These particular proxies have their own deployment and configuration methods. Please read their documents for more information. It's advised to set the outbound IP of VM-Runners as the only whitelist of the Remote Proxy port on VM-Proxy to avoid active detection from the firewall.

Before setup runners on VM-Runners, we need a Local Proxy on VM-Runners.

Usually firstly we need to setup the client of selected Remote Proxy which exposes a SOCKS5 proxy on VM-Runners (this local SOCKS5 localhost port will unconditionally forward all traffics to VM-Proxy), and then setup a privoxy on top of previous local SOCKS5 for domain based forwarding. These configurations are complex and prone to errors. Via Clash, we can combine both client of Remote Proxy and domain based forwarding into only one Local Proxy. The example configuration file and startup script of Clash and given in this repo's clash.example/ directory.

Assume the Local Proxy was launched as socks5h://localhost:7890, we can test it via following commands on VM-Runners:

# Without *Local Proxy*, it will print outbound public IP of *VM-Runners*
curl -s -4 icanhazip.com

# With *Local Proxy*, it will print outbound public IP of *VM-Proxy* !!!
all_proxy=socks5h://localhost:7890 curl -s -4 icanhazip.com

When Local Proxy is ready, we start self-hosted runners' setup on VM-Runners.

As VM-Runners Can NOT access GitHub directly or stably, use Local Proxy to clone this repository:

all_proxy=socks5h://localhost:7890 git clone https://github.com/vbem/multi-runners
cd multi-runners

As self-hosted runners' tar package downloading and registration-token fetching also requires communication with GitHub, we also configure Local Proxy for this application:

cat > .env <<- __
    MR_GITHUB_PAT='<paste-for-GitHub-PAT-here>'
    all_proxy='socks5h://localhost:7890'
__

To download the self-hosted runners' tar package from GitHub:

./mr.bash download

To validate your PAT has sufficient permissions for self-hosted runners registration on your GitHub organization https://github.com/<ORG-NAME>:

./mr.bash pat2token --org <ORG-NAME>

To setup two self-hosted runners on VM-Runners for your GitHub organization:

./mr.bash add --org <ORG-NAME> --dotenv 'all_proxy=socks5h://localhost:7890'
./mr.bash add --org <ORG-NAME> --dotenv 'all_proxy=socks5h://localhost:7890'

To check the status of self-hosted runners:

./mr.bash list

To check the Local Proxy works well in your runners' process, you can add a simple workflow .github/workflows/test-local-proxy.yml in your repository. If icanhazip.com was configured as a following-to-remote domain, the workflow run will print outbound public IP of VM-Proxy, even though this workflow actually runs on VM-Proxy.

---
name: Test Local Proxy works in my self-hosted runners
on:
  workflow_dispatch:
jobs:
  test:
    runs-on: [self-hosted, '${{ github.repository }}']
    steps:
      - run: |
          curl -s -4 icanhazip.com
...