Skip to content

Commit

Permalink
Espenhgn/issue12 (#14)
Browse files Browse the repository at this point in the history
* Push singularity .sif files to ghcr.io with tags
Fixes #12

* rename action

* name

* readme updates
  • Loading branch information
espenhgn authored May 2, 2024
1 parent 2c78524 commit 8bf44f0
Show file tree
Hide file tree
Showing 5 changed files with 144 additions and 46 deletions.
99 changes: 99 additions & 0 deletions .github/workflows/container_build_push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: container build push

on:
push:
branches:
- 'main'
tags:
- "v*.*.*"

jobs:
docker:
runs-on: ubuntu-latest
steps:
-
name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/${{ github.repository_owner }}/ldpred2
tags: |
type=ref,event=branch
type=semver,pattern={{version}}
type=sha
-
name: Set up QEMU
uses: docker/setup-qemu-action@v3
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
-
name: Login to GitHub Container Registry
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
-
name: Build and push
uses: docker/build-push-action@v5
with:
context: "{{defaultContext}}:docker"
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
file:
./dockerfiles/ldpred2/Dockerfile
platforms: linux/amd64

build-push-singularity-container:
needs: docker
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
container:
image: quay.io/singularity/singularity:v4.1.0
options: --privileged
name: Pull-push Singularity Container
steps:
- name: Check out code for the container builds
uses: actions/checkout@v4
- name: Install dependencies
run: |
apk add --no-cache curl
VERSION="1.1.0"
curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz"
mkdir -p oras-install/
tar -zxf oras_${VERSION}_*.tar.gz -C oras-install/
sudo mv oras-install/oras /usr/local/bin/
rm -rf oras_${VERSION}_*.tar.gz oras-install/
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/${{ github.repository_owner }}/ldpred2
tags: |
type=semver,pattern={{version}}
type=ref,event=branch
type=sha
- name: Pull and push Singularity container
run: |
tags="${{ steps.meta.outputs.tags }}"
tags_array=
old_ifs="$IFS"
IFS=$'\n'
for tag in $tags; do
tags_array="$tags_array $tag"
done
IFS="$old_ifs"
echo ${{ secrets.GITHUB_TOKEN }} | oras login --username ${{ github.repository_owner }} --password-stdin ghcr.io
for tag in $tags_array; do
echo "processing tag: $tag"
singularity pull ldpred2.sif docker://"$tag"
oras push "$tag" --artifact-type application/vnd.acme.rocket.config ldpred2.sif
rm ldpred2.sif
done
shell: sh
33 changes: 0 additions & 33 deletions .github/workflows/docker_build_push.yml

This file was deleted.

10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,13 @@ git pull
## Build status

[![License](http://img.shields.io/:license-GPLv3+-green.svg)](http://www.gnu.org/licenses/gpl-3.0.html)
[![Documentation Status](https://readthedocs.org/projects/container-template/badge/?version=latest)](https://container-template.readthedocs.io/en/latest/?badge=latest)
[![Documentation Status](https://readthedocs.org/projects/ldpred2_standalone/badge/?version=latest)](https://ldpred2_standalone.readthedocs.io/en/latest/?badge=latest)
[![Flake8 lint](https://github.com/comorment/ldpred2_standalone/actions/workflows/python.yml/badge.svg)](https://github.com/comorment/ldpred2_standalone/actions/workflows/python.yml)
[![Dockerfile lint](https://github.com/comorment/ldpred2_standalone/actions/workflows/docker.yml/badge.svg)](https://github.com/comorment/ldpred2_standalone/actions/workflows/docker.yml)

## Description of available containers

* ``ldpred2.sif`` - a minimal Singularity container with R based on [rocker/r-ver](https://rocker-project.org/images/versioned/r-ver.html), [PLINK](https://www.cog-genomics.org/plink/) version 1.9 and 2.0, and [PRSice-2](https://choishingwan.github.io/PRSice/), and R dependencies for running PGS using [LDpred2](https://privefl.github.io/bigsnpr/articles/LDpred2.html).
* ``ldpred2.sif`` - a minimal Singularity container with R and RStudio based on [rocker/r-ver](https://rocker-project.org/images/versioned/r-ver.html), [Python](https://python.org) with some common numerics and plotting packages, [PLINK](https://www.cog-genomics.org/plink/) version 1.9 and 2.0, and [PRSice-2](https://choishingwan.github.io/PRSice/), and R dependencies for running PGS using [LDpred2](https://privefl.github.io/bigsnpr/articles/LDpred2.html).


## Using
Expand All @@ -38,12 +38,12 @@ Please keep up to date when pushing new container builds:

## Building/rebuilding containers

For instructions on how to build or rebuild containers using [Docker](https://www.docker.com) and [Singularity](https://docs.sylabs.io) refer to [`<ldpred2_standalone>/docker/README.md`](https://github.com/comorment/ldpred2_standalone/blob/main/docker/README.md).
For instructions on how to build or rebuild containers using [Docker](https://www.docker.com) and [Singularity](https://docs.sylabs.io) or [Apptainer](https://apptainer.org) refer to [`<ldpred2_standalone>/docker/README.md`](https://github.com/comorment/ldpred2_standalone/blob/main/docker/README.md).

## Build the documentation

Within this repository, the html-documentation can be built from source files put here using [Sphinx](https://www.sphinx-doc.org/en/master/index.html).
To do so, install Sphinx and some additional packages in python using [Conda](https://docs.conda.io/en/latest/) by issuing:
To do so, install Sphinx and some additional packages in Python using [Conda](https://docs.conda.io/en/latest/) by issuing:

```
cd <ldpred2_standalone>/docs/source
Expand All @@ -60,7 +60,7 @@ The documentation may also be hosted online on [readthedocs.org](https://readthe
## SLURM jobscript example

A basic job script example for running a Singularity container in an HPC setting with the [SLURM](https://slurm.schedmd.com) job scheduler is provided in the file [singularity_slurm_job.sh](https://github.com/comorment/ldpred2_standalone/blob/main/scripts/singularity_slurm_job.sh), and should be modified as needed.
It expects a few environment variables, and can be submitted as
It expects a few environment variables and can be submitted as

```
export JOBNAME=ldpred2_standalone
Expand Down
16 changes: 10 additions & 6 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,21 @@

## Singularity containers

This repository is used to develop and document [Docker](https://www.docker.com) and [Singularity](https://docs.sylabs.io) containers with various software and analytical tools.
This repository directory is used to develop and document [Docker](https://www.docker.com) and [Singularity](https://docs.sylabs.io) containers with various software and analytical tools.

## Software versions

Below is the list of tools included in the different Dockerfiles and installer bash scripts for each container.
Please keep up to date (and update the main `<ldpred2_standalone>/README.md` when pushing new container builds):

### ldpred2.sif
### `ldpred2` container

| OS/tool | Version/Git tag | License | Source
| --------------------- | ----------------------------- | ----------------- | -------------
| Ubuntu | 22.04 (LTS) | [Creative Commons CC-BY-SA version 3.0 UK licence](https://ubuntu.com/legal/intellectual-property-policy) | [Ubuntu.com](https://ubuntu.com) |
| Rocker/r-ver | 3.2.1 | [GPL](https://github.com/rocker-org/rocker-versioned2/blob/master/LICENSE) | [rocker-project.org](https://rocker-project.org)
| R | 4.3.1 (2023-06-16) | [GPL-*](https://www.r-project.org/Licenses/) | [r-project.org](https://www.r-project.org) |
| Python | 3.11 | [PSF](https://docs.python.org/3/license.html) | [python.org](https://python.org) |
| PLINK-1.9 | [v2.00a4.5](https://github.com/chrchang/plink-ng/releases/tag/v2.00a4.5) | [GPL-3](https://github.com/chrchang/plink-ng/blob/master/1.9/LICENSE) | [https://www.cog-genomics.org/plink/](https://www.cog-genomics.org/plink/) |
| PLINK-2.0 | [v2.00a4.5](https://github.com/chrchang/plink-ng/releases/tag/v2.00a4.5) | [GPL-3](https://github.com/chrchang/plink-ng/blob/master/2.0/COPYING) | [https://www.cog-genomics.org/plink/](https://www.cog-genomics.org/plink/) |

Expand All @@ -25,11 +26,14 @@ If you face any issues, or if you need additional software, please let us know b

## Build instructions

For most users there is little reason to build the container from scratch, as pre-built containers are available from the [GitHub Container Registry](https://github.com/orgs/comorment/packages/container/package/ldpred2).
The following instructions are for those who wish to build the container locally or to modify the container.

### The easy(er) way

For convenience, a `Makefile` is provided in this directory in order to build [Singularity](https://docs.sylabs.io) containers from Dockerfiles (as `<ldpred2_standalone/src/dockerfiles/ldpred2/Dockerfile>`).
Using this files assumes that a working [Docker](https://www.docker.com) and [Singularity](https://docs.sylabs.io) installation, as well as the [`GNU make`](https://www.gnu.org/software/make/) utility is available on the host computer/build system.
On Debian-based Linux OS, this utility can usually be installed by issuing`apt-get install -y make`; on MacOS with [Homebrew](https://brew.sh) as`brew install make`. Prefix`sudo` if necessary.
Using this file assumes that a working [Docker](https://www.docker.com) and [Singularity](https://docs.sylabs.io) or [Apptainer](https://apptainer.org) installation, as well as the [`GNU make`](https://www.gnu.org/software/make/) utility is available on the host computer/build system.
On Debian-based Linux OS, this utility can usually be installed by issuing `apt-get install -y make`; on MacOS with [Homebrew](https://brew.sh) as`brew install make`. Prefix`sudo` if necessary.

Then, the container can be built by issuing:

Expand All @@ -46,7 +50,7 @@ sudo make ldpred2.sif

### Manual builds

In order to build the container manually, this is possible via the following steps
To build the container manually, this is possible via the following steps

```
docker build -t ldpred2 -f dockerfiles/ldpred2/Dockerfile . # build docker container
Expand All @@ -69,7 +73,7 @@ bash scripts/scripts/move_singularity_file.sh.sh ldpred2 # put ldpred2.sif file
```

Again, super-user (`sudo`) privileges may be required on the host computer. In that case, prefix `sudo` on the line(s) that fail.
For further details on the commands within each bash file, open the ``.sh`` files in a code editor.
For further details on the commands that are run within each bash file, open the ``.sh`` files in a code editor.

### Clean up

Expand Down
32 changes: 30 additions & 2 deletions scripts/LDpred2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,37 @@ If needed, please obtain them as described in the official [documentation](https

## Usage with ``ldpred2`` container

Minimal example invoking the R scripts provided here using:
Minimal example invoking the R scripts provided here using either Apptainer/Singularity or Docker is shown below.
The latter is recommended for MacOS users with M1/M2 and newer chipsets.

### Singularity
### Apptainer/Singularity

Apptainer [https://apptainer.org](https://apptainer.org) is a tool to run Singularity containers with a simple command line interface.

#### Pulling

The container can be pulled from the GitHub Container Registry [ghcr.io](https://ghcr.io) on hosts with internet access by issuing:

```
$ apptainer pull ghcr.io/comorment/ldpred2:latest
```

or

```
$ singularity pull docker://ghcr.io/comorment/ldpred2:latest
```

or using the [ORAS CLI](https://oras.land)

```
$ oras pull ghcr.io/comorment/ldpred2:latest
```

Either of the above will put the container in the current directory as `ldpred2.sif`.
The recommended location is `<path/to/ldpred2_standalone>/containers/ldpred2.sif`.

#### Using

```
$ export CONTAINER=<path/to/ldpred2_standalone>/containers/ldpred2.sif
Expand Down

0 comments on commit 8bf44f0

Please sign in to comment.