03_run_nfcore_rnaseq_with_nfquilt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 3: Run Nextflow pipelines with `nf-quilt` plugin\n",
    "\n",
    "The Nextflow `nf-core/rnaseq pipeline`, in conjunction with `nf-quilt` was used to process raw sequencing data (fastqs) and generate per sample expression values. Samples were processed together in batches (called \"runs\"), mirroring common practice in NGS centers when multiple samples on a sequencing flow cell are pre-processed at the same time. The `nf-quilt` plugin automatically packages Nextflow pipeline output into a Quilt package, and appends detailed pipeline run metadata to the package.\n",
    "\n",
    "In this demo, we will use the raw data packages containing raw fastqs and sample-level package metadata as input to the `nf-core/rnaseq` pipeline to process RNA-sequencing data and generate sample-level expression measurements. \n",
    "\n",
    "\n",
    "For more detailed information on using `nf-quilt` plugin for Nextflow pipelines, please refer to the docs at https://github.com/nextflow-io/nf-quilt. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "# 1. Create a sample sheet\n",
    "\n",
    "The primary input to `nf-core/rnaseq` pipeline is a `samplesheet.csv`, which tells the pipeline where the input files are located. For more information on samplesheets, please refer to `nf-core` documentation: https://nf-co.re/rnaseq/3.14.0. \n",
    "\n",
    "Since we want to process all samples from a single batch together in the same pipeline run, we will create 1 sample sheet per sample batch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# user-specified params for specific batches of raw data to include in nf-core run\n",
    "registry = \"s3://quilt-example-bucket\"\n",
    "batch = \"ccle/20190225_PRJNA523380\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get list of packages from the project\n",
    "samples = []\n",
    "fastq1 = []\n",
    "fastq2 = []\n",
    "\n",
    "# get locations of fastqs from raw quilt packages\n",
    "for p in packages:\n",
    "    if batch in p:\n",
    "        print(\">>>>> \" + p)\n",
    "        pkg = quilt3.Package.browse(p, registry)\n",
    "        samples = samples + [pkg.meta[\"CellLine\"] + \"__\" + p.replace(\"ccle/20190225_PRJNA523380_\", \"\")]\n",
    "        pkg_files = list(pkg)\n",
    "        for file in pkg_files:\n",
    "            print(file)\n",
    "            if \"_1.fastq.gz\" in file:\n",
    "                fastq1 = fastq1 + [registry + \"/\" + batch + \"_\" + p.replace(\"ccle/20190225_PRJNA523380_\", \"\") + \"/\" + file]\n",
    "            if \"_2.fastq.gz\" in file:\n",
    "                fastq2 = fastq2 + [registry + \"/\" + batch +  \"_\" + p.replace(\"ccle/20190225_PRJNA523380_\", \"\") + \"/\" + file]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create sample sheet\n",
    "sample_sheet = pd.DataFrame({\"sample\": samples, \"fastq_1\": fastq1, \"fastq_2\": fastq2})\n",
    "sample_sheet.insert(sample_sheet.shape[1], \"strandedness\", \"auto\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save sample sheets locally, could also be to a location on s3\n",
    "outpath = \"~/data/nfcore\"\n",
    "sample_sheet.to_csv(outpath + \"/\" + batch.replace(\"ccle/\", \"\") + \"__samplesheet.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# upload sample sheet to quilt package\n",
    "# this quilt package also house nextflow pipeline outputs\n",
    "\n",
    "# define pkg name\n",
    "pname = \"ccle/20190225_PRJNA523380_nfcore_rnaseq\"\n",
    "\n",
    "# create the quilt package\n",
    "p = quilt3.Package()\n",
    "\n",
    "# stage sample sheet in pkg\n",
    "p.set(batch.replace(\"ccle/\", \"\") + \"__samplesheet.csv\",\n",
    "      outpath + \"/\" + batch.replace(\"ccle/\", \"\") + \"__samplesheet.csv\"\n",
    "     )\n",
    "\n",
    "# push sample sheet to bucket \n",
    "p.push(pname, registry='s3://quilt-example-bucket', message='upload sample sheet for batch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "# 2. Run `nf-core/rnaseq` with `nf-quilt`\n",
    "\n",
    "Now that we have the input sample sheet prepared, we can run the Nextflow pipeline with the `nf-quilt` plugin to package pipeline outputs automatically & append detailed pipeline run parameters as metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## 2.1 Seqera Tower\n",
    "\n",
    "\n",
    "Below we have listed the three places to make modifications to Nextflow runs in Seqera Tower Launchpad to enable packaging outputs with `nf-quilt` plugin. \n",
    "\n",
    "\n",
    "**`Pipeline parameters`**\n",
    "\n",
    "- Ensure the `--input` points to sample sheets created above; accepts both an s3 path or Quilt package URI\n",
    "- Ensure `--outdir` is a Quilt package URI beginning with `quilt+s3://`\n",
    "\n",
    "```json\n",
    "{\"input\":\"s3://quilt-example-bucket/ccle/20190224_PRJNA523380_nfcore_rnaseq/20190224_PRJNA523380__samplesheet.csv\",\n",
    "\"outdir\":\"quilt+s3://quilt-example-bucket#package=ccle/20190224_PRJNA523380_nfcore_rnaseq\",\n",
    "...\n",
    "}\n",
    "```\n",
    "\n",
    "**`Advanced options > Nextflow config file`**\n",
    "\n",
    "- You can specify a specific version of nf-quilt by using `'nf-quilt@0.7.12'`\n",
    "- We recommend using the latest version\n",
    "\n",
    "```yml\n",
    "plugins {\n",
    "    id 'nf-quilt'\n",
    "}\n",
    "```\n",
    "\n",
    "**`Advanced options > Pre-run script`**\n",
    "\n",
    "- paste the following text into the pre-run script field to install `nf-quilt` plugin requirements for the run\n",
    "\n",
    "```bash\n",
    "yum install python3-pip -y\n",
    "yum install git -y\n",
    "pip3 install quilt3\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "... and that's it! You can run your Nextflow pipeline!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## 2.2 Command-line\n",
    "\n",
    "Alternatively to Seqera Tower, you can launch Nextflow pipelines with the `nf-quilt` plugin directly from the command line. An example is show below. Similar to with Tower, be sure your output directory is a Quilt package URI, and specify the use of `nf-quilt` with the `plugins` option (plugin version is optional, if not specified will run with the latest version).  \n",
    "\n",
    "```bash\n",
    "nextflow run 'https://github.com/nf-core/rnaseq' \n",
    "\t-r 3.14.0\n",
    "\t-profile docker\n",
    "\t-input \"s3://quilt-example-bucket/ccle/20190224_PRJNA523380_nfcore_rnaseq/20190224_PRJNA523380__samplesheet.csv\"\n",
    "\t-outdir \"quilt+s3://quilt-example-bucket#package=ccle/20190224_PRJNA523380_nfcore_rnaseq\"\n",
    "\t-plugins nf-quilt@0.7.12\n",
    "\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}