notebook/200401_Run_template.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RUN: {Name for run}\n",
    "\n",
    "```\n",
    "Lead     : {your name / github id} (ii = initials)\n",
    "Issue    : {github issue # if applicable}\n",
    "Version  : {git rev-parse HEAD}\n",
    "start    : YYYY MM DD\n",
    "complete : YYYY MM DD\n",
    "files    : {~/serratus/notebook/YYMMDD_ii/}           # Small repo files / plots\n",
    "s3_files : {s3://serratus-public/notebook/YYMMDD_ii/} # Bucket containing large (>500 kb) data files\n",
    "output   : {s3://serratus-public/out/YYMMDD_ii/}      # Output data files (bam/flagstat/etc...)\n",
    "```\n",
    "\n",
    "### Objectives\n",
    "- {Primary objective of this run}\n",
    "- {Secondary objective of this run}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serratus Initialization\n",
    "\n",
    "Local system initialization procedures for `serratus`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize local workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Serratus commit version\n",
    "SERRATUS=\"/home/artem/serratus\"\n",
    "cd $SERRATUS\n",
    "git rev-parse HEAD # commit version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create local run directory\n",
    "WORK=\"$SERRATUS/notebook/200423_ab\"\n",
    "mkdir -p $WORK; cd $WORK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SRA RunInfo Table for run\n",
    "aws s3 cp s3://serratus-public/sra/testing_SraRunInfo.csv ./\n",
    "RUNINFO=\"$WORK/testing_SraRunInfo.csv\"\n",
    "cat $RUNINFO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Packer / AMI Initialization (optional)\n",
    "Does not need to be ran each time if you have access to the AMI already.\n",
    "\n",
    "Current Build: `us-east-1: ami-046baafb2ee438b69`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd $SERRATUS/packer\n",
    "packer build docker-ami.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Serratus containers (optional)\n",
    "Serratus containers are available on the `serratusbio` dockerhub. If you wish to deploy your own containers, you will have to build them from the `serratus` repository and upload them to your own dockerhub account.\n",
    "\n",
    "This can be done with the `build.sh` script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd $SERRATUS\n",
    "\n",
    "# If you want to upload containers to your repository\n",
    "# include this.\n",
    "export DOCKERHUB_USER='serratusbio' # optional\n",
    "sudo docker login # optional\n",
    "\n",
    "# Build all containers and upload them docker hub repo\n",
    "# (if available)\n",
    "./build.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Terraform Initialization\n",
    "The Global Variables for Terraform file must be modified to initialize for your system.\n",
    "\n",
    "File: `$SERRATUS/terarform/main/terraform.tfvars`\n",
    "\n",
    "This step must be done manually in a text editor currently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Change these parameters in $SERRATUS/terarform/main/terraform.tfvars\n",
    "\n",
    "# Your public IP followed by \"/32\"\n",
    "LOCALIP=\"75.155.242.67/32\" #dev_cidrs\n",
    "# Your AWS key name\n",
    "KEYNAME=\"serratus\"         #key_name\n",
    "# Dockerhub account containing serratus containers\n",
    "DOCKERHUB_USER='serratusbio'    #dockerhub_account (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize terraform\n",
    "cd $SERRATUS/terraform/main\n",
    "terraform init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Launch Terraform Cluster\n",
    "# Initialize the serratus cluster with minimal nodes\n",
    "terraform apply -auto-approve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Serratus \n",
    "Upload the run data, scale-out the cluster, monitor performance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Monitors & Upload table\n",
    "\n",
    "Open SSH tunnels to monitor node then open monitors in browser\n",
    "\n",
    "- [Scheduler Table](localhost:8000/jobs/)\n",
    "- [Cluster Monitor: Grafana](http://localhost:3000/?orgId=1)\n",
    "- [Cluster Monitor: Prometheus](http://localhost:9090)\n",
    "\n",
    "\n",
    "#### Empty Scheduler Table (localhost:8000/jobs/)\n",
    "![Empty Table Load Screen](200423_ab/empty_scheduler.png)\n",
    "\n",
    "#### Ready Scheduler Table (localhost:8000/jobs/)\n",
    "![Empty Table Load Screen](200423_ab/ready_scheduler.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd $SERRATUS/terraform/main\n",
    "\n",
    "# Open SSH tunnels to the monitor\n",
    "./create_tunnels.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load SRA Run Info into scheduler (READY)\n",
    "curl -s -X POST -T $RUNINFO localhost:8000/jobs/add_sra_run_info/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scale up the cluster\n",
    "This will set-up 10 download, 10 align and 2 merge nodes to process data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "./dl_set_capacity.sh 10\n",
    "./align_set_capacity.sh 10\n",
    "./merge_set_capacity.sh 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can track the progress of accessions in the scheduler:\n",
    "\n",
    "![Running Scheduler](200423_ab/running_scheduler2.png)\n",
    "\n",
    "And monitor the performance of the cluster in the monitor:\n",
    "\n",
    "![Running Monitor](200423_ab/running_monitor.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When all downloading/splitting is done,\n",
    "# scale-in the downloaders\n",
    "./dl_set_capacity.sh 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When all alignment is done,\n",
    "# scale-in the aligners\n",
    "./align_set_capacity.sh 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When all merging is done,\n",
    "# scale in the mergers\n",
    "./merge_set_capacity.sh 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dump the Scheduler SQLITE table to a local file\n",
    "curl localhost:8000/db > \\\n",
    "  $SERRATUS/notebook/200423_ab/schedDump.sqlite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shutting down procedures\n",
    "\n",
    "Closing up shop.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save output of runs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output files are in two folders:\n",
    "# Bam and Bai files\n",
    "aws s3 ls s3://tf-serratus-work-<id>/out/bam/\n",
    "# Flagstat and RefCount files\n",
    "aws s3 ls s3://tf-serratus-work-<id>/out/flagstat/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy output to a permenant bucket\n",
    "# TODO: automatically transfer final outputs\n",
    "# to the permenant bucket\n",
    "aws s3 sync \\\n",
    "  s3://tf-serratus-work-<id>/out \\\n",
    "  s3://serratus-public/out/YYMMDD_ii/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Destroy Cluster\n",
    "\n",
    "Close out all resources with terraform (will take a few minutes).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terraform destroy -auto-approve\n",
    "# WARNING this will also delete the standard output bucket/data\n",
    "# Save data prior to destroy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run Notes\n",
    "Notes on this run for future reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Errors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}