tutorials/train_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model, you first need to convert your sequences and targets into the input HDF5 format. Check out my tutorials for how to do that; they're linked from the [main page](../README.md).\n",
    "\n",
    "For this tutorial, grab a small example HDF5 that I constructed here with 10% of the training sequences and only GM12878 targets for various DNase-seq, ChIP-seq, and CAGE experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os, subprocess\n",
    "\n",
    "if not os.path.isfile('data/heart_l131k.h5'):\n",
    "    subprocess.call('curl -o data/heart_l131k.h5 https://storage.googleapis.com/basenji_tutorial_data/heart_l131k.h5', shell=True)\n",
    "    subprocess.call('curl -o data/heart_l131k.bed https://storage.googleapis.com/basenji_tutorial_data/heart_l131k.bed', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you need to decide what sort of architecture to use. This grammar probably needs work; my goal was to enable hyperparameter searches to write the parameters to file so that I could run parallel training jobs to explore the hyperparameter space. I included an example set of parameters that will work well with this data in models/params_small.txt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, run [basenji_train.py](https://github.com/calico/basenji/blob/master/bin/basenji_train.py) to train a model. The program will offer training feedback via stdout and write the model output files to the prefix given by the *-s* parameter.\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --augment_rc | | Process even-numbered epochs as forward, odd-numbered as reverse complemented. |\n",
    "| --ensemble_rc | | Average forward and reverse complemented predictions on validation set. |\n",
    "| --augment_shifts | \"1,0,-1\" | Rotate epochs over small sequence shifts. |\n",
    "| --logdir | models/heart | Directory to save training logs and model checkpoints. |\n",
    "| --params | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| --data | data/heart_l131k.h5 | HDF5 file containing the training and validation input and output datasets as generated by [basenji_hdf5_single.py](https://github.com/calico/basenji/blob/master/bin/basenji_hdf5_single.py) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to train, uncomment the following line and run it. Depending on your hardware, it may require several hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ! basenji_train.py --augment_rc --ensemble_rc --augment_shifts \"1,0,-1\" --logdir models/heart --params models/params_small.txt --data data/heart_l131k.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can just download a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if not os.path.isdir('models/heart'):\n",
    "    os.mkdir('models/heart')\n",
    "if not os.path.isfile('models/heart/model_best.tf.meta'):\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.index https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.index', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.meta https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.meta', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.data-00000-of-00001 https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.data-00000-of-00001', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "models/heart/model_best.tf will now specify the name of your saved model to be provided to other programs.\n",
    "\n",
    "To further benchmark the accuracy (e.g. computing significant \"peak\" accuracy), use [basenji_test.py](https://github.com/calico/basenji/blob/master/bin/basenji_test.py).\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --rc | | Average the forward and reverse complement to form prediction. |\n",
    "| -o | output/heart_test | Output directory. |\n",
    "| --ai | 0,1,2 | Make accuracy scatter plots for targets 0, 1, and 2. |\n",
    "| --ti | 3,4,5 | Make BigWig tracks for targets 3, 4, and 5. |\n",
    "| -t | data/heart_l131k.bed | BED file describing sequence regions for BigWig track output. |\n",
    "| params_file | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| model_file | models/heart/model_best.tf | Trained saved model prefix. |\n",
    "| data_file | data/heart_l131k.h5 | HDF5 file containing the test input and output datasets as generated by [basenji_hdf5_single.py](https://github.com/calico/basenji/blob/master/bin/basenji_hdf5_single.py) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/davidkelley/anaconda3/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5\n",
      "  return f(*args, **kwds)\n",
      "{'adam_beta2': 0.98, 'adam_beta1': 0.97, 'target_pool': 128, 'cnn_filters': [128, 128, 192, 256, 256, 32, 32, 32, 32, 32, 32, 384], 'cnn_dilation': [1, 1, 1, 1, 1, 2, 4, 8, 16, 32, 64, 1], 'batch_size': 4, 'batch_buffer': 4096, 'loss': 'poisson', 'cnn_filter_sizes': [20, 7, 7, 7, 3, 3, 3, 3, 3, 3, 3, 1], 'cnn_dropout': 0.1, 'learning_rate': 0.002, 'cnn_dense': [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0], 'optimizer': 'adam', 'num_targets': 3, 'link': 'softplus', 'cnn_pool': [2, 4, 4, 4, 1, 0, 0, 0, 0, 0, 0, 0]}\n",
      "Targets pooled by 128 to length 1024\n",
      "Convolution w/ 3 384x1 filters to final targets\n",
      "Model building time 13s\n",
      "2018-06-05 18:27:30.205759: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n"
     ]
    }
   ],
   "source": [
    "! basenji_test.py --rc -o output/heart_test --ai 0,1,2 -t data/heart_l131k.bed --ti 0,1,2 --peaks models/params_small.txt models/heart/model_best.tf data/heart_l131k.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*data/gm12878_test/acc.txt* is a table specifiying the loss function value, R2, R2 after log2, and Spearman correlation for each dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   0  0.39378  0.17944  0.44621  0.46011  \n",
      "   1  0.20366  0.47488  0.70148  0.59639  \n",
      "   2  0.75878  0.22110  0.48164  0.43180  \n"
     ]
    }
   ],
   "source": [
    "! cat output/heart_test/acc.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*output/heart_test/peak.txt* is a table specifiying the number of peaks called, AUROC, and AUPRC for each dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   0     678  0.86775  0.27891\n",
      "   1     386  0.94903  0.48325\n",
      "   2     782  0.85143  0.27553\n"
     ]
    }
   ],
   "source": [
    "! cat output/heart_test/peaks.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directories *pr*, *roc*, *violin*, and *scatter* in *data/heart_test* contain plots for the targets indexed by 0, 1, and 2 as specified by the --ai option above.\n",
    "\n",
    "E.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"600\"\n",
       "            height=\"500\"\n",
       "            src=\"output/heart_test/pr/t0.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x108557278>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame('output/heart_test/pr/t0.pdf', width=600, height=500)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}