notebook/200423_Run_cov2r_pilot_test_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RUN: COV2R Pilot Run\n",
    "\n",
    "```\n",
    "Lead     : ababaian\n",
    "Issue    : n/a\n",
    "Version  : af64c2da1fdb3815dcabb0db3aafd2443c0c470d <git rev-parse HEAD> \n",
    "start    : 2020 04 23\n",
    "complete : 2020 04 23\n",
    "files    : ~/serratus/notebook/200423_ab/\n",
    "s3)files : n/a\n",
    "output   : s3://serratus-public/out/200423_ab_cov2r/\n",
    "```\n",
    "\n",
    "### Objectives\n",
    "- Create a re-usable template for running `serratus`\n",
    "- Run the 49 SRA test datasets with the current standard `serratus` against the `cov2r` pan-genome.\n",
    "- Compare `cov2r` alignment statistics to `cov0r` alignments\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serratus Initialization\n",
    "Prerequisites for running Serratus\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize local workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "af64c2da1fdb3815dcabb0db3aafd2443c0c470d\r\n"
     ]
    }
   ],
   "source": [
    "# Serratus commit version\n",
    "SERRATUS=\"/home/artem/serratus\"\n",
    "cd $SERRATUS\n",
    "git rev-parse HEAD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "# Create local run directory\n",
    "WORK=\"$SERRATUS/notebook/200423_ab\"\n",
    "mkdir -p $WORK; cd $WORK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 23.1 KiB/23.1 KiB with 1 file(s) remaining\r",
      "download: s3://serratus-public/sra/testing_SraRunInfo.csv to ./testing_SraRunInfo.csv\r\n",
      "Run,ReleaseDate,LoadDate,spots,bases,spots_with_mates,avgLength,size_MB,AssemblyName,download_path,Experiment,LibraryName,LibraryStrategy,LibrarySelection,LibrarySource,LibraryLayout,InsertSize,InsertDev,Platform,Model,SRAStudy,BioProject,Study_Pubmed_id,ProjectID,Sample,BioSample,SampleType,TaxID,ScientificName,SampleName,g1k_pop_code,source,g1k_analysis_group,Subject_ID,Sex,Disease,Tumor,Affection_Status,Analyte_Type,Histological_Type,Body_Site,CenterName,Submission,dbgap_study_accession,Consent,RunHash,ReadHash\r\n",
      "SRR11454614,2020-04-02 00:08:41,2020-04-01 00:45:40,5758629,1736681196,5758629,301,634,,https://sra-download.ncbi.nlm.nih.gov/traces/sra60/SRR/011186/SRR11454614,SRX8032203,HBCDC-HB-01/2019,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,PAIRED,0,0,ILLUMINA,Illumina MiSeq,SRP254688,PRJNA616446,,616446,SRS6404538,SAMN14479128,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-01/2019,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,05229E4AAAAB625773B0D0DA94254AF3,21F247CFE19CCCBEC0534387577A4B2D\r\n",
      "SRR11454608,2020-04-02 00:08:41,2020-04-01 00:42:25,2690410,725883101,2690410,269,221,,https://sra-download.ncbi.nlm.nih.gov/traces/sra58/SRR/011186/SRR11454608,SRX8032209,HBCDC-HB-01/2020,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,PAIRED,0,0,ILLUMINA,Illumina iSeq 100,SRP254688,PRJNA616446,,616446,SRS6404544,SAMN14479134,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-01/2020,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,3A10EFA7FFBF58F2DE67285D4D0C0CF1,9E93E6B73250BD01865563C96F7CAE8B\r\n",
      "SRR11454615,2020-04-02 00:08:41,2020-04-01 00:46:06,5229309,1576262654,5229309,301,583,,https://sra-download.ncbi.nlm.nih.gov/traces/sra63/SRR/011186/SRR11454615,SRX8032202,HBCDC-HB-02/2019,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,PAIRED,0,0,ILLUMINA,Illumina MiSeq,SRP254688,PRJNA616446,,616446,SRS6404537,SAMN14479127,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-02/2019,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,E5B530697787EB189A5800D65123050C,EF7F003465B9861230D6B8AAABCB824E\r\n",
      "SRR11454609,2020-04-02 00:08:41,2020-04-01 00:44:06,17121629,1284122175,0,75,493,,https://sra-download.ncbi.nlm.nih.gov/traces/sra64/SRR/011186/SRR11454609,SRX8032208,HBCDC-HB-02/2020,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 550,SRP254688,PRJNA616446,,616446,SRS6404543,SAMN14479133,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-02/2020,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,01C04D313F4A7C03A868857C73656D3C,C16910274209E3B973FA2A2495FEE1DF\r\n",
      "SRR11454613,2020-04-02 00:08:41,2020-04-01 00:46:16,5418689,1633652238,5418689,301,602,,https://sra-download.ncbi.nlm.nih.gov/traces/sra60/SRR/011186/SRR11454613,SRX8032204,HBCDC-HB-03/2019,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,PAIRED,0,0,ILLUMINA,Illumina MiSeq,SRP254688,PRJNA616446,,616446,SRS6404541,SAMN14479129,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-03/2019,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,3EA59A52EA9218E19790AA4245FBF3DE,FB830CA5A238EAE60C76CB8B83EDD172\r\n",
      "SRR11454610,2020-04-02 00:08:41,2020-04-01 00:46:21,14337950,1075346250,0,75,394,,https://sra-download.ncbi.nlm.nih.gov/traces/sra63/SRR/011186/SRR11454610,SRX8032207,HBCDC-HB-03/2020,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 550,SRP254688,PRJNA616446,,616446,SRS6404542,SAMN14479132,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-03/2020,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,30838BD1CC100381C127343CB1898778,CDF9AB45497010D07904F336B4B25945\r\n",
      "SRR11454612,2020-04-02 00:08:41,2020-04-01 00:44:01,3755251,1131947433,3755251,301,413,,https://sra-download.ncbi.nlm.nih.gov/traces/sra64/SRR/011186/SRR11454612,SRX8032205,HBCDC-HB-04/2019,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,PAIRED,0,0,ILLUMINA,Illumina MiSeq,SRP254688,PRJNA616446,,616446,SRS6404539,SAMN14479130,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-04/2019,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,D5301502CDE8652E796A631122D46750,95A4A16FB59F6805C8F80495A628F40F\r\n",
      "SRR11454611,2020-04-02 00:08:41,2020-04-01 00:41:37,1405559,105416855,0,74,37,,https://sra-download.ncbi.nlm.nih.gov/traces/sra62/SRR/011186/SRR11454611,SRX8032206,HBCDC-HB-04/2020,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 550,SRP254688,PRJNA616446,,616446,SRS6404540,SAMN14479131,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-04/2020,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,425F5E923790B892A78321AD3CF21E2F,D6D88B2A5A39DD1BD6DAA63FCBF83685\r\n",
      "SRR11454607,2020-04-02 00:08:41,2020-04-01 00:44:08,4407436,1157794996,4407436,262,394,,https://sra-download.ncbi.nlm.nih.gov/traces/sra61/SRR/011186/SRR11454607,SRX8032210,HBCDC-HB-06/2020,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,PAIRED,0,0,ILLUMINA,Illumina MiniSeq,SRP254688,PRJNA616446,,616446,SRS6404545,SAMN14479135,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-06/2020,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,3A92BC1F2B549FA64625FBDB30D27AF4,AB126143BEC511E36B120FE414040C07\r\n",
      "SRR11454606,2020-04-02 00:08:41,2020-04-01 00:43:41,5668472,1350168827,5668472,238,466,,https://sra-download.ncbi.nlm.nih.gov/traces/sra63/SRR/011186/SRR11454606,SRX8032211,HBCDC-HB-07/2020,RNA-Seq,RANDOM PCR,TRANSCRIPTOMIC,PAIRED,0,0,ILLUMINA,Illumina MiniSeq,SRP254688,PRJNA616446,,616446,SRS6404546,SAMN14479136,simple,2697049,Severe acute respiratory syndrome coronavirus 2,HBCDC-HB-07/2020,,,,,,,no,,,,,HUBEI PROVINCIAL CENTER FOR DISEASE CONTROL AND PREVENTION,SRA1061047,,public,423C44B00186A82E728E46591BB9C393,0AEBDF4A44C6E7DD36F7328855AA4FA6\r\n",
      "ERR2906838,2019-10-28 07:04:12,2019-10-28 07:16:32,10183588,514326940,0,50,201,,https://sra-download.ncbi.nlm.nih.gov/traces/era4/ERR/ERR2906/ERR2906838,ERX2909998,HCT116 siNT5 1_s,RNA-Seq,Oligo-dT,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 4000,ERP112143,PRJEB29794,,579812,ERS2906299,SAMEA5094961,simple,9606,Homo sapiens,E-MTAB-7393:HCT116 siNT5 1,,,,,,,no,,,,,\"Transcriptome and Genome Analysis Laboratory (TAL), Gottingen, Germany.\",ERA1659712,,public,F7BAF6D62015FEE75A4363870D33CB7F,DB91A0D7A3D5C23AE896FBB2B74EFE00\r\n",
      "ERR2906839,2019-10-28 07:04:12,2019-10-28 07:17:02,15136628,764334853,0,50,296,,https://sra-download.ncbi.nlm.nih.gov/traces/era4/ERR/ERR2906/ERR2906839,ERX2909999,HCT116 siNT5 2_s,RNA-Seq,Oligo-dT,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 4000,ERP112143,PRJEB29794,,579812,ERS2906300,SAMEA5094962,simple,9606,Homo sapiens,E-MTAB-7393:HCT116 siNT5 2,,,,,,,no,,,,,\"Transcriptome and Genome Analysis Laboratory (TAL), Gottingen, Germany.\",ERA1659712,,public,7C46428151C98157959BDF2E46F4607D,533B175010FFEFAE9793A667A1AD29C9\r\n",
      "ERR2906840,2019-10-28 07:04:12,2019-10-28 07:17:22,12065191,609181927,0,50,239,,https://sra-download.ncbi.nlm.nih.gov/traces/era4/ERR/ERR2906/ERR2906840,ERX2910000,HCT116 siNT5 3_s,RNA-Seq,Oligo-dT,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 4000,ERP112143,PRJEB29794,,579812,ERS2906301,SAMEA5094963,simple,9606,Homo sapiens,E-MTAB-7393:HCT116 siNT5 3,,,,,,,no,,,,,\"Transcriptome and Genome Analysis Laboratory (TAL), Gottingen, Germany.\",ERA1659712,,public,04AAC3FA7A8B1C5C560D0FAFCD49F453,E213B9D96B4D8FF4034DA6143C81CEF4\r\n",
      "ERR2906841,2019-10-28 07:04:12,2019-10-28 07:17:52,19246751,970086787,0,50,377,,https://sra-download.ncbi.nlm.nih.gov/traces/era4/ERR/ERR2906/ERR2906841,ERX2910001,HCT116 siUSP22 1_s,RNA-Seq,Oligo-dT,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 4000,ERP112143,PRJEB29794,,579812,ERS2906302,SAMEA5094964,simple,9606,Homo sapiens,E-MTAB-7393:HCT116 siUSP22 1,,,,,,,no,,,,,\"Transcriptome and Genome Analysis Laboratory (TAL), Gottingen, Germany.\",ERA1659712,,public,3A834B5131754BED2B5A57A234CBB698,510F7A751B4A172C8B32D449BFBAD588\r\n",
      "ERR2906842,2019-10-28 07:04:12,2019-10-28 07:18:01,17903244,904293936,0,50,351,,https://sra-download.ncbi.nlm.nih.gov/traces/era4/ERR/ERR2906/ERR2906842,ERX2910002,HCT116 siUSP22 2_s,RNA-Seq,Oligo-dT,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 4000,ERP112143,PRJEB29794,,579812,ERS2906303,SAMEA5094965,simple,9606,Homo sapiens,E-MTAB-7393:HCT116 siUSP22 2,,,,,,,no,,,,,\"Transcriptome and Genome Analysis Laboratory (TAL), Gottingen, Germany.\",ERA1659712,,public,BAE90B0E9547FB29E9423207CF6865BD,E420E0229E741A2E88DF755C4790C7DB\r\n",
      "ERR2906843,2019-10-28 07:04:12,2019-10-28 07:18:25,23077186,1163479767,0,50,451,,https://sra-download.ncbi.nlm.nih.gov/traces/era4/ERR/ERR2906/ERR2906843,ERX2910003,HCT116 siUSP22 3_s,RNA-Seq,Oligo-dT,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 4000,ERP112143,PRJEB29794,,579812,ERS2906304,SAMEA5094966,simple,9606,Homo sapiens,E-MTAB-7393:HCT116 siUSP22 3,,,,,,,no,,,,,\"Transcriptome and Genome Analysis Laboratory (TAL), Gottingen, Germany.\",ERA1659712,,public,5745B9F4298C4ADE96F2DAA89AA4BB21,73F8E2CE7A16E9F1FF4176C65EFBB4FA\r\n",
      "SRR9658356,2020-02-20 21:50:32,2019-07-09 09:41:33,2979196,225293505,0,75,99,,https://sra-download.ncbi.nlm.nih.gov/traces/sra17/SRR/009431/SRR9658356,SRX6419553,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076497,SAMN12235745,simple,9541,Macaca fascicularis,GSM3932647,,,,,,,no,,,,,GEO,SRA918841,,public,8549A30E71B794BD61B8B5F00D7578C1,EEA1F1BB34AEF30214FE397683604171\r\n",
      "SRR9658357,2020-02-20 21:50:32,2019-07-09 09:41:50,3346739,253003131,0,75,110,,https://sra-download.ncbi.nlm.nih.gov/traces/sra49/SRR/009431/SRR9658357,SRX6419554,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076498,SAMN12235744,simple,9541,Macaca fascicularis,GSM3932648,,,,,,,no,,,,,GEO,SRA918841,,public,55C29647F9AFBDF6242B666F1FA3F787,53A4BDB0226CF5E947B4CD72AD2BE97E\r\n",
      "SRR9658358,2020-02-20 21:50:32,2019-07-09 09:41:52,3658747,276511355,0,75,120,,https://sra-download.ncbi.nlm.nih.gov/traces/sra65/SRR/009431/SRR9658358,SRX6419555,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076499,SAMN12235743,simple,9541,Macaca fascicularis,GSM3932649,,,,,,,no,,,,,GEO,SRA918841,,public,447F3F277A36098011DAC99629AA73AF,39D03ED5AF20B77B8DC75CB72E0B6FC3\r\n",
      "SRR9658359,2020-02-20 21:50:32,2019-07-09 09:42:09,4494934,339777663,0,75,149,,https://sra-download.ncbi.nlm.nih.gov/traces/sra51/SRR/009431/SRR9658359,SRX6419556,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076500,SAMN12235742,simple,9541,Macaca fascicularis,GSM3932650,,,,,,,no,,,,,GEO,SRA918841,,public,EEF1EA9D068E3C7261CF508E8169E730,8BFC2E2CA78CBFEA99451CA6D2759348\r\n",
      "SRR9658360,2020-02-20 21:50:32,2019-07-09 09:41:40,2930452,221519261,0,75,96,,https://sra-download.ncbi.nlm.nih.gov/traces/sra46/SRR/009431/SRR9658360,SRX6419557,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076501,SAMN12235741,simple,9541,Macaca fascicularis,GSM3932651,,,,,,,no,,,,,GEO,SRA918841,,public,BE08D8C2C2295E44ED1F96D846AFA12D,4A518B8CCBEE066C574BDCE102450F93\r\n",
      "SRR9658361,2020-02-20 21:50:32,2019-07-09 09:41:43,3044284,230270144,0,75,101,,https://sra-download.ncbi.nlm.nih.gov/traces/sra75/SRR/009431/SRR9658361,SRX6419558,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076502,SAMN12235740,simple,9541,Macaca fascicularis,GSM3932652,,,,,,,no,,,,,GEO,SRA918841,,public,0B19CD2F1EA3839846F734CF0FE9A188,BD7076F534796403DDB171FEC930B6C0\r\n",
      "SRR9658362,2020-02-20 21:50:32,2019-07-09 09:41:42,2899664,219094355,0,75,95,,https://sra-download.ncbi.nlm.nih.gov/traces/sra68/SRR/009431/SRR9658362,SRX6419559,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076503,SAMN12235739,simple,9541,Macaca fascicularis,GSM3932653,,,,,,,no,,,,,GEO,SRA918841,,public,40D240C0976A716A9DE888FDF430928B,18D3F3602CCFCA14E3DD9EEB6ECA4BB3\r\n",
      "SRR9658363,2020-02-20 21:50:32,2019-07-09 09:41:55,4072481,307763244,0,75,133,,https://sra-download.ncbi.nlm.nih.gov/traces/sra73/SRR/009431/SRR9658363,SRX6419560,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076504,SAMN12235738,simple,9541,Macaca fascicularis,GSM3932654,,,,,,,no,,,,,GEO,SRA918841,,public,6F69A0C37C1EF499F2BF2D741940A3BC,DB54473CD721223C2A439A4BB466E718\r\n",
      "SRR9658364,2020-02-20 21:50:32,2019-07-09 09:41:54,3606532,272532727,0,75,125,,https://sra-download.ncbi.nlm.nih.gov/traces/sra31/SRR/009431/SRR9658364,SRX6419561,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076505,SAMN12235737,simple,9541,Macaca fascicularis,GSM3932655,,,,,,,no,,,,,GEO,SRA918841,,public,ADF318644FB913E8E38F045452EE134A,53EC66EDD0BD0A4BEC0D65408AA44F91\r\n",
      "SRR9658365,2020-02-20 21:50:32,2019-07-09 09:41:35,2655241,200639326,0,75,92,,https://sra-download.ncbi.nlm.nih.gov/traces/sra27/SRR/009431/SRR9658365,SRX6419562,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,NextSeq 500,SRP213845,PRJNA553361,3,553361,SRS5076506,SAMN12235736,simple,9541,Macaca fascicularis,GSM3932656,,,,,,,no,,,,,GEO,SRA918841,,public,43E7F99675F02EB2251BF93494DC4682,995318E9FEAC2DB87057846E71A5B7F8\r\n",
      "SRR9658384,2020-02-20 21:50:33,2019-07-09 09:40:52,593100,29655000,0,50,22,,https://sra-download.ncbi.nlm.nih.gov/traces/sra2/SRR/009432/SRR9658384,SRX6419581,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076525,SAMN12235759,simple,10090,Mus musculus,GSM3932675,,,,,,,no,,,,,GEO,SRA918841,,public,D0EF24E4FD41711D3EFA72DA5571C57F,F86E03B979E8C2BD7AA35B83CA46CDAD\r\n",
      "SRR9658385,2020-02-20 21:50:33,2019-07-09 09:40:56,534609,26730450,0,50,19,,https://sra-download.ncbi.nlm.nih.gov/traces/sra56/SRR/009432/SRR9658385,SRX6419581,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076525,SAMN12235759,simple,10090,Mus musculus,GSM3932675,,,,,,,no,,,,,GEO,SRA918841,,public,4F9FF2C4435A32C7F926B991CC621A63,C2B9F3B176FA54CAC8A725C596DC111D\r\n",
      "SRR9658386,2020-02-20 21:50:33,2019-07-09 09:40:58,593110,29655500,0,50,21,,https://sra-download.ncbi.nlm.nih.gov/traces/sra41/SRR/009432/SRR9658386,SRX6419581,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076525,SAMN12235759,simple,10090,Mus musculus,GSM3932675,,,,,,,no,,,,,GEO,SRA918841,,public,76A14854464E3C5C36E9EF9D86398067,5ED7DE2A1435C7DBD537E7F9AE06B6B5\r\n",
      "SRR9658387,2020-02-20 21:50:33,2019-07-09 09:40:48,583074,29153700,0,50,21,,https://sra-download.ncbi.nlm.nih.gov/traces/sra46/SRR/009432/SRR9658387,SRX6419581,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076525,SAMN12235759,simple,10090,Mus musculus,GSM3932675,,,,,,,no,,,,,GEO,SRA918841,,public,C78044F69FA06D725672D217505ED650,4436FC4CFD96968EDBC2FB6CB3B377B7\r\n",
      "SRR9658388,2020-02-20 21:50:33,2019-07-09 09:40:56,582683,29134150,0,50,21,,https://sra-download.ncbi.nlm.nih.gov/traces/sra72/SRR/009432/SRR9658388,SRX6419581,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076525,SAMN12235759,simple,10090,Mus musculus,GSM3932675,,,,,,,no,,,,,GEO,SRA918841,,public,D54EA88660DDE61EF0C508AE73E62021,91A372C91749F8E90AA7A674C91A0E2D\r\n",
      "SRR9658389,2020-02-20 21:50:33,2019-07-09 09:41:31,621766,31088300,0,50,22,,https://sra-download.ncbi.nlm.nih.gov/traces/sra0/SRR/009432/SRR9658389,SRX6419581,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076525,SAMN12235759,simple,10090,Mus musculus,GSM3932675,,,,,,,no,,,,,GEO,SRA918841,,public,79586D77BB0ED536566B174642ECDF8E,9882579A9C1CC509EA0B3946E578B08E\r\n",
      "SRR9658390,2020-02-20 21:50:33,2019-07-09 09:40:56,643875,32193750,0,50,24,,https://sra-download.ncbi.nlm.nih.gov/traces/sra27/SRR/009432/SRR9658390,SRX6419582,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076526,SAMN12235758,simple,10090,Mus musculus,GSM3932676,,,,,,,no,,,,,GEO,SRA918841,,public,EC9EE12AB008D771E6DF442A5E3E86A7,815DC4133960B3DA153E3453DD2DA208\r\n",
      "SRR9658391,2020-02-20 21:50:33,2019-07-09 09:40:54,598755,29937750,0,50,22,,https://sra-download.ncbi.nlm.nih.gov/traces/sra59/SRR/009432/SRR9658391,SRX6419582,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076526,SAMN12235758,simple,10090,Mus musculus,GSM3932676,,,,,,,no,,,,,GEO,SRA918841,,public,B1C62DAC1123A0490D5D5D7344132F4B,170EFEFCF1C013FD3071D8A85771959E\r\n",
      "SRR9658392,2020-02-20 21:50:33,2019-07-09 09:40:50,653917,32695850,0,50,24,,https://sra-download.ncbi.nlm.nih.gov/traces/sra26/SRR/009432/SRR9658392,SRX6419582,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076526,SAMN12235758,simple,10090,Mus musculus,GSM3932676,,,,,,,no,,,,,GEO,SRA918841,,public,57F7BD730E28FCFC29A4348E667355E1,EB77B4FC77D58D38707CF37BACC5CD67\r\n",
      "SRR9658393,2020-02-20 21:50:33,2019-07-09 09:40:59,656890,32844500,0,50,24,,https://sra-download.ncbi.nlm.nih.gov/traces/sra27/SRR/009432/SRR9658393,SRX6419582,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076526,SAMN12235758,simple,10090,Mus musculus,GSM3932676,,,,,,,no,,,,,GEO,SRA918841,,public,13CF9103D589EB0A84F061C80361811D,4CD5395AC0126A27B0E8853153178EAC\r\n",
      "SRR9658394,2020-02-20 21:50:33,2019-07-09 09:40:52,646160,32308000,0,50,24,,https://sra-download.ncbi.nlm.nih.gov/traces/sra5/SRR/009432/SRR9658394,SRX6419582,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ABI_SOLID,AB 5500xl Genetic Analyzer,SRP213845,PRJNA553361,3,553361,SRS5076526,SAMN12235758,simple,10090,Mus musculus,GSM3932676,,,,,,,no,,,,,GEO,SRA918841,,public,F307DE2AC2E00FBA7D7FDB454BE551A5,E21613DD4D826D1A0C5E3862C238347B\r\n",
      "SRR6639047,2018-06-13 19:43:16,2018-01-30 04:20:52,24390039,2449550734,0,100,367,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639047/SRR6639047.1,SRX3626699,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893803,SAMN08432512,simple,9685,Felis catus,GSM2970233,,,,,,,no,,,,,GEO,SRA653382,,public,32BEF1199E4651D7AC981E8314E29398,1B6D1ECA89E7FA2A4B641460D5434E7B\r\n",
      "SRR6639048,2018-06-13 19:43:16,2018-01-30 04:22:13,31886603,3193751931,0,100,488,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639048/SRR6639048.1,SRX3626700,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893807,SAMN08432511,simple,9685,Felis catus,GSM2970234,,,,,,,no,,,,,GEO,SRA653382,,public,EE71371915B2D6BC6B5FB1B891EB939F,2B343DB44271A718BBBE80F1ED04D843\r\n",
      "SRR6639049,2018-06-13 19:43:16,2018-01-30 04:19:52,28022864,2812395660,0,100,428,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639049/SRR6639049.1,SRX3626701,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893806,SAMN08432510,simple,9685,Felis catus,GSM2970235,,,,,,,no,,,,,GEO,SRA653382,,public,48D8858D2F7FE8E224A31FFB6036B2AC,028CDEEDFD4FBD63BE48494CD05AD95D\r\n",
      "SRR6639050,2018-06-13 19:43:16,2018-01-30 04:22:17,28952259,2905264247,0,100,436,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639050/SRR6639050.1,SRX3626702,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893805,SAMN08432509,simple,9685,Felis catus,GSM2970236,,,,,,,no,,,,,GEO,SRA653382,,public,054A037D645B3D5DED720C263118DFB1,EDDA20A08D4F958ECD77632B33ECABA1\r\n",
      "SRR6639051,2018-06-13 19:43:16,2018-01-30 04:27:56,30416721,3053539416,0,100,462,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639051/SRR6639051.1,SRX3626703,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893810,SAMN08432508,simple,9685,Felis catus,GSM2970237,,,,,,,no,,,,,GEO,SRA653382,,public,A4BC02C6E9E2D29D9EFA330497625A53,95643C87623C0D615F023A12C2E0FA49\r\n",
      "SRR6639052,2018-06-13 19:43:16,2018-01-30 04:21:21,26628178,2670441791,0,100,409,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos2/sra-pub-run-13/SRR6639052/SRR6639052.1,SRX3626704,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893808,SAMN08432507,simple,9685,Felis catus,GSM2970238,,,,,,,no,,,,,GEO,SRA653382,,public,A9A57D9FDF01E3427FE6A5AF3CC0A46F,F87DC0CF2A86D58E071A9644B19CCF52\r\n",
      "SRR6639053,2018-06-13 19:43:16,2018-01-30 04:19:00,15281316,1531853873,0,100,241,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos2/sra-pub-run-13/SRR6639053/SRR6639053.1,SRX3626705,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893811,SAMN08432506,simple,9685,Felis catus,GSM2970239,,,,,,,no,,,,,GEO,SRA653382,,public,799CBE36F192B9A47D6352FE3AA4C63A,7D3578A3E52BAADF92E56F4CD55E2B3C\r\n",
      "SRR6639054,2018-06-13 19:43:16,2018-01-30 04:18:56,13442786,1346587875,0,100,214,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639054/SRR6639054.1,SRX3626706,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893812,SAMN08432505,simple,9685,Felis catus,GSM2970240,,,,,,,no,,,,,GEO,SRA653382,,public,0DE03FD341D85A79DD317A323D2A5068,B3B4A27810EB89400117879A79DA8CD9\r\n",
      "SRR6639055,2018-06-13 19:43:16,2018-01-30 04:19:34,14027987,1404171539,0,100,224,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos2/sra-pub-run-13/SRR6639055/SRR6639055.1,SRX3626707,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893809,SAMN08432504,simple,9685,Felis catus,GSM2970241,,,,,,,no,,,,,GEO,SRA653382,,public,FAAEBE2DA88D7402A23989434DA8060C,7034B5EDC5FC546FAE083E4B5C12C599\r\n",
      "SRR6639056,2018-06-13 19:43:17,2018-01-30 04:18:56,16287160,1633399500,0,100,251,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos2/sra-pub-run-13/SRR6639056/SRR6639056.1,SRX3626708,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893813,SAMN08432503,simple,9685,Felis catus,GSM2970242,,,,,,,no,,,,,GEO,SRA653382,,public,D4BE2D29E7B491091A9BF5A4B2693145,BAF2627A278CF5F1EAE788838B46D698\r\n",
      "SRR6639057,2018-06-13 19:43:17,2018-01-30 04:21:49,15464572,1549205897,0,100,232,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639057/SRR6639057.1,SRX3626709,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893864,SAMN08432502,simple,9685,Felis catus,GSM2970243,,,,,,,no,,,,,GEO,SRA653382,,public,79A26393F286A782B1C0096CA69FD12B,D2784BE2EF4B064F0485C5596AD96324\r\n",
      "SRR6639058,2018-06-13 19:43:17,2018-01-30 04:19:01,16316683,1635856593,0,100,254,GCA_000181335.2,https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-2/SRR6639058/SRR6639058.1,SRX3626710,,RNA-Seq,cDNA,TRANSCRIPTOMIC,SINGLE,0,0,ILLUMINA,Illumina HiSeq 2500,SRP131692,PRJNA432069,2,432069,SRS2893814,SAMN08432501,simple,9685,Felis catus,GSM2970244,,,,,,,no,,,,,GEO,SRA653382,,public,1BB2ED120282F1F2AC388FB821B20A34,624AC8E60E8EF108D6179B78C4EEC523"
     ]
    }
   ],
   "source": [
    "# SRA RunInfo Table for run\n",
    "aws s3 cp s3://serratus-public/sra/testing_SraRunInfo.csv ./\n",
    "RUNINFO=\"$WORK/testing_SraRunInfo.csv\"\n",
    "cat $RUNINFO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Packer / AMI Initialization\n",
    "Does not need to be ran each time if you have access to the AMI already.\n",
    "\n",
    "Current Build: `us-east-1: ami-046baafb2ee438b69`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;32mamazon-ebs: output will be in this color.\u001b[0m\r\n",
      "\r\n",
      "\u001b[1;32m==> amazon-ebs: Prevalidating any provided VPC information\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Prevalidating AMI Name: packer-amazon-linux-2-docker-005\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Found Image ID: ami-0323c3dd2da7fb37d\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Creating temporary keypair: packer_5ea203b1-f549-bf3e-1007-c202f84bf7a6\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Creating temporary security group for this instance: packer_5ea203b4-b2cc-7d4b-e1cb-191ce719da7b\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Launching a source AWS instance...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Adding tags to source instance\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Adding tag: \"Name\": \"Packer Builder\"\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Instance ID: i-00f674bf01a53eacd\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Waiting for instance (i-00f674bf01a53eacd) to become ready...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Using ssh communicator to connect: 54.226.209.208\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Waiting for SSH to become available...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Connected to SSH!\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Provisioning with shell script: /tmp/packer-shell125212686\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Delta RPMs disabled because /usr/bin/applydeltarpm not installed.\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Existing lock /var/run/yum.pid: another copy is running as pid 2939.\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory :  94 M RSS (309 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:18 2020 - 00:02 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Running, pid: 2939\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Examining /var/tmp/yum-root-UA2mwB/epel-release-latest-7.noarch.rpm: epel-release-7-12.noarch\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Marking /var/tmp/yum-root-UA2mwB/epel-release-latest-7.noarch.rpm to be installed\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Resolving Dependencies\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --> Running transaction check\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ---> Package epel-release.noarch 0:7-12 will be installed\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --> Finished Dependency Resolution\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Dependencies Resolved\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ================================================================================\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:  Package          Arch       Version    Repository                         Size\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ================================================================================\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Installing:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:  epel-release     noarch     7-12       /epel-release-latest-7.noarch      24 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Transaction Summary\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ================================================================================\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Install  1 Package\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Total size: 24 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Installed size: 24 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Downloading packages:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Running transaction check\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Running transaction test\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Transaction test succeeded\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Running transaction\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Installing : epel-release-7-12.noarch                                     1/1\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Verifying  : epel-release-7-12.noarch                                     1/1\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Installed:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   epel-release.noarch 0:7-12\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Complete!\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ================================== repo: epel ==================================\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: [epel]\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: async = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: bandwidth = 0\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: base_persistdir = /var/lib/yum/repos/x86_64/2\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: baseurl =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: cache = 0\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: cachedir = /var/cache/yum/x86_64/2/epel\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: check_config_file_age = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: compare_providers_priority = 80\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: cost = 1000\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: deltarpm_metadata_percentage = 100\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: deltarpm_percentage =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: enabled = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: enablegroups = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: exclude =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: failovermethod = priority\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ftp_disable_epsv = False\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: gpgcadir = /var/lib/yum/repos/x86_64/2/epel/gpgcadir\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: gpgcakey =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: gpgcheck = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: gpgdir = /var/lib/yum/repos/x86_64/2/epel/gpgdir\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: hdrdir = /var/cache/yum/x86_64/2/epel/headers\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: http_caching = all\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: includepkgs =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ip_resolve =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: keepalive = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: keepcache = False\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: mddownloadpolicy = sqlite\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: mdpolicy = group:small\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: mediaid =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: metadata_expire = 21600\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: metadata_expire_filter = read-only:present\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: metalink = https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=$infra&content=$contentdir\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: minrate = 0\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: mirrorlist =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: mirrorlist_expire = 86400\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: name = Extra Packages for Enterprise Linux 7 - x86_64\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: old_base_cache_dir =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: password =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: persistdir = /var/lib/yum/repos/x86_64/2/epel\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: pkgdir = /var/cache/yum/x86_64/2/epel/packages\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: priority = 99\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: proxy = False\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: proxy_dict =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: proxy_password =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: proxy_username =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: repo_gpgcheck = False\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: report_instanceid = False\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: retries = 7\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: skip_if_unavailable = False\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ssl_check_cert_permissions = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: sslcacert =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: sslclientcert =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: sslclientkey =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: sslverify = True\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: throttle = 0\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: timeout = 5.0\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ui_id = epel/x86_64\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ui_repoid_vars = releasever,\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:    basearch\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: username =\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Existing lock /var/run/yum.pid: another copy is running as pid 3263.\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory :  49 M RSS (338 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:23 2020 - 00:01 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Sleeping, pid: 3263\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory :  49 M RSS (338 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:23 2020 - 00:03 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Sleeping, pid: 3263\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory :  49 M RSS (338 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:23 2020 - 00:05 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Sleeping, pid: 3263\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory :  49 M RSS (338 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:23 2020 - 00:07 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Sleeping, pid: 3263\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory : 200 M RSS (489 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:23 2020 - 00:09 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Running, pid: 3263\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory : 247 M RSS (536 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:23 2020 - 00:11 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Running, pid: 3263\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Another app is currently holding the yum lock; waiting for it to exit...\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:   The other application is: yum\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Memory : 264 M RSS (567 MB VSZ)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     Started: Thu Apr 23 21:09:23 2020 - 00:13 ago\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:     State  : Running, pid: 3263\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: 192 packages excluded due to repository priority protections\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Resolving Dependencies\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --> Running transaction check\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ---> Package iftop.x86_64 0:1.0-0.21.pre4.el7 will be installed\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --> Processing Dependency: libncursesw.so.5()(64bit) for package: iftop-1.0-0.21.pre4.el7.x86_64\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --> Processing Dependency: libtinfo.so.5()(64bit) for package: iftop-1.0-0.21.pre4.el7.x86_64\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ---> Package iotop.noarch 0:0.6-4.amzn2 will be installed\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ---> Package nload.x86_64 0:0.7.4-4.el7 will be installed\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --> Running transaction check\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ---> Package ncurses-compat-libs.x86_64 0:6.0-8.20170212.amzn2.1.3 will be installed\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --> Finished Dependency Resolution\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Dependencies Resolved\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ================================================================================\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:  Package               Arch     Version                      Repository    Size\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ================================================================================\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Installing:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:  iftop                 x86_64   1.0-0.21.pre4.el7            epel          53 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:  iotop                 noarch   0.6-4.amzn2                  amzn2-core    53 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:  nload                 x86_64   0.7.4-4.el7                  epel          70 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Installing for dependencies:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:  ncurses-compat-libs   x86_64   6.0-8.20170212.amzn2.1.3     amzn2-core   308 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Transaction Summary\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ================================================================================\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Install  3 Packages (+1 Dependent package)\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Total download size: 484 k\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Installed size: 1.3 M\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Downloading packages:\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: warning: /var/cache/yum/x86_64/2/epel/packages/iftop-1.0-0.21.pre4.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Public key for iftop-1.0-0.21.pre4.el7.x86_64.rpm is not installed\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: --------------------------------------------------------------------------------\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Total                                              1.8 MB/s | 484 kB  00:00\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Importing GPG key 0x352C64E5:\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:  Userid     : \"Fedora EPEL (7) <epel@fedoraproject.org>\"\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:  Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:  Package    : epel-release-7-12.noarch (installed)\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs:  From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Running transaction check\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Running transaction test\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Transaction test succeeded\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Running transaction\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Installing : ncurses-compat-libs-6.0-8.20170212.amzn2.1.3.x86_64          1/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Installing : nload-0.7.4-4.el7.x86_64                                     2/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Installing : iftop-1.0-0.21.pre4.el7.x86_64                               3/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Installing : iotop-0.6-4.amzn2.noarch                                     4/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Verifying  : nload-0.7.4-4.el7.x86_64                                     1/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Verifying  : iftop-1.0-0.21.pre4.el7.x86_64                               2/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Verifying  : ncurses-compat-libs-6.0-8.20170212.amzn2.1.3.x86_64          3/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   Verifying  : iotop-0.6-4.amzn2.noarch                                     4/4\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Installed:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   iftop.x86_64 0:1.0-0.21.pre4.el7          iotop.noarch 0:0.6-4.amzn2\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   nload.x86_64 0:0.7.4-4.el7\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Dependency Installed:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:   ncurses-compat-libs.x86_64 0:6.0-8.20170212.amzn2.1.3\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Complete!\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Provisioning with shell script: setup_node_exporter.sh\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: + curl -sL https://github.com/prometheus/node_exporter/releases/download/v1.0.0-rc.0/node_exporter-1.0.0-rc.0.linux-amd64.tar.gz -o node_exporter.tgz\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: + tar -xvzf node_exporter.tgz\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: node_exporter-1.0.0-rc.0.linux-amd64/\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: node_exporter-1.0.0-rc.0.linux-amd64/LICENSE\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: node_exporter-1.0.0-rc.0.linux-amd64/NOTICE\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: node_exporter-1.0.0-rc.0.linux-amd64/node_exporter\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: + sudo cp node_exporter-1.0.0-rc.0.linux-amd64/node_exporter /usr/bin/\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: + sudo useradd -s /bin/false prometheus\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: + sudo tee /etc/systemd/system/node_exporter.service\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: [Unit]\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Description=Prometheus Node Exporter\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: + sudo systemctl enable node_exporter.service\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Wants=network-online.target\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: After=network-online.target\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: [Service]\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: User=prometheus\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: ExecStart=/usr/bin/node_exporter\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs:\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: [Install]\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: WantedBy=default.target\u001b[0m\r\n",
      "\u001b[1;31m==> amazon-ebs: Created symlink from /etc/systemd/system/default.target.wants/node_exporter.service to /etc/systemd/system/node_exporter.service.\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Stopping the source instance...\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: Stopping instance\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Waiting for the instance to stop...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Creating AMI packer-amazon-linux-2-docker-005 from instance i-00f674bf01a53eacd\u001b[0m\r\n",
      "\u001b[0;32m    amazon-ebs: AMI: ami-046baafb2ee438b69\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Waiting for AMI to become ready...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Terminating the source AWS instance...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Cleaning up any extra volumes...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: No volumes to clean up, skipping\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Deleting temporary security group...\u001b[0m\r\n",
      "\u001b[1;32m==> amazon-ebs: Deleting temporary keypair...\u001b[0m\r\n",
      "\u001b[1;32mBuild 'amazon-ebs' finished.\u001b[0m\r\n",
      "\r\n",
      "==> Builds finished. The artifacts of successful builds are:\r\n",
      "--> amazon-ebs: AMIs were created:\r\n",
      "us-east-1: ami-046baafb2ee438b69\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "cd $SERRATUS/packer\n",
    "packer build docker-ami.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Build Serratus containers (optional)\n",
    "Serratus containers are available on the `serratusbio` dockerhub. If you wish to deploy your own containers, you will have to build them from the `serratus` repository and upload them to your own dockerhub account.\n",
    "\n",
    "This can be done with the `build.sh` script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "cd $SERRATUS\n",
    "\n",
    "# If you want to upload containers to your repository\n",
    "# include this.\n",
    "export DOCKERHUB_USER='serratusbio' # optional\n",
    "sudo docker login # optional\n",
    "\n",
    "# Build all containers and upload them docker hub repo\n",
    "# (if available)\n",
    "./build.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "NOTE: The genome version is currently hard-set as part of `scheduler/flask_app/jobs.py` on line 172\n",
    "```\n",
    "    response['genome'] = \"cov1r\"\n",
    "```\n",
    "changed to \n",
    "```\n",
    "    response['genome'] = \"cov2r\"\n",
    "```\n",
    "\n",
    "And containers re-built. This variable needs to be moved to terraform to allow control of genome versions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Terraform Initialization\n",
    "The Global Variables for Terraform file must be modified to initialize for your system.\n",
    "\n",
    "File: `$SERRATUS/terarform/main/terraform.tfvars`\n",
    "\n",
    "This step must be done manually in a text editor currently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your public IP followed by \"/32\"\n",
    "LOCALIP=\"75.155.242.67/32\" #dev_cidrs\n",
    "# Your AWS key name\n",
    "KEYNAME=\"serratus\"         #key_name\n",
    "# Dockerhub account containing serratus containers\n",
    "DOCKERHUB_USER='serratusbio'    #dockerhub_account (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[1mInitializing modules...\u001b[0m\r\n",
      "\r\n",
      "\u001b[0m\u001b[1mInitializing the backend...\u001b[0m\r\n",
      "\r\n",
      "\u001b[0m\u001b[1mInitializing provider plugins...\u001b[0m\r\n",
      "\r\n",
      "The following providers do not have any version constraints in configuration,\r\n",
      "so the latest version was installed.\r\n",
      "\r\n",
      "To prevent automatic upgrades to new major versions that may contain breaking\r\n",
      "changes, it is recommended to add version = \"...\" constraints to the\r\n",
      "corresponding provider blocks in configuration, with the constraint strings\r\n",
      "suggested below.\r\n",
      "\r\n",
      "* provider.local: version = \"~> 1.4\"\r\n",
      "\r\n",
      "\u001b[0m\u001b[1m\u001b[32mTerraform has been successfully initialized!\u001b[0m\u001b[32m\u001b[0m\r\n",
      "\u001b[0m\u001b[32m\r\n",
      "You may now begin working with Terraform. Try running \"terraform plan\" to see\r\n",
      "any changes that are required for your infrastructure. All Terraform commands\r\n",
      "should now work.\r\n",
      "\r\n",
      "If you ever set or change modules or backend configuration for Terraform,\r\n",
      "rerun this command to reinitialize your working directory. If you forget, other\r\n",
      "commands will detect it and remind you to do so if necessary.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "# Initialize terraform\n",
    "cd $SERRATUS/terraform/main\n",
    "terraform init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[1mmodule.align.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.data.aws_ami.ecs: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.data.aws_availability_zones.all: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.data.aws_availability_zones.all: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.data.aws_availability_zones.all: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_cloudwatch_log_group.scheduler: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_cluster.monitor: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role.role: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role.role: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_cloudwatch_log_group.g: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_instance_profile.monitor: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role.role: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_cloudwatch_log_group.g: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1maws_security_group.internal: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role.role: Creation complete after 1s [id=SerratusIamRole-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role.task_role: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role.role: Creation complete after 1s [id=SerratusIamRole-scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role.role: Creation complete after 1s [id=SerratusIamRole-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role.role: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_cloudwatch_log_group.g: Creation complete after 1s [id=serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy.cloudwatch: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_cloudwatch_log_group.g: Creation complete after 1s [id=serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_instance_profile.profile: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_cloudwatch_log_group.scheduler: Creation complete after 1s [id=scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_instance_profile.profile: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_instance_profile.monitor: Creation complete after 1s [id=profile-serratus-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role_policy.cloudwatch: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role.task_role: Creation complete after 0s [id=SerratusIamRole-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_cloudwatch_log_group.g: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Creation complete after 1s [id=SerratusIamRole-serratus-dl-20200423221044120900000002]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role.role: Creation complete after 1s [id=SerratusIamRole-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_instance_profile.profile: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy.cloudwatch: Creation complete after 1s [id=SerratusIamRole-serratus-dl:CloudWatchLogsCreate-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy.cloudwatch: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role_policy.cloudwatch: Creation complete after 1s [id=SerratusIamRole-scheduler:CloudWatchLogsCreate-scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy_attachment.attachment: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_instance_profile.profile: Creation complete after 1s [id=profile-scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy.cloudwatch: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_instance_profile.profile: Creation complete after 1s [id=profile-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Creation complete after 0s [id=SerratusIamRole-serratus-merge-20200423221044968200000003]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy.cloudwatch: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_cloudwatch_log_group.g: Creation complete after 2s [id=serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy.cloudwatch: Creation complete after 1s [id=SerratusIamRole-serratus-merge:CloudWatchLogsCreate-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_instance_profile.profile: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy_attachment.attachment: Creation complete after 1s [id=SerratusIamRole-monitor-20200423221045368800000004]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_instance_profile.profile: Creation complete after 1s [id=profile-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy.cloudwatch: Creation complete after 1s [id=SerratusIamRole-monitor:CloudwatchGetMetrics]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Creation complete after 1s [id=SerratusIamRole-serratus-align-20200423221045750000000005]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy.cloudwatch: Creation complete after 1s [id=SerratusIamRole-serratus-align:CloudWatchLogsCreate-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1maws_security_group.internal: Creation complete after 4s [id=sg-0b1c9a028ea3bb7f0]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_instance_profile.profile: Creation complete after 1s [id=profile-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Creation complete after 7s [id=tf-serratus-work-20200423221042780000000001]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"fq-blocks\"]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_iam_role_policy.s3_write: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"out\"]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_write: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.full: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"bam-blocks\"]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_delete[0]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_delete[0]: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"out\"]: Creation complete after 1s [id=tf-serratus-work-20200423221042780000000001:prefix-out]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_write: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_iam_role_policy.s3_write: Creation complete after 1s [id=SerratusIamRole-serratus-dl:S3WriteData-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_delete[0]: Creation complete after 1s [id=SerratusIamRole-serratus-align:S3DeleteData-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"bam-blocks\"]: Creation complete after 1s [id=tf-serratus-work-20200423221042780000000001:prefix-bam-blocks]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.full: Creation complete after 1s [id=tf-serratus-work-20200423221042780000000001:full]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_delete[0]: Creation complete after 1s [id=SerratusIamRole-serratus-merge:S3DeleteData-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_write: Creation complete after 1s [id=SerratusIamRole-serratus-align:S3WriteData-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"fq-blocks\"]: Creation complete after 1s [id=tf-serratus-work-20200423221042780000000001:prefix-fq-blocks]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_write: Creation complete after 1s [id=SerratusIamRole-serratus-merge:S3WriteData-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_cluster.monitor: Still creating... [10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_cluster.monitor: Creation complete after 11s [id=arn:aws:ecs:us-east-1:797308887321:cluster/serratus-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_instance.monitor: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still creating... [10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_instance.monitor: Still creating... [10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still creating... [20s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Creation complete after 21s [id=i-0139403d7159eb53d]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_eip.sch: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_task_definition.monitor: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_task_definition.monitor: Creation complete after 1s [id=monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Creation complete after 1s [id=arn:aws:ecs:us-east-1:797308887321:service/serratus-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_eip.sch: Creation complete after 4s [id=eipalloc-09bc27fef18639a4c]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_launch_configuration.worker: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_launch_configuration.worker: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_launch_configuration.worker: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_launch_configuration.worker: Creation complete after 2s [id=tf-serratus-merge-20200423221112408900000008]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_launch_configuration.worker: Creation complete after 2s [id=tf-serratus-align-20200423221112402100000007]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_autoscaling_group.worker: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_instance.monitor: Creation complete after 20s [id=i-086e77637ebd22c38]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_eip.monitor: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_launch_configuration.worker: Creation complete after 2s [id=tf-serratus-dl-20200423221112630800000009]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_autoscaling_group.worker: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_autoscaling_group.worker: Creation complete after 2s [id=tf-asg-tf-serratus-align-20200423221112402100000007]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.align_set_capacity: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.align_set_capacity: Creation complete after 0s [id=06cf46b01dfabc5f2a61f5b51d43ee3a2ed357b0]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_autoscaling_group.worker: Creation complete after 2s [id=tf-asg-tf-serratus-dl-20200423221112630800000009]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.dl_set_capacity: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.dl_set_capacity: Creation complete after 0s [id=7dff4901b65f09a5603318e2597fe3dd01a1935f]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_eip.monitor: Creation complete after 3s [id=eipalloc-04ea14ff82fbc0d41]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.hosts: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.create_tunnel: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.create_tunnel: Creation complete after 0s [id=46e450a0e9dca822ad7e6d6015a0bb658af1e97c]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.hosts: Creation complete after 0s [id=7705e128fb4eeaf15c17804a91633e44e185e807]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Still creating... [10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Still creating... [20s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Still creating... [30s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Still creating... [40s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Creation complete after 40s [id=tf-asg-tf-serratus-merge-20200423221112408900000008]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.merge_set_capacity: Creating...\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.merge_set_capacity: Creation complete after 0s [id=63c6450a668d1c442aff7f2b60d0202bef3f1d8e]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1m\u001b[32m\r\n",
      "Apply complete! Resources: 52 added, 0 changed, 0 destroyed.\u001b[0m\r\n",
      "\u001b[0m\u001b[1m\u001b[32m\r\n",
      "Outputs:\r\n",
      "\r\n",
      "align_asg_name = tf-asg-tf-serratus-align-20200423221112402100000007\r\n",
      "dl_asg_name = tf-asg-tf-serratus-dl-20200423221112630800000009\r\n",
      "help = Run ./create_tunnels.sh to create SSH tunnels for all services.\r\n",
      "\r\n",
      "merge_asg_name = tf-asg-tf-serratus-merge-20200423221112408900000008\r\n",
      "monitor_dns = ec2-100-26-25-117.compute-1.amazonaws.com\r\n",
      "scheduler_dns = ec2-35-170-19-36.compute-1.amazonaws.com\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "# Launch Terraform Cluster\n",
    "# Initialize the serratus cluster with minimal nodes\n",
    "terraform apply -auto-approve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Serratus \n",
    "Upload the run data, scale-out the cluster, monitor performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rwxrwxr-x 1 artem artem 194 Apr 23 15:11 align_set_capacity.sh\r\n",
      "-rwxrwxr-x 1 artem artem 405 Apr 23 15:11 create_tunnels.sh\r\n",
      "-rwxrwxr-x 1 artem artem 191 Apr 23 15:11 dl_set_capacity.sh\r\n",
      "-rwxrwxr-x 1 artem artem 194 Apr 23 15:11 merge_set_capacity.sh\r\n"
     ]
    }
   ],
   "source": [
    "# Terraform will have created four scripts to control\n",
    "# serratus\n",
    "ls -alh *.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Monitors & Upload table\n",
    "\n",
    "Open SSH tunnels to monitor node then open monitors in browser\n",
    "\n",
    "- [Scheduler Table](localhost:8000/jobs/)\n",
    "- [Cluster Monitor: Grafana](http://localhost:3000/?orgId=1)\n",
    "- [Cluster Monitor: Prometheus](http://localhost:9090)\n",
    "\n",
    "\n",
    "#### Empty Scheduler Table (localhost:8000/jobs/)\n",
    "![Empty Table Load Screen](200423_ab/empty_scheduler.png)\n",
    "\n",
    "#### Ready Scheduler Table (localhost:8000/jobs/)\n",
    "![Empty Table Load Screen](200423_ab/ready_scheduler.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tunnels created:\r\n",
      "Warning: Permanently added 'ec2-100-26-25-117.compute-1.amazonaws.com,100.26.25.117' (ECDSA) to the list of known hosts.\r",
      "\r\n",
      "    localhost:3000 -- grafana\r\n",
      "Warning: Permanently added 'ec2-100-26-25-117.compute-1.amazonaws.com,100.26.25.117' (ECDSA) to the list of known hosts.\r",
      "\r\n",
      "    localhost:9090 -- prometheus\r\n",
      "Warning: Permanently added 'ec2-35-170-19-36.compute-1.amazonaws.com,35.170.19.36' (ECDSA) to the list of known hosts.\r",
      "\r\n",
      "    localhost:8000 -- scheduler\r\n"
     ]
    }
   ],
   "source": [
    "cd $SERRATUS/terraform/main\n",
    "\n",
    "# Open SSH tunnels to the monitor\n",
    "./create_tunnels.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"inserted_rows\": 49, \r\n",
      "  \"total_rows\": 49\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "# Load SRA Run Info into scheduler (READY)\n",
    "curl -s -X POST -T $RUNINFO localhost:8000/jobs/add_sra_run_info/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scale up the cluster\n",
    "This will set-up 10 download, 10 align and 2 merge nodes to process data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+ export AWS_REGION=us-east-1\r\n",
      "+ AWS_REGION=us-east-1\r\n",
      "+ aws autoscaling set-desired-capacity --auto-scaling-group-name tf-asg-tf-serratus-dl-20200423221112630800000009 --desired-capacity 10\r\n",
      "+ export AWS_REGION=us-east-1\r\n",
      "+ AWS_REGION=us-east-1\r\n",
      "+ aws autoscaling set-desired-capacity --auto-scaling-group-name tf-asg-tf-serratus-align-20200423221112402100000007 --desired-capacity 10\r\n",
      "+ export AWS_REGION=us-east-1\r\n",
      "+ AWS_REGION=us-east-1\r\n",
      "+ aws autoscaling set-desired-capacity --auto-scaling-group-name tf-asg-tf-serratus-merge-20200423221112408900000008 --desired-capacity 2\r\n"
     ]
    }
   ],
   "source": [
    "./dl_set_capacity.sh 10\n",
    "./align_set_capacity.sh 10\n",
    "./merge_set_capacity.sh 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can track the progress of accessions in the scheduler:\n",
    "\n",
    "![Running Scheduler](200423_ab/running_scheduler.png)\n",
    "\n",
    "And monitor the performance of the cluster in the monitor:\n",
    "\n",
    "![Running Monitor](200423_ab/running_monitor.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+ export AWS_REGION=us-east-1\r\n",
      "+ AWS_REGION=us-east-1\r\n",
      "+ aws autoscaling set-desired-capacity --auto-scaling-group-name tf-asg-tf-serratus-dl-20200423221112630800000009 --desired-capacity 0\r\n"
     ]
    }
   ],
   "source": [
    "# When all downloading/splitting is done,\n",
    "# scale-in the downloaders\n",
    "./dl_set_capacity.sh 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+ export AWS_REGION=us-east-1\r\n",
      "+ AWS_REGION=us-east-1\r\n",
      "+ aws autoscaling set-desired-capacity --auto-scaling-group-name tf-asg-tf-serratus-align-20200423221112402100000007 --desired-capacity 0\r\n",
      "+ export AWS_REGION=us-east-1\r\n",
      "+ AWS_REGION=us-east-1\r\n",
      "+ aws autoscaling set-desired-capacity --auto-scaling-group-name tf-asg-tf-serratus-merge-20200423221112408900000008 --desired-capacity 0\r\n"
     ]
    }
   ],
   "source": [
    "# When all alignment is done,\n",
    "# scale-in the aligners\n",
    "./align_set_capacity.sh 0\n",
    "\n",
    "# When all merging is done,\n",
    "# scale in the mergers\n",
    "./merge_set_capacity.sh 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\r\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      " 93  188k   93  176k    0     0   276k      0 --:--:-- --:--:-- --:--:--  276k\r",
      "100  188k  100  188k    0     0   291k      0 --:--:-- --:--:-- --:--:--  291k\r\n"
     ]
    }
   ],
   "source": [
    "# Dump the Scheduler SQLITE table to a local file\n",
    "curl localhost:8000/db > \\\n",
    "  $SERRATUS/notebook/200423_ab/schedDump.sqlite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shutting down procedures\n",
    "\n",
    "Closing up shop.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save output of runs\n",
    "\n",
    "output directory: `s3://serratus-public/out/200423_ab_cov2r/`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output files are in two folders:\n",
    "# Bam and Bai files\n",
    "aws s3 ls s3://tf-serratus-work-20200423221042780000000001/out/bam/\n",
    "# Flagstat and RefCount files\n",
    "aws s3 ls s3://tf-serratus-work-20200423221042780000000001/out/flagstat/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 262.8 KiB/~197.2 MiB with ~77 file(s) remaining (calculating...)\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906839.bam to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906839.bam\r\n",
      "Completed 262.8 KiB/~197.2 MiB with ~77 file(s) remaining (calculating...)\r",
      "Completed 592.5 KiB/~197.2 MiB with ~120 file(s) remaining (calculating...)\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906838.bam to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906838.bam\r\n",
      "Completed 592.5 KiB/~197.2 MiB with ~122 file(s) remaining (calculating...)\r",
      "Completed 907.2 KiB/~197.2 MiB with ~122 file(s) remaining (calculating...)\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906843.bam to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906843.bam\r\n",
      "Completed 907.2 KiB/~197.2 MiB with ~123 file(s) remaining (calculating...)\r",
      "Completed 1.5 MiB/~197.2 MiB with ~127 file(s) remaining (calculating...)  \r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906841.bam to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906841.bam\r\n",
      "Completed 1.5 MiB/~197.2 MiB with ~128 file(s) remaining (calculating...)\r",
      "Completed 1.6 MiB/~197.2 MiB with ~133 file(s) remaining (calculating...)\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906838.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906838.bam.bai\r\n",
      "Completed 1.6 MiB/~197.2 MiB with ~135 file(s) remaining (calculating...)\r",
      "Completed 1.7 MiB/~197.2 MiB with ~139 file(s) remaining (calculating...)\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906840.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906840.bam.bai\r\n",
      "Completed 1.7 MiB/~197.2 MiB with ~138 file(s) remaining (calculating...)\r",
      "Completed 1.8 MiB/197.2 MiB with 142 file(s) remaining                   \r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906839.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906839.bam.bai\r\n",
      "Completed 1.8 MiB/197.2 MiB with 141 file(s) remaining\r",
      "Completed 2.0 MiB/197.2 MiB with 141 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906842.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906842.bam.bai\r\n",
      "Completed 2.0 MiB/197.2 MiB with 140 file(s) remaining\r",
      "Completed 2.1 MiB/197.2 MiB with 140 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906841.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906841.bam.bai\r\n",
      "Completed 2.1 MiB/197.2 MiB with 139 file(s) remaining\r",
      "Completed 2.3 MiB/197.2 MiB with 139 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906842.bam to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906842.bam\r\n",
      "Completed 2.3 MiB/197.2 MiB with 138 file(s) remaining\r",
      "Completed 2.6 MiB/197.2 MiB with 138 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906840.bam to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906840.bam\r\n",
      "Completed 2.6 MiB/197.2 MiB with 137 file(s) remaining\r",
      "Completed 2.7 MiB/197.2 MiB with 137 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454606.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454606.bam.bai\r\n",
      "Completed 2.7 MiB/197.2 MiB with 136 file(s) remaining\r",
      "Completed 2.8 MiB/197.2 MiB with 136 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/ERR2906843.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/ERR2906843.bam.bai\r\n",
      "Completed 2.8 MiB/197.2 MiB with 135 file(s) remaining\r",
      "Completed 2.9 MiB/197.2 MiB with 135 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454607.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454607.bam.bai\r\n",
      "Completed 2.9 MiB/197.2 MiB with 134 file(s) remaining\r",
      "Completed 5.5 MiB/197.2 MiB with 134 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454606.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454606.bam\r\n",
      "Completed 5.5 MiB/197.2 MiB with 133 file(s) remaining\r",
      "Completed 5.6 MiB/197.2 MiB with 133 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454608.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454608.bam.bai\r\n",
      "Completed 5.6 MiB/197.2 MiB with 132 file(s) remaining\r",
      "Completed 5.7 MiB/197.2 MiB with 132 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454609.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454609.bam.bai\r\n",
      "Completed 5.7 MiB/197.2 MiB with 131 file(s) remaining\r",
      "Completed 5.9 MiB/197.2 MiB with 131 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454611.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454611.bam.bai\r\n",
      "Completed 5.9 MiB/197.2 MiB with 130 file(s) remaining\r",
      "Completed 6.2 MiB/197.2 MiB with 130 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454611.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454611.bam\r\n",
      "Completed 6.2 MiB/197.2 MiB with 129 file(s) remaining\r",
      "Completed 7.1 MiB/197.2 MiB with 129 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454612.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454612.bam\r\n",
      "Completed 7.1 MiB/197.2 MiB with 128 file(s) remaining\r",
      "Completed 11.2 MiB/197.2 MiB with 128 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454610.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454610.bam\r\n",
      "Completed 11.2 MiB/197.2 MiB with 127 file(s) remaining\r",
      "Completed 11.3 MiB/197.2 MiB with 127 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454610.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454610.bam.bai\r\n",
      "Completed 11.3 MiB/197.2 MiB with 126 file(s) remaining\r",
      "Completed 13.9 MiB/197.2 MiB with 126 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454609.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454609.bam\r\n",
      "Completed 13.9 MiB/197.2 MiB with 125 file(s) remaining\r",
      "Completed 15.4 MiB/197.2 MiB with 125 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454607.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454607.bam\r\n",
      "Completed 15.4 MiB/197.2 MiB with 124 file(s) remaining\r",
      "Completed 15.5 MiB/197.2 MiB with 124 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454612.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454612.bam.bai\r\n",
      "Completed 15.5 MiB/197.2 MiB with 123 file(s) remaining\r",
      "Completed 15.6 MiB/197.2 MiB with 123 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454614.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454614.bam.bai\r\n",
      "Completed 15.6 MiB/197.2 MiB with 122 file(s) remaining\r",
      "Completed 15.8 MiB/197.2 MiB with 122 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454613.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454613.bam.bai\r\n",
      "Completed 15.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 23.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 31.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 39.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 47.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 55.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 63.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 71.8 MiB/197.2 MiB with 121 file(s) remaining\r",
      "Completed 79.6 MiB/197.2 MiB with 121 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454608.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454608.bam\r\n",
      "Completed 79.6 MiB/197.2 MiB with 120 file(s) remaining\r",
      "Completed 79.7 MiB/197.2 MiB with 120 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454615.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454615.bam.bai\r\n",
      "Completed 79.7 MiB/197.2 MiB with 119 file(s) remaining\r",
      "Completed 83.1 MiB/197.2 MiB with 119 file(s) remaining\r",
      "Completed 91.1 MiB/197.2 MiB with 119 file(s) remaining\r",
      "Completed 91.3 MiB/197.2 MiB with 119 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658356.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658356.bam\r\n",
      "Completed 91.3 MiB/197.2 MiB with 118 file(s) remaining\r",
      "Completed 99.3 MiB/197.2 MiB with 118 file(s) remaining\r",
      "Completed 99.7 MiB/197.2 MiB with 118 file(s) remaining\r",
      "Completed 99.9 MiB/197.2 MiB with 118 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658357.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658357.bam\r\n",
      "Completed 99.9 MiB/197.2 MiB with 117 file(s) remaining\r",
      "Completed 107.9 MiB/197.2 MiB with 117 file(s) remaining\r",
      "Completed 115.9 MiB/197.2 MiB with 117 file(s) remaining\r",
      "Completed 116.0 MiB/197.2 MiB with 117 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658357.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658357.bam.bai\r\n",
      "Completed 116.0 MiB/197.2 MiB with 116 file(s) remaining\r",
      "Completed 124.0 MiB/197.2 MiB with 116 file(s) remaining\r",
      "Completed 124.1 MiB/197.2 MiB with 116 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658356.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658356.bam.bai\r\n",
      "Completed 124.1 MiB/197.2 MiB with 115 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454613.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454613.bam\r\n",
      "Completed 124.1 MiB/197.2 MiB with 114 file(s) remaining\r",
      "Completed 124.3 MiB/197.2 MiB with 114 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658359.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658359.bam\r\n",
      "Completed 124.3 MiB/197.2 MiB with 113 file(s) remaining\r",
      "Completed 124.4 MiB/197.2 MiB with 113 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658358.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658358.bam.bai\r\n",
      "Completed 124.4 MiB/197.2 MiB with 112 file(s) remaining\r",
      "Completed 124.6 MiB/197.2 MiB with 112 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658359.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658359.bam.bai\r\n",
      "Completed 124.6 MiB/197.2 MiB with 111 file(s) remaining\r",
      "Completed 132.6 MiB/197.2 MiB with 111 file(s) remaining\r",
      "Completed 132.7 MiB/197.2 MiB with 111 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658360.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658360.bam.bai\r\n",
      "Completed 132.7 MiB/197.2 MiB with 110 file(s) remaining\r",
      "Completed 132.9 MiB/197.2 MiB with 110 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658358.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658358.bam\r\n",
      "Completed 132.9 MiB/197.2 MiB with 109 file(s) remaining\r",
      "Completed 133.0 MiB/197.2 MiB with 109 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658360.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658360.bam\r\n",
      "Completed 133.0 MiB/197.2 MiB with 108 file(s) remaining\r",
      "Completed 141.0 MiB/197.2 MiB with 108 file(s) remaining\r",
      "Completed 149.0 MiB/197.2 MiB with 108 file(s) remaining\r",
      "Completed 149.2 MiB/197.2 MiB with 108 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658361.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658361.bam.bai\r\n",
      "Completed 149.2 MiB/197.2 MiB with 107 file(s) remaining\r",
      "Completed 149.4 MiB/197.2 MiB with 107 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658361.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658361.bam\r\n",
      "Completed 149.4 MiB/197.2 MiB with 106 file(s) remaining\r",
      "Completed 157.4 MiB/197.2 MiB with 106 file(s) remaining\r",
      "Completed 165.4 MiB/197.2 MiB with 106 file(s) remaining\r",
      "Completed 165.5 MiB/197.2 MiB with 106 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658362.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658362.bam\r\n",
      "Completed 165.5 MiB/197.2 MiB with 105 file(s) remaining\r",
      "Completed 165.7 MiB/197.2 MiB with 105 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658362.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658362.bam.bai\r\n",
      "Completed 165.7 MiB/197.2 MiB with 104 file(s) remaining\r",
      "Completed 165.8 MiB/197.2 MiB with 104 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658363.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658363.bam\r\n",
      "Completed 165.8 MiB/197.2 MiB with 103 file(s) remaining\r",
      "Completed 173.8 MiB/197.2 MiB with 103 file(s) remaining\r",
      "Completed 174.0 MiB/197.2 MiB with 103 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658364.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658364.bam.bai\r\n",
      "Completed 174.0 MiB/197.2 MiB with 102 file(s) remaining\r",
      "Completed 174.1 MiB/197.2 MiB with 102 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658365.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658365.bam\r\n",
      "Completed 174.1 MiB/197.2 MiB with 101 file(s) remaining\r",
      "Completed 174.3 MiB/197.2 MiB with 101 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658364.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658364.bam\r\n",
      "Completed 174.3 MiB/197.2 MiB with 100 file(s) remaining\r",
      "Completed 177.6 MiB/197.2 MiB with 100 file(s) remaining\r",
      "Completed 177.8 MiB/197.2 MiB with 100 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658365.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658365.bam.bai\r\n",
      "Completed 177.8 MiB/197.2 MiB with 99 file(s) remaining\r",
      "Completed 177.9 MiB/197.2 MiB with 99 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658363.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658363.bam.bai\r\n",
      "Completed 177.9 MiB/197.2 MiB with 98 file(s) remaining\r",
      "Completed 178.1 MiB/197.2 MiB with 98 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658384.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658384.bam\r\n",
      "Completed 178.1 MiB/197.2 MiB with 97 file(s) remaining\r",
      "Completed 178.2 MiB/197.2 MiB with 97 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658384.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658384.bam.bai\r\n",
      "Completed 178.2 MiB/197.2 MiB with 96 file(s) remaining\r",
      "Completed 178.3 MiB/197.2 MiB with 96 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658385.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658385.bam.bai\r\n",
      "Completed 178.3 MiB/197.2 MiB with 95 file(s) remaining\r",
      "Completed 178.5 MiB/197.2 MiB with 95 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658385.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658385.bam\r\n",
      "Completed 178.5 MiB/197.2 MiB with 94 file(s) remaining\r",
      "Completed 178.7 MiB/197.2 MiB with 94 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658386.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658386.bam\r\n",
      "Completed 178.7 MiB/197.2 MiB with 93 file(s) remaining\r",
      "Completed 178.8 MiB/197.2 MiB with 93 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658387.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658387.bam.bai\r\n",
      "Completed 178.8 MiB/197.2 MiB with 92 file(s) remaining\r",
      "Completed 178.9 MiB/197.2 MiB with 92 file(s) remaining\r",
      "Completed 179.1 MiB/197.2 MiB with 92 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658386.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658386.bam.bai\r\n",
      "Completed 179.1 MiB/197.2 MiB with 91 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658387.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658387.bam\r\n",
      "Completed 179.1 MiB/197.2 MiB with 90 file(s) remaining\r",
      "Completed 179.2 MiB/197.2 MiB with 90 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658388.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658388.bam.bai\r\n",
      "Completed 179.2 MiB/197.2 MiB with 89 file(s) remaining\r",
      "Completed 179.4 MiB/197.2 MiB with 89 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658388.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658388.bam\r\n",
      "Completed 179.4 MiB/197.2 MiB with 88 file(s) remaining\r",
      "Completed 179.6 MiB/197.2 MiB with 88 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658390.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658390.bam\r\n",
      "Completed 179.6 MiB/197.2 MiB with 87 file(s) remaining\r",
      "Completed 179.7 MiB/197.2 MiB with 87 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658390.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658390.bam.bai\r\n",
      "Completed 179.7 MiB/197.2 MiB with 86 file(s) remaining\r",
      "Completed 187.7 MiB/197.2 MiB with 86 file(s) remaining\r",
      "Completed 187.9 MiB/197.2 MiB with 86 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658391.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658391.bam\r\n",
      "Completed 187.9 MiB/197.2 MiB with 85 file(s) remaining\r",
      "Completed 188.0 MiB/197.2 MiB with 85 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658393.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658393.bam.bai\r\n",
      "Completed 188.0 MiB/197.2 MiB with 84 file(s) remaining\r",
      "Completed 188.0 MiB/197.2 MiB with 84 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906838.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906838.refCount\r\n",
      "Completed 188.0 MiB/197.2 MiB with 83 file(s) remaining\r",
      "Completed 188.1 MiB/197.2 MiB with 83 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658389.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658389.bam.bai\r\n",
      "Completed 188.1 MiB/197.2 MiB with 82 file(s) remaining\r",
      "Completed 188.3 MiB/197.2 MiB with 82 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658389.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658389.bam\r\n",
      "Completed 188.3 MiB/197.2 MiB with 81 file(s) remaining\r",
      "Completed 188.4 MiB/197.2 MiB with 81 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658394.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658394.bam.bai\r\n",
      "Completed 188.4 MiB/197.2 MiB with 80 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454614.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454614.bam\r\n",
      "Completed 188.4 MiB/197.2 MiB with 79 file(s) remaining\r",
      "Completed 188.5 MiB/197.2 MiB with 79 file(s) remaining\r",
      "Completed 188.6 MiB/197.2 MiB with 79 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658392.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658392.bam.bai\r\n",
      "Completed 188.6 MiB/197.2 MiB with 78 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658391.bam.bai to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658391.bam.bai\r\n",
      "Completed 188.6 MiB/197.2 MiB with 77 file(s) remaining\r",
      "Completed 188.6 MiB/197.2 MiB with 77 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906839.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906839.flagstat\r\n",
      "Completed 188.6 MiB/197.2 MiB with 76 file(s) remaining\r",
      "Completed 188.6 MiB/197.2 MiB with 76 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906838.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906838.flagstat\r\n",
      "Completed 188.6 MiB/197.2 MiB with 75 file(s) remaining\r",
      "Completed 188.8 MiB/197.2 MiB with 75 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658392.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658392.bam\r\n",
      "Completed 188.8 MiB/197.2 MiB with 74 file(s) remaining\r",
      "Completed 189.0 MiB/197.2 MiB with 74 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658393.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658393.bam\r\n",
      "Completed 189.0 MiB/197.2 MiB with 73 file(s) remaining\r",
      "Completed 189.0 MiB/197.2 MiB with 73 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906839.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906839.refCount\r\n",
      "Completed 189.0 MiB/197.2 MiB with 72 file(s) remaining\r",
      "Completed 189.0 MiB/197.2 MiB with 72 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906840.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906840.refCount\r\n",
      "Completed 189.0 MiB/197.2 MiB with 71 file(s) remaining\r",
      "Completed 189.0 MiB/197.2 MiB with 71 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906840.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906840.flagstat\r\n",
      "Completed 189.0 MiB/197.2 MiB with 70 file(s) remaining\r",
      "Completed 189.0 MiB/197.2 MiB with 70 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906841.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906841.refCount\r\n",
      "Completed 189.0 MiB/197.2 MiB with 69 file(s) remaining\r",
      "Completed 189.0 MiB/197.2 MiB with 69 file(s) remaining\r",
      "Completed 189.2 MiB/197.2 MiB with 69 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906842.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906842.flagstat\r\n",
      "Completed 189.2 MiB/197.2 MiB with 68 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR9658394.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR9658394.bam\r\n",
      "Completed 189.2 MiB/197.2 MiB with 67 file(s) remaining\r",
      "Completed 189.2 MiB/197.2 MiB with 67 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906842.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906842.refCount\r\n",
      "Completed 189.2 MiB/197.2 MiB with 66 file(s) remaining\r",
      "Completed 189.2 MiB/197.2 MiB with 66 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454606.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454606.flagstat\r\n",
      "Completed 189.2 MiB/197.2 MiB with 65 file(s) remaining\r",
      "Completed 189.2 MiB/197.2 MiB with 65 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 65 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 65 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454606.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454606.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 64 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906843.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906843.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 63 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 63 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454607.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454607.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 62 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 62 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454607.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454607.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 61 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 61 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454608.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454608.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 60 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 60 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906843.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906843.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 59 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 59 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/ERR2906841.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/ERR2906841.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 58 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 58 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454609.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454609.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 57 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 57 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 57 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 57 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454610.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454610.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 56 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454609.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454609.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 55 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454610.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454610.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 54 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 54 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454611.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454611.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 53 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 53 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454608.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454608.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 52 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/bam/SRR11454615.bam to s3://serratus-public/out/200423_ab_cov2r/bam/SRR11454615.bam\r\n",
      "Completed 197.2 MiB/197.2 MiB with 51 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 51 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454612.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454612.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 50 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 50 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454612.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454612.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 49 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 49 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454614.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454614.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 48 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 48 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454615.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454615.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 47 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 47 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454615.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454615.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 46 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 46 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658356.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658356.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 45 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 45 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454613.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454613.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 44 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 44 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658356.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658356.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 43 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 43 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 43 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454614.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454614.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 42 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658357.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658357.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 41 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 41 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454613.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454613.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 40 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 40 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658358.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658358.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 39 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 39 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658359.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658359.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 38 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 38 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658359.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658359.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 37 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 37 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658358.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658358.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 36 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 36 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 36 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658360.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658360.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 35 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658361.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658361.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 34 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 34 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR11454611.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR11454611.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 33 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 33 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658361.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658361.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 32 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 32 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658362.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658362.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 31 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 31 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658360.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658360.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 30 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 30 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658357.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658357.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 29 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 29 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658362.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658362.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 28 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 28 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658363.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658363.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 27 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 27 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658364.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658364.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 26 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 26 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658365.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658365.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 25 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 25 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658365.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658365.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 24 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 24 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658384.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658384.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 23 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 23 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658363.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658363.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 22 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 22 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 22 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658385.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658385.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 21 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658364.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658364.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 20 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 20 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658386.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658386.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 19 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 19 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658385.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658385.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 18 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 18 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658384.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658384.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 17 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 17 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658387.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658387.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 16 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 16 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658388.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658388.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 15 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 15 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658388.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658388.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 14 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 14 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658389.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658389.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 13 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 13 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658389.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658389.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 12 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 12 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658387.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658387.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 11 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 11 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658390.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658390.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 10 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 10 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658391.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658391.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 9 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 9 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658386.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658386.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 8 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 8 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658392.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658392.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 7 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 7 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658393.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658393.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 6 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 6 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658391.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658391.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 5 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 5 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658390.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658390.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 4 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 4 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658392.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658392.refCount\r\n",
      "Completed 197.2 MiB/197.2 MiB with 3 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 3 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658393.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658393.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 2 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 2 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658394.flagstat to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658394.flagstat\r\n",
      "Completed 197.2 MiB/197.2 MiB with 1 file(s) remaining\r",
      "Completed 197.2 MiB/197.2 MiB with 1 file(s) remaining\r",
      "copy: s3://tf-serratus-work-20200423221042780000000001/out/flagstat/SRR9658394.refCount to s3://serratus-public/out/200423_ab_cov2r/flagstat/SRR9658394.refCount\r\n"
     ]
    }
   ],
   "source": [
    "# Copy output to a permenant bucket\n",
    "# TODO: automatically transfer final outputs\n",
    "# to the permenant bucket\n",
    "aws s3 sync \\\n",
    "  s3://tf-serratus-work-20200423221042780000000001/out \\\n",
    "  s3://serratus-public/out/200423_ab_cov2r/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Destroy Cluster\n",
    "\n",
    "Close out all resources with terraform (will take a few minutes).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[1mmodule.download.aws_cloudwatch_log_group.g: Refreshing state... [id=serratus-dl]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_cloudwatch_log_group.g: Refreshing state... [id=serratus-merge]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role.role: Refreshing state... [id=SerratusIamRole-serratus-align]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_cloudwatch_log_group.scheduler: Refreshing state... [id=scheduler]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Refreshing state... [id=tf-serratus-work-20200423221042780000000001]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.data.aws_region.current: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1maws_security_group.internal: Refreshing state... [id=sg-0b1c9a028ea3bb7f0]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_cloudwatch_log_group.g: Refreshing state... [id=serratus-align]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.data.aws_availability_zones.all: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.data.aws_ami.amazon_linux_2: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.data.aws_availability_zones.all: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role.task_role: Refreshing state... [id=SerratusIamRole-monitor]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.data.aws_ami.ecs: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.data.aws_availability_zones.all: Refreshing state...\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role.role: Refreshing state... [id=SerratusIamRole-serratus-dl]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_instance_profile.monitor: Refreshing state... [id=profile-serratus-monitor]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role.role: Refreshing state... [id=SerratusIamRole-scheduler]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role.role: Refreshing state... [id=SerratusIamRole-serratus-merge]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_cluster.monitor: Refreshing state... [id=arn:aws:ecs:us-east-1:797308887321:cluster/serratus-monitor]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_instance_profile.profile: Refreshing state... [id=profile-serratus-align]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy.cloudwatch: Refreshing state... [id=SerratusIamRole-serratus-align:CloudWatchLogsCreate-serratus-align]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Refreshing state... [id=SerratusIamRole-serratus-align-20200423221045750000000005]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy_attachment.attachment: Refreshing state... [id=SerratusIamRole-monitor-20200423221045368800000004]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy.cloudwatch: Refreshing state... [id=SerratusIamRole-monitor:CloudwatchGetMetrics]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Refreshing state... [id=SerratusIamRole-serratus-dl-20200423221044120900000002]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_instance_profile.profile: Refreshing state... [id=profile-serratus-dl]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy.cloudwatch: Refreshing state... [id=SerratusIamRole-serratus-dl:CloudWatchLogsCreate-serratus-dl]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role_policy.cloudwatch: Refreshing state... [id=SerratusIamRole-scheduler:CloudWatchLogsCreate-scheduler]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_instance_profile.profile: Refreshing state... [id=profile-scheduler]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy.cloudwatch: Refreshing state... [id=SerratusIamRole-serratus-merge:CloudWatchLogsCreate-serratus-merge]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_instance_profile.profile: Refreshing state... [id=profile-serratus-merge]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Refreshing state... [id=SerratusIamRole-serratus-merge-20200423221044968200000003]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_instance.monitor: Refreshing state... [id=i-086e77637ebd22c38]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Refreshing state... [id=i-0139403d7159eb53d]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_eip.monitor: Refreshing state... [id=eipalloc-04ea14ff82fbc0d41]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.full: Refreshing state... [id=tf-serratus-work-20200423221042780000000001:full]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"bam-blocks\"]: Refreshing state... [id=tf-serratus-work-20200423221042780000000001:prefix-bam-blocks]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"fq-blocks\"]: Refreshing state... [id=tf-serratus-work-20200423221042780000000001:prefix-fq-blocks]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"out\"]: Refreshing state... [id=tf-serratus-work-20200423221042780000000001:prefix-out]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_delete[0]: Refreshing state... [id=SerratusIamRole-serratus-align:S3DeleteData-serratus-align]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_write: Refreshing state... [id=SerratusIamRole-serratus-merge:S3WriteData-serratus-merge]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_iam_role_policy.s3_write: Refreshing state... [id=SerratusIamRole-serratus-dl:S3WriteData-serratus-dl]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_delete[0]: Refreshing state... [id=SerratusIamRole-serratus-merge:S3DeleteData-serratus-merge]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_write: Refreshing state... [id=SerratusIamRole-serratus-align:S3WriteData-serratus-align]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_eip.sch: Refreshing state... [id=eipalloc-09bc27fef18639a4c]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_task_definition.monitor: Refreshing state... [id=monitor]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.hosts: Refreshing state... [id=7705e128fb4eeaf15c17804a91633e44e185e807]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.create_tunnel: Refreshing state... [id=46e450a0e9dca822ad7e6d6015a0bb658af1e97c]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_launch_configuration.worker: Refreshing state... [id=tf-serratus-merge-20200423221112408900000008]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_launch_configuration.worker: Refreshing state... [id=tf-serratus-dl-20200423221112630800000009]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_launch_configuration.worker: Refreshing state... [id=tf-serratus-align-20200423221112402100000007]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Refreshing state... [id=arn:aws:ecs:us-east-1:797308887321:service/serratus-monitor]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_autoscaling_group.worker: Refreshing state... [id=tf-asg-tf-serratus-dl-20200423221112630800000009]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Refreshing state... [id=tf-asg-tf-serratus-merge-20200423221112408900000008]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_autoscaling_group.worker: Refreshing state... [id=tf-asg-tf-serratus-align-20200423221112402100000007]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.merge_set_capacity: Refreshing state... [id=63c6450a668d1c442aff7f2b60d0202bef3f1d8e]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.align_set_capacity: Refreshing state... [id=06cf46b01dfabc5f2a61f5b51d43ee3a2ed357b0]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.dl_set_capacity: Refreshing state... [id=7dff4901b65f09a5603318e2597fe3dd01a1935f]\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.create_tunnel: Destroying... [id=46e450a0e9dca822ad7e6d6015a0bb658af1e97c]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.merge_set_capacity: Destroying... [id=63c6450a668d1c442aff7f2b60d0202bef3f1d8e]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.align_set_capacity: Destroying... [id=06cf46b01dfabc5f2a61f5b51d43ee3a2ed357b0]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.hosts: Destroying... [id=7705e128fb4eeaf15c17804a91633e44e185e807]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.align_set_capacity: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.hosts: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.create_tunnel: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.dl_set_capacity: Destroying... [id=7dff4901b65f09a5603318e2597fe3dd01a1935f]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.merge_set_capacity: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mlocal_file.dl_set_capacity: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.full: Destroying... [id=tf-serratus-work-20200423221042780000000001:full]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role_policy.cloudwatch: Destroying... [id=SerratusIamRole-scheduler:CloudWatchLogsCreate-scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"out\"]: Destroying... [id=tf-serratus-work-20200423221042780000000001:prefix-out]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_write: Destroying... [id=SerratusIamRole-serratus-merge:S3WriteData-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_delete[0]: Destroying... [id=SerratusIamRole-serratus-merge:S3DeleteData-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Destroying... [id=SerratusIamRole-serratus-align-20200423221045750000000005]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy.cloudwatch: Destroying... [id=SerratusIamRole-serratus-dl:CloudWatchLogsCreate-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Destroying... [id=SerratusIamRole-serratus-dl-20200423221044120900000002]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Destroying... [id=arn:aws:ecs:us-east-1:797308887321:service/serratus-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_iam_role_policy.s3_write: Destroying... [id=SerratusIamRole-serratus-dl:S3WriteData-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_write: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_iam_role_policy.s3_write: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy.cloudwatch: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy.cloudwatch: Destroying... [id=SerratusIamRole-serratus-merge:CloudWatchLogsCreate-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_iam_role_policy.s3_delete[0]: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role_policy.cloudwatch: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy_attachment.attachment: Destroying... [id=SerratusIamRole-monitor-20200423221045368800000004]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_delete[0]: Destroying... [id=SerratusIamRole-serratus-align:S3DeleteData-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_write: Destroying... [id=SerratusIamRole-serratus-align:S3WriteData-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"out\"]: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.full: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"fq-blocks\"]: Destroying... [id=tf-serratus-work-20200423221042780000000001:prefix-fq-blocks]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"bam-blocks\"]: Destroying... [id=tf-serratus-work-20200423221042780000000001:prefix-bam-blocks]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Destroying... [id=SerratusIamRole-serratus-merge-20200423221044968200000003]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy.cloudwatch: Destroying... [id=SerratusIamRole-monitor:CloudwatchGetMetrics]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Destroying... [id=tf-asg-tf-serratus-merge-20200423221112408900000008]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy.cloudwatch: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy.cloudwatch: Destroying... [id=SerratusIamRole-serratus-align:CloudWatchLogsCreate-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy_attachment.attachment: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_delete[0]: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_autoscaling_group.worker: Destroying... [id=tf-asg-tf-serratus-align-20200423221112402100000007]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_iam_role_policy.s3_write: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_eip.monitor: Destroying... [id=eipalloc-04ea14ff82fbc0d41]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_autoscaling_group.worker: Destroying... [id=tf-asg-tf-serratus-dl-20200423221112630800000009]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"fq-blocks\"]: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket_metric.prefix[\"bam-blocks\"]: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role_policy_attachment.attachment[\"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"]: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role_policy.cloudwatch: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role_policy.cloudwatch: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_eip.monitor: Destruction complete after 2s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_instance.monitor: Destroying... [id=i-086e77637ebd22c38]\u001b[0m\u001b[0m\r\n",
      "Connection to ec2-100-26-25-117.compute-1.amazonaws.com closed by remote host.\r",
      "\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Still destroying... [id=arn:aws:ecs:us-east-1:797308887321:service/serratus-monitor, 10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Still destroying... [id=tf-asg-tf-serratus-merge-20200423221112408900000008, 10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_autoscaling_group.worker: Still destroying... [id=tf-asg-tf-serratus-align-20200423221112402100000007, 10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_autoscaling_group.worker: Still destroying... [id=tf-asg-tf-serratus-dl-20200423221112630800000009, 10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_instance.monitor: Still destroying... [id=i-086e77637ebd22c38, 10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_instance.monitor: Destruction complete after 15s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_instance_profile.monitor: Destroying... [id=profile-serratus-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_instance_profile.monitor: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_autoscaling_group.worker: Destruction complete after 19s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Still destroying... [id=arn:aws:ecs:us-east-1:797308887321:service/serratus-monitor, 20s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_launch_configuration.worker: Destroying... [id=tf-serratus-merge-20200423221112408900000008]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_autoscaling_group.worker: Destruction complete after 19s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_launch_configuration.worker: Destroying... [id=tf-serratus-align-20200423221112402100000007]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_launch_configuration.worker: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_cloudwatch_log_group.g: Destroying... [id=serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_instance_profile.profile: Destroying... [id=profile-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_launch_configuration.worker: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_instance_profile.profile: Destroying... [id=profile-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_cloudwatch_log_group.g: Destroying... [id=serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.aws_cloudwatch_log_group.g: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_autoscaling_group.worker: Still destroying... [id=tf-asg-tf-serratus-dl-20200423221112630800000009, 20s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_instance_profile.profile: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role.role: Destroying... [id=SerratusIamRole-serratus-merge]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.aws_cloudwatch_log_group.g: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_instance_profile.profile: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role.role: Destroying... [id=SerratusIamRole-serratus-align]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.merge.module.iam_role.aws_iam_role.role: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.align.module.iam_role.aws_iam_role.role: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Still destroying... [id=arn:aws:ecs:us-east-1:797308887321:service/serratus-monitor, 30s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_autoscaling_group.worker: Destruction complete after 30s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_launch_configuration.worker: Destroying... [id=tf-serratus-dl-20200423221112630800000009]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_launch_configuration.worker: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_instance_profile.profile: Destroying... [id=profile-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_cloudwatch_log_group.g: Destroying... [id=serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Destroying... [id=tf-serratus-work-20200423221042780000000001]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_eip.sch: Destroying... [id=eipalloc-09bc27fef18639a4c]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.aws_cloudwatch_log_group.g: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_instance_profile.profile: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role.role: Destroying... [id=SerratusIamRole-serratus-dl]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.download.module.iam_role.aws_iam_role.role: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_eip.sch: Destruction complete after 2s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_service.monitor: Destruction complete after 38s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_cluster.monitor: Destroying... [id=arn:aws:ecs:us-east-1:797308887321:cluster/serratus-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_task_definition.monitor: Destroying... [id=monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_task_definition.monitor: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role.task_role: Destroying... [id=SerratusIamRole-monitor]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Destroying... [id=i-0139403d7159eb53d]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_ecs_cluster.monitor: Destruction complete after 0s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.monitoring.aws_iam_role.task_role: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Still destroying... [id=tf-serratus-work-20200423221042780000000001, 10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still destroying... [id=i-0139403d7159eb53d, 10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Still destroying... [id=tf-serratus-work-20200423221042780000000001, 20s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still destroying... [id=i-0139403d7159eb53d, 20s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Still destroying... [id=tf-serratus-work-20200423221042780000000001, 30s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still destroying... [id=i-0139403d7159eb53d, 30s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Still destroying... [id=tf-serratus-work-20200423221042780000000001, 40s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still destroying... [id=i-0139403d7159eb53d, 40s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Still destroying... [id=tf-serratus-work-20200423221042780000000001, 50s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still destroying... [id=i-0139403d7159eb53d, 50s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Still destroying... [id=tf-serratus-work-20200423221042780000000001, 1m0s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.work_bucket.aws_s3_bucket.work: Destruction complete after 1m1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still destroying... [id=i-0139403d7159eb53d, 1m0s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Still destroying... [id=i-0139403d7159eb53d, 1m10s elapsed]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_instance.scheduler: Destruction complete after 1m13s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_instance_profile.profile: Destroying... [id=profile-scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_cloudwatch_log_group.scheduler: Destroying... [id=scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1maws_security_group.internal: Destroying... [id=sg-0b1c9a028ea3bb7f0]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.aws_cloudwatch_log_group.scheduler: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_instance_profile.profile: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role.role: Destroying... [id=SerratusIamRole-scheduler]\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1maws_security_group.internal: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1mmodule.scheduler.module.iam_role.aws_iam_role.role: Destruction complete after 1s\u001b[0m\u001b[0m\r\n",
      "\u001b[0m\u001b[1m\u001b[32m\r\n",
      "Destroy complete! Resources: 52 destroyed.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "terraform destroy -auto-approve\n",
    "# WARNING this will also delete the standard output bucket/data\n",
    "# Save data prior to destroy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run Notes\n",
    "\n",
    "## Errors\n",
    "\n",
    "Accessions: `SRR6639047` - `SRR6639058` all suffered from `split_err` (download fault).\n",
    "\n",
    "With example error:\n",
    "\n",
    "```\n",
    "+ fastq-dump --split-e SRR9658359\n",
    "Rejected 3658747 READS because of filtering out non-biological READS\n",
    "Read 3658747 spots for SRR9658358\n",
    "Written 3658747 spots for SRR9658358\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}