Arteria delivery & final NGI-rnaseq

• Anaconda now separated into an unique role
• UGC delivery is now part of the deployment
• Weekly snapshots of Cassandra's database are now performed
• Standalone scripts repo now auto-fetches latest version, and has been added to $PATH
• Nougat has been version bumped to support additional features
• NGI-Pipeline has been version bumped to the latest stable version
• TACA has been version bumped to the latest stable version
• Corrected an outdated reference genome for fastq_screen
• Removed legacy test_devel role

NGI-rnaseq related

• Reversed back to local (non-module) nextflow installation
• Several bugfixes for nextflow implementation
• ngi-rnaseq now uses an updated configuration
• The alias rnaseq is now again in use
• ngi-rnaseq workfiles are now back to ngi-sw access, rather than ngi
• Due to a requirement change, R packages are now installed

Arteria delivery related

• Arteria-delivery webservice, for handling deliveries via Mover and SUPR, is part of the deployment
• Various arteria-delivery webservice bugfixes
• Arteria-delivery seperated into unique python3 environment in order to solve a critical async problem

TACA Delivery related

• Ign_reports is now a requirement for Uppsala deliveries, and removed for stockholm
• md5 files are now forcibly recreated when create_ngi_pipeline_dirs.sh is ran
• Fixed a NGI_Reports collision due to $PATH
• NGI_visualizations repo version updated
• Fixed a wildcard related issue

NGI Pipeline Update

• NGI Pipeline updated to version 0.1.6 with, amongst other things, support for the NextFlow Rna-Seq pipeline
• MultiQC and MultiQC NGI updated to version 0.9 and 0.3.1 respectively to graph some new features and fix some bugs.
• uppnex_datadeliver script has been included, for data deliveries (primarily used by Uppsala for now)
• The standalone_scripts repo has been included, for small production ready scripts with hardly any dependencies
• R dependencies removed from NGI_Pipeline
• NextFlow variables added to properly support the module load
• Piper module load is reinstated in order to run NGI_Pipeline. Removal of this will be saved for a latter version.

Bianca deliveries and bugfixes

• Staging version now submits jobs to the testcore nodes
• Wildwest folder is now writable by anyone
• E-mails are now seperated for Stockholm and Uppsala
• TACA-ngi-pipeline updated to deliver to more clusters (bianca and mosler)
• TACA updated to include new test command
• NGI_Reports & NGI visualizations updated to completely remove PDF generation
• Nextflow is now removed; rely on the module system instead
• MultiQC now properly sends its reports to another server (tools)
• Folder name for denovo projects changed

Updated MultiQC installation

General updates

• MultiQC upgraded to version v0.8
• MultiQC NGI plugin upgraded to version v0.3
• MultiQC now pushes reports to the genomics.www server
• NouGAT now runs in the NGI environment per default
• Bootstrap script to more easily set up initial deployment
• Github readme has been improved to better accomdate for the new deployment schedule
• The GATK key is now provided under the resources/piper folder
• create_ngi_pipeline_dirs.sh now also creates a denovo directory under NGI/nobackup/denovo

Staging specific updates

• NGI Pipeline now runs on reserved test cores
• Automatically linked open-source testing data is now available under the incoming directories

New version of TACA-NGI

This release of the Irma environment introduces a new version of TACA-NGI that supports copy of NGI reports to an outbox, from where they can be rsynced down to a local site. Enabled for Uppsala.

It also adds a couple of minor corrections vs previous release:

• adds the Piper module to the interactive shell
• fixes the loading of the Piper module in the ngi_pipeline config
• cleanup of Uppsala crontab
• fixes a bug where proj_root got set to /proj for all environments
• explicitly sets the Bash prompt so that it works in both production env for func user and staging env for personal user at the same time
• updates the Arteria web services so that they use the Irma environment's staging/production features instead of rolling their own

Bug fixes for production environment

This release fixes a couple of issues discovered when trying to deploy v1.1 inside production environment.

Mainly it adds the missing TACA config files, but also shortens the Bash prompt, setups the correct symlinks in create_ngi_pipeline_dirs.sh script, and adds some clarifying comments in the README. It also adds a minor package PyVCF to be used for jointcalling.

Piper 1.5.0 via module system

This release of the Irma environment switches over to use Piper 1.5.0 from the module system instead of 1.4.0 that was previously deployed via our own Ansible role.

It also:

• pulls in a new version of ngi_pipeline that includes some fixes that makes it pick up environment variables set by the Piper module, instead of having them hard coded in the environment configs
• prefixes the bash prompt with an (irma-env: <version>) for easier irma environment identification
• changes the log structure from /proj/.../private/ngi_pipeline/{db,log}/ to /proj/.../private/{db,log}, as some of the files ended up in un-intuitive locations before

First proper release

This is the first proper release that we will start using when provisioning the production environment on Irma.

Introduces the support of several parallel installations of the environment according to:

• /lupus/ngi/irma3/devel-root/<username>_<branch name>/{conf,resources,sw} on irma3 for development purposes
• /lupus/ngi/staging/<version>/{conf,resources,sw} for staging tests
• /lupus/ngi/production/<version>/{conf,resources,sw} for production usage

/lupus/ngi/staging/latest and /lupus/ngi/production/latest will be symlinks to the latest deployed version of the respective type.

Both staging and devel introduces a directory wildwest under /lupus/ngi/irma3/devel-root/wildwest and /lupus/ngi/staging/wildwest that acts as the new root path for the proj dir structure. I.e. it replaces /proj in all normal paths.

Production deployments should only be done with releases published here on Github. Staging deployments can be done with local branches checkout on irma3, or with releases published here on Github if several feature branches are supposed to be tested at the same time. More details in the README.

First test pre-release

This initial pre-release is to help testing the devel -> staging -> production workflow.