Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revising trycycler and select assembly implementations #61

Open
wants to merge 34 commits into
base: main
Choose a base branch
from

Conversation

fredjaya
Copy link
Member

@fredjaya fredjaya commented Nov 4, 2024

Tested and works on all Vibrio and Tenacibaculum barcodes. Currently on /scratch/er01/fj9712/2411_wholetest - to be moved.

Things to discuss and address either in this PR or later ones:

  1. How should the results directory be structured? What files should be output here? (e.g. Medaka output for consensus chromosomes is split into 2 subdirectories #28)
  2. Should the select assembly stats be reported?
  3. What should go in the MultiQC report?
  4. Does it work on the cat and dog data? (Fred currently testing)

Implementation

image

Reference-free chromosome assembly selection

Addresses #54, #23

For the chromosomal assembly, every barcode is assembled by flye and unicycler, and polished. The single "best" polished assembly out of flye, unicycler, and optionally trycycler (consensus assembly), is selected for downstream annotation and analyses.

To avoid biasing assemblies to published references, the assembly with the most complete BUSCOs is considered the best one. This now allows unicycler assemblies to be considered too. QUAST is also run but not used for selecting the best assembly.

Now only has a implementation for chromosomal assembly, instead of two independent ones, to make updating the criteria for selecting an assembly easier. For example, to incorporate QUAST outputs, or add additional tools like Merqury.

Trycycler implementation

Addresses #43, #60

Trycycler processes are now self-contained. Additional assemblers can be implemented easier to generate better consensus assemblies if required.

Added more error handling for too-few-contigs (trycycler cluster filters more out). If trycycler correctly fails at any point, the pipeline will still continue and select either the flye or unicycler assembly for downstream processes.

Input/output process definitions are more explicit (i.e. specific files instead of globs) for better error handling. A lot more operators and groovy in the workflow scope as a result.

Modularising assembly steps for incoming revision of select_assembly.

Current implementation branches channels according to the trycycler_cluster output. Move counting of contigs per assembly prior to trycycler so all trycycler (sub)processes can be run directly one after the other.
Refactoring iteratively so everything doesn't break
TIL the clustering step filters out small contigs (default < 5000 nt)
and will run into < 2 contigs error in trycycler_classify again.
Consider flye and unicycler assemblies as "de novo". This impacts which
data are grouped together for processing and channelling.

Previously, medaka was hardcoded to polish only trycycler and flye
assemblies. This commit adds an assembly-agnostic module for medaka
polishing (WIP).

Introduce val(assembler_name) for tagging, reporting etc.
Rename channels and comments to clarify differences between de novo vs. consensus assemblies.

combined assemblies should be both de novo and consensus (all).
Add more flexible quast module, some temp medaka changes to keep old
implementation running for now
To better align with best practices and readability. Be more explicit
with error strategy and process script def. This required some
additional groovy in workflow{} though
- Remove manual file moving or bash conditionals in process script def
- Remove if/else channel operators/groovy
- Output dir no longer has barcode, might re-add later, but might be ok
because it's output with the barcode tuple
Change process outputs to recurse through barcode and cluster
directories (e.g. **/out_file)
Mainly tidying medaka denovo and consensus implementations to look for
the polished assembly in the process outputs.
Fixes inconsistent publishing for assemblies and qc results
Diffs I have intentionally kept separate - a lot of things add temporary
tweaks to get this current version running during development.
Also fix `trycycler_reconcile_new` inconsistent `2_all_seqs.fasta`
process output
For trycycler and flye-specific downstream processes, modules, and
config. Commenting out existing "chromosome" implementations and will re-add progressively.
`select_assembly_new` didn't cache properly as it was outputing a
`stdout` - best assembly now stored in a text file.

Update bakta and amrfinderplus processes for chromosome annotation to
handle new metadata, reduce `mkdir` and file movement within script, and
decouple output definitions from hardcoded paths etc.

No longer need `helper.patch`
Clarify map syntax, module tags, publishDir handling
@fredjaya fredjaya marked this pull request as ready for review November 4, 2024 00:18
@fredjaya
Copy link
Member Author

fredjaya commented Nov 4, 2024

This is what the current results/ folder looks like for a single barcode:

results
├── annotations
│   ├── barcode01
│   │   ├── abricate
│   │   │   └── barcode01_consensus_chr.txt
│   │   ├── amrfinderplus
│   │   │   └── barcode01_consensus_chr.tsv
│   │   ├── bakta
│   │   │   ├── barcode01_consensus_chr.faa
│   │   │   └── barcode01_consensus_chr.txt
│   │   └── plasmids
│   │       └── barcode01_bakta
├── assemblies
│   ├── barcode01_consensus
│   │   └── consensus.fasta
│   ├── barcode01_flye
│   │   └── consensus.fasta
│   ├── barcode01_plassembler
│   │   ├── flye_output
│   │   ├── logs
│   │   ├── plassembler_1730446699.3410256.log
│   │   ├── plassembler_plasmids.fasta
│   │   ├── plassembler_plasmids.gfa
│   │   ├── plassembler_summary.tsv
│   │   └── unicycler_output
│   ├── barcode01_unicycler
│   │   └── consensus.fasta
├── quality_control
│   ├── barcode01
│   │   ├── barcode01_consensus
│   │   ├── barcode01_consensus_busco
│   │   ├── barcode01_consensus.tsv
│   │   ├── barcode01_flye
│   │   ├── barcode01_flye_busco
│   │   ├── barcode01_flye.tsv
│   │   ├── barcode01_unicycler
│   │   ├── barcode01_unicycler_busco
│   │   └── barcode01_unicycler.tsv
│   ├── barcode01_kraken2
│   │   └── barcode01.k2report
├── report
│   ├── barcode01_consensus
│   ├── barcode01_flye
│   ├── barcode01_unicycler
├── run_info
│   ├── dag.svg
│   ├── gadi-nf-core-trace-*.txt
│   ├── report.html
│   └── timeline.html
├── taxonomy
│   ├── abricate_vfdb_output.txt
│   ├── amrfinderplus_output.txt
│   ├── barcode_species_table_mqc.txt
│   ├── combined_plot_mqc.png
│   └── phylogeny
└── tree

350 directories, 234 files

Suggestions:

  • publish only the selected chromosomal assembly, remove assembler from name
  • ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant