Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for using cross-organisational components #2816

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,9 @@ There are four additional flags that you can use when installing a subworkflow:
- `--force`: Overwrite a previously installed version of the subworkflow.
- `--prompt`: Select the subworkflow version using a cli prompt.
- `--sha <commit_sha>`: Install the subworkflow at a specific commit.

:::note
You can use the `meta.yml` file to specify different remotes for the components
in your subworkflow. This is useful when you want to use modules from organisations
different from the one where the subworkflow is located. For more information, [see our tutorial](/docs/tutorials/external_usage/cross_organisational_subworkflows).
:::
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
---
title: 'Using subworkflows with modules from multiple organisations'
subtitle: Guidance on how to use nf-core/tools with subworkflows that use modules from more than one organisation.
---

In order to use cross-organisational subworkflows, you can specify individual remotes to be used for each module within the `meta.yml`
file in that subworkflow.

For example, let's say you have a subworkflow in your own organisation's `modules` repository that uses FastQC and fastp.
But, instead of using the fastp module from your own organisation, you want to use nf-core's fastp module. This is the case for the `fastq_trim_fastp_fastqc` subworkflow in the [nf-core-test](https://github.com/nf-core-test/modules/) organization.

In order to have this subworkflow, with a FastQC module from your organisation and fastp from nf-core, you'd define the
components section of this subworkflow's `meta.yml` file as such:

```yaml title="meta.yml" {11-12}
name: 'fastq_trim_fastp_fastqc'
description: Read QC, fastp trimming and read qc
keywords:
- qc
- quality_control
- adapters
- trimming
- fastq
components:
- fastqc
- fastp:
git_remote: https://github.com/nf-core/modules.git
```

By using the structure above, `nf-core/tools` knows that it should install FastQC from the same repository that the subworkflow is located
in, but fastp should come from the `nf-core/modules` remote.

:::note
Be aware that to install subworkflows from organisations other than nf-core you must
specify `--git-remote` in the `nf-core subworkflows` command, e.g. `nf-core subworkflows install fastq_trim_fastp_fastqc --git-remote $ORG_NAME{:bash}`
:::

If you're using cross-organisational subworkflows in your repository, be aware that you must also specify a different JSON schema
to lint the `meta.yml` files for your subworkflows.
To do this, just change the `components` section type to allow both objects and strings.

```json
"components": {
"type": "array",
"description": "Modules and subworkflows used in the subworkflow",
"items": {
"type": ["object", "string"]
},
"minItems": 0
},
Comment on lines +38 to +50
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's easier to just add these changes to the schema in the modules repo

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It won`t be added to the main modules repo, I believe, but I am adding it to the nf-core-test repo. I can remove this section and just link to the file there.

```

:::tip
To get the full content for the schema, see the corresponding
file in the [nf-core-test](https://github.com/nf-core-test/modules/) modules repository.
:::
Loading