Skip to content

Commit

Permalink
docs: add draft on executors
Browse files Browse the repository at this point in the history
  • Loading branch information
kreczko committed Oct 12, 2023
1 parent b1dd37e commit 5e8b6b7
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 1 deletion.
27 changes: 27 additions & 0 deletions docs/executors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Executors - run your workflows

Since `fasthep-flow` is built on top of Apache Airflow, it supports all of the
executors that Airflow supports. The default executor is the
[LocalExecutor](#localexecutor), which runs all tasks in parallel on the local
machine. A full list of executors can be found in the
[Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html).

Since Apache Airflow is not widely used in High Energy Particle Physics, let's
go over the executors that are most relevant to us.

## LocalExecutor

The `LocalExecutor` (see
[Airflow docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html))
runs each task in a separate process on the local machine. This is the default
executor for `fasthep-flow`.

## DaskExecutor

The `DaskExecutor` (see
[Airflow docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html))
runs each task in a separate process on a Dask cluster. A Dask cluster can be
run on a local machine or as a distributed cluster using a batch system (e.g.
HTCondor, LSF, PBS, SGE, SLURM) or other distributed systems such as LHCb's
DIRAC. This is the recommended executor for running `fasthep-flow` workflows on
distributed resources.
3 changes: 2 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,13 @@ YAML file. More on this under [Configuration](./configuration.md).
## Documentation

```{toctree}
changelog.md
installation.md
concepts.md
configuration.md
operators.md
executors.md
examples/index.md
advanced_examples/index.md
developers_corner.md
```

0 comments on commit 5e8b6b7

Please sign in to comment.