Skip to content

Commit

Permalink
docs: add draft on operators
Browse files Browse the repository at this point in the history
  • Loading branch information
kreczko committed Oct 12, 2023
1 parent f0c6ba4 commit b1dd37e
Showing 1 changed file with 19 additions and 4 deletions.
23 changes: 19 additions & 4 deletions docs/operators.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,22 @@
# Operators

Operators are used here as a general term of callable code blocks that operate
on data. In Airflow, operators are used to define tasks in a DAG. In
`fasthep-flow`, operators are used to define stages in a workflow. The operators
are defined in the YAML file, and then mapped to Airflow operators when the DAG
is generated.
on data. In Airflow, operators are used to define tasks in a
[DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph). In `fasthep-flow`,
operators are used to define stages in a workflow. The operators are defined in
the YAML file, and then mapped to Airflow operators when the DAG is generated.
One `fasthep-flow` operator can map to multiple Airflow operators.

## Operator Types

There are four types of operators:

1. **Data Input**: These are a special set that does not require any input data,
and instead generates data. These are used to start a workflow.
2. **Data Output**: These are a special set that does not require any output
data, and instead consumes data. These are used to end a workflow.
3. **Data Transform**: These are the most common operators, and are used to
transform data or add data to the workflow.
4. **Filter**: These are used to filter data. They are similar to data transform
operators, but instead of adding data, they restrict part of the data to
continue in the workflow.

0 comments on commit b1dd37e

Please sign in to comment.