Skip to content

Commit

Permalink
workflow readme details on kfp and misc tests
Browse files Browse the repository at this point in the history
Signed-off-by: David Wood <[email protected]>
  • Loading branch information
daw3rd committed Sep 18, 2024
1 parent e064e00 commit 3a81d21
Showing 1 changed file with 40 additions and 15 deletions.
55 changes: 40 additions & 15 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,29 @@
Here we have the start of a system to automatically generated github workflows (currently only for transforms).
In general, the design is to use templates and `make` to generate/update the workflows.

Goals
#### Goals
1. Run only tests for a given transform when only the transform changes.
Includes python, ray, spark and kfp_ray as available.
2. When the core dpk lib components files changes, test all transforms
3. When the shared kfp components changes, test a randomly selected transform test
(We would like to avoid running all transform kfp tests in one PR)
4. Extra credit: If .md or other non-code changes are made, run no tests.

Assumptions:
#### Assumptions
1. All transforms will have test workflows. A transform can disable its tests locally
(temporarily?) by renaming its transforms/universal/noop/Makefile.disabled.
```
git clone ....
...
git checkout -b new-branch
make # Creates new test*.yml workflows
git commit -a -s -m "update workflows"
git push --set-upstream origin new-branch
```
should be sufficient.
(temporarily?) by renaming its Makefile. For example,
`cp transforms/universal/noop/Makefile transforms/universal/noop/Makefile.disabled`.

## DPK libraries (`data-processing-lib` directory)
The DPK libraries, in data-processing-lib/{python,ray,spark}, are tested
via the fixed
[test-lib.yml](test-lib.yml)
file and is triggered when any code files in that tree change.

The transforms test workflows also depend on this directory tree and so
changes made here will trigger transform tests.

## Transforms
## Transforms (`transforms` directory tree)
We define a unique test workflow for each transform, based on a common template [test-transform.template](test-transform.template).
The Makefile is used to (re)generate all workflows a necessary. By design, workflows for a given transform should run when

Expand All @@ -37,6 +38,30 @@ When a new transform is added to the repository,
1. Run `make` in this directory to create the new test .yml for all transforms found in transforms/{universal,code,language} directories
1. commit and push the change to your branch with the new transform.

## KFP
Something like the following:
```
git clone ....
...
git checkout -b new-branch
make # Creates new test*.yml workflows
git commit -a -s -m "update workflows"
git push --set-upstream origin new-branch
```

## KFP (`kfp` directory tree)

Like DPK core libs, kfp tests are defined in
[test-kfp.yml](test-kfp.yml) and run whenever changes are made in
the `kfp` directory tree. Tests currently include

1. test kfp on randomly selected transform.

Eventually we would like to enable the transform-specific kfp test
when only the transform code is modified or maybe when only
the `kfp_ray` directory contents is modified.

## Miscellaneous
[test-misc.yml](test-misc.yml) defines some repo consistency tests including

## DPK libraries
1. Make sure `set-versions` make target can be run recursively throughout the repo
2. Makes sure there is a test workflow for each transform in the repo.

0 comments on commit 3a81d21

Please sign in to comment.