From d78d7b2b83507ca649e954c8cd5a6ca4b0423688 Mon Sep 17 00:00:00 2001 From: Revital Sur Date: Sun, 22 Sep 2024 14:36:58 +0300 Subject: [PATCH] Change README.md. Signed-off-by: Revital Sur --- .github/workflows/README.md | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index faf8139b4..d4a09b1c3 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,20 +1,23 @@ # Workflow Management -Here we have the start of a system to automatically generated github workflows (currently only for transforms). +Here we have the start of a system to automatically generated github workflows. In general, the design is to use templates and `make` to generate/update the workflows. #### Goals 1. Run only tests for a given transform when only the transform changes. Includes python, ray, spark and kfp_ray as available. 2. When the core dpk lib components files changes, test all transforms -3. When the shared kfp components changes, test a randomly selected transform test - (We would like to avoid running all transform kfp tests in one PR) +3. When the shared kfp components changes or core dpk lib components files changes, + test a randomly selected transform test. Otherwise run kfp test for the changed transforms. 4. Extra credit: If .md or other non-code changes are made, run no tests. #### Assumptions 1. All transforms will have test workflows. A transform can disable its tests locally (temporarily?) by renaming its Makefile. For example, `cp transforms/universal/noop/Makefile transforms/universal/noop/Makefile.disabled`. +A github action for kfp testing will not be generated if it appears in `KFP_BLACK_LIST` +in the [Makefile](./Makefile). + ## DPK libraries (`data-processing-lib` directory) The DPK libraries, in data-processing-lib/{python,ray,spark}, are tested @@ -26,18 +29,18 @@ The transforms test workflows also depend on this directory tree and so changes made here will trigger transform tests. ## Transforms (`transforms` directory tree) -We define a unique test workflow for each transform, based on a common -template [test-transform.template](test-transform.template). -The [Makefile](Makefile) is used to (re)generate all workflows a necessary. -By design, workflows for a given transform should run when +We define two test workflows for each transform: one is based on a common +template [test-transform.template](test-transform.template) and the other, for kfp testing, +is based on a common template [test-kfp-transform.template](test-kfp-transform.template). +The [Makefile](Makefile) is used to (re)generate all workflows as necessary. +By design, non kfp workflows for a given transform should run when * anything of substance effecting operation is modified in the transform's directory tree. * anything in the core libraries in this repo (e.g., data-processing/lib) assuming the transform depends on these. -Note that the kfp tests (in kfp_ray/Makefile workflow-test) for a given transform are -**not** currently being run when the transform's tests are run. -Currently these are run randomly via the [test-kfp.yml](test-kfp.yml). -We expect to fix this is in the future. +The generated kfp workflows should run when anything of substance effecting operation is modified in the transform's directory tree +and non of the core libraries in this repo nor the kfp components were changed. +Otherwise a randomly chosen transform will undergo KFP testing, triggered by the [test-kfp.yml](test-kfp.yml) workflow. When a new transform is added to the repository, @@ -58,16 +61,11 @@ git push --set-upstream origin new-branch Like DPK core libs, kfp tests are defined in [test-kfp.yml](test-kfp.yml) and run whenever changes are made in -the `kfp` directory tree. Tests currently include - -1. test kfp on randomly selected transform. - -Eventually we would like to enable the transform-specific kfp test -when only the transform code is modified or maybe when only -the `kfp_ray` directory contents is modified. +the `kfp` directory tree as well as in the DPK core libs. Tests currently include +test kfp on randomly selected transform. ## Miscellaneous [test-misc.yml](test-misc.yml) defines some repo consistency tests including 1. Make sure `set-versions` make target can be run recursively throughout the repo -2. Makes sure there is a test workflow for each transform in the repo. \ No newline at end of file +2. Makes sure there is a test workflow for each transform in the repo.