From fea9370b3b27301af05f6d06bb18d3e9179395e8 Mon Sep 17 00:00:00 2001 From: Abdullah Yildirim Date: Mon, 25 Mar 2024 10:13:34 -0400 Subject: [PATCH] chore: refactor developer guide (#112) Signed-off-by: a3hadi --- development.md | 57 +++++++++---------- pkg/mapper/examples/even_odd/go.mod | 2 +- pkg/mapper/examples/flatmap/go.mod | 2 +- pkg/mapper/examples/forward_message/go.mod | 2 +- pkg/mapper/examples/retry/go.mod | 2 +- pkg/mapper/examples/tickgen/go.mod | 2 +- .../examples/flatmap_stream/go.mod | 2 +- pkg/reducer/examples/counter/go.mod | 2 +- pkg/reducer/examples/sum/go.mod | 2 +- pkg/reducestreamer/examples/counter/go.mod | 2 +- pkg/reducestreamer/examples/sum/go.mod | 2 +- pkg/sessionreducer/examples/counter/go.mod | 2 +- .../examples/simple-sideinput/go.mod | 2 +- .../examples/simple-sideinput/udf/go.mod | 2 +- pkg/sinker/examples/log/go.mod | 2 +- pkg/sourcer/examples/simple_source/go.mod | 2 +- .../examples/assign_event_time/go.mod | 2 +- .../examples/event_time_filter/go.mod | 2 +- 18 files changed, 44 insertions(+), 47 deletions(-) diff --git a/development.md b/development.md index 1849baff..505305c4 100644 --- a/development.md +++ b/development.md @@ -1,50 +1,47 @@ -# Development +# Developer Guide This document explains the development process for the Numaflow Go SDK. - ### Testing The SDK uses local references in the `go.mod` file, i.e. the `github.com/numaproj/numaflow-go` dependency is pointing to your local `numaflow-go` directory. Thus, you can automatically test your SDK changes without the need to push to your forked repository or modify the `go.mod` file. -Simply make your changes, build and push the desired example image (with an appropriate tag, such as `test`), and you are ready to use it in any pipeline. +Simply make your changes, build the desired example image, and you are ready to use it in any pipeline. -Each example directory has a Makefile which can be used to build, tag, and push images. In most cases, the `image-push` target should be used. -However, `buildx`, which is used to support multiple architectures, does not make a built image available locally. In the case that a developer -wants this functionality, they can use the `image` target. +Each example directory has a Makefile which can be used to build, tag, and push images. +If you want to build and tag the image and then immediately push it to quay.io, use the `image-push` target. +If you want to build and tag a local image without pushing, use the `image` target. -After making changes to the SDK, if you want to build all the example images at once, in the root directory you can run: +If you want to build and push all the example images at once, you can run: ```shell ./hack/update_examples.sh -bp -t ``` -The default tag is `stable`, but it is recommended you specify your own for testing purposes, as the Github Actions CI uses the `stable` tag. Note: do not forget to clean up testing tags -in the registry, i.e. delete them, once you are done testing. +The default tag is `stable`, but it is recommended you specify your own for testing purposes, as the Github Actions CI uses the `stable` tag. +This consistent tag name is used so that the tags in the [E2E test pipelines](https://github.com/numaproj/numaflow/tree/main/test) do not need to be updated each time an SDK change is made. -You can alternatively build a specific example image by running the following in the root directory and providing the path to the Dockerfile (relative to root): +You can alternatively build and push a specific example image by running the following: ```shell -./hack/update_examples.sh -bpe -t +./hack/update_examples.sh -bpe -t ``` This is essentially equivalent to running `make image-push TAG=` in the example directory itself. +Note: before running the script, ensure that through the CLI, you are logged into quay.io. + ### Deploying -1. Create a PR for your changes. Once merged, it will trigger a workflow to build and push the images for all the examples, -with the tag `stable`. This consistent tag name is used so that the tags in the [E2E test pipelines](https://github.com/numaproj/numaflow/tree/main/test) do not need to be -updated each time a new change is made. -2. If the change that you have introduced is a breaking change, then after merge you should update the dependency management files so that the version -displayed in the `go.mod` file reflects the commit SHA of the merged changes. This can be done by getting the -commit SHA of the merged changes and using it with the update script. Alternatively, you can provide the specific version that you would like to update to, or even -pass in `latest` to fetch the latest version from the remote repository: +After confirming that your changes pass local testing: +1. Clean up testing artifacts +2. Create a PR. Once your PR has been merged, a Github Actions workflow (Docker Publish) will be triggered, to build, tag (with `stable`), and push + all example images. This ensures that all example images are using the most up-to-date version of the SDK, i.e. the one including your + changes. + +### After release + +Once a new version has been released, and its corresponding version tag exists on the remote repo, you want to update the `go.mod` +files to reflect this new version: ```shell -./hack/update_examples.sh -u -``` -After running the script, create another PR for these changes. Ideally, the update script should only be need to run when a new version is released, i.e. provide a version or `latest` to it, -or when a breaking change is merged before the next release, i.e. provide a commit SHA to it. If your merged change is a small chore, then it is unnecessary to run the update script as we want to -avoid flooding the commit history with dependency updates. - -Updating the version may not seem necessary since we are using local references. However, the client prints -out information related to the server, which includes the SDK version, which is retrieved from the `go.mod` file. -After a change is merged/new release, even though the images will be using the most up to date SDK -version, if the dependency management files are not updated, the logs will print the previous version as the current SDK version. -Thus, in order for the correctness of the server information, consistency, and to avoid future confusion, it is recommended -to update the `numaflow-go` dependency version across all the example directories, after a large/breaking change or new release. +./hack/update_examples.sh -u + ``` +After running the above, create a PR for the changes that the script made. Once merged, it will trigger the Docker Publish workflow. +As a result, the correct SDK version will always be printed in the server information logs, +and the example images will always be using the latest changes (due to the local references). \ No newline at end of file diff --git a/pkg/mapper/examples/even_odd/go.mod b/pkg/mapper/examples/even_odd/go.mod index 1e27d242..a90a11ef 100644 --- a/pkg/mapper/examples/even_odd/go.mod +++ b/pkg/mapper/examples/even_odd/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/mapper/examples/flatmap/go.mod b/pkg/mapper/examples/flatmap/go.mod index c16fc9dd..b92c7843 100644 --- a/pkg/mapper/examples/flatmap/go.mod +++ b/pkg/mapper/examples/flatmap/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/mapper/examples/forward_message/go.mod b/pkg/mapper/examples/forward_message/go.mod index 1e27d242..a90a11ef 100644 --- a/pkg/mapper/examples/forward_message/go.mod +++ b/pkg/mapper/examples/forward_message/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/mapper/examples/retry/go.mod b/pkg/mapper/examples/retry/go.mod index 809603ac..f23d6bd1 100644 --- a/pkg/mapper/examples/retry/go.mod +++ b/pkg/mapper/examples/retry/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/mapper/examples/tickgen/go.mod b/pkg/mapper/examples/tickgen/go.mod index 8e9a570b..f5dbf906 100644 --- a/pkg/mapper/examples/tickgen/go.mod +++ b/pkg/mapper/examples/tickgen/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/mapstreamer/examples/flatmap_stream/go.mod b/pkg/mapstreamer/examples/flatmap_stream/go.mod index 05f60e9a..ca0e15cf 100644 --- a/pkg/mapstreamer/examples/flatmap_stream/go.mod +++ b/pkg/mapstreamer/examples/flatmap_stream/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/reducer/examples/counter/go.mod b/pkg/reducer/examples/counter/go.mod index a185a623..a3d73e82 100644 --- a/pkg/reducer/examples/counter/go.mod +++ b/pkg/reducer/examples/counter/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/reducer/examples/sum/go.mod b/pkg/reducer/examples/sum/go.mod index 208c27bf..b6c66010 100644 --- a/pkg/reducer/examples/sum/go.mod +++ b/pkg/reducer/examples/sum/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/reducestreamer/examples/counter/go.mod b/pkg/reducestreamer/examples/counter/go.mod index a185a623..a3d73e82 100644 --- a/pkg/reducestreamer/examples/counter/go.mod +++ b/pkg/reducestreamer/examples/counter/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/reducestreamer/examples/sum/go.mod b/pkg/reducestreamer/examples/sum/go.mod index 208c27bf..b6c66010 100644 --- a/pkg/reducestreamer/examples/sum/go.mod +++ b/pkg/reducestreamer/examples/sum/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/sessionreducer/examples/counter/go.mod b/pkg/sessionreducer/examples/counter/go.mod index cbce3830..228bc09b 100644 --- a/pkg/sessionreducer/examples/counter/go.mod +++ b/pkg/sessionreducer/examples/counter/go.mod @@ -5,7 +5,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. require ( - github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 + github.com/numaproj/numaflow-go v0.6.0 go.uber.org/atomic v1.11.0 ) diff --git a/pkg/sideinput/examples/simple-sideinput/go.mod b/pkg/sideinput/examples/simple-sideinput/go.mod index b2c3daa2..449a225c 100644 --- a/pkg/sideinput/examples/simple-sideinput/go.mod +++ b/pkg/sideinput/examples/simple-sideinput/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/sideinput/examples/simple-sideinput/udf/go.mod b/pkg/sideinput/examples/simple-sideinput/udf/go.mod index b2e9caea..53d5de3d 100644 --- a/pkg/sideinput/examples/simple-sideinput/udf/go.mod +++ b/pkg/sideinput/examples/simple-sideinput/udf/go.mod @@ -6,7 +6,7 @@ replace github.com/numaproj/numaflow-go => ../../../../.. require ( github.com/fsnotify/fsnotify v1.6.0 - github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 + github.com/numaproj/numaflow-go v0.6.0 ) require ( diff --git a/pkg/sinker/examples/log/go.mod b/pkg/sinker/examples/log/go.mod index a22cf0f9..5820da2a 100644 --- a/pkg/sinker/examples/log/go.mod +++ b/pkg/sinker/examples/log/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/sourcer/examples/simple_source/go.mod b/pkg/sourcer/examples/simple_source/go.mod index c46157e5..75fc81d3 100644 --- a/pkg/sourcer/examples/simple_source/go.mod +++ b/pkg/sourcer/examples/simple_source/go.mod @@ -6,7 +6,7 @@ replace github.com/numaproj/numaflow-go => ../../../.. require ( github.com/google/uuid v1.3.0 - github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 + github.com/numaproj/numaflow-go v0.6.0 github.com/stretchr/testify v1.8.1 ) diff --git a/pkg/sourcetransformer/examples/assign_event_time/go.mod b/pkg/sourcetransformer/examples/assign_event_time/go.mod index 72254972..41b0603d 100644 --- a/pkg/sourcetransformer/examples/assign_event_time/go.mod +++ b/pkg/sourcetransformer/examples/assign_event_time/go.mod @@ -4,7 +4,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. -require github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 +require github.com/numaproj/numaflow-go v0.6.0 require ( github.com/golang/protobuf v1.5.3 // indirect diff --git a/pkg/sourcetransformer/examples/event_time_filter/go.mod b/pkg/sourcetransformer/examples/event_time_filter/go.mod index c2f33951..949a8d93 100644 --- a/pkg/sourcetransformer/examples/event_time_filter/go.mod +++ b/pkg/sourcetransformer/examples/event_time_filter/go.mod @@ -5,7 +5,7 @@ go 1.20 replace github.com/numaproj/numaflow-go => ../../../.. require ( - github.com/numaproj/numaflow-go v0.6.1-0.20240221033908-7c1198bc89f1 + github.com/numaproj/numaflow-go v0.6.0 github.com/stretchr/testify v1.8.1 )