From eb0acb7725c51772407b37c11bb2487512b2f9b6 Mon Sep 17 00:00:00 2001 From: cosmicBboy Date: Thu, 18 Apr 2024 11:36:52 -0400 Subject: [PATCH 1/2] use sphinx-design directives instead of sphinx-panels Signed-off-by: cosmicBboy --- .../running_a_workflow_locally.md | 172 ++++++++++++++++++ examples/dbt_plugin/README.md | 2 +- examples/onnx_plugin/README.md | 37 ++-- 3 files changed, 192 insertions(+), 19 deletions(-) create mode 100644 docs/getting_started/running_a_workflow_locally.md diff --git a/docs/getting_started/running_a_workflow_locally.md b/docs/getting_started/running_a_workflow_locally.md new file mode 100644 index 000000000..b07afd09e --- /dev/null +++ b/docs/getting_started/running_a_workflow_locally.md @@ -0,0 +1,172 @@ +--- +jupytext: + formats: md:myst + text_representation: + extension: .md + format_name: myst +--- + +# Running a workflow locally + +You can run a workflow locally in two ways: + +* **{ref}`In a local Python environment `:** To develop and test your code quickly without the overhead of setting up a local Flyte cluster, you can run your workflow in a local Python environment. +* **{ref}`In a local Flyte cluster `:** To test your code in a more production-like setting, you can run your workflow in a local cluster, such as the demo Flyte cluster. + +(getting_started_running_workflow_local_python_environment)= + +## Running a workflow in a local Python environment + +### Prerequisites + +* {doc}`Install development tools ` +* {doc}`Create a Flyte project ` + +### Steps + +1. On the command line, navigate to the workflows directory of your Flyte project: +```{prompt} bash $ +cd my_project/workflows +``` +2. Run the workflow with `pyflyte run`: +```{prompt} bash $ +pyflyte run example.py wf +``` + +:::{note} +While you can run the example file like a Python script with `python example.py`, we recommend using `pyflyte run` instead. To run the file like a Python script, you would have to add a `main` module conditional at the end of the script: +```python +if __name__ == "__main__": + print(wf()) +``` + +Your code would become even more verbose if you wanted to pass arguments to the workflow: +```python +if __name__ == "__main__": + from argparse import ArgumentParser + + parser = ArgumentParser() + parser.add_argument("--name", type=str) + + args = parser.parse_args() + print(wf(name=args.name)) +``` +::: + +(getting_started_running_workflow_local_cluster)= + +## Running a workflow in a local Flyte cluster + +### Prerequisites + +#### 1. Install development tools + +If you have not already done so, follow the steps in {doc}`"Installing development tools" ` to install Python, Flytekit, and optionally, conda. + +#### 2. Create a Flyte project + +If you have not already done so, follow the steps in {doc}`"Creating a Flyte project" ` to create a Flyte project. + +#### 3. Install Docker + +Follow the steps in the [Docker installation guide](https://docs.docker.com/get-docker/) to install Docker. + +Flyte supports any [OCI-compatible](https://opencontainers.org/) container technology (like [Podman](https://podman.io/), [LXD](https://linuxcontainers.org/lxd/introduction/), and [Containerd](https://containerd.io/)), but for the purpose of this documentation, `flytectl` uses Docker to start a local Kubernetes cluster that you can interact with on your machine. + +#### 4. Install `flytectl` + +You must install `flytectl` to start and configure a local Flyte cluster, as well as register workflows to a local or remote Flyte cluster. + +::::{tab-set} + +:::{tab-item} macOS +To use Homebrew, on the command line, run the following: + +```{prompt} bash $ +brew install flyteorg/homebrew-tap/flytectl +``` + +To use `curl`, on the command line, run the following: + +```{prompt} bash $ +curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin +``` + +To download manually, see the [flytectl releases](https://github.com/flyteorg/flytectl/releases). +::: + +:::{tab-item} Linux +To use `curl`, on the command line, run the following: + +```{prompt} bash $ +curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin +``` + +To download manually, see the [flytectl releases](https://github.com/flyteorg/flytectl/releases). +;:: + +:::{tab-item} Windows +To use `curl`, in a Linux shell (such as [WSL](https://learn.microsoft.com/en-us/windows/wsl/install)), on the command line, run the following: + +```{prompt} bash $ +curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin +``` + +To download manually, see the [flytectl releases](https://github.com/flyteorg/flytectl/releases). +::: + +:::: + +### Steps + +1. Export the `FLYTECTL_CONFIG` environment variable in your shell: + +```{prompt} bash $ +export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml +``` +2. Start the Docker daemon. +3. Start the demo cluster: + +```{prompt} bash $ +flytectl demo start +``` +4. Create a project on the demo cluster to correspond to your local Flyte project: +```{prompt} bash $ +flytectl create project \ + --id "my-project" \ + --labels "my-label=my-project" \ + --description "My Flyte project" \ + --name "My project" +``` +3. On the command line, navigate to the workflows directory of your Flyte project: +```{prompt} bash $ +cd my_project/workflows +``` +4. Run the workflow on the Flyte cluster with `pyflyte run` using the `--remote` flag and additional parameters for the project name and domain. In this example, you can also optionally pass a `name` parameter to the workflow: +```{prompt} bash $ +pyflyte run --remote -p my-project -d development example.py wf --name Ada +``` + +You should see a URL to the workflow execution on your demo Flyte cluster, where `` is a unique identifier for the workflow execution: + +```{prompt} bash $ +Go to http://localhost:30080/console/projects/flytesnacks/domains/development/executions/ to see execution in the console. +``` + +### Inspecting a workflow run in the FlyteConsole web interface + +You can inspect the results of a workflow run by navigating to the URL produced by `pyflyte run` for the workflow execution. You should see FlyteConsole, the web interface used to manage Flyte entities such as tasks, workflows, and executions. The default execution view shows the list of tasks executing in sequential order. + +![Landing page of Flyte UI showing two successful tasks run for one workflow execution, along with Nodes, Graph, and Timeline view switcher links](https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/getting_started/flyteconsole_default.png) + +#### Task panel + +Clicking on a single task will open a panel that shows task logs, inputs, outputs, and metadata. + +![Single task panel showing task logs, rerun task button, and executions, inputs, outputs, and task metadata sections](https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/getting_started/flyteconsole_task_panel.png) + +#### Graph view + +The **Graph** view shows the execution graph of the workflow, providing visual information about the topology of the graph and the state of each node as the workflow progresses. + +![Graph view of single workflow execution showing directed acyclic graph of start node, say_hello_node, greeting_length node, and end node](https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/getting_started/flyteconsole_graph_view.png) diff --git a/examples/dbt_plugin/README.md b/examples/dbt_plugin/README.md index 6d6e9b8b5..2acd6060e 100644 --- a/examples/dbt_plugin/README.md +++ b/examples/dbt_plugin/README.md @@ -85,7 +85,7 @@ This image is built using the following `Dockerfile` and contains: - A postgres database. ````{dropdown} See Dockerfile -:title: text-muted +:color: muted This Dockerfile can be found in the ``flytesnacks/examples`` directory under the filepath listed in the code block title below. diff --git a/examples/onnx_plugin/README.md b/examples/onnx_plugin/README.md index 688bc46ea..9a4375e71 100644 --- a/examples/onnx_plugin/README.md +++ b/examples/onnx_plugin/README.md @@ -11,36 +11,37 @@ and deep learning models. It enables interoperability between different framewor The flytekit onnx type plugin comes in three flavors: -```{eval-rst} -.. tabbed:: ScikitLearn - - .. code-block:: +::::{tab-set} - pip install flytekitplugins-onnxpytorch +:::{tab-item} ScikitLearn - This plugin enables the conversion from scikitlearn models to ONNX models. +```{code-block} +pip install flytekitplugins-onnxpytorch ``` -```{eval-rst} -.. tabbed:: TensorFlow - - .. code-block:: +This plugin enables the conversion from scikitlearn models to ONNX models. +::: - pip install flytekitplugins-onnxtensorflow +:::{tab-item} TensorFlow - This plugin enables the conversion from tensorflow models to ONNX models. +```{code-block} +pip install flytekitplugins-onnxtensorflow ``` -```{eval-rst} -.. tabbed:: PyTorch - - .. code-block:: +This plugin enables the conversion from tensorflow models to ONNX models. +::: - pip install flytekitplugins-onnxpytorch +:::{tab-item} PyTorch - This plugin enables the conversion from pytorch models to ONNX models. +```{code-block} +pip install flytekitplugins-onnxpytorch ``` +This plugin enables the conversion from pytorch models to ONNX models. +::: + +:::: + :::{note} If you'd like to add support for a new framework, please create an issue and submit a pull request to the flytekit repo. You can find the ONNX plugin source code [here](https://github.com/flyteorg/flytekit/tree/master/plugins). From a78b15153257e330e8b8a592824e7c40785a91c4 Mon Sep 17 00:00:00 2001 From: cosmicBboy Date: Fri, 19 Apr 2024 14:42:22 -0400 Subject: [PATCH 2/2] delete getting started file Signed-off-by: cosmicBboy --- .../running_a_workflow_locally.md | 172 ------------------ 1 file changed, 172 deletions(-) delete mode 100644 docs/getting_started/running_a_workflow_locally.md diff --git a/docs/getting_started/running_a_workflow_locally.md b/docs/getting_started/running_a_workflow_locally.md deleted file mode 100644 index b07afd09e..000000000 --- a/docs/getting_started/running_a_workflow_locally.md +++ /dev/null @@ -1,172 +0,0 @@ ---- -jupytext: - formats: md:myst - text_representation: - extension: .md - format_name: myst ---- - -# Running a workflow locally - -You can run a workflow locally in two ways: - -* **{ref}`In a local Python environment `:** To develop and test your code quickly without the overhead of setting up a local Flyte cluster, you can run your workflow in a local Python environment. -* **{ref}`In a local Flyte cluster `:** To test your code in a more production-like setting, you can run your workflow in a local cluster, such as the demo Flyte cluster. - -(getting_started_running_workflow_local_python_environment)= - -## Running a workflow in a local Python environment - -### Prerequisites - -* {doc}`Install development tools ` -* {doc}`Create a Flyte project ` - -### Steps - -1. On the command line, navigate to the workflows directory of your Flyte project: -```{prompt} bash $ -cd my_project/workflows -``` -2. Run the workflow with `pyflyte run`: -```{prompt} bash $ -pyflyte run example.py wf -``` - -:::{note} -While you can run the example file like a Python script with `python example.py`, we recommend using `pyflyte run` instead. To run the file like a Python script, you would have to add a `main` module conditional at the end of the script: -```python -if __name__ == "__main__": - print(wf()) -``` - -Your code would become even more verbose if you wanted to pass arguments to the workflow: -```python -if __name__ == "__main__": - from argparse import ArgumentParser - - parser = ArgumentParser() - parser.add_argument("--name", type=str) - - args = parser.parse_args() - print(wf(name=args.name)) -``` -::: - -(getting_started_running_workflow_local_cluster)= - -## Running a workflow in a local Flyte cluster - -### Prerequisites - -#### 1. Install development tools - -If you have not already done so, follow the steps in {doc}`"Installing development tools" ` to install Python, Flytekit, and optionally, conda. - -#### 2. Create a Flyte project - -If you have not already done so, follow the steps in {doc}`"Creating a Flyte project" ` to create a Flyte project. - -#### 3. Install Docker - -Follow the steps in the [Docker installation guide](https://docs.docker.com/get-docker/) to install Docker. - -Flyte supports any [OCI-compatible](https://opencontainers.org/) container technology (like [Podman](https://podman.io/), [LXD](https://linuxcontainers.org/lxd/introduction/), and [Containerd](https://containerd.io/)), but for the purpose of this documentation, `flytectl` uses Docker to start a local Kubernetes cluster that you can interact with on your machine. - -#### 4. Install `flytectl` - -You must install `flytectl` to start and configure a local Flyte cluster, as well as register workflows to a local or remote Flyte cluster. - -::::{tab-set} - -:::{tab-item} macOS -To use Homebrew, on the command line, run the following: - -```{prompt} bash $ -brew install flyteorg/homebrew-tap/flytectl -``` - -To use `curl`, on the command line, run the following: - -```{prompt} bash $ -curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin -``` - -To download manually, see the [flytectl releases](https://github.com/flyteorg/flytectl/releases). -::: - -:::{tab-item} Linux -To use `curl`, on the command line, run the following: - -```{prompt} bash $ -curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin -``` - -To download manually, see the [flytectl releases](https://github.com/flyteorg/flytectl/releases). -;:: - -:::{tab-item} Windows -To use `curl`, in a Linux shell (such as [WSL](https://learn.microsoft.com/en-us/windows/wsl/install)), on the command line, run the following: - -```{prompt} bash $ -curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin -``` - -To download manually, see the [flytectl releases](https://github.com/flyteorg/flytectl/releases). -::: - -:::: - -### Steps - -1. Export the `FLYTECTL_CONFIG` environment variable in your shell: - -```{prompt} bash $ -export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml -``` -2. Start the Docker daemon. -3. Start the demo cluster: - -```{prompt} bash $ -flytectl demo start -``` -4. Create a project on the demo cluster to correspond to your local Flyte project: -```{prompt} bash $ -flytectl create project \ - --id "my-project" \ - --labels "my-label=my-project" \ - --description "My Flyte project" \ - --name "My project" -``` -3. On the command line, navigate to the workflows directory of your Flyte project: -```{prompt} bash $ -cd my_project/workflows -``` -4. Run the workflow on the Flyte cluster with `pyflyte run` using the `--remote` flag and additional parameters for the project name and domain. In this example, you can also optionally pass a `name` parameter to the workflow: -```{prompt} bash $ -pyflyte run --remote -p my-project -d development example.py wf --name Ada -``` - -You should see a URL to the workflow execution on your demo Flyte cluster, where `` is a unique identifier for the workflow execution: - -```{prompt} bash $ -Go to http://localhost:30080/console/projects/flytesnacks/domains/development/executions/ to see execution in the console. -``` - -### Inspecting a workflow run in the FlyteConsole web interface - -You can inspect the results of a workflow run by navigating to the URL produced by `pyflyte run` for the workflow execution. You should see FlyteConsole, the web interface used to manage Flyte entities such as tasks, workflows, and executions. The default execution view shows the list of tasks executing in sequential order. - -![Landing page of Flyte UI showing two successful tasks run for one workflow execution, along with Nodes, Graph, and Timeline view switcher links](https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/getting_started/flyteconsole_default.png) - -#### Task panel - -Clicking on a single task will open a panel that shows task logs, inputs, outputs, and metadata. - -![Single task panel showing task logs, rerun task button, and executions, inputs, outputs, and task metadata sections](https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/getting_started/flyteconsole_task_panel.png) - -#### Graph view - -The **Graph** view shows the execution graph of the workflow, providing visual information about the topology of the graph and the state of each node as the workflow progresses. - -![Graph view of single workflow execution showing directed acyclic graph of start node, say_hello_node, greeting_length node, and end node](https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/getting_started/flyteconsole_graph_view.png)