diff --git a/README.md b/README.md index d187c23a0..44d42a8bb 100644 --- a/README.md +++ b/README.md @@ -61,68 +61,11 @@ Model name | Model recipes | Serving apps SDXL | [`sharktank/sharktank/models/punet/`](https://github.com/nod-ai/SHARK-Platform/tree/main/sharktank/sharktank/models/punet) | [`shortfin/python/shortfin_apps/sd/`](https://github.com/nod-ai/SHARK-Platform/tree/main/shortfin/python/shortfin_apps/sd) llama | [`sharktank/sharktank/models/llama/`](https://github.com/nod-ai/SHARK-Platform/tree/main/sharktank/sharktank/models/llama) | [`shortfin/python/shortfin_apps/llm/`](https://github.com/nod-ai/SHARK-Platform/tree/main/shortfin/python/shortfin_apps/llm) -## Development tips -Each sub-project has its own developer guide. If you would like to work across -projects, these instructions should help you get started: +## SHARK Users -### Setup a venv +If you're looking to use SHARK check out our [User Guide](docs/README.md). -We recommend setting up a Python -[virtual environment (venv)](https://docs.python.org/3/library/venv.html). -The project is configured to ignore `.venv` directories, and editors like -VSCode pick them up by default. +## SHARK Developers -```bash -python -m venv .venv -source .venv/bin/activate -``` - -### Install PyTorch for your system - -If no explicit action is taken, the default PyTorch version will be installed. -This will give you a current CUDA-based version, which takes longer to download -and includes other dependencies that SHARK does not require. To install a -different variant, run one of these commands first: - -* *CPU:* - - ```bash - pip install -r pytorch-cpu-requirements.txt - ``` - -* *ROCM:* - - ```bash - pip install -r pytorch-rocm-requirements.txt - ``` - -* *Other:* see instructions at . - -### Install development packages - -```bash -# Install editable local projects. -pip install -r requirements.txt -e sharktank/ shortfin/ - -# Optionally clone and install the latest editable iree-turbine dep in deps/, -# along with nightly versions of iree-base-compiler and iree-base-runtime. -pip install -f https://iree.dev/pip-release-links.html --upgrade --pre \ - iree-base-compiler iree-base-runtime --src deps \ - -e "git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine" -``` - -See also: [`docs/nightly_releases.md`](./docs/nightly_releases.md). - -### Running tests - -```bash -pytest sharktank -pytest shortfin -``` - -### Optional: pre-commits and developer settings - -This project is set up to use the `pre-commit` tooling. To install it in -your local repo, run: `pre-commit install`. After this point, when making -commits locally, hooks will run. See https://pre-commit.com/ +If you're looking to develop SHARK, check out our [Developer Guide](docs/developer.md). \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..858780611 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,118 @@ +# SHARK User Guide + +> [!WARNING] +> This is still pre-release so the artifacts listed here may be broken +> + +These instructions cover the usage of the latest stable release of SHARK. For a more bleeding edge release please install the [nightly releases](nightly_releases.md). + +## Prerequisites + +Our current user guide requires that you have: +- Access to a computer with an installed AMD Instinct™ MI300x Series Accelerator +- Installed a compatible version of Linux and ROCm on the computer (see the [ROCm compatability matrix](https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html)) + + +## Set up Environment + +You will need a recent version of Python. We recommend also setting up a Python environment. + +Officially we support Python versions: 3.11, 3.12, 3.13, 3.13t + +The rest of this guide assumes you are using Python 3.11. + +### Install Python +To install Python 3.11 on Ubuntu: + +```bash +sudo apt install python3.11 python3.11-dev python3.11-venv + +which python3.11 +# /usr/bin/python3.11 +``` + +### Create a Python Environment + +This guide assumes you'll be using pyenv. Setup your pyenv with the following commands: + +```bash +# Set up a virtual environment to isolate packages from other envs. +python3.11 -m venv 3.11.venv +source 3.11.venv/bin/activate +``` + +## Install SHARK and its dependencies + +```bash +pip install transformers +pip install dataclasses-json +pip install pillow +pip install shark-ai +``` + +Temporarily, you may need an update to your `shortfin` install. +Install the latest pre-release with: +``` +pip install shortfin --upgrade --pre -f https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels +``` + +### Test the installation. + +``` +python -m shortfin_apps.sd.server --help +``` + +## Quickstart + +### Run the SD Server + +Run the [SD Server](../shortfin/python/shortfin_apps/sd/README.md#Start SD Server) + +### Run the SD Client + +``` +python -m shortfin_apps.sd.simple_client --interactive +``` + +Congratulations!!! At this point you can play around with the server and client based on your usage. + +### Update flags + +Please see --help for both the server and client for usage instructions. Here's a quick snapshot. + +#### Update server options: + +| Flags | options | +|---|---| +|--host HOST | +|--port PORT | server port | +|--root-path ROOT_PATH | +|--timeout-keep-alive | +|--device | local-task,hip,amdgpu | amdgpu only supported in this release +|--target | gfx942,gfx1100 | gfx942 only supported in this release +|--device_ids | +|--tokenizers | +|--model_config | +| --workers_per_device | +| --fibers_per_device | +| --isolation | per_fiber, per_call, none | +| --show_progress | +| --trace_execution | +| --amdgpu_async_allocations | +| --splat | +| --build_preference | compile,precompiled | +| --compile_flags | +| --flagfile FLAGFILE | +| --artifacts_dir ARTIFACTS_DIR | Where to store cached artifacts from the Cloud | + +#### Update client with different options: + +| Flags |options| +|---|--- +|--file | +|--reps | +|--save | Whether to save image generated by the server | +|--outputdir| output directory to store images generated by SDXL | +|--steps | +|--interactive | +|--port| port to interact with server | \ No newline at end of file diff --git a/docs/developer.md b/docs/developer.md new file mode 100644 index 000000000..832466688 --- /dev/null +++ b/docs/developer.md @@ -0,0 +1,65 @@ +# SHARK Developer Guide + +Each sub-project has its own developer guide. If you would like to work across +projects, these instructions should help you get started: + +### Setup a venv + +We recommend setting up a Python +[virtual environment (venv)](https://docs.python.org/3/library/venv.html). +The project is configured to ignore `.venv` directories, and editors like +VSCode pick them up by default. + +```bash +python -m venv .venv +source .venv/bin/activate +``` + +### Install PyTorch for your system + +If no explicit action is taken, the default PyTorch version will be installed. +This will give you a current CUDA-based version, which takes longer to download +and includes other dependencies that SHARK does not require. To install a +different variant, run one of these commands first: + +* *CPU:* + + ```bash + pip install -r pytorch-cpu-requirements.txt + ``` + +* *ROCM:* + + ```bash + pip install -r pytorch-rocm-requirements.txt + ``` + +* *Other:* see instructions at . + +### Install development packages + +```bash +# Install editable local projects. +pip install -r requirements.txt -e sharktank/ shortfin/ + +# Optionally clone and install the latest editable iree-turbine dep in deps/, +# along with nightly versions of iree-base-compiler and iree-base-runtime. +pip install -f https://iree.dev/pip-release-links.html --upgrade --pre \ + iree-base-compiler iree-base-runtime --src deps \ + -e "git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine" +``` + +See also: [nightly_releases.md](nightly_releases.md). + +### Running tests + +```bash +pytest sharktank +pytest shortfin +``` + +### Optional: pre-commits and developer settings + +This project is set up to use the `pre-commit` tooling. To install it in +your local repo, run: `pre-commit install`. After this point, when making +commits locally, hooks will run. See https://pre-commit.com/ diff --git a/shortfin/python/shortfin_apps/sd/README.md b/shortfin/python/shortfin_apps/sd/README.md index 0bb5db511..5ade5d27e 100644 --- a/shortfin/python/shortfin_apps/sd/README.md +++ b/shortfin/python/shortfin_apps/sd/README.md @@ -3,28 +3,12 @@ This directory contains a SD inference server, CLI and support components. -## Quick start +## Install -In your shortfin environment, -``` -pip install transformers -pip install dataclasses-json -pip install pillow -pip install shark-ai - -``` - -Temporarily, you may need an update to your `shortfin` install. -Install the latest pre-release with: -``` -pip install shortfin --upgrade --pre -f https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels -``` +For [nightly releases](../../../../docs/nightly_releases.md) +For our [stable release](../../../../docs/README.md) -``` -python -m shortfin_apps.sd.server --help -``` - -# Run on MI300x +## Start SD Servier The server will prepare runtime artifacts for you. By default, the port is set to 8000. If you would like to change this, use `--port` in each of the following commands. @@ -39,6 +23,8 @@ python -m shortfin_apps.sd.server --device=amdgpu --device_ids=0 --build_prefere INFO - Application startup complete. INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) ``` +## Run the SD Client + - Run a CLI client in a separate shell: ``` python -m shortfin_apps.sd.simple_client --interactive