WARNING: This is an early preview that is in progress. It is not ready for general use.
The SHARK Tank sub-project contains a collection of model recipes and conversion tools to produce inference-optimized programs.
- See the SHARK Tank Programming Guide for information about core concepts, the development model, dataset management, and more.
- See Direct Quantization with SHARK Tank for information about quantization support.
The shortfin sub-project is SHARK's high performance inference library and serving engine.
- API documentation for shortfin is available on readthedocs.
The Tuner sub-project assists with tuning program performance by searching for optimal parameter configurations to use during model compilation.
| Model name | Model recipes | Serving apps |
|---|---|---|
| SDXL | sharktank/sharktank/models/punet/ |
shortfin/python/shortfin_apps/sd/ |
| llama | sharktank/sharktank/models/llama/ |
shortfin/python/shortfin_apps/llm/ |
Use this as a guide to get started developing the project using pinned, pre-release dependencies. You are welcome to deviate as you see fit, but these canonical directions mirror what the CI does.
We recommend setting up a virtual environment (venv). The project is configured
to ignore .venv directories, and editors like VSCode pick them up by default.
python -m venv --prompt sharktank .venv
source .venv/bin/activate
If no explicit action is taken, the default PyTorch version will be installed. This will give you a current CUDA-based version. Install a different variant by doing so explicitly first:
CPU:
pip install -r pytorch-cpu-requirements.txt
ROCM:
pip install -r pytorch-rocm-requirements.txt
# Clone and install editable iree-turbine dep in deps/
pip install -f https://iree.dev/pip-release-links.html --src deps \
-e "git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine"
# Install editable local projects.
pip install -r requirements.txt -e sharktank/ shortfin/
pytest sharktank
pytest shortfin
This project is set up to use the pre-commit tooling. To install it in
your local repo, run: pre-commit install. After this point, when making
commits locally, hooks will run. See https://pre-commit.com/