Important
Development is still in progress for several project components. See the notes below for which workflows are best supported.
The shortfin sub-project is SHARK's high performance inference library and serving engine.
- API documentation for shortfin is available on readthedocs.
The SHARK Tank sub-project contains a collection of model recipes and conversion tools to produce inference-optimized programs.
Warning
SHARK Tank is still under development. Experienced users may want to try it out, but we currently recommend most users download pre-exported or pre-compiled model files for serving with shortfin.
- See the SHARK Tank Programming Guide for information about core concepts, the development model, dataset management, and more.
- See Direct Quantization with SHARK Tank for information about quantization support.
The Tuner sub-project assists with tuning program performance by searching for optimal parameter configurations to use during model compilation.
Warning
SHARK Tuner is still in early development. Interested users may want to try it out, but the tuner is not ready for general use yet. Check out the readme for more details.
| Model name | Model recipes | Serving apps |
|---|---|---|
| SDXL | sharktank/sharktank/models/punet/ |
shortfin/python/shortfin_apps/sd/ |
| llama | sharktank/sharktank/models/llama/ |
shortfin/python/shortfin_apps/llm/ |
If you're looking to use SHARK check out our User Guide.
If you're looking to develop SHARK, check out our Developer Guide.