v0.9.0-alpha5 - text embed cache data backend configuration

[bghira]

Breaking changes

Your text embed cache location now must be specified in the multidatabackend.json configuration file. There is an example in the repository. The current text embed cache directory can be used, it should work as it did before.

Text embeds can now be stored on S3 buckets, leaving next to zero local storage in use for training other than model checkpoints.

What's Changed

• torch compile fixes
• DeepSpeed save state fixes by @bghira in torch compile fixes | DeepSpeed save state fixes #263
• updates for next release by @bghira in updates for next release #264
• collate_fn: multi-threaded retrieval of SDXL text embeds by @bghira in collate_fn: multi-threaded retrieval of SDXL text embeds #265
• text embedding cache should write embeds in parallel by @bghira in text embedding cache should write embeds in parallel #266
• text embedding cache should stop writing and kill the thread when we finish by @bghira in text embedding cache should stop writing and kill the thread when we finish #267
• text embedding cache: optimise the generation of embeds by @bghira in text embedding cache: optimise the generation of embeds  #268
• multiple text embed caches
• cache the text embed lists and only process meaningful prompts by @bghira in multiple text embed caches | cache the text embed lists and only process meaningful prompts #269

Full Changelog: v0.9.0-alpha4...v0.9.0-alpha5

---

This discussion was created from the release v0.9.0-alpha5 - text embed cache data backend configuration.