CV-VAE: A Compatible Video VAE for Latent Generative Video Models

Sijie Zhao · Yong Zhang* · Xiaodong Cun · Shaoshu Yang · Muyao Niu

Xiaoyu Li · Wenbo Hu · Ying Shan

^*Corresponding Authors

TL; DR: A video VAE for latent generative video models, which is compatible with pretrained image and video models, e.g., SD 2.1 and SVD

News

• 2024-10-14 🤗 We have updated the training code of CV-VAE.

• 2024-10-14 We have released the inference code and model weights of CV-VAE-SD3 which is compatible with SD3 and SD3.5.

• 2024-10-14 We have updated the CV-VAE with better performance, please check cv-vae-v1-1.

• 2024-09-25 CV-VAE is accepted by NeurIPS 2024.

• 2024-06-03 We have released the inference code and model weights of CV-VAE.

• 2024-05-30 We have updated the arXiv preprint.

Usage

Dependencies

• Python >= 3.8 (Recommend to use Anaconda)
• PyTorch >= 1.13.0
• NVIDIA GPU + CUDA

Video reconstruction

Download the model weight from Hugging Face

python3 cvvae_inference_video.py \
  --vae_path MODEL_PATH \
  --video_path INPUT_VIDEO_PATH \
  --save_path VIDEO_SAVE_PATH \
  --height HEIGHT \
  --width WIDTH 

😉 Citation

@article{zhao2024cvvae,
  title={CV-VAE: A Compatible Video VAE for Latent Generative Video Models},
  author={Zhao, Sijie and Zhang, Yong and Cun, Xiaodong and Yang, Shaoshu and Niu, Muyao and Li, Xiaoyu and Hu, Wenbo and Shan, Ying},
  journal={https://arxiv.org/abs/2405.20279},
  year={2024}
}