🌰 SEED Multimodal

Powered by CV Center, Tencent AI Lab, and ARC Lab, Tencent PCG.

The repository provides the official implementation of SEED, SEED-LLaMA. For any inquiries, please email [email protected].

News

🍻 We are actively looking for self-motivated interns. Please feel free to reach out if you are interested. 🍻

• 2024-04-27 🤗 We have released an upgraded version SEED-X based on continuous visual embeddings, which supports multi-granularity comprehension and generation.

• 2024-02-24 🤗 We have released the training code of SEED-LLaMa, including SEED tokenizer, Multimodal LLM pretraining and instruction tuning. Our Multimodal LLM training codebase supports 1. large-scale multi-node training with deepspeed 2. highly-efficient multiple training datapipes.

• 2024-01-16 🤩 Our SEED-LLaMA (arXiv) has been accepted by ICLR 2024 (OpenReview), see you in Vienna!

• 2023-11-03 🤗 We have released the demo of seed-llama-v2-1, and the quality of generated images has been greatly improved, feel free to use it by yourself.

  

• 2023-10-23 🤗 We have optimized the memory overhead. Through 8bit quantization and dynamic loading, SEED-LLaMA 8b/14B can run on single 16GB/24GB GPU.

• 2023-10-23 🤗 All model weights will be downloaded automatically when starting the demo.

• 2023-10-20 🤗 We release the checkpoints and code of the SEED-2 tokenizer, and SEED-LLaMA-8B/14B.

• 2023-10-20 👾 We release an online gradio demo, feel free to use it by yourself.

• 2023-10-02 📎 We release the technical report of SEED-LLaMA on arXiv, which is empowered by the improved SEED-2 tokenizer.

• 2023-07-29 We release the checkpoint of the SEED tokenizer and its inference code. Check it out via SEED-1.

• 2023-07-16 📎 We release the technical report of SEED on arXiv.

Stay tuned for the updates!

Brief Introduction

It is recommended to check out our papers for technical details.

💬 What can SEED-LLaMA do?

SEED-LLaMA is capable of both multimodal comprehension and generation, exhibiting compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant. [Compare to SOTA] [More examples on X]

💡 How does SEED-LLaMA achieve it?

The core of SEED-LLaMA is the tailored SEED tokenizer, which properly quantized visual signals into discrete visual tokens, capturing necessary semantics while being produced under 1D causal dependence. [SEED-2 vs. SEED-1]

Hightlights

1. We use GPT-4 to rewrite the instructions in the InstructPix2Pix dataset, such as transforming "add a storm" into "Can you add a storm effect to the image?" and responding with "Sure, I have successfully added a storm effect to the image.". The instruction tuned model can generate informative text and images in a single response, as shown in the figure below (this is also an emergent ability).

2. Given a starting image and story, the instruction tuned model can generate the following story and multiple images in one go.

3. We use GPT-4 to generate instruction based on the text content of MMC4. The instruction tuned model can generate image-text interleaved content (Our released sft model does not possess this feature as we separately instruction tune the pre-trained model on MMC4).

Usage

Dependencies

• Python >= 3.8 (Recommend to use Anaconda)
• PyTorch >= 1.11.0
• NVIDIA GPU + CUDA

Installation

Clone the repo and install dependent packages

git clone https://github.com/AILab-CVC/SEED.git
cd SEED
pip install -r requirements.txt

Model Weights

We release the pretrained SEED Tokenizer and De-Tokenizer, pretrained and instruction tuned SEED-LLaMA-8B and SEED-LLaMA-14B as below,

• Check the SEED tokenizer weights in AILab-CVC/seed-tokenizer-2
• Check the SEED LLaMA(pre-trained 8B) weights in AILab-CVC/seed-llama-8b-pretrain
• Check the SEED LLaMA(sft 8B) weights in AILab-CVC/seed-llama-8b-sft
• Check the SEED LLaMA(pre-trained 14B) weights in AILab-CVC/seed-llama-14b-pretrain
• Check the SEED LLaMA(sft 14B) weights in AILab-CVC/seed-llama-14b-sft

The model weights of unCLIP SD-UNet which are used to reconstruct the image will be downloaded automatically.

Inference for visual tokenization and de-tokenization

To discretize an image to 1D visual codes with causal dependency, and reconstruct the image from the visual codes using the off-the-shelf unCLIP SD-UNet:

cd ..   # SEED/ 
python scripts/seed_tokenizer_inference.py

Inference for SEED-LLaMA

Given that SEED-LLaMA-8B is based on Vicuna-7B and SEED-LLaMA-14B based on LLaMA2-Chat-13B, we use Vicuna-7B's ("USER:", "ASSISTANT:") and LLaMA2-Chat-13B's ([INST] [/INST]) prompts for respective instruction tuning.

# Inference for SEED-LLaMA-8B
python scripts/seed_llama_inference_8B.py

# Inference for SEED-LLaMA-14B
python scripts/seed_llama_inference_14B.py

Launching Gradio Demo of SEED-LLaMA-14B Locally

1. Building the local demo of SEED-LLaMA-14B currently requires single 24GB GPU.

# SEED/
# in first terminal
bash scripts/start_backend_14b.sh
# in second terminal
bash scripts/start_frontend_14b.sh

2. Building the local demo of SEED-LLaMA-8B currently requires single 16GB GPU.

# SEED/
# in first terminal
bash scripts/start_backend_8b.sh
# in second terminal
bash scripts/start_frontend_8b.sh

Then the demo can be accessed through http://127.0.0.1:80

Training SEED-LLaMA

Training SEED Tokenization based on LAVIS

1. Installation

cd SEED/SEED_Tokenizer
sh install.sh

2. Download pre-trained Q-Former from BLIP-2 and put the checkpoint under the folder "pretrained".

3. Training Causal Q-Former

sh train_scripts/causal_qformer.sh

4. Download CLIP for unCLIP-SD and put the checkpoint under the folder "pretrained".

5. Training SEED Tokenizer and De-Tokenizer

sh train_scripts/codebook.sh

6. After training, you can tokenize an image into discrete tokens and decode the discrete tokens into a realistic image via unclip SD

# You need to load the pre-trained ckpt.
python3 eval/seed_inference.py

Multimodal LLM Pre-training and instruction truning

1. Installation

cd SEED
pip install -r requirements.txt
cd MultiModalLLM

2. Download the pre-trained LLM (for example, vicuna-7b-v1.1) and SEED Tokenizer, and put them under the folder "pretrained".

3. Pre-process the pre-training data by converting images into discrete tokens. For example,

python3 src/tools/extract_image_ids_to_torchdata_parallel.py \
  --tokenizer configs/tokenizer/seed_llama_tokenizer.yaml \
  --image_transform configs/processer/blip_transform.yaml \
  --data configs/data/caption_torchdata_preprocess.yaml \
  --save_dir dataset/seed_llama/caption/unsplash_cc3m/ \
  --batch_size 1024 --num_workers 8 --gpus 8

4. Pre-training Multimodal LLM with SEED tokens using lora.

sh scripts/train_a100_lora_multi_node_pretrain.sh

5. Merge the lora checkpoint with the original LLM.

python3 src/tools/merge_lora_weights.py \
  --model_cfg configs/model/vicuna_7b_lora_pretrained.yaml \
  --tokenizer_cfg configs/tokenizer/seed_llama_tokenizer.yaml \ 
  --base_model pretrained/vicuna-7b-v1.1 \
  --lora_model log/seed_vicuna-7b_lora_pretrain/checkpoint-10000 \
  --save_path log/seed_vicuna-7b_lora_pretrain/checkpoint-merged-10000 

6. Pre-process the instruction tuning data by converting images into discrete tokens. (You first need to convert the data into JSON format, with each line of the JSON containing "image" (the path of the image), "question", and "answer".)

python3 src/tools/extract_image_ids_to_torchdata_parallel_qa.py \
  --tokenizer configs/tokenizer/seed_llama_tokenizer.yaml \
  --image_transform configs/processer/blip_transform.yaml \
  --data configs/data/question_answer_torchdata_eval.yaml \
  --save_dir  data/VQAv2 \
  --batch_size 512 --num_workers 8 --gpus 8

7. Instruction tuning Multimodal LLM with SEED tokens using lora.

sh scripts/train_a100_lora_multi_node_sft.sh

Citation

If you find the work helpful, please consider citing:

@article{ge2023making,
  title={Making LLaMA SEE and Draw with SEED Tokenizer},
  author={Ge, Yuying and Zhao, Sijie and Zeng, Ziyun and Ge, Yixiao and Li, Chen and Wang, Xintao and Shan, Ying},
  journal={arXiv preprint arXiv:2310.01218},
  year={2023}
}

@article{ge2023planting,
  title={Planting a seed of vision in large language model},
  author={Ge, Yuying and Ge, Yixiao and Zeng, Ziyun and Wang, Xintao and Shan, Ying},
  journal={arXiv preprint arXiv:2307.08041},
  year={2023}
}

The project is still in progress.

License

SEED is released under Apache License Version 2.0.

SEED-LLaMA is released under the original License of LLaMA2.

Acknowledgement

We thank the great work from unCLIP SD and BLIP2.