Learning to Use Tools For Creating Multimodal Agents.
[Project Page] [Arxiv] [Demo] [Data] [Model Zoo]
Note: Some sections of the code are currently being prepared and updated. Please stay tuned.
- [11/11] π₯ We released LLaVA-Plus: Large Language and Vision Assistants that Plug and Learn to Use Skills. Enable LMM to use tools for general vision tasks! Checkout the paper and demo.
If you are not using Linux, do NOT proceed, see instructions for macOS and Windows from LLaVA.
- Clone this repository and navigate to the LLaVA-Plus folder
git clone https://github.com/LLaVA-VL/LLaVA-Plus-Codebase LLaVA-Plus
cd LLaVA-Plus- Install Package
conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip # enable PEP 660 support
pip install -e .- Install additional packages for training cases
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
We are still preparing the part. Stay tuned!
Please check out our Model Zoo for all public LLaVA-Plus checkpoints, and the instructions on how to use the weights.
To run our demo, you have four steps.
- Launch a controller: enable to control different works.
- Launch a model worker: core llava-plus model.
- Launch tool workers: the tools you want to call.
- Launch a gradio web server: a front end page for users.
python -m llava.serve.controller --host 0.0.0.0 --port 20001This is the actual worker that performs the inference on the GPU. Each worker is responsible for a single model specified in --model-path.
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:20001 --port 40000 --worker http://localhost:40000 --model-path <huggingface or local path>Wait until the process finishes loading the model and you see "Uvicorn running on ...". Now, refresh your Gradio web UI, and you will see the model you just launched in the model list.
If you are using an Apple device with an M1 or M2 chip, you can specify the mps device by using the --device flag: --device mps.
Multiple works
You can launch as many workers as you want, and compare between different model checkpoints in the same Gradio interface. Please keep the `--controller` the same, and modify the `--port` and `--worker` to a different port number for each worker. ```Shell python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:20001 --port --worker http://localhost: --model-path ```Launch a model worker (Multiple GPUs, when GPU VRAM <= 24GB)
If the VRAM of your GPU is less than 24GB (e.g., RTX 3090, RTX 4090, etc.), you may try running it with multiple GPUs. Our latest code base will automatically try to use multiple GPUs if you have more than one GPU. You can specify which GPUs to use with CUDA_VISIBLE_DEVICES. Below is an example of running with the first two GPUs.
CUDA_VISIBLE_DEVICES=0,1 python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:20001 --port 40000 --worker http://localhost:40000 --model-path <huggingface or local path>You need to open different tool works, as shown in the figure above, which means you need to prepare codes from other projects.
We provide a detailed guideline for different projects.
python -m llava.serve.gradio_web_server_llava_plus --controller http://localhost:20001 --model-list-mode reloadYou just launched the Gradio web interface. Now, you can open the web interface with the URL printed on the screen.
LLaVA training consists of two stages: (1) feature alignment stage, and (2) visual instruction tuning stage.
Our llava-plus is trained from the llava-stage-1-pre-trained projectors.
Training cost
LLaVA-Plus is trained on 4/8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the `per_device_train_batch_size` and increase the `gradient_accumulation_steps` accordingly. Always keep the global batch size the same: `per_device_train_batch_size` x `gradient_accumulation_steps` x `num_gpus`.Download Vicuna checkpoints (automatically)
Our base model Vicuna v1.5, which is an instruction-tuned chatbot, will be downloaded automatically when you run our provided training scripts. No action is needed.
Download pre-trained projector directly as we did.
Or you may train the projector following the guideline in LLaVA.
- Prepare data
Please download the training data, and download the images from constituting datasets:
- Start training!
Training script with DeepSpeed ZeRO-2: training_llava_plus_v0_7b.sh or training_llava_plus_v1.3_7b.sh
If you do not have enough GPU memory:
- Use LoRA. See LLaVA repo for more details. - Replace `zero2.json` with `zero3.json` or `zero3_offload.json`.If you are interested in finetuning LLaVA(LLaVA-Plus) model to your own task/data:
please check out [`Finetune_Custom_Data.md`](https://github.com/haotian-liu/LLaVA/blob/main/docs/Finetune_Custom_Data.md)γSome explainations of options:
--data_path path/to/llava-150k-tool-aug.json,path/to/llava-plus-v1-117k-tool-merge.json: You may pass multiple data files with,separated.--image_folder /path/to/coco/train2017/,/path/to/hiertext/train,/path/to/infoseek/infoseek_images,/path/to/instruct-pix2pix/clip-filtered-dataset,/path/to/goldg/vg_mdetr/images: You may pass multiple image folders with,separated. Note that it may cause problems if multiple folders have images with the same name.--mm_projector_type mlp2x_gelu: the two-layer MLP vision-language connector.--vision_tower openai/clip-vit-large-patch14-336: CLIP ViT-L/14 336px.--image_aspect_ratio pad: this pads the non-square images to square, instead of cropping them; it slightly reduces hallucination.--group_by_modality_length True: this should only be used when your instruction tuning dataset contains both language (e.g. ShareGPT) and multimodal (e.g. LLaVA-Instruct). It makes the training sampler only sample a single modality (either image or language) during training, which we observe to speed up training by ~25%, and does not affect the final outcome.
See LLaVA's Instruction on model evaluations.
If you find LLaVA useful for your research and applications, please cite using this BibTeX:
@article{liu2023llavaplus,
title={LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents},
author={Liu, Shilong and Cheng, Hao and Liu, Haotian and Zhang, Hao and Li, Feng and Ren, Tianhe and Zou, Xueyan and Yang, Jianwei and Su, Hang and Zhu, Jun and Zhang, Lei and Gao, Jianfeng and Li, Chunyuan},
journal={arXiv:2311.05437},
year={2023}
}
@misc{liu2023llava,
title={Visual Instruction Tuning},
author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
publisher={arXiv:2304.08485},
year={2023}
}Included Tools
- Grounding DINO
- Grounded-Segment-Anything
- Recognize Anythging
- Segment-Anything.
- SEEM: Segment Everything Everywhere All at Once
- Semantic-SAM
- Segment-Anything
For future project ideas, please check out:
