Could you describe the procedure of reproduce the GeoChat?

[Amazingren]

Dear @salman-h-khan ,

Thanks for your fantastic work GeoChat, I am really interested in it. And the ckpt provided by you works for me.

However, when I tried to reproduce it as a beginner of the LLMs. It turns out a bit confusing for me to conduct all the training/finturning step by step.

Could you please specify where am I wrong when regarding what I did:

• Step1: I prepared all the datasets and set the finetune_lora.sh as follows and run it

################## VICUNA ##################
PROMPT_VERSION=v1
MODEL_VERSION="vicuna-v1.5-7b"
gpu_ids=0,1,2,3
################## VICUNA ##################

 deepspeed --master_port=$((RANDOM + 10000)) --include localhost:$gpu_ids geochat/train/train_mem.py \
    --deepspeed ./scripts/zero2.json \
    --lora_enable True \
    --model_name_or_path /data/.../geochat/llava-v1.5-7b \
    --version $PROMPT_VERSION \
    --data_path /data/.../geochat/GeoChat_Instruct.json \
    --image_folder /data/.../geochat/final_images_llava  \
    --vision_tower openai/clip-vit-large-patch14-336 \
    --mm_projector_type mlp2x_gelu \
    --pretrain_mm_mlp_adapter /data/.../geochat/llava-v1.5-mlp2x-336px-pretrain-vicuna-7b-v1.5/mm_projector.bin \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --image_aspect_ratio pad \
    --bf16 True \
    --output_dir /data/.../outckpts/geochat_reproduce \
    --num_train_epochs 1 \
    --per_device_train_batch_size 18 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 2 \
    --evaluation_strategy "no" \
    --save_strategy "epoch" \
    --save_steps 7000 \
    --save_total_limit 1 \
    --learning_rate 2e-4 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --dataloader_num_workers 16 \
    --report_to wandb

then I got the following output folder geochat_reproduce with the following files:

• Step 2: I try to merge the ckpts achieved in step1, with the original llava-v1.5-7b ckpts as follows:

python scripts/merge_lora_weights.py \
    --model-path /data/.../geochat/outckpts/geochat_reproduce \
    --model-base /data/.../geochat/llava-v1.5-7b \
    --save-model-path /data/.../geochat/outckpts/merged

• Step 3: I tried to run the demo with the reproduced ckpts.

python geochat_demo.py \
    --model-path /data/.../geochat/outckpts/merged

It then turns out lots of errors as follows:

To this end, I would like to ask if there are some mistakes in my reproduction or if some other steps are missing.

It would be super nice to receive some guidance from you.

Best regards and have a nice day,

[krishna-exe]

hey, were you able to run the demo successfully ?