多卡训练的显存占用

[Moon-404]

Reminder

• I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.2.dev0
• Platform: Linux-5.15.0-126-generic-x86_64-with-glibc2.35
• Python version: 3.10.15
• PyTorch version: 2.5.1 (GPU)
• Transformers version: 4.45.2
• Datasets version: 2.19.1
• Accelerate version: 1.0.1
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA GeForce RTX 4090
• DeepSpeed version: 0.15.4

Reproduction

llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml

Expected behavior

使用 lora 训练时发现多卡并非将模型切分到多卡上训练，而是每个卡上都占用了一个模型并行训练。

llama-factory 是否有将模型切分到多卡上进行训练的功能？

Others

No response

[hiyouga]

https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/README_zh.md#%E4%BD%BF%E7%94%A8-deepspeed-zero-3-%E5%B9%B3%E5%9D%87%E5%88%86%E9%85%8D%E6%98%BE%E5%AD%98