Cannot reproduce the results of Llama7B dora_r32.

[xiaoshingshing2]

First of all, using the official checkpoint is ok. The results on BoolQ is 69.63 while the official result is 69.7.

However, when I try to reproduce the results, I encounter two problems.

The first is about Llama7B dora_r32 without dora_simple. I change three commands in llama_7B_Dora.sh. For example, the micro_batch_size from 16 to 4, learning_rate from 2e-4 to 1e-4, and add --dora_simple False to avoid using dora_simple. I use the command line sh llama_7B_Dora.sh 32 64 ./finetuned_result/dora_r32 0, and the results are

BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
69.3	78.9	78.3	54.3	80.0	82.6	66.1	81.0	73.8

which are worse than the official results.

The second is that when I delete the --dora_simple False to accelerate the training process with dora_simple, the results are even worse:

BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
32.9	75.5	71.8	9.9	41.3	81.9	66.3	75.8	56.9

[xiaoshingshing2]

This is the log of the training process and adapter_config with --dora_simple False
trainer_state.json
adapter_config.json

[nbasyl]

Did you install all the packages following requirements.txt?

[xiaoshingshing2]

Hi, I donot install bitsandbytes and my pytorch version is 2.1.0. The transformers package is installed with pip install transformers==4.36.0. Other packages are the same as requirements.txt.

Will that hurt the performance?

The packages I use are listed below:

Package Version

---

accelerate 0.25.0
aiofiles 23.2.1
aiohttp 3.9.5
aiosignal 1.3.1
altair 5.3.0
annotated-types 0.7.0
anyio 4.4.0
appdirs 1.4.4
asttokens 2.4.1
async-timeout 4.0.3
attrs 23.2.0
black 23.12.0
certifi 2024.6.2
charset-normalizer 3.3.2
click 8.1.7
cmake 3.29.6
contourpy 1.2.1
cycler 0.12.1
datasets 2.15.0
decorator 5.1.1
dill 0.3.7
dnspython 2.6.1
email_validator 2.2.0
exceptiongroup 1.2.1
executing 2.0.1
fastapi 0.111.0
fastapi-cli 0.0.4
ffmpy 0.3.2
filelock 3.15.4
fire 0.5.0
fonttools 4.53.0
frozenlist 1.4.1
fsspec 2023.10.0
gradio 4.9.0
gradio_client 0.7.2
h11 0.14.0
httpcore 1.0.5
httptools 0.6.1
httpx 0.27.0
huggingface-hub 0.23.4
idna 3.7
importlib_resources 6.4.0
ipython 8.25.0
jedi 0.19.1
Jinja2 3.1.4
jsonschema 4.22.0
jsonschema-specifications 2023.12.1
kiwisolver 1.4.5
lit 18.1.8
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.9.0
matplotlib-inline 0.1.7
mdurl 0.1.2
mpmath 1.3.0
multidict 6.0.5
multiprocess 0.70.15
mypy-extensions 1.0.0
networkx 3.3
numpy 1.26.4
orjson 3.10.5
packaging 24.1
pandas 2.2.2
parso 0.8.4
pathspec 0.12.1
pexpect 4.9.0
pillow 10.3.0
pip 24.0
platformdirs 4.2.2
prompt_toolkit 3.0.47
protobuf 5.27.2
psutil 6.0.0
ptyprocess 0.7.0
pure-eval 0.2.2
pyarrow 16.1.0
pyarrow-hotfix 0.6
pydantic 2.7.4
pydantic_core 2.18.4
pydub 0.25.1
Pygments 2.18.0
pyparsing 3.1.2
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.9
pytorch-triton-rocm 2.1.0
pytz 2024.1
PyYAML 6.0.1
referencing 0.35.1
regex 2024.5.15
requests 2.32.3
rich 13.7.1
rpds-py 0.18.1
safetensors 0.4.3
scipy 1.11.4
semantic-version 2.10.0
sentencepiece 0.1.99
setuptools 69.5.1
shellingham 1.5.4
six 1.16.0
sniffio 1.3.1
stack-data 0.6.3
starlette 0.37.2
sympy 1.12.1
termcolor 2.4.0
tokenize-rt 5.2.0
tokenizers 0.15.2
tomli 2.0.1
tomlkit 0.12.0
toolz 0.12.1
torch 2.1.0+rocm5.6
tqdm 4.66.4
traitlets 5.14.3
transformers 4.36.0
typer 0.12.3
typing_extensions 4.12.2
tzdata 2024.1
ujson 5.10.0
urllib3 2.2.2
uvicorn 0.30.1
uvloop 0.19.0
watchfiles 0.22.0
wcwidth 0.2.13
websockets 11.0.3
wheel 0.43.0
xxhash 3.4.1
yarl 1.9.4

I am trying to use exactly the same package as in requirements.txt, and will update my results when the finetuning and testing process finish.

[xiaoshingshing2]

I use exactly the packages in requirements.txt, and the results are:

with dora simple:

BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
69.1	82.8	78.8	86.2	81.0	82.1	66.1	79.2	78.2

without dora simple:

BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
68.7	83.3	79.4	85.5	81.3	80.8	66.0	78.8	78.0

which has 0.4% average accuracy gap between the results reported in readme.

[xiaoshingshing2]

New updates:

I use exactly the packages in requirements.txt, and the results on r=[8,16] still have a large gap with the results reported in readme, while the results on r=[4,64] are better and the result on r=[32] is roughly equal.

Average acc:

r	original	reproduce
4	61.9	65.2
8	77.9	72.5
16	77.5	62.7
32	78.4	78.2
64	76.8	77.9

Is this a normal result?

[zhanqiqi77]

@xiaoshingshing2 I have encountered a similar issue. Did you manage to resolve it? Could you provide your package versions?

[xiaoshingshing2]

@xiaoshingshing2 I have encountered a similar issue. Did you manage to resolve it? Could you provide your package versions?

In the latest update, I used exactly the packages in requirements.txt with the same versions. I still have the problems.