reproduce lora

[PancakeCookie]

There is a difference in the results of the two lora with completely identical settings, just changing the output folder name.
(95% similar, but different details)
I fixed the seed and also turned off the shuffle caption option.
Is it impossible to reproduce lora perfectly? Or is there some randomness that I'm missing?

[Serallan]

Did most of my training in a nearly year old Kohya version, so I can't say this didn't change in newer ones, but I didn't experienced this issue then. I've recreated some loras a couple of times without any changes in them, and I did change output folders between sessions.

Can you post your full settings?

Besides Kohya version, one thing that definitively changes the training output is torch/xformers/cuda version, so you need to ensure you're using the same Kohya and packages versions to reproduce a lora.

[PancakeCookie]

To test lora reproducibility, I experimented by simply changing the output folder on one PC(so, torch/xformers/cuda is exactly same).
My kohya_ss version is
Kohya_ss GUI version: v23.0.5
Torch 2.1.2+cu118
Torch backend: nVidia CUDA 11.8 cuDNN 8700
Torch detected GPU: NVIDIA GeForce RTX 4090 VRAM 24564 Arch (8, 9) Cores 128
Python version is 3.10.11
(bf16 install option can be problem?)

and training setting is
{
"LoRA_type": "Standard",
"LyCORIS_preset": "full",
"adaptive_noise_scale": 0,
"additional_parameters": "",
"block_alphas": "",
"block_dims": "",
"block_lr_zero_threshold": "",
"bucket_no_upscale": false,
"bucket_reso_steps": 64,
"cache_latents": true,
"cache_latents_to_disk": false,
"caption_dropout_every_n_epochs": 0.0,
"caption_dropout_rate": 0,
"caption_extension": ".txt",
"clip_skip": 2,
"color_aug": false,
"constrain": 0.0,
"conv_alpha": 1,
"conv_block_alphas": "1",
"conv_block_dims": "8",
"conv_dim": 1,
"debiased_estimation_loss": false,
"decompose_both": false,
"dim_from_weights": false,
"down_lr_weight": "",
"enable_bucket": true,
"epoch": 5,
"factor": -1,
"flip_aug": false,
"fp8_base": false,
"full_bf16": false,
"full_fp16": false,
"gpu_ids": "",
"gradient_accumulation_steps": "1",
"gradient_checkpointing": false,
"keep_tokens": 1,
"learning_rate": 0.0001,
"logging_dir": "",
"lora_network_weights": "",
"lr_scheduler": "cosine_with_restarts",
"lr_scheduler_args": "",
"lr_scheduler_num_cycles": "",
"lr_scheduler_power": "",
"lr_warmup": 10,
"max_bucket_reso": 1280,
"max_data_loader_n_workers": "0",
"max_grad_norm": 1,
"max_resolution": "512,512",
"max_timestep": 1000,
"max_token_length": "75",
"max_train_epochs": "",
"max_train_steps": "",
"mem_eff_attn": false,
"mid_lr_weight": "",
"min_bucket_reso": 320,
"min_snr_gamma": 0,
"min_timestep": 0,
"mixed_precision": "fp16",
"model_list": "custom",
"module_dropout": 0,
"multi_gpu": false,
"multires_noise_discount": 0,
"multires_noise_iterations": 0,
"network_alpha": 8,
"network_dim": 64,
"network_dropout": 0,
"noise_offset": 0,
"noise_offset_type": "Original",
"num_cpu_threads_per_process": 2,
"num_machines": 1,
"num_processes": 1,
"optimizer": "AdamW8bit",
"optimizer_args": "",
"output_dir": "C:/webui/stable-diffusion-webui/models/Lora/lora_reproduce2",
"output_name": "lora_reproduce_2",
"persistent_data_loader_workers": false,
"pretrained_model_name_or_path": "C:/webui/stable-diffusion-webui/models/Stable-diffusion/basemodel/flat2danime.ckpt",
"prior_loss_weight": 1.0,
"random_crop": false,
"rank_dropout": 0,
"rank_dropout_scale": false,
"reg_data_dir": "",
"rescaled": false,
"resume": "",
"sample_every_n_epochs": 0,
"sample_every_n_steps": 0,
"sample_prompts": "",
"sample_sampler": "euler_a",
"save_every_n_epochs": 1,
"save_every_n_steps": 0,
"save_last_n_steps": 0,
"save_last_n_steps_state": 0,
"save_model_as": "safetensors",
"save_precision": "fp16",
"save_state": false,
"scale_v_pred_loss_like_noise_pred": false,
"scale_weight_norms": 0,
"sdxl": false,
"sdxl_cache_text_encoder_outputs": false,
"sdxl_no_half_vae": true,
"seed": "42",
"shuffle_caption": true,
"stop_text_encoder_training": 0,
"text_encoder_lr": 5e-05,
"train_batch_size": 2,
"train_data_dir": "C:/Users//Desktop/lora/lora_reproduce/img",
"train_norm": false,
"train_on_input": true,
"training_comment": "",
"unet_lr": 0.0005,
"unit": 1,
"up_lr_weight": "",
"use_cp": false,
"use_scalar": false,
"use_tucker": false,
"use_wandb": false,
"v2": false,
"v_parameterization": false,
"v_pred_like_loss": 0,
"vae": "",
"vae_batch_size": 0,
"wandb_api_key": "",
"weighted_captions": false,
"xformers": "xformers"
}

[Serallan]

I was wrong, there is a change in my files. I didn't notice because it's very subtle in earlier epochs, which I'm using, and only shows up in some seeds. I used some spare time I had on a jupyter notebook to run some tests, but couldn't try everything I'd like so you'll have test more to confirm.

I saw changes in many situations, not only with different output directories but filename (A1111 reports a different hash even when you just rename a lora, changing results as consequence). It's not the bf16, I'm not using it. I'd like to try with xformers off, but the notebook can't handle my dataset then.

TL;DR, I saw changes caused by two factors:

1. The ones leading to a change the hash, like a different file name. Counterintuitive but makes sense.
2. Different output directories. Doesn't make sense, looks like a bug to me.

I wonder if we'd also see changes with same dataset but in different dir path, etc.

[PancakeCookie]

I just change this section, everything else is the same, including the dataset
So I created two lora with completely identical output folder paths.
After that, the model hash values of both loras became the same.

We can see two icons in webui, all items from the first i button are match.
And in the next tool button, all items are the same except for the lora creation time.
However, there are still subtle differences in the results of the two loras!!!
I just can't figure it out. There must be randomness somewhere. (Dataloader?)