can resume loras

[kohler19]

i dont know how.. im newbie.. when resume from check point ,( 2024-03-13 20:12:34 INFO resume training from local state: C:\Users\cheap\Desktop\Train train_util.py:3510
data\For_Test\Train\model\Siti_Khatijah_Halim.safetensors
Traceback (most recent call last):
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\sd-scripts\sdxl_train_network.py", line 184, in
trainer.train(args)
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\sd-scripts\train_network.py", line 460, in train
train_util.resume_from_local_or_hf_if_specified(accelerator, args)
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\sd-scripts\library\train_util.py", line 3511, in resume_from_local_or_hf_if_specified
accelerator.load_state(args.resume)
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 2790, in load_state
raise ValueError(f"Tried to find {input_dir} but folder does not exist")
ValueError: Tried to find C:\Users\cheap\Desktop\Train data\For_Test\Train\model\Siti_Khatijah_Halim.safetensors but folder does not exist
Traceback (most recent call last):
File "C:\Users\cheap\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\cheap\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\venv\Scripts\accelerate.exe_main.py", line 7, in
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss\venv\Scripts\python.exe', 'C:\Users\cheap\Documents\KOHYA-GUI\kohya_ss/sd-scripts/sdxl_train_network.py', '--bucket_no_upscale', '--bucket_reso_steps=64', '--cache_latents', '--cache_latents_to_disk', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--gradient_checkpointing', '--learning_rate=0.0004', '--logging_dir=C:/Users/cheap/Desktop/Train data/For_Test/Train\log', '--network_weights=C:\Users\cheap\Desktop\Train data\For_Test\Train\model\Siti_Khatijah_Halim.safetensors', '--lr_scheduler=constant_with_warmup', '--lr_scheduler_num_cycles=1', '--lr_warmup_steps=11360', '--max_data_loader_n_workers=0', '--max_grad_norm=1', '--resolution=1024,1024', '--max_train_steps=11360', '--mixed_precision=bf16', '--network_alpha=1', '--network_dim=256', '--network_module=networks.lora', '--no_half_vae', '--optimizer_args', 'scale_parameter=False', 'relative_step=False', 'warmup_init=False', '--optimizer_type=Adafactor', '--output_dir=C:/Users/cheap/Desktop/Train data/For_Test/Train\model', '--output_name=Siti', '--pretrained_model_name_or_path=C:/Users/cheap/Documents/Stable_Diffusion/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors', '--reg_data_dir=C:/Users/cheap/Desktop/Train data/For_Test/Train\reg', '--resume=C:\Users\cheap\Desktop\Train data\For_Test\Train\model\Siti_Khatijah_Halim.safetensors', '--save_every_n_epochs=1', '--save_model_as=safetensors', '--save_precision=fp16', '--save_state', '--text_encoder_lr=0.0001', '--train_batch_size=1', '--train_data_dir=C:/Users/cheap/Desktop/Train data/For_Test/Train\img', '--unet_lr=0.0001', '--xformers']' returned non-zero exit status 1.) how to fix ?

[Serallan]

I could be wrong since I never had to resume a training session so I never tried these options, but --resume and --network_weights are different methods of resuming training. You're supposed to pick one.

With --network_weights you can just throw your lora output. Results might not look that similar to what you were training before.

For --resume you need to train using --save_state true. It'll generate large last-state files, but the resumed training should be more predictable, similar to what you were getting in the previous session.

Btw, why are your paths so bipolar? 😂 Pick one slash direction and stick to it. \ will escape characters in the command line, so it's wise to use / for paths.

[kohler19]

thanks so much , btw can u give some example photo or video to set it up ? i tried many time for this thing,but not solve, yeah im newbie ,tha why my path so bipolar haha, i hope you can help me, give some tutorial for setup --save_state true,and where to paste this coment in kohya?.. i hope you can help me .. :(

[Serallan]

I can't, I'm sorry. I use the command line script on an online notebook that can't run gradio, so I've never trained on the GUI. You'll have to look it up, I'm sure there's a tutorial somewhere.

[kohler19]

now that fix.. i add ("resume": --save_state true,) to json.. it work fine.. thanks ..cheers..