How to working this?

[GiyeopKim1993]

Hello. I was planning to generate images using SDXL.
I loaded the model noobaiXLNAIXL_vPred075SVersion and used it as follows:

To use the GPU, I built CUBLAS.
The CPU is an Intel i7 11th generation, and the GPU is an RTX 4060.

stable-diffusion.cpp:497 - total params memory size = 12327.02MB (VRAM 12327.02MB, RAM 0.00MB): clip 3119.36MB(VRAM), unet 9113.19MB(VRAM), vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)

/bin/sd -m ./noobaiXLNAIXL_vPred075SVersion.safetensors -p "looking at viewer" -o "looking at viewer.png" --vae ./sdxl_vae.safetensors

It took 1 hour and 25 minutes, and I only got noise.
I am attaching the noisy image.

[Green-Sky]

Run with -v and provide the log. Also, when running with cuda, it should not take that long.

[GiyeopKim1993]

OK, i will when tomorrow go to work.

tanks your kind.

but GPU USED 99% AND VRAM 12GB used.

[GiyeopKim1993]

I tried it on another PC to see if it would work, but I obtained the same noise. However, the speed was faster.

cpu : i7-9700F
ram : 16gb
gpu rtx 2060 super 8gb

bin/sd -m ./noobaiXLNAIXL_vPred075SVersion.safetensors -p "looking at viewer" -o "looking at viewer.png" --vae ./sdxl_vae.safetensors -v
Option:
n_threads: 4
mode: txt2img
model_path: ./noobaiXLNAIXL_vPred075SVersion.safetensors
wtype: unspecified
clip_l_path:
clip_g_path:
t5xxl_path:
diffusion_model_path:
vae_path: ./sdxl_vae.safetensors
taesd_path:
esrgan_path:
controlnet_path:
embeddings_path:
stacked_id_embeddings_path:
input_id_images_path:
style ratio: 20.00
normalize input image : false
output_path: looking at viewer.png
init_img:
control_image:
clip on cpu: false
controlnet cpu: false
vae decoder on cpu:false
diffusion flash attention:false
strength(control): 0.90
prompt: looking at viewer
negative_prompt:
min_cfg: 1.00
cfg_scale: 7.00
slg_scale: 0.00
guidance: 3.50
clip_skip: -1
width: 512
height: 512
sample_method: euler_a
schedule: default
sample_steps: 20
strength(img2img): 0.75
rng: cuda
seed: 42
batch_count: 1
vae_tiling: false
upscale_repeats: 1
System Info:
SSE3 = 1
AVX = 1
AVX2 = 1
AVX512 = 0
AVX512_VBMI = 0
AVX512_VNNI = 0
FMA = 1
NEON = 0
ARM_FMA = 0
F16C = 1
FP16_VA = 0
WASM_SIMD = 0
VSX = 0
[DEBUG] stable-diffusion.cpp:159 - Using CUDA backend
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 2060 SUPER, compute capability 7.5, VMM: yes
[INFO ] stable-diffusion.cpp:191 - loading model from './noobaiXLNAIXL_vPred075SVersion.safetensors'
[INFO ] model.cpp:888 - load ./noobaiXLNAIXL_vPred075SVersion.safetensors using safetensors format
[DEBUG] model.cpp:959 - init from './noobaiXLNAIXL_vPred075SVersion.safetensors'
[INFO ] stable-diffusion.cpp:226 - loading vae from './sdxl_vae.safetensors'
[INFO ] model.cpp:888 - load ./sdxl_vae.safetensors using safetensors format
[DEBUG] model.cpp:959 - init from './sdxl_vae.safetensors'
[INFO ] stable-diffusion.cpp:238 - Version: SDXL
[INFO ] stable-diffusion.cpp:271 - Weight type: f32
[INFO ] stable-diffusion.cpp:272 - Conditioner weight type: f32
[INFO ] stable-diffusion.cpp:273 - Diffusion model weight type: f32
[INFO ] stable-diffusion.cpp:274 - VAE weight type: f32
[DEBUG] stable-diffusion.cpp:276 - ggml tensor size = 400 bytes
[DEBUG] clip.hpp:171 - vocab size: 49408
[DEBUG] clip.hpp:182 - trigger word img already in vocab
[DEBUG] ggml_extend.hpp:1071 - clip params backend buffer size = 469.44 MB(VRAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1071 - clip params backend buffer size = 2649.92 MB(VRAM) (517 tensors)
[DEBUG] ggml_extend.hpp:1071 - unet params backend buffer size = 9113.19 MB(VRAM) (1680 tensors)
[DEBUG] ggml_extend.hpp:1071 - vae params backend buffer size = 94.47 MB(VRAM) (140 tensors)
[DEBUG] stable-diffusion.cpp:413 - loading weights
[DEBUG] model.cpp:1645 - loading tensors from ./noobaiXLNAIXL_vPred075SVersion.safetensors
[INFO ] model.cpp:1809 - unknown tensor 'v_pred | f32 | 1 [0, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1809 - unknown tensor 'ztsnr | f32 | 1 [0, 1, 1, 1, 1]' in model file
[DEBUG] model.cpp:1645 - loading tensors from ./sdxl_vae.safetensors
[INFO ] stable-diffusion.cpp:497 - total params memory size = 12327.02MB (VRAM 12327.02MB, RAM 0.00MB): clip 3119.36MB(VRAM), unet 9113.19MB(VRAM), vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:516 - loading model from './noobaiXLNAIXL_vPred075SVersion.safetensors' completed, taking 118.42s
[INFO ] stable-diffusion.cpp:546 - running in eps-prediction mode
[DEBUG] stable-diffusion.cpp:590 - finished loaded file
[DEBUG] stable-diffusion.cpp:1464 - txt2img 512x512
[DEBUG] stable-diffusion.cpp:1194 - prompt after extract and remove lora: "looking at viewer"
[INFO ] stable-diffusion.cpp:673 - Attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:1199 - apply_loras completed, taking 0.00s
[DEBUG] conditioner.hpp:329 - parse 'looking at viewer' to [['looking at viewer', 1], ]
[DEBUG] clip.hpp:311 - token length: 77
[DEBUG] ggml_extend.hpp:1023 - clip compute buffer size: 1.40 MB(VRAM)
[DEBUG] ggml_extend.hpp:1023 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] ggml_extend.hpp:1023 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] conditioner.hpp:457 - computing condition graph completed, taking 813 ms
[DEBUG] conditioner.hpp:329 - parse '' to [['', 1], ]
[DEBUG] clip.hpp:311 - token length: 77
[DEBUG] ggml_extend.hpp:1023 - clip compute buffer size: 1.40 MB(VRAM)
[DEBUG] ggml_extend.hpp:1023 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] ggml_extend.hpp:1023 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] conditioner.hpp:457 - computing condition graph completed, taking 131 ms
[INFO ] stable-diffusion.cpp:1332 - get_learned_condition completed, taking 980 ms
[INFO ] stable-diffusion.cpp:1355 - sampling using Euler A method
[INFO ] stable-diffusion.cpp:1359 - generating image: 1/1 - seed 42
[DEBUG] ggml_extend.hpp:1023 - unet compute buffer size: 132.05 MB(VRAM)
|==================================================| 20/20 - 5.44s/it
[INFO ] stable-diffusion.cpp:1395 - sampling completed, taking 112.79s
[INFO ] stable-diffusion.cpp:1403 - generating 1 latent images completed, taking 113.78s
[INFO ] stable-diffusion.cpp:1406 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1023 - vae compute buffer size: 1664.00 MB(VRAM)
[DEBUG] stable-diffusion.cpp:1045 - computing vae [mode: DECODE] graph completed, taking 0.76s
[INFO ] stable-diffusion.cpp:1416 - latent 1 decoded, taking 0.76s
[INFO ] stable-diffusion.cpp:1420 - decode_first_stage completed, taking 0.76s
[INFO ] stable-diffusion.cpp:1539 - txt2img completed in 115.53s
save result image to 'looking at viewer.png'

[stduhpf]

[INFO ] model.cpp:1809 - unknown tensor 'v_pred | f32 | 1 [0, 1, 1, 1, 1]' in model file

[INFO ] stable-diffusion.cpp:546 - running in eps-prediction mode

Oh. This could be the reason.

Base SDXL is trained to do eps-prediction, but it seems that this model is supposed to do v-prediction, which, I believe, is only (poorly) supported for SD2 models for now...

[GiyeopKim1993]

You are Goat(Greatest of all time)!

If you're willing, I'd like to research and learn about this and write the v_pred sdxl code.