Support for shuttle-3-diffusion

[bombless]

I downloaded the weights from https://huggingface.co/shuttleai/shuttle-3-diffusion, the program loaded the weights and exit for no error message.

I debugged the program, it seems that the problem is the call to ggml_type_name() with parameter 34, which is missing in the ggml.h file. Checking the llama.cpp repo, it should be GGML_TYPE_TQ1_0, added from ggml-org/llama.cpp#8151

[stduhpf]

In the version of ggml used by sdcpp, type 34 is invalid: https://github.com/ggerganov/ggml/blob/21d3a308fcb7f31cb9beceaeebad4fb622f3c337/include/ggml.h#L396

Shuttle-3-diffusion should work just like Flux Schnell in theory.

Can you share the command you used to launch it?

[bombless]

Thank you for your reply!
The command is bin/Debug/sd.exe -m e:/Downloads/shuttle-3-diffusion.safetensors -p "red cat" -v

[bombless]

FYI the real problem is in https://github.com/ggerganov/ggml/blob/21d3a308fcb7f31cb9beceaeebad4fb622f3c337/src/ggml.c#L1015, the entry for 34 is missing.

[stduhpf]

Thank you for your reply! The command is bin/Debug/sd.exe -m e:/Downloads/shuttle-3-diffusion.safetensors -p "red cat" -v

I don't think shuttle-3-diffusion has built-in vae and text encoders. You should download t5xxl, clip_L (from here ) and the vae from the shuttle-3-diffusion repo, then use bin/Debug/sd.exe -m e:/Downloads/shuttle-3-diffusion.safetensors --vae path/to/diffusion_pytorch_model.safetensors --t5xxl path/to/t5xxl_fp16.safetensors --clip_l path/to/clip_l.safetensors -p "red cat" -v instead

Also, GGML_TYPE_TQ1_0 is missing in your link.

FYI the real problem is in https://github.com/ggerganov/ggml/blob/21d3a308fcb7f31cb9beceaeebad4fb622f3c337/src/ggml.c#L1015, the entry for 34 is missing.

I don't think it's related. Sdcpp uses an older version of GGML that doesn't support TQ1_0 type yet. This shouldn't cause any issues.

[bombless]

Thank you for the pointers.

I tried bin/Debug/sd.exe -m e:/Downloads/shuttle-3-diffusion.safetensors --vae path/to/diffusion_pytorch_model.safetensors --t5xxl path/to/t5xxl_fp16.safetensors --clip_l path/to/clip_l.safetensors -p "red cat" -v and nothing changed, no error message, output.png isn't there.

I believe sdcpp should at least imply that the process didn't finish.

[stduhpf]

That's weird

[bombless]

Okay it worked!
I was missing a parameter.
Any idea why it didn't emit an error when some tensors are missing?

[stduhpf]

Any idea why it didn't emit an error when some tensors are missing?

I don't have a definitive answer, but it probably have something to do with the call to ggml_type_name(34) you noticed. i think ggml_type_name() Is only ever used in some error logs, so what could have happend is it crashed when trying to print the error because the invaild type isn't properly handled.

Glad you got it working though