Stable Diffusion 2

[vvvm23]

Model/Pipeline/Scheduler description

Perhaps you already have something in the works for this, but I couldn't see an existing issue for this.

Stable Diffusion 2 just got released on a separate repo. There are a few variations including the base, depth2img, and inpainting models. Though useable in the stable diffusion repo now, it would be really awesome to have support in diffusers to allow for ease of us in other applications!

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Model code and inference scripts can be found here

Model weights seem already available on 🤗 Hub at stabilityai/stable-diffusion-2-***

[0xdevalias]

Semi-Related:

• Please make a Dockerfile Stability-AI/stablediffusion#4
• Add an example Google Colab / Jupyter Notebook file Stability-AI/stablediffusion#9
• [Feature Request]: SD 2.0 Support AUTOMATIC1111/stable-diffusion-webui#5009 [dupe]
• [Feature Request]: Support for new 2.0 models | 768x768 resolution + new 512x512 + depth + inpainting AUTOMATIC1111/stable-diffusion-webui#5011
• Add support for Stable Diffusion 2.0 models Haidra-Org/AI-Horde#86
• Add support for Stable Diffusion 2.0 models JoePenna/Dreambooth-Stable-Diffusion#112
• Add stable diffusion 2 TheLastBen/fast-stable-diffusion#599
• Add support for Stable Diffusion 2.0 models ShivamShrirao/diffusers#143
• https://github.com/Sygil-Dev/nataili/issues/67
• Add support for Stable Diffusion 2.0 models Sygil-Dev/sygil-webui#1686

[0xdevalias]

rudimentary support for stable diffusion 2.0

MrCheeze/stable-diffusion-webui@069591b

Originally posted by @152334H in AUTOMATIC1111/stable-diffusion-webui#5011 (comment)

[0xdevalias]

https://github.com/hafriedlander/diffusers/blob/stable_diffusion_2/scripts/convert_original_stable_diffusion_to_diffusers.py

Notes:

• Only tested on the two txt2img models, not inpaint / depth2img / upscaling
• You will need to change your text embedding to use the penultimate layer too
• It spits out a bunch of warnings about vision_model, but that's fine
• I have no idea if this is right or not. It generates images, no guarantee beyond that. (Hence no PR - if you're patient, I'm sure the Diffusers team will do a better job than I have)

Originally posted by @hafriedlander in #1388 (comment)

---

Here's an example of accessing the penultimate text embedding layer https://github.com/hafriedlander/stable-diffusion-grpcserver/blob/b34bb27cf30940f6a6a41f4b77c5b77bea11fd76/sdgrpcserver/pipeline/text_embedding/basic_text_embedding.py#L33

Originally posted by @hafriedlander in #1388 (comment)

---

doesn't seem to work for me on the 768-v model using the v2 config for v

TypeError: EulerDiscreteScheduler.init() got an unexpected keyword argument 'prediction_type'

Originally posted by @devilismyfriend in #1388 (comment)

---

You need to use the absolute latest Diffusers and merge this PR (or use my branch which has it in it) #1386

Originally posted by @hafriedlander in #1388 (comment)

---

(My branch is at https://github.com/hafriedlander/diffusers/tree/stable_diffusion_2)

Originally posted by @hafriedlander in #1388 (comment)

[averad]

🤗 Diffusers with Stable Diffusion 2 is live!

anton-l commented (#1388 (comment))
diffusers==0.9.0 with Stable Diffusion 2 are live! https://github.com/huggingface/diffusers/releases/tag/v0.9.0

📰 News

• Weights for the 768x768 model is up! - Averad

  • https://huggingface.co/stabilityai/stable-diffusion-2

• add v prediction #1386 - by patil-suraj and patrickvonplaten has been merged into --main by patil-suraj

• Weights for the 512x512 models are up! - patrickvonplaten

  • https://huggingface.co/stabilityai/stable-diffusion-2-base
  • https://huggingface.co/stabilityai/stable-diffusion-2-inpainting

✏️ Notes & Information

Related huggingface/diffusers Pull Requests:

• SD2 docs #1424
• added initial v-pred support to DPM-solver #1421
• Add tests for Stable Diffusion 2 V-prediction 768x768 #1420
• Allow to set config params directly in init #1419
• Fix ONNX conversion and inference #1416
• Fix all stable diffusion #1415
• fix clip guided #1414
• Add SD2 inpainting integration tests #1412
• [MPS] call contiguous after permute #1411
• fix stable-diffusion V2 on MPS #1410
• Upscaling fixed #1402
• Add the new SD2 attention params to the VD text unet #1400
• Support SD2 attention slicing #1397
• StableDiffusionUpscalePipeline #1396
• add v prediction #1386 - by patil-suraj and patrickvonplaten

👇 Quick Links:

• https://github.com/Stability-AI/stablediffusion
  • requirements.txt
  • txt2img
  • img2img
  • Gradio Scripts
  • Streamlit Scripts
• https://huggingface.co/stabilityai/stable-diffusion-2
• https://huggingface.co/stabilityai/stable-diffusion-2-base
• https://huggingface.co/stabilityai/stable-diffusion-2-depth
• https://huggingface.co/stabilityai/stable-diffusion-2-inpainting
• https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler

👁️ User Submitted Resources:

• Test Version of convert_original_stable_diffusion_to_diffusers.py by hafriedlander
• Example of accessing the penultimate text embedding layer by hafriedlander
• NovelAI Improvements on Stable Diffusion

💭 User Story

Stable Diffusion 2.0 has recently been released. When you run convert_original_stable_diffusion_to_diffusers.py on the new Stability-AI/stablediffusion models the following errors occur.

convert_original_stable_diffusion_to_diffusers.py --checkpoint_path="./512-inpainting-ema.ckpt" --dump_path="./512-inpainting-ema_diffusers"

Output:

Traceback (most recent call last):
File "convert_original_stable_diffusion_to_diffusers.py", line 720, in <module> 
        unet.load_state_dict(converted_unet_checkpoint)
File "lib\site-packages\torch\nn\modules\module.py", line 1667, in load_state_dict
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
        size mismatch for down_blocks.0.attentions.0.proj_in.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
        size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
        size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
        size mismatch for down_blocks.0.attentions.0.proj_out.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
.... blocks.1.attentions blocks.2.attentions etc. etc.

[0xdevalias]

testing in progress on the horde https://github.com/Sygil-Dev/nataili/tree/v2
try it out Stable Diffusion 2.0 on our UI's

https://tinybots.net/artbot
https://aqualxx.github.io/stable-ui/
https://dbzer0.itch.io/lucid-creations

https://sigmoid.social/@stablehorde/109398715339480426

SD 2.0

• Initial implementation ready for testing
• img2img
• inpainting
• k_diffusers support

Originally posted by @AlRlC in https://github.com/Sygil-Dev/nataili/issues/67#issuecomment-1326385645

[0xdevalias]

• TheLastBen/fast-stable-diffusion@11fd38b

  • Create pathsV2.py

• TheLastBen/fast-stable-diffusion@fe445d9

  • Support for SD V.2

• TheLastBen/fast-stable-diffusion@da9b380

  • fix

• TheLastBen/fast-stable-diffusion@6c84728

  • fix

• TheLastBen/fast-stable-diffusion@04ba92b

  • fix

• TheLastBen/fast-stable-diffusion@ebea134

  • Create sd_hijackV2.py

• TheLastBen/fast-stable-diffusion@88496f5

  • Create sd_samplersV2.py

• TheLastBen/fast-stable-diffusion@f324b3d

  • fix V2

Originally posted by @0xdevalias in TheLastBen/fast-stable-diffusion#599 (comment)

[0xdevalias]

Should work now, make sure you check the box "redownload original model" when choosing V2

https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast_stable_diffusion_AUTOMATIC1111.ipynb

Requires more than 12GB of RAM for now, so free colab probably won't suffice.

Originally posted by @TheLastBen in TheLastBen/fast-stable-diffusion#599 (comment)

[vvvm23]

From @pcuenca on the HF discord:

We are busy preparing a new release of diffusers to fully support Stable Diffusion 2. We are still ironing things out, but the basics already work from the main branch in github. Here's how to do it:

• Install diffusers from github alongside its dependencies:

pip install --upgrade git+https://github.com/huggingface/diffusers.git transformers accelerate scipy

• Use the code in this script to run your predictions:

from diffusers import DiffusionPipeline, EulerDiscreteScheduler
import torch

repo_id = "stabilityai/stable-diffusion-2"
device = "cuda"

scheduler = EulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler", prediction_type="v_prediction")
pipe = DiffusionPipeline.from_pretrained(repo_id, torch_dtype=torch.float16, revision="fp16", scheduler=scheduler)
pipe = pipe.to(device)

prompt = "High quality photo of an astronaut riding a horse in space"
image = pipe(prompt, width=768, height=768, guidance_scale=9).images[0]
image.save("astronaut.png")