Skip to content

Conversation

@patil-suraj
Copy link
Contributor

What does this PR do?

This PR adds support for LCM in:

  • StableDiffusionControlNetPipeline
  • StableDiffusionXLControlNetPipeline
  • StableDiffusionAdapterPipeline
  • StableDiffusionXLAdapterPipeline

Also added fast tests for LCMs with these pipelines.

(((4 * 2 + 1) * 16),),
# (dim=160) The internal feature map will be 5x5 after the first T2I down block (downscaled x32).
((4 * 1 + 1) * 32),
(((4 * 1 + 1) * 32),),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change on purpose?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, without this, the parametrized tests were erroing.

[
# (dim=264) The internal feature map will be 33x33 after initial pixel unshuffling (downscaled x8).
((4 * 8 + 1) * 8),
(((4 * 8 + 1) * 8),),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change on purpose?

Copy link
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@patil-suraj
Copy link
Contributor Author

The failing test is unrelated. Merging now.

@patil-suraj patil-suraj merged commit 64cbd8e into main Nov 16, 2023
@patil-suraj patil-suraj deleted the lcm-in-other-pipes branch November 16, 2023 13:59
@patil-suraj patil-suraj mentioned this pull request Nov 16, 2023
@tvytlx
Copy link

tvytlx commented Dec 3, 2023

@patil-suraj Hi

class StableDiffusionAdapterPipeline(DiffusionPipeline):

Seems like we need a LoraLoaderMixin here?

https://github.com/huggingface/diffusers/blob/main/src/diffusers/loaders/lora.py#L69

Because when I use Lora in StableDiffusionAdapterPipeline

   pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
   pipe.load_lora_weights(lcm_lora_id)

it causes AttributeError: 'StableDiffusionAdapterPipeline' object has no attribute 'load_lora_weights'

@sayakpaul
Copy link
Member

That is happening because it doesn't inherit the LoraLoaderMixin class:

class StableDiffusionAdapterPipeline(DiffusionPipeline):

Unlike StableDiffusionXLAdapterPipeline:

https://github.com/huggingface/diffusers/blob/d486f0e84669447b178569ad499eeb86c739b99e/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L171C7-L171C39

Would you like to open a PR for this?

yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request Dec 25, 2023
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants