vq diffusion classifier free sampling #1294
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
d0d5beb to
4ee1e06
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
4ee1e06 to
cac658d
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
cac658d to
f5fbbbe
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
f5fbbbe to
8746de8
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
8746de8 to
fe8db41
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
fe8db41 to
bfc4459
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
bfc4459 to
40dc3ff
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
40dc3ff to
10e2ea3
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
williamberman
force-pushed
the
will/vq-diffusion-classifier-free-guidance
branch
from
10e2ea3 to
08984ab
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
| "https://huggingface.co/datasets/williamberman/misc/resolve/main" | ||
| "/vq_diffusion/teddy_bear_pool_classifier_free_sampling.png" |
There was a problem hiding this comment.
Should be moved to the huggingface testing dataset. FWIW you might have to also regenerate the image because I get a different image on VQDiffusionPipelineIntegrationtests#test_vq_diffusion.
There was a problem hiding this comment.
Very cool :-) I'll move it!
williamberman
changed the title
src/diffusers/models/embeddings.py
Outdated
| class LearnedClassifierFreeSamplingEmbeddings(ModelMixin, ConfigMixin): | ||
| """ | ||
| Utility class for storing learned text embeddings for classifier free sampling | ||
| """ | ||
|
|
||
| @register_to_config | ||
| def __init__(self, learnable: bool, hidden_size: Optional[int] = None, length: Optional[int] = None): | ||
| super().__init__() | ||
|
|
||
| self.learnable = learnable | ||
|
|
||
| if self.learnable: | ||
| assert hidden_size is not None, "learnable=True requires `hidden_size` to be set" | ||
| assert length is not None, "learnable=True requires `length` to be set" | ||
|
|
||
| embeddings = torch.zeros(length, hidden_size) | ||
| else: | ||
| embeddings = None | ||
|
|
||
| self.embeddings = nn.Parameter(embeddings) |
There was a problem hiding this comment.
Not sure if this is the preferred way to add the learned embeddings to the pipeline. An alternative might be to add the additional vector to the scheduler instead
There was a problem hiding this comment.
It's very model specific, so moving it to the pipeline here directly :-)
Think that's a bit cleaner! The model works much better now though - thanks!
| tokenizer: CLIPTokenizer, | ||
| transformer: Transformer2DModel, | ||
| scheduler: VQDiffusionScheduler, | ||
| learned_classifier_free_sampling_embeddings: LearnedClassifierFreeSamplingEmbeddings, |
There was a problem hiding this comment.
That's definitely the right way to do it - it's quite specific to vq-diffusion IMO though, so will move it here :-)
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
|
Very nice job @williamberman ! |
* vq diffusion classifier free sampling * correct * uP Co-authored-by: Patrick von Platen <[email protected]>
3 participants
Adds classifier free sampling to VQ diffusion. This results in significantly better image quality.
The pipeline now has a default guidance_scale of 5.0
Additionally, the ithq dataset uses a learned parameter for the classifier free embeddings. We modify the convert script to add this parameter to the ported model. Weights will have to be reuploaded
Prompts: "teddy bear playing in the pool" and "horse"
Diffusers VQ diffusion with classifier free sampling
Diffusers VQ diffusion without classifier free sampling
Original VQ diffusion implementation with classifier free sampling
Original VQ diffusion implementation without classifier free sampling