Add LDM Super Resolution pipeline #1116

duongna21 · 2022-11-03T11:18:42Z

4x Super Resolution by Latent Diffusion Model (original checkpoint here). Might fixes #463 and fixes #146.

How to use:

pip install git+https://github.com/duongna21/diffusers.git@add-sr-pipeline

from diffusers import LDMSuperResolutionPipeline
from PIL import Image

pipe = LDMSuperResolutionPipeline.from_pretrained('duongna/ldm-super-resolution')
pipe.to('cuda')

img = Image.open('low_resolution.jpg')
super_img = pipe(img, num_inference_steps=100, eta=1)
super_img['images'][0]

->

cc @patrickvonplaten @patil-suraj @pcuenca

HuggingFaceDocBuilderDev · 2022-11-03T11:21:50Z

The documentation is not available anymore as the PR was closed or merged.

patil-suraj

Great PR @duongna21 and super cool addition! Tried the pipeline and it works super well already.

I left some comments, we need to address a few things before we can merge this.
Mainly

handling dtype and device
handling different schedulers
add doc page for the pipeline
add tests for the pipeline in tests/pipelines/ldm_superresolition

Let me know if you need help with any of this. Great work!

src/diffusers/__init__.py

src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

src/diffusers/utils/dummy_torch_and_transformers_objects.py

src/diffusers/pipelines/latent_diffusion/__init__.py

patrickvonplaten

Super cool addition - think we only need to solve some nits and then this is good to go :-)

…sion_superresolution.py Co-authored-by: Patrick von Platen <[email protected]>

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

into add-sr-pipeline

duongna21 · 2022-11-06T10:47:37Z

@patil-suraj @patrickvonplaten Thanks you very much for the detailed comments. I learned a lot about the library when trying to address them. Please check out the fixes.
Also, can you help me fix the test at tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py?

patil-suraj · 2022-11-07T11:05:28Z

Hey @duongna21 super cool! This PR also fixes typos in other pipelines. It would be best to open a separate PR for this and keep this PR only for super-resolution pipeline. It's better to have single purpose PR, so it's easy to test and review. Hope you understand :)

duongna21 · 2022-11-07T11:40:42Z

It would be best to open a separate PR for this and keep this PR only for super-resolution pipeline.

@patil-suraj Sure, indeed we should do that. Unfixed the typo.

tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py

pcuenca · 2022-11-07T13:09:14Z

src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

+
+def preprocess(image):
+    w, h = image.size
+    w, h = map(lambda x: x - x % 32, (w, h))  # resize to integer multiple of 32


Why is this necessary? An alternative would be to pad and then crop the upscaled image. Not sure if it's worth it, slightly worried that this might skew images a little bit.

@pcuenca This is how other pipelines resize the image so it can successfully forward over UNet (agree that it might skew the image). Really sorry I can't fully understand your suggestion, could you kindly push a commit for it?

Here the preprocessing should be similar to how it's done in the original repo, since the model is trained on the preprocessed image. @duongna21 could post a link to the original inference code ?

@patil-suraj Look at this and this. It works great with varying img size. But I can't spend time on this in the next few days.

Thanks and no worries. We'll try to take a look at this, we can merge the PR without that also.

src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

…solution.py Co-authored-by: Pedro Cuenca <[email protected]>

into add-sr-pipeline

duongna21 · 2022-11-08T09:03:35Z

@pcuenca Thanks a lot for helpful suggestions. The tests look good now.

patil-suraj

The PR is looking good! Thank you for addressing the comments :)

Will run the slow tests and upload the checkpoint under CompVis org on the hub.

One last thing to verify is to check if the preprocessing code is similar to how it's done in the original repo.

Then this should be good to merge :)

patil-suraj · 2022-11-08T09:38:17Z

src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

+
+def preprocess(image):
+    w, h = image.size
+    w, h = map(lambda x: x - x % 32, (w, h))  # resize to integer multiple of 32


Here the preprocessing should be similar to how it's done in the original repo, since the model is trained on the preprocessed image. @duongna21 could post a link to the original inference code ?

src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

patil-suraj · 2022-11-09T12:42:08Z

Thanks a lot @duongna21 ! Uploaded the checkpoint under official account.

https://huggingface.co/CompVis/ldm-super-resolution-4x-openimages

patrickvonplaten · 2022-11-09T21:46:44Z

tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import random


nice tests!

* Add ldm super resolution pipeline * style * fix copies * style * fix doc * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Patrick von Platen <[email protected]> * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Patrick von Platen <[email protected]> * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Suraj Patil <[email protected]> * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Suraj Patil <[email protected]> * add doc * address comments * address comments * fix doc * minor * add tests * add tests * load text encoder from subfolder * fix test * fix test * style * style * handle mps latents * unfix typo * unfix typo * Update tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py Co-authored-by: Pedro Cuenca <[email protected]> * fix set_timesteps mps * fix set_timesteps mps * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Suraj Patil <[email protected]> * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Suraj Patil <[email protected]> * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Suraj Patil <[email protected]> * Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Co-authored-by: Suraj Patil <[email protected]> * style * test 64x64 instead of 256x256 Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Suraj Patil <[email protected]> Co-authored-by: Pedro Cuenca <[email protected]>

duongna21 added 2 commits November 3, 2022 17:23

Add ldm super resolution pipeline

7664874

style

2d3c98a

duongna21 added 2 commits November 3, 2022 19:47

fix copies

0c44672

style

5519da2

duongna21 changed the title ~~Add Super Resolution pipeline~~ Add LDM Super Resolution pipeline Nov 3, 2022

patil-suraj self-assigned this Nov 3, 2022

fix doc

8af0ade

patil-suraj reviewed Nov 4, 2022

View reviewed changes

patil-suraj assigned patrickvonplaten and unassigned patrickvonplaten Nov 4, 2022

patil-suraj requested a review from patrickvonplaten November 4, 2022 10:51

patrickvonplaten reviewed Nov 4, 2022

View reviewed changes

src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Nov 4, 2022

View reviewed changes

src/diffusers/utils/dummy_torch_and_transformers_objects.py Show resolved Hide resolved

patrickvonplaten reviewed Nov 4, 2022

View reviewed changes

src/diffusers/pipelines/latent_diffusion/__init__.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Nov 4, 2022

View reviewed changes

duongna21 and others added 14 commits November 5, 2022 10:24

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

82623e0

…sion_superresolution.py Co-authored-by: Patrick von Platen <[email protected]>

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

9ef5ba1

…sion_superresolution.py Co-authored-by: Patrick von Platen <[email protected]>

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

9977636

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

226bbc0

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

add doc

d52704f

Merge branch 'add-sr-pipeline' of https://github.com/duongna21/diffusers

b3e9cff

into add-sr-pipeline

address comments

16584f7

address comments

003e185

fix doc

e360ce6

minor

d189eea

add tests

b2d5e21

add tests

4ca74e8

load text encoder from subfolder

69daedc

fix test

ac78735

duongna21 added 4 commits November 6, 2022 09:45

fix test

9c5134c

style

7115557

style

afc4462

handle mps latents

5708a2c

duongna21 added 2 commits November 7, 2022 18:37

unfix typo

b4fbb2b

unfix typo

f02b34b

pcuenca reviewed Nov 7, 2022

View reviewed changes

duongna21 and others added 4 commits November 8, 2022 15:32

Update tests/pipelines/latent_diffusion/test_latent_diffusion_superre…

9606d01

…solution.py Co-authored-by: Pedro Cuenca <[email protected]>

fix set_timesteps mps

dc7de80

Merge branch 'add-sr-pipeline' of https://github.com/duongna21/diffusers

11e3d7b

into add-sr-pipeline

fix set_timesteps mps

6f98543

patil-suraj approved these changes Nov 8, 2022

View reviewed changes

patil-suraj reviewed Nov 8, 2022

View reviewed changes

tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py Show resolved Hide resolved

duongna21 and others added 7 commits November 8, 2022 18:42

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

3f6e1fa

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

ef0c091

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

47593f2

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

Update src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffu…

1f808a1

…sion_superresolution.py Co-authored-by: Suraj Patil <[email protected]>

style

5308ff5

test 64x64 instead of 256x256

6f122a7

Merge branch 'main' into add-sr-pipeline

903bba0

patil-suraj merged commit 5a59f9b into huggingface:main Nov 9, 2022

patrickvonplaten reviewed Nov 9, 2022

View reviewed changes

skirsten mentioned this pull request Nov 16, 2022

[Flax] Fix loading scheduler from subfolder #1319

Merged

Add LDM Super Resolution pipeline #1116

Add LDM Super Resolution pipeline #1116

Uh oh!

Conversation

duongna21 commented Nov 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Nov 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patil-suraj left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

duongna21 commented Nov 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patil-suraj commented Nov 7, 2022

Uh oh!

duongna21 commented Nov 7, 2022

Uh oh!

Uh oh!

pcuenca Nov 7, 2022

Choose a reason for hiding this comment

Uh oh!

duongna21 Nov 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patil-suraj Nov 8, 2022

Choose a reason for hiding this comment

Uh oh!

duongna21 Nov 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patil-suraj Nov 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

duongna21 commented Nov 8, 2022

Uh oh!

patil-suraj left a comment

Choose a reason for hiding this comment

Uh oh!

patil-suraj Nov 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patil-suraj commented Nov 9, 2022

Uh oh!

patrickvonplaten Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

duongna21 commented Nov 3, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Nov 3, 2022 •

edited

Loading

patil-suraj left a comment •

edited

Loading

duongna21 commented Nov 6, 2022 •

edited

Loading

duongna21 Nov 8, 2022 •

edited

Loading

duongna21 Nov 8, 2022 •

edited

Loading