[Question][Solved] How can I use color correction .pt in diffusers

[camenduru]

https://huggingface.co/Linaqruf/anything-v3.0/blob/main/Anything-V3.0.vae.pt
this is for color correction how can I use this with diffusers

[apolinario]

Hi @camenduru , that 'color correction .pt' is the anythingV3.0 VAE. VAE is the variational auto-encoder that encodes/decodes images for models like Stable Diffusion

While it would be possible to convert that pt to the diffusers format, there's no specific VAE-only conversion script, so you would need to adapt the existing script a little bit to convert the VAE only.

However, the good thing is that if the VAE is also in the entire model's CKPT, as the entire anything-v3.0 model is in the diffusers format as well, you can load the anything-v3.0 VAE (or any other arbitrary VAE, for e.g. the fine-tuned one by Stability AI) in any Stable Diffusion model with diffusers, in the following way:

from diffusers.models import AutoencoderKL
from diffusers import StableDiffusionPipeline
model = "CompVis/stable-diffusion-v1-4" #Replace this to any model you would like to use this VAE with
vae = AutoencoderKL.from_pretrained("Linaqruf/anything-v3.0", subfolder="vae")
pipe = StableDiffusionPipeline.from_pretrained(model, vae=vae)

[camenduru]

While it would be possible to convert that pt to the diffusers format, there's no specific VAE-only conversion script, so you would need to adapt the existing script a little bit to convert the VAE only.

oh no 😐

[camenduru]

I did it 🎉

[patrickvonplaten]

Wuhuu super cool! Would you like to maybe add a conversion script to diffusers for this? :-)

[camenduru]

hi @patrickvonplaten 👋 ok I will add after this #1900

[camenduru]

If someone is here for information on how to please check the swap ckpt vae inside https://huggingface.co/spaces/camenduru/converter