Skip to content

Conversation

kim-sangyeon
Copy link
Contributor

What does this PR do?

Partially fixes #6545 regarding InstructPix2Pix SDXL.

Before submitting

Who can review?

@sayakpaul

@sayakpaul
Copy link
Member

I think the PR is not yet ready (conflict).

@kim-sangyeon kim-sangyeon force-pushed the modularize_instructpix2pix_inferencing branch 2 times, most recently from 0b7748e to 837ba90 Compare January 17, 2024 16:53
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@kim-sangyeon
Copy link
Contributor Author

@sayakpaul Thanks for letting me know.
I resolved the conflict and I think it is ready for review now.

@kim-sangyeon kim-sangyeon force-pushed the modularize_instructpix2pix_inferencing branch from 837ba90 to 29f9cbb Compare January 17, 2024 17:47
@kim-sangyeon kim-sangyeon force-pushed the modularize_instructpix2pix_inferencing branch from 29f9cbb to 7652efb Compare January 17, 2024 17:49
Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks cool. Do you have a training command for me to test this with?

Do the changes work as expected?

@kim-sangyeon
Copy link
Contributor Author

kim-sangyeon commented Jan 19, 2024

@sayakpaul Yes, it works as expected.
Here is the sequence of validation images with the prompt, "make it in japan" :)
When making the GIF file, I resized the images to 256x256 to comply with the upload size limitations.

Original Edited

Regarding the training command, I used the one below.

export DATASET_ID="fusing/instructpix2pix-1000-samples"

accelerate launch train_instruct_pix2pix_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --dataset_name=$DATASET_ID \
    --use_ema \
    --enable_xformers_memory_efficient_attention \
    --resolution=512 --random_flip \
    --train_batch_size=4 --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=15000 \
    --checkpointing_steps=5000 --checkpoints_total_limit=1 \
    --learning_rate=5e-05 --lr_warmup_steps=0 \
    --conditioning_dropout_prob=0.05 \
    --seed=42 \
    --val_image_url_or_path="https://datasets-server.huggingface.co/assets/fusing/instructpix2pix-1000-samples/--/fusing--instructpix2pix-1000-samples/train/23/input_image/image.jpg" \
    --validation_prompt="make it in japan" \
    --report_to=wandb \
    --push_to_hub \

@sayakpaul
Copy link
Member

Lovely. I am gonna go ahead and merge. Thanks so much for this valuable contribution.

@sayakpaul sayakpaul merged commit a9288b4 into huggingface:main Jan 19, 2024
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Tracker] modularize inferencing during and after training in the example scripts

3 participants