Add state checkpointing to other training scripts #1687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

pcuenca merged 8 commits into main from resume-checkpoints

Dec 15, 2022

Member

pcuenca commented Dec 13, 2022 •

edited

Loading

Bring dreambooth changes in #1668 to the rest of the training scripts.
Fixes #666.

pcuenca added 2 commits

December 13, 2022 15:18


          Add state checkpointing to other training scripts

8b9439c


          Fix first_epoch

6a351b1

pcuenca requested a review from patrickvonplaten

December 13, 2022 14:24

HuggingFaceDocBuilderDev commented Dec 13, 2022 •

edited

Loading

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten reviewed

View reviewed changes

examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed

View reviewed changes

examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py Show resolved Hide resolved

patrickvonplaten reviewed

View reviewed changes

examples/text_to_image/train_text_to_image.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed

View reviewed changes

examples/textual_inversion/textual_inversion.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed

View reviewed changes

examples/unconditional_image_generation/train_unconditional.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed

View reviewed changes

examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed

View reviewed changes

Contributor

patrickvonplaten left a comment

Would maybe rephrase the help message a bit and possible even add an example of how to use intermediate checkpoints for inference

Contributor

patrickvonplaten commented Dec 13, 2022

BTW @pcuenca , accelerate already merged the "only last checkpoitns" features to "main": huggingface/accelerate#916

Let's maybe allow people to directly use it depending on the accelerate version?


          Apply suggestions from code review

976bfdc

Co-authored-by: Patrick von Platen <[email protected]>

Member Author

pcuenca commented Dec 13, 2022

BTW @pcuenca , accelerate already merged the "only last checkpoitns" features to "main": huggingface/accelerate#916

Let's maybe allow people to directly use it depending on the accelerate version?

Yes!

patrickvonplaten and others added 4 commits

December 14, 2022 13:05


          Merge branch 'main' into resume-checkpoints

917c706


          Update Dreambooth checkpoint help message.

1be8470


          Merge remote-tracking branch 'origin/main' into resume-checkpoints

26a5234


          Dreambooth docs: checkpoints, inference from a checkpoint.

66a738c

patrickvonplaten reviewed

View reviewed changes

docs/source/training/dreambooth.mdx

    
                --max_train_steps=800

              ```

              ### Saving checkpoints while training

Contributor

patrickvonplaten Dec 15, 2022

Very nice!

patrickvonplaten approved these changes

View reviewed changes

Contributor

patrickvonplaten left a comment

Perf thanks!


          make style

98fee97

pcuenca merged commit badddee into main

pcuenca deleted the resume-checkpoints branch

December 15, 2022 18:49

allo- mentioned this pull request

[textual inversion] Load learned_embeddings.bin if it exists in the target folder #1597

Closed

sliard pushed a commit to sliard/diffusers that referenced this pull request


          Add state checkpointing to other training scripts (huggingface#1687)

a7c93f5

* Add state checkpointing to other training scripts

* Fix first_epoch

* Apply suggestions from code review

Co-authored-by: Patrick von Platen <[email protected]>

* Update Dreambooth checkpoint help message.

* Dreambooth docs: checkpoints, inference from a checkpoint.

* make style

Co-authored-by: Patrick von Platen <[email protected]>

sanghyun0927 commented Jan 2, 2023

Sorry for that I am newbie in github

Just want you to know that this line may have to be changed

examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py

675 for epoch in range(first_epoch, args.num_epochs):
==>
675 for epoch in range(first_epoch, args.num_train_epochs):

Member Author

pcuenca commented Jan 2, 2023

I think you are right, @sanghyun0927, just opened #1885 to fix it. Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet