Improve reproduceability 2/3 #1906
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
patrickvonplaten
changed the title
src/diffusers/utils/torch_utils.py
Outdated
|
|
||
|
|
||
| def torch_randn( | ||
| def rand_tensorn( |
There was a problem hiding this comment.
It seems like there is simply no way to make things fully deterministic even if we generate all tensors on CPU.
|
|
||
| causal_attention_mask = torch.full( | ||
| [num_embeddings + additional_embeddings, num_embeddings + additional_embeddings], float("-inf") | ||
| [num_embeddings + additional_embeddings, num_embeddings + additional_embeddings], -10000.0 |
There was a problem hiding this comment.
For consistency, doesn't change the outputs here. Actually surprised that PyTorch seems to be able to handle + float("-inf") in newer versions, but think in older versions it wasn't the case.
| images[0].save("/home/patrick_huggingface_co/diffusers-images/unclip/karlo_v1_alpha_horse_image.png") | ||
|
|
||
| assert image.shape == (256, 256, 3) | ||
| assert np.abs(expected_image - image).max() < 1e-1 |
There was a problem hiding this comment.
Even on CPU I cannot get a higher precision
There was a problem hiding this comment.
@anton-l I'd say we make this a nightly ? It takes 5 min on CPU
There was a problem hiding this comment.
Yeah, it should be ok to move the tests for less frequently updated pipelines completely to nightly IMO
| # Here we just check that the image doesn't deviate more than 10 pixels from the reference image on average | ||
| avg_diff = np.abs(image - expected_image).mean() | ||
|
|
||
| assert avg_diff < 10, f"Error image deviates {avg_diff} pixels on average" |
There was a problem hiding this comment.
Pretty high avg difference, but it doesn't seem like we can get closer than this.
patrickvonplaten
changed the title
|
All slow tests passing -> Good enough for merge. Will complete the PR series tomorrow 🥳 |
* [Repro] Correct reproducability * up * up * uP * up * need better image * allow conversion from no state dict checkpoints * up * up * up * up * check tensors * check tensors * check tensors * check tensors * next try * up * up * better name * up * up * Apply suggestions from code review * correct more * up * replace all torch randn * fix * correct * correct * finish * fix more * up
2nd PR of the series after #1902.
This PR allows one to pass CPU generators even when the model is run on GPU to improve the reproducability of models since this way the latents are not dependent on the CUDA / GPU hardware (see: pytorch/pytorch#87992).