models downloaded by diffusers 0.10 do not appear in scan-cache

[keturn]

Describe the bug

I downloaded several models with diffusers.StableDiffusionPipeline.from_pretrained. They download successfully and work just fine, but then when I go to try to figure out how to recover some disk space, I see they do not appear in the output of huggingface-cli scan-cache, neither as line-items nor accounted for in the reported total size on disk.

The output does say Got 11 warning(s) while scanning. Use -vvv to print details.

Repo path is not a directory: 🏠/.cache/huggingface/hub/version.txt
Snapshots dir doesn't exist in cached repo: 🏠/.cache/huggingface/hub/models--wavymulder--wavyfusion/snapshots
Reference(s) refer to missing commit hashes: {'f15bc7606314c6fa957b4267bee417ee866c0b84': {'main'}, '3a5394eabe61df7c1b14a554772a48307987bada': {'flax'}} (🏠/.cache/huggingface/hub/models--CompVis--stable-diffusion-v1-4).

the rest of those 11 warnings being of the "Snapshots dir doesn't exist in cached repo" type.

Reproduction

import diffusers, torch
diffusers.StableDiffusionPipeline.from_pretrained("wavymulder/wavyfusion", torch_dtype=torch.float16)

huggingface-cli scan-cache -vvv

Logs

No response

System info

- huggingface_hub version: 0.11.1
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: 🏠/.huggingface/token
- Has saved token ?: True
- Who am I ?: keturn
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 1.13.0
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A

[Wauplin]

Hi @keturn, thank you for reporting. Could you print the output of

tree house/.cache/huggingface/hub/models--wavymulder--wavyfusion

please ? Not sure yet what could have happened here that makes the snapshot directory missing.

cc @patrickvonplaten @pcuenca also. Could it be that diffusers use the "old cache" somewhere ? Or that the weights are not downloaded from HF Hub but from somewhere else when having a community pipeline ?

[keturn]

the directory under hub is empty, but there's another directory under diffusers with stuff in it

directory tree

hub/models--wavymulder--wavyfusion/
diffusers/models--wavymulder--wavyfusion/
├── blobs
│   ├── 0d9d33b883843d1b370da781f3943051067e1b2c
│   ├── 12fd0e318364aada830239b20003e37f8367e63867a47aeb7a4a4559aaed6da2
│   ├── 16d28f2b37109f222cdc33620fdd262102ac32112be0352a7f77e9614b35a394
│   ├── 2165bdb8ad70303632ba0120d7744afe79fd698b40de7a2090a90f50f1a4c996
│   ├── 22c3fb762f46de16b6123feb536df65d673f3588
│   ├── 2c2130b544c0c5a72d5d00da071ba130a9800fb2
│   ├── 2c35cda84a4cbda06638649d96170add762bf300
│   ├── 30c465e533a4a3372363509a484eb2127fa54ccdae931be5a6d47c3db4c6e8dc
│   ├── 469be27c5c010538f845f518c4f5e8574c78f7c8
│   ├── 55ad9628530430eed28c01dcf3c971a29ffe5d657856cb054728b6c956da3037
│   ├── 5ba7bf706515bc60487ad0e1816b4929b82542d6
│   ├── 76e821f1b6f0a9709293c3b6b51ed90980b3166b
│   ├── 8fdc7f5cfe5fbb5da315c33eda3c3b689748ab6a
│   ├── c208b5265ccfd5d14e5c33e2223bd16b2228cef5
│   ├── dcf4507d99b88db73f3916e2a20169fe74ada6b5582e9af56cfa80f5f3141765
│   ├── dd1f095afdb20b5e3cb44e8eea607e0fc178324d
│   └── f01a58e0fd9864a28e7d4b89f24b9e42d3128cdf
├── refs
│   └── main
└── snapshots
    └── f5a728ccfc63dcff9814347f5e95739939515cf0
        ├── feature_extractor
        │   └── preprocessor_config.json -> ../../../blobs/0d9d33b883843d1b370da781f3943051067e1b2c
        ├── model_index.json -> ../../blobs/c208b5265ccfd5d14e5c33e2223bd16b2228cef5
        ├── safety_checker
        │   ├── config.json -> ../../../blobs/dd1f095afdb20b5e3cb44e8eea607e0fc178324d
        │   └── pytorch_model.bin -> ../../../blobs/16d28f2b37109f222cdc33620fdd262102ac32112be0352a7f77e9614b35a394
        ├── scheduler
        │   └── scheduler_config.json -> ../../../blobs/f01a58e0fd9864a28e7d4b89f24b9e42d3128cdf
        ├── text_encoder
        │   ├── config.json -> ../../../blobs/8fdc7f5cfe5fbb5da315c33eda3c3b689748ab6a
        │   └── pytorch_model.bin -> ../../../blobs/12fd0e318364aada830239b20003e37f8367e63867a47aeb7a4a4559aaed6da2
        ├── tokenizer
        │   ├── merges.txt -> ../../../blobs/76e821f1b6f0a9709293c3b6b51ed90980b3166b
        │   ├── special_tokens_map.json -> ../../../blobs/2c2130b544c0c5a72d5d00da071ba130a9800fb2
        │   ├── tokenizer_config.json -> ../../../blobs/5ba7bf706515bc60487ad0e1816b4929b82542d6
        │   └── vocab.json -> ../../../blobs/469be27c5c010538f845f518c4f5e8574c78f7c8
        ├── unet
        │   ├── config.json -> ../../../blobs/22c3fb762f46de16b6123feb536df65d673f3588
        │   ├── diffusion_pytorch_model.bin -> ../../../blobs/30c465e533a4a3372363509a484eb2127fa54ccdae931be5a6d47c3db4c6e8dc
        │   └── diffusion_pytorch_model.safetensors -> ../../../blobs/2165bdb8ad70303632ba0120d7744afe79fd698b40de7a2090a90f50f1a4c996
        └── vae
            ├── config.json -> ../../../blobs/2c35cda84a4cbda06638649d96170add762bf300
            ├── diffusion_pytorch_model.bin -> ../../../blobs/dcf4507d99b88db73f3916e2a20169fe74ada6b5582e9af56cfa80f5f3141765
            └── diffusion_pytorch_model.safetensors -> ../../../blobs/55ad9628530430eed28c01dcf3c971a29ffe5d657856cb054728b6c956da3037

[Wauplin]

Oh I see. Then you can run

huggingface-cli scan-cache --dir ~/.cache/huggingface/diffusers
huggingface-cli delete-cache --dir ~/.cache/huggingface/diffusers

@patrickvonplaten @pcuenca is this the default behavior in diffusers to store things in a separate cache folder ? We should think about moving it to the default cache so that downloaded weights are also part of the "normal" cache (and avoid reported bug like this one). I can help with this if you want :)

In any case, @keturn feel free to close this issue once it's resolved on your side :)

[pcuenca]

Thanks for the ping @Wauplin! I noticed the problem with scan-cache the other day but failed to follow up on it. I just opened an issue in diffusers to assess whether we can safely change it.

[Wauplin]

Closing this issue as it is now tracked in diffusers directly.