Store cached models in location compatible with `huggingface-cli`

[pcuenca]

Reference: huggingface/huggingface_hub#1259

huggingface-cli scan-cache doesn't see cached diffusers models. Are there any drawbacks to changing the cache folder?

[patrickvonplaten]

Yes we could indeed change the cache here! This was a copy-paste back then from the transformers repo! Would you like to open a PR to change the default cache @pcuenca ? Otherwise happy to look into it in a couple of days.

[Wauplin]

I support the idea to get everything under $HF_HOME/hub/ so that users can benefit from huggingface_hub tool. Also beneficial if a user downloads some weights outside of diffusers for some reason, the cache will be shared. This is not the case at the moment.

Also note that if there are some stuff cached but not from the Hub (preprocessed data, community weights,...) it would make sense to use assets cache (see docs). I can help with that if needed :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

cc @pcuenca would you like to tackle this one? Otherwise, I should be able to find some time

[pcuenca]

Yeah, I'll take a look today.