[Bugfix] fix error of peft lora when xformers enabled#5697

patrickvonplaten merged 3 commits intohuggingface:mainhuggingface/diffusers:mainfrom

okotaku:bugfix/peft_loraokotaku/diffusers:bugfix/peft_loraCopy head branch name to clipboard