AdaLora + bnb not working

[BenjaminBossan]

System Info

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

The issue is this line:

peft/src/peft/tuners/adalora/bnb.py

Line 144 in 49ddefa

compute_dtype = lora_A.weight.dtype

In AdaLoRA, lora_A and lora_B are not ModuleDicts but ParameterDicts, so lora_A[adapter_name].weight.dtype does not exist, it should just be lora_A[adapter_name].dtype.

Furthermore, using AdaLoRA with 8bit bnb gives NaNs for me for opt-125m.

Expected behavior

AdaLoRA + bnb should work.