Add LoRA+ to optimizer params

[rockerBOO]

Supports LoRA+ in Kohya.

Willing to update the code, values, naming, flow and so on if required.

I was guessing at the values for LoHa and LoKr which I think is accurate and works according to my testing. Haven't tried like GLoRA yet.

Related PR for Kohya-ss: kohya-ss/sd-scripts#1233

https://arxiv.org/abs/2402.12354
https://github.com/nikhil-ghosh-berkeley/loraplus

[KohakuBlueleaf]

1. Please use old method/param groups when lora_plus_ratio is None
2. Please remove commented code

[rockerBOO]

1. Please use old method/param groups when lora_plus_ratio is None

What do you mean by use the old method/param groups? You want 2 methods for prepare_optimizer_params where it is the current and lora_plus_ratio version separated?

Or you mean the lora/plus "naming"? Those are not used past the original separation. Could be combined instead of 2 steps.

Will remove the commented out code, and have some bug fixes coming as well. Thank you.

[KohakuBlueleaf]

1. Please use old method/param groups when lora_plus_ratio is None

What do you mean by use the old method/param groups? You want 2 methods for prepare_optimizer_params where it is the current and lora_plus_ratio version separated?

Or you mean the lora/plus "naming"? Those are not used past the original separation. Could be combined instead of 2 steps.

Will remove the commented out code, and have some bug fixes coming as well. Thank you.

Basically
plz ensure your code can work with "None" as input of lora_plus params.
For now it will not work.

And what I want is basically: send single param group (or 2~3 for UNet/TE1/TE2) when lora_plus params are None

[rockerBOO]

Made some fixes and added GLoRA. Should now properly support single param group, only TE and only UNet and all LoRA+ and LR permutations.

The following test script I made to make sure it is working as expected. Willing to fix anything you find incorrect.

https://gist.github.com/rockerBOO/7e58b268d266c6f55a7f90085a1d1dfe

[rockerBOO]

Being reworked on Kohya's side a little bit. Will rehash this once that is complete.