Add support for `OptimizerParamGroup`

[lostmsu]

OptimizerParamGroup is used to set different learning rates, weight decay rates, and other optimizer properties differently for different model parameters.

Exmaple: AdamW(vector<OptimizerParamGroup> param_groups, AdamWOptions defaults)

[NiklasGustafsson]

@lostmsu -- I'm going to pick up work on this next. It'll take a little bit, because it affects all the optimizers, but the work itself is rather straight-forward.

[NiklasGustafsson]

I took a tangent to fix a few other issues that seemed related and more significant -- errors, rather than missing features. Anyway, I'm working on this one now.

[NiklasGustafsson]

@lostmsu -- I'm experimenting with the API.

Here's the current idea:

var lin1 = Linear(1000, 100);
var lin2 = Linear(100, 10);
var seq = Sequential(("lin1", lin1), ("relu1", ReLU()), ("lin2", lin2));

var pgs = new Modules.ParameterGroup[] {
      new Modules.ParameterGroup { Parameters = lin1.parameters(), Options = new ASGDOptimizer.OptimizerOptions { LearningRate = 0.005 } },
      new Modules.ParameterGroup { Parameters = lin2.parameters() } };

var optimizer = torch.optim.ASGD(pgs);

Thoughts?

[lostmsu]

@NiklasGustafsson LGTM

Just thoughts on cleaning up usage:

1. In the array creation you should be able to use target-typed new
2. Maybe have a generic ParameterGroup<TOptions> and initialize the corresponding Options property so that you could write new { Parameters = Parameters, Options = { LearningRate = 0.005 } }. Optionally ASGD would take specific instantination of ParameterGroup<TOptions> then.
3. Have a ParameterGroup constructor, that accepts parameters and optional options - there does not seem to be any possibility for parameter confusion because types are different. E.g. new ParameterGroup(params, new ASGDOptimizer.Options(...))

[NiklasGustafsson]

The cleanest I seem able to get it with my knowledge of how target-typed context works seems to be:

            var pgs = new [] {
                new ParamsGroup<ASGDOptimizer.Options> { Parameters = lin1.parameters(), Options = { LearningRate = 0.005f } },
                new ParamsGroup<ASGDOptimizer.Options> { Parameters = lin2.parameters() }
            };

The compiler doesn't like my removing the new ParamsGroup<ASGDOptimizer.Options> bit inside an array initializer.

[lostmsu]

@NiklasGustafsson try new () { Parameters = ... }

E.g. this works:

new List<string>[] {
  new (){"a", "b"}
};

but this does not

new List<string>[]{
    new {"a", "b"} // <- no () after new
};

[NiklasGustafsson]

Yes, the following works, and is slightly simpler:

var pgs = new ParamsGroup<ASGDOptimizer.Options>[]
            {
                new () { Parameters = lin1.parameters(), Options = { LearningRate = 0.005f } },
                new () { Parameters = lin2.parameters() }
            };

[NiklasGustafsson]

Which can be further simplified by renaming the optimizer class. This should be alright, since the name of the factory method that should be the natural way of creating an optimizer doesn't change:

            var pgs = new ParamsGroup<ASGD.Options>[]
            {
                new () { Parameters = lin1.parameters(), Options = { LearningRate = 0.005f } },
                new () { Parameters = lin2.parameters() }
            };

I'll need to check how this looks in F#, too. Just as important to get the F# UX right...

[lostmsu]

@NiklasGustafsson what about constructors?

[NiklasGustafsson]

@NiklasGustafsson what about constructors?

That also works:

var pgs = new[]
            {
                new ParamsGroup<ASGD.Options>(lin1.parameters(), new () { LearningRate = 0.005f }),
                new ParamsGroup<ASGD.Options>(lin2.parameters())
            };

[dsyme]

I suggested an alternative here: NiklasGustafsson#2 (comment)
F#:

    let optimizer = SGD([|
        SGD.Parameters(model.parameters(), lr = 0.005, momentum = 1.0, dampening = 0.5);
        SGD.Parameters(model.parameters(), momentum = 1.0, dampening = 0.5)
    |], lr)

C#

            var pgs = new SGD.Parameters[] {
                new (lin1.parameters(), lr: 0.005f),
                new (lin2.parameters()) };

            var x = torch.randn(new long[] { 64, 1000 });
            var y = torch.randn(new long[] { 64, 10 });

            double learning_rate = 0.00004f;
            var optimizer = torch.optim.SGD(pgs, learning_rate);

[NiklasGustafsson]

@dsyme -- I agree, that is definitely cleaner than what I came up with.

[lostmsu]

Looks confusing to me, as those are not really parameters, but parameter groups.