gaussian_nll_loss doesn't work on GPU

[pkese]

TorchSharp/src/TorchSharp/NN/Losses.cs

Line 422 in ba2fa75

if ((variance < 0).any().item<bool>()) throw new ArgumentException("variance has negative entry/entries");

I had to change

if ((variance < 0).any().item<bool>())
    throw new ArgumentException("variance has negative entry/entries");

into

if ((variance < 0).any().to(DeviceType.CPU).item<bool>())
    throw new ArgumentException("variance has negative entry/entries");

in order to make it work on a GPU, because one can't extract an item() from tensor unless the tensor resides on CPU.

However when looking at this code, I think this whole line is unnecessary (if variance is less than zero it gets clamped to eps in the next line) as well as slow, as it requires synchronizing GPU state and copying data back to CPU which is inefficient and causes extra I/O latency.

Also

variance = variance.clone().maximum(torch.tensor(eps))

causes an unnecessary cloning of tensor (maximum is supposed to return a new tensor anyway) and tensor allocation for eps.
I suggest to replace that with:

variance = variance.clamp_min(eps);

[pkese]

It seems however that Python implementation is also similarly inefficient

https://github.com/pytorch/pytorch/blob/5953fd9133c0bdcc0158acf1472fac403bc5f636/torch/nn/functional.py#L2819

... so the inefficiency thing is up for discussion (how about pushing such costly checks into #if DEBUG),
but the GPU error still needs to be fixed.

[NiklasGustafsson]

However when looking at this code, I think this whole line is unnecessary

Yes, unless you want to flag as an error that the variance is negative -- it might be a sign of something bad going on. That said, I'm happy to get rid of any inefficiency, and thanks for the clamp_min tip.

[NiklasGustafsson]

@pkese - I have a new release going out tonight (urgent, due to missing dependency). I'll add the clamp_min and cpu() fix, but I won't remove the error checking. If you think that is wrong, please open another issue.

[pkese]

Sure, that is fine.

I'll try to make a small demo project to measure how much such additional checking affects performance and then open a discussion.