LazyArrays doesn't work with CuArrays 2.1.0

[simonbyrne]

Description

LazyArrays doesn't play nice with CuArrays 2.1.0

Reproducible example

using CuArrays, LazyArrays
CuArrays.allowscalar(false)
a = CuArray(rand(10,3))
W = CuArray(rand(1,3))
reduce(+, @~ W .* a; init=0.0)

See Slurm failures on #1004.

We can either fix the issue or find another way to write kernels similar to the above.

cc: @lcw @vchuravy @leios

For CLIMA Developers

• This feature can be added (if it cannot be, explain why in a comment below -- lack of technical expertise, not relevant to the scope of this project, too ambitious)
• There is a timeline for when this feature can be implemented
• The feature has been (or will be) implemented (Please link the PR)

[mwarusz]

I looked at this for a sec and based on this comment it was fairly easy to make the above example work but that approach requires a PR that was merged into Julia master this Thursday.

[simonbyrne]

The good news is that it made the feature freeze of Julia 1.5, but we're probably still a month or two away from that release.

[simonbyrne]

@mwarusz I wasn't able to run it directly on current Julia master: how did you do it?

[simonbyrne]

Upstream: JuliaGPU/GPUArrays.jl#268

[mwarusz]

I had to make a few changes to both GPUArrays and CuArrays, mainly just making functions accept Broadcasted arguments. Here they are, obviously these are far from complete and even have some code commented out.