WIP: The Plan

[iamed2]

This issue is still being written

See discussion in JuliaArrays/AxisArrays.jl#84 for background

Concepts

AxisArrays.jl is currently a mashup of distinct and mostly orthogonal indexing behaviour. These are, at the highest level:

• associating a name with a dimension
• associating an index mapping with a dimension
• looking up an index in an index mapping
• translating a getindex call using some transformation into a getindex call on an underlying structure, then constructing a wrapped result

Named Dimensions

Index Lookup

Indexed Dimensions

getindex Translation

[oxinabox]

Cc : @blegat , @juan-pablo-vielma

[iamed2]

I plan to continue adding to this as soon as I make some progress on my current task.

[oxinabox]

Named Dimensions

These are wanted for Neural networks mostly, to make sure that you are summing and batching etc over the dimension you think you are.
See: http://nlp.seas.harvard.edu/NamedTensor
(In contrast Indexed Dimensions, are wanted for making sure the whole DataScience pipeline is correct)

• Wrapper Type, that wraps an AbstractArray giving names to the indexes. The index named can go in the type params
• Mapping function: name2dim(::NamedDimsArray, ::Symbol)::Int that will take an array with named dimentsions and dimention name, and return its dimension index.
• Overloads for common functions that take a dims argument. Like sum and mean
• kwarg style indexing feature request: kwargs for getindex JuliaLang/julia#25631

Most of this functionality should be implemented in a kind of traitish way so that
we can easily write code for the various different combinations of
Indexed Dimension, Named Dimension and Indexed Named Dimension

[Tokazama]

I just started looking through the NamedDims repo, so I apologize if this is already documented but I missed it. Is NamedDims ultimately intended to be integrated into AxisArrays or is it suppose to be a dependency or something else entirely?

Also, I think https://github.com/JuliaDiffEq/LabelledArrays.jl implements a really simple method of indexing for symbols that can be adapted to indices. I know the comments in https://github.com/invenia/NamedDims.jl/blob/master/src/name_core.jl#L66 indicate can run at compile time but I haven't really stress tested NamedDims for speed yet.

I have a mostly finished version of all the statically typed ranges in base and tested against a lot of the range tests in Base https://github.com/Tokazama/StaticRanges.jl. I'm hoping to figure out a mix of dynamic and static indexing in a week or two when classes finish up here. I figured this may be helpful given how much of AxisArrays's code is dedicated to figuring out indexing.

[oxinabox]

I just started looking through the NamedDims repo, so I apologize if this is already documented but I missed it. Is NamedDims ultimately intended to be integrated into AxisArrays or is it suppose to be a dependency or something else entirely?

NamedDims.jl is intended to A) Be used on its own,
and B) Be integrated into a future package along with IndexedDims.jl (name pending, Indexes.jl?),
and that future package will replace AxisArrays.jl. (Like how StaticArrays.jl, replaced FixedSizedArrays.jl)

So if you only want Named Dimensions and nothing else.
then it is fine to just used NamedDim.jl

Also, I think https://github.com/JuliaDiffEq/LabelledArrays.jl implements a really simple method of indexing for symbols that can be adapted to indices.

Probably. It is a different thing again to either of the packages we're talk about (as I think you know.)
(as an aside: one does have to be careful about referring to index vs dim.)
But Indexes.jl (name pending), needs to support a lot more than symbols so I am not so much seeing what ideas you think are worth adapting.
But I've not really looked at the source.

I know the comments in https://github.com/invenia/NamedDims.jl/blob/master/src/name_core.jl#L66 indicate can run at compile time but I haven't really stress tested NamedDims for speed yet.

As of
invenia/NamedDims.jl@e60f5ea
basically everything is effectively zero-cost
No overhead beyond that of the underlying operations, as they would have been on the wrapped array type.

I have a mostly finished version of all the statically typed ranges in base and tested against a lot of the range tests in Base https://github.com/Tokazama/StaticRanges.jl. I'm hoping to figure out a mix of dynamic and static indexing in a week or two when classes finish up here. I figured this may be helpful given how much of AxisArrays's code is dedicated to figuring out indexing.

This looks interesting and might fill some, or all the pieces that the proposed Indexes.jl wants to do.

---

In general I am currently focussing on NamedDims, because they solve the bugs that had in my code related to various incorrect dimensions being summed and then std dev'd over.
(The sum of the std devs, is not the std dev of the sums 😬)

Indexes.jl will to solve the other kinds of mistake.

[iamed2]

On Monday, @oxinabox pointed to https://github.com/andyferris/AcceleratedArrays.jl as a candidate for the index lookup aspect and I think it's a great choice and I've been running with it on what I'm calling IndexedDims.jl. My goal for the JuliaCon hackathon tomorrow is to get a reasonable prototype of the indexed dimensions aspect functional, which may not have all of its necessary performance optimizations and may not actually overload getindex yet.

Key issues that require attention with this approach:

• selecting a subset of a dimension requires reindexing (re-accelerating), meaning the original index is lost and a new one is constructed anew. It would be nice if perhaps AcceleratedArrays supported intelligent reindexing, so that a new accelerated array can be constructed through indexing without duplicating work in some scenarios (like indexing into an UniqueSortIndex-ed UnitRange with a UnitRange)
• trying to split indexing into two separate styles is difficult, especially since there's no way to notate another indexing style without macros (due to setindex being a special kind of parsed expression). We do need to be able to index into the underlying array with Ints to participate in the generic AbstractArray interface, but this makes Ints special and hard to handle in the special indexing algorithm (same story for logical masks). Not sure how to solve this yet.
• it's theoretically possible to handle APL-style multidimensional indexing but I'm punting on that for now as AxisArrays doesn't do it and it's hard. But what should happen when unsupported generic indexing patterns are used? Should we decay to generic AbstractArray indexing, or give a method error?
• How much can we avoid implementing? Are we okay with just getindex and similar, or do we need to handle stuff like copyto! as well? How about broadcasting; what do we need?
• How much type stability do we need? Is Julia better at dealing with partial type stability than in the 0.4 era? In which cases (user-facing method calls) can we not know the output type based on the input type?

[c42f]

It's great to see this issue and some progress on replacing AxisArrays with a set of more minimal orthogonal functionality. Lately I've done a couple of cleanups over at AxisArrays but the package is basically unmaintained and I don't really want to be the new maintainer :-) Partly it's a matter of time, but even more importantly I can't see how the current AxisArrays design can move forward in a clear way. At least, not without removing features.

[rafaqz]

This might be of interest: https://github.com/rafaqz/DimensionalData.jl

It's a mix of AxisArays.jl and NamedDims.jl. Basically as @c42f said above the things I wanted aren't possible without a serious refactor of AxisArrays. Mostly its easier to extend and the syntax is more flexible for using in other packages. I wrote it for spatial data, and its extended in https://github.com/rafaqz/GeoData.jl

Edit: The list of concepts above is basically identical to the concepts in DimensionalData.jl. The main point of difference is the stress on always using abstract types and method dispatch in DimensionalData - inherited behaviour instead of wrapper types.

A few other points relating to the list of @iamed2 and DimensionalData :

• Using AcceleratedArrays would be great, but I have the same concern about the rebuild - it would be preferable if it was lazy or subsetted intelligently.
• In DD normal getindex is the primary indexing method, the dimensions and selectors just return indices to getindex which works as usual. Extending packages don't need to know about named dims or selectors, they just need to support getindex and a dims() method.
• DD methods seem to be type stable more often than the Base methods, as it has more type information
• Multiple indexing styles coexist by using type wrappers - there are 4 styles of getindex:
  • regular getindex a[1, 2:3]
  • dim indexing where order doesn't matter a[X(2:3), Y(1)]
  • value selector indexing a[At(34.5), Between(70, 90)]
  • dim/value selector indexing a[X<|Between(70, 90), Y<|Near(34)]
    (<| just reduces parentheses)

Its a little verbose in the last case. but I can't think of how to simplify it while maintaining the flexibility.

Also I've just registered DD as I have a whole chain of packages depending on it, but it would be great not to maintain it on my own in future, and I'm keen to integrate the functionality into a future AxisArrays if it's possible.

[kcajf]

@rafaqz DimensionalData looks nice, have been playing around with it a bit.

I've started a discussion over at AcceleratedArrays - andyferris/AcceleratedArrays.jl#4 - to try and make rebuilding, slicing and dicing AcceleratedArrays cheaper. I currently have my own shoddy equivalents of UniqueHashIndex and SortedIndex but it would be great to have a common layer on which to build, and AcceleratedArrays seems the furthest ahead so far afaict.