Use value indexing by default

[andreasnoack]

In the single use case I have for AxisArrays right now, it would be very convenient if the default indexing of an AxisArray used the values of the indices instead of the indices of the indices. E.g. if I do

x = AxisArray(Axis{:x}(-3:3))
x[[-2,1]]

it would be handy if it returned the second and fifth element of the AxisVector. You can always refer to the indices of the indices by using the Axis type. Could this work?

[mlubin]

I've started hacking on this on a fork here.

[mbauman]

I'm coming to the conclusion that supporting both by-axis-value and by-axis-index within the A[] is fraught with trouble. I think it's crucial to look at prior art here and decisions others have made.

Pandas used to support four different ways of indexing into their dataframes: df[], df.ix[], df.iloc, and df.loc. iloc is by index and loc is by value. ix used to do both, changing based upon the type of the index. They found this too confusing and decided to deprecate it completely. The more similar project for Python, xarray, has been developed more recently, and they, too, decided to completely detangle things here: http://xarray.pydata.org/en/stable/indexing.html.

If we decide to completely detangle these things we have two decisions to make:

• What should the default A[] syntax do? Indices or labels?
• How should we toggle between the two? We have several options here: it could be per-dimension with a macro or wrapper (like @v), or now that we have getproperty we could do the pythonic thing and use a "index accessor" property like A.iloc[]. The downside there is that it applies to all dimensions. When I dealt with the Pandas deprecation of ix, I found that it was most commonly used in cases like df.ix['label', 0] — that is, one dimension by label and one dimension by offset.

[Petr-Hlavenka]

I'm an user of AxisArrays and I like its approach a lot. On the other hand I'm just an average user with little knowledge of the inner functionality, so regarding the architectural decisions i fully trust the people like M. Bauman and Tim Holy. This post is only to show how I'm using them and why I find this package great.

I'm almost always using the AxisArrays package in combination with Unitful or with Images. In both cases I use AxisArray as a way to annotate physical meaning of the Integer indexes and dimensions. The huge advantage for me is to use AxisArray for steering the dispatch into dedicated functions, and the dispatch is really pleasure to write thanks to rdeits suggestion in discourse with constructs like

const QAxis{Name, Dim} = Axis{Name, AX} where AX <: AbstractArray{Q} where Q <: Dim
const TimeAxis = QAxis{S, Unitful.Time} where S;
const TimeArray{T, N, V} = AxisArray{T, N, V, <:Tuple{TimeAxis, Vararg{Axis}}};

const QAxisW{Name, Dim} = Axis{Name, AX} where AX <: Range{Q} where Q <: Dim
const TimeAxisW = QAxisW{S, Unitful.Time} where S;
const TimeWaveformArray{T, N, V} = AxisArray{T, N, V, <:Tuple{TimeAxisW, Vararg{Axis}}};
const TimeWaveform{T, V} = TimeWaveformArray{T, 1, V}

const EnergyAxis = QAxis{S, Unitful.Energy} where S;
const EnergySpectrumArray{T, N, V} = AxisArray{T, N, V, <:Tuple{EnergyAxis, Vararg{Axis}}};

and this definitions I use to define specialized functions - some of them expecting Axis of StepRange kind (FIR, image processing, ...), other can work with arbitrary axis values (after e.g removal of outliers) such as plotting. I know I can define my own type and do the dispatch on its type - but this way it is for free, all playing just well with the rest of Julia.

A second usecase is when I use it instead of DataFrames - just that accessing any column of my measurement dataset is so simple, my main axis (time, or energy, or length) is still present and propagated - but I can pass it to any function expecting an AbstractArray and it just works without explicitly writing any conversions, any macro, just pure math-like syntax.

So my opinion is:

• AxisArray is a AbstractArray thus default indexing without axis specification arr[1, 1], arr[:, 1], arr[1:end-5, 1] should be of the underlying array (I've even an use case I use a sparse array for spectrum representation in a huge datacube - works great!)
• AxisArray is accessed by "dimension name" be it :Red, :time -> kind of DataFrames-light use case
• AxisArray is accessed by "dimension name" and an Integer index -> again index into the underlying array
• Any access to index or range through the Axis mapping is an opt-in for methods that expect and rely on the added information provided by the Axes, can dispatch only on a specific type of Axis and can opt-in a different way of indexing, e.g. what you propose with index accessor arr.ax[:time, 5u"ns"], arr.ax[5u"ns"], arr.ax[5u"ns"..15u"ns"] for explicit axis at given value, implicit axis on value and for implicit axis by range.

My feeling is, the three characters .ax to make sure indexing is by label is wort typing and not causing big pain for readability.

But again, don;t put too much weight to my examples and suggestions.

[timholy]

Since the upgrade-to-0.7 train is at full speed now and the exported name axes conflicts with Base, this issue has gone critical. Time to make a decision.

My proposal is to orthogonalize the concepts as much as possible, using the following new packages:

• NamedAxisIndexing: all this does is map a name to a dimension slot. It's effectively like Base.PermutedDimsArray except the axis names can be anything. Any other behavior is a property of the object you wrap. This package could have one wrapper NamedAxisArray <: AbstractArray (for integer-indexing) and another NamedAxisDict <: AbstractDict (for value indexing). (pity we don't have traits, then we could have just one wrapper...)
• RangedAxisArrays: this adds range metadata to each axis. These are still AbstractArrays and you index them with integers. The axis metadata is indexed by slot number rather than symbol or anything else.
• People who want to index by value can create a new package or standardize on one of the others (AssociativeArrays, NamedArrays, DataCubes, etc.)

Either of the latter two can be combined with NamedAxisIndexing at the user's or package author's option. That way everyone gets to leverage common functionality without there being so much angst over what things should actually mean.

We might need a small glue package (AnnotatedAxisArrays?) that defines trivial implementations of core trait functions that all of these packages can extend for their specific types.

[iamed2]

I'm not sure what the difference between NamedAxisArray and NamedAxisDict is. Are these for the data arrays or for axes?

[timholy]

The key point is the subtype relationship, NamedAxisArray <: AbstractArray. If you create a container that indexes by axis-range values (e.g., a[3.2s, 4.4mm]), it's not so obvious that you should be calling it a subtype of AbstractArray since we make the assumption in lots of places that you can index them by integers. But you could call that kind of object an AbstractDict.

[iamed2]

It sounds like NamedAxis* are for mapping dimension names to dimension numbers only. In that case, shouldn't the wrapper not care how indexing works on the wrapped structure, and just generated reordered calls to getindex and setindex? Or is there extra functionality in this hypothetical package that I'm missing?

[timholy]

You have to make a decision about subtyping when you define the structure:

struct NamedAxisArray{T,N,A<:AbstractArray,S<:NTuple{N,Symbol}} <: AbstractArray{T,N}
    data::A
end

# Anything that isn't an Array but supports getindex/setindex!
struct NamedAxisDict{K,V,D,S<:Tuple{Vararg{Symbol}} <: AbstractDict{K,V}
    data::D
end

but the implementations of getindex and setindex! are identical for both types. It's really just so that the wrapped objects have the appropriate isa relationships.

Like I said, if we were trait-based rather than inheritance-based then this wouldn't be an issue, since the isarray trait could be defined isarray(W::NamedAxisWrapper) = isarray(W.data).

[timholy]

Hmm, though I realized there's still nothing to prevent someone for asking for value-based lookup with a RangedAxisArray. Really, is there anything wrong with things the way they are now? We have atvalue and atindex, these do allow one to index by value unambiguously and mix integer-indexing and value-indexing freely. Now I'm a little less certain we need to do the split in #84 (comment).

Overall I have the perception seems that existential angst about this package not quite being right is holding it back. Personally, I don't share this angst because I really do think this package should just be an axis metadata wrapper.