Prior Art

Summary

These represent a range of features, including wrappers around or implementations of BLAS-like features and generic multidimensional array libraries.

Some of these projects are actively developed, some are relative static, while others are historical in nature.

C++ Universe

• Eigen
• Armadillo
• Boost.uBLAS
• Boost.MultiArray
• Blaze
• Elemental
• TBLIS
• Cyclops Tensor Framework (CTF)
• Kokkos View
• Kokkos Kernels
• Blitz++
• FreePOOMA
• LAPACK++
• Matrix Template Library (MTL4)
• GLAS: Generic Linear Algebra Software
• FLENS
• Seldon
• taco
• ALGLIB (also supports C# and Delphi)
• OpenGL Mathematics (GLM)
• Boost QVM: Quaternions, Vectors, Matrices
• linalg
• rarray
• ArrayFire
• xtensor
• PyTorch C++ API
• iTensor
• ChainerX

Other Universes

• Numpy (Python)
• PETSc (C89-based, with Matlab, Fortran, and other language bindings)
• BLIS (C-based)
• libflame (C-based)
• LAPACK (legacy Fortran)
• ScaLAPACK (legacy Fortran)
• PLASMA
• MAGMA
• MIR-GLAS (D)
• Matlab
• Fortran 95+ (sorry, I couldn't find a better website)
• TensorFlow (C and Python)
• PyTorch (Python)
• Numba (Python)

If you believe Wikipedia, see this.

Existing C++ Linear Algebra Projects Synopsis

I'd like us to give just a short synopsis of capabilities in existing C++ based Linear Algebra Libraries which address issues we listed as important for this project. I am mostly interested in design principles not completeness of math functionality.

Kokkos-Kernels

Provides BLAS, Sparse and Graph kernels on node using Kokkos. As far as I know this may be the closest existing thing to what we want to do since it already incorporates a number of key concepts (such as that it is using multi dimensional arrays with layouts as input parameters and is valid for any scalar type combination).

• C++ based interface using Kokkos::View (i.e. what mdspan is mostly based off).

  • template<class AViewType,
         class XViewType,
         class YViewType>
    void gemv (const char trans[],
      typename AViewType::const_value_type& alpha, const AViewType& A,
      const XViewType& x, typename YViewType::const_value_type& beta, const YViewType& y);

• Can accept any Scalar type combination (i.e. AViewType, XViewType, YViewType can have different Scalar types)
• Can accept any layouts (e.g. XViewType can be a non-contigues subarray for example a row in a column major matrix)
• Is memory space aware. Kokkos::View has a memory space template parameter, which in mdspan could be incorporated via the accessor_property. Chooses execution mechanism (CUDA,OpenMP,etc.) based on memory space.
• Can call TPLs (MKL,CUBLAS etc.) if scalar type, data layout and memory space match the TPL capabilities.
• Starts to provide nested BLAS. e.g. calling BLAS with a CUDA-Block, or an OpenMP team of threads.

   gemv(team_handle, 'N', alpha, A, x, beta, y);

• Biggest gap from our perspective (i.e. this activity here): no expression template stuff at this point, that means no kernel fusion other than the ones which are explicitly provided.

Blitz++

Blitz++ was the first C++ library to use expression templates and set the ground for other libraries like uBLAS. Expression templates are a mechanism to construct the algebraic expression syntax tree during compilation. The syntax tree is available to the library implementation for performing certain optimizations. The optimizations have been shown to directly rival Fortran array operation performance: link. Most of the popular high performing C++ linear algebra libraries today use an expression templates mechanism.