Deliver pyarrow as a standalone crate

[brunal]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I have a codebase that heavily uses arrow. I offer bindings in multiple languages.

For python, this means I depend on the pyarrow feature. But for other languages (e.g. C++), this is very undesirable: I don't want my C++ library to depend on pyarrow and pyo3 and a python shared library. Due to feature unification, when I build everything at once, my C++ bindings end up with a dependency on python.

Describe the solution you'd like

If pyarrow were a standalone crate, then my python bindings would depend on it, but nothing else would.

Describe alternatives you've considered

Vendor arrow-rs, providing myself pyarrow as a crate in my codebase.

Additional context

For a glance at the pyarrow feature, its scope is relatively limited, and extracting it into its own crate is relatively easy. pyarrow could remain a feature of the arrow crate, and users could either use it just like now or as a standalone crate.

I have a fork that does it at https://github.com/brunal/arrow-rs/commits/standalone-pyarrow/. LMK if a pull request would be welcome.

[linhr]

+1 for the idea!

I feel having a standalone crate for PyArrow code would help Rust+Python projects as well.

My use case depends on Arrow and DataFusion and uses PyO3 for the Python binding. I have multiple Python environments in my development flow. Every time I switch Python environment, the PyO3 package is rebuilt. Since I use the pyarrow feature in the arrow crate, the arrow crate is rebuilt as well, which in turn causes DataFusion to be rebuilt. If we have a separate crate for PyArrow code, the compilation time would be greatly reduced in this case.

[kylebarron]

You might also be interested in my standalone crate pyo3-arrow. There's some documentation about what it does differently than the pyarrow feature:

• arrow-rs's Python FFI integration does not support Arrow extension types, because it omits field metadata when constructing an Arc<dyn Array>. pyo3-arrow gets around this by storing both an ArrayRef (Arc<dyn Array>) and a FieldRef (Arc<Field>) in a PyArray struct.
• arrow-rs has no ability to work with an Arrow stream of bare arrays that are not record batches, and so arrow-rs has no way to interop with a pyarrow.ChunkedArray or polars.Series.
• pyo3-arrow will automatically convert Python buffer protocol objects (such as numpy arrays, memoryviews, bytes objects, and more) to PyArray. This conversion is zero copy.
• arrow-rs only supports returning data to pyarrow. Pyarrow is a very large dependency (its unpacked Linux wheels are 130MB, not including a required dependency on Numpy) and some projects may wish not to use it. Now that the Arrow PyCapsule interface exists, it's possible to have a modular approach, where a very small library contains core Arrow objects, and works seamlessly with other libraries.
• pyo3-arrow implements schema negotiation for the PyCapsule Interface on data export, allowing other Python libraries to request Arrow data types they know how to handle.
• In my opinion arrow-rs is too tightly connected to pyo3 and pyarrow. pyo3 releases don't line up with arrow-rs's release cadence, which means it could be a bit of a wait to use the latest pyo3 version with arrow-rs, especially with arrow-rs waiting longer to release breaking changes.

[alamb]

See PR here:

• Define a "arrow-pyrarrow" crate to implement the "pyarrow" feature. #7694

[alamb]

label_issue.py automatically added labels {'arrow'} from #7694