`test_to_pyarrow` tests fail during release verification

[alamb]

Describe the bug
When verifying the release candidate

• Release arrow-rs / parquet Minor version 55.2.0 (June 2025) #7394

running 2 tests
test test_to_pyarrow_byte_view ... FAILED
test test_to_pyarrow ... FAILED

failures:

---- test_to_pyarrow_byte_view stdout ----
input: RecordBatch { schema: Schema { fields: [Field { name: "string_view", data_type: Utf8View, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "binary_view", data_type: BinaryView, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [StringViewArray
[
  "inlined",
], BinaryViewArray
[
  [105, 110, 108, 105, 110, 101, 100],
]], row_count: 1 }

thread 'test_to_pyarrow_byte_view' panicked at arrow-pyarrow/tests/pyarrow.rs:69:10:
called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'pyarrow'"), traceback: None }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- test_to_pyarrow stdout ----
input: RecordBatch { schema: Schema { fields: [Field { name: "a", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "b", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "c", data_type: Utf8View, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [PrimitiveArray<Int32>
[
  1,
  2,
], StringArray
[
  "a",
  "b",
], StringViewArray
[
  "short",
  "a very long string",
]], row_count: 2 }

thread 'test_to_pyarrow' panicked at arrow-pyarrow/tests/pyarrow.rs:43:6:
called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'pyarrow'"), traceback: None }


failures:
    test_to_pyarrow
    test_to_pyarrow_byte_view

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

error: test failed, to rerun pass `-p arrow-pyarrow --test pyarrow`
+ cleanup
+ '[' no = yes ']'
+ echo 'Failed to verify release candidate. See /var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/arrow-55.2.0.XXXXX.85iutp2Fct for details.'
Failed to verify release candidate. See /var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/arrow-55.2.0.XXXXX.85iutp2Fct for details.

To Reproduce

./dev/release/verify-release-candidate.sh 55.2.0 1

Expected behavior
Tests should pass

Additional context
I suspect this is related to the following changes:

• Define a "arrow-pyrarrow" crate to implement the "pyarrow" feature. #7694

[alamb]

It fails locally with this command:

cargo test -p arrow-pyarrow --test pyarrow

[brunal]

This is likely caused by a "cargo test": that will run the tests of all crates in the workspace, including "pyarrow". However "pyarrow" needs specific setup (pip install pyarrow). In .github/workflows/rust.yml, I had to update the "cargo test" to "cargo test --workspace --exclude arrow-pyarrow" to fix that.

It was not an issue before the MR the "pyarrow" feature had to be enabled for "cargo test" to run those tests.

I'm not sure where the rogue "cargo test" is though -- I didn't see anyrthing suspicious under .github/.

[alamb]

I think the cargo test in question is run via this

cargo test --all

arrow-rs/dev/release/verify-release-candidate.sh

Line 116 in df702cf

cargo test --all

I am trying to figure out how to get the test to pass for me locally / document how to do so and then I'll probably update the release verification script.

Thanks @brunal

[brunal]

Indeed! Replacing it with cargo --workspace --exclude arrow-pyarrow will fix the issue (and restores the prévious behavior).

I have a commit with it on my fork, but i can't send a proper PR before tonight.

[alamb]

Indeed! Replacing it with cargo --workspace --exclude arrow-pyarrow will fix the issue (and restores the prévious behavior).

I have a commit with it on my fork, but i can't send a proper PR before tonight.

Thank you @brunal

I have a slightly more involved fix here:

• Move arrow-pyarrow tests that require pyarrow to be installed into arrow-pyarrow-testing crate #7742

I will also make a small PR to exclude the arrow-pyarrow package from the verification script to unblock this release

[alamb]

Here is a PR to fix the script:

• [55_maintenance] Fix verify-release-candidate.sh by skipping arrow-pyarrow and parquet-variant #7745