Add configuration option for test executor arguments #2636

freakboy3742 · 2025-10-22T07:14:26Z

iOS and Android test suites are executed using a testbed app. This app needs to be started, and the test arguments (the value of test_command) is passed to that testbed.

However, the testbed app itself has configuration arguments, and it can sometimes be useful to be able to use these arguments.

This issue has become mildly critical as a result of the recent changes to the macos-15 test runner. At present, the simulator that is picked by default by the iOS test runner is entirely valid from the perspective of Xcode, but then refuses to launch.

This is a known issue with the macOS-15 runner that GitHub is investigating; but it's not a once-off issue. When Xcode 16.4 was released, it removed the "iPhone SE" simulator as a default image, and as there was no way to configure the simulator used at runtime, a full update of the iOS support packages and a cibuildwheel release was necessary to resolve the issue.

So - this PR adds a CIBW_TEST_EXECUTION_ARGS setting (with related platform-specific alternatives). These arguments are passed to the testbed runner on iOS and Android.

As Android requires explicitly specifying the simulator to use as part of the runner arguments, the value of CIBW_TEST_EXECUTION_ARGS is lightly processed. If there's a --managed or --connected value in CIBW_TEST_EXECUTION_ARGS, that value will be used by the testbed runner; if no such argument exists, --managed maxVersion will be used by the test runner.

I'm not entirely sure how best to test this; it's in a portion of the code that can't be easily tested without actually starting simulators, and... well... the problem that led to this PR is that we can't use other simulators :-) So - I've gone with testing this empirically through the Github Actions configuration, passing in the necessary arguments so that on GitHub, simulators are specified on iOS and Android x86 simulator. On other platforms, the default behavior will continue to operate.

cibuildwheel/platforms/android.py

mhsmith · 2025-10-22T13:45:47Z

docs/options.md

+### `test-execution-args` {: #test-execution-args toml env-var }
+
+> Define additional arguments that will be passed to the command that runs the tests.


test-execution-args still sounds like they might be arguments to the tests themselves. Would test-runner-args be a clearer name?

I picked test-execution-args based on the discussion that lead to this PR; but test-runner-args would also work. The exact naming isn't a hill I'll die on, and it's easy to change; would be interested in hearing other opinions.

I had considered test-runner-... too, but I didn't like it because to my mind 'pytest' is a test runner, and we're not talking about that. Ideally, the name would evoke more 'environment' or 'context' in its flavour, but nothing fit that didn't already have a meaning. So I landed on 'execution', which feels higher-level than runner(?).

Here are some other options:

"testbed" is the term we've used for both Android and iOS, but that wouldn't be appropriate for a Docker container, as mentioned in the linked discussion.

"container" is what Docker would call it, but that implies a certain level of isolation which we don't necessarily provide.

"launcher" seems like a fairly generic way of saying "the thing that starts (but doesn't "run") the test suite".

On the other hand, if this option could have additional sub-keys in the future as suggested below, then maybe a general name like "execution" is OK.

Yeah, if I'm completely honest, I'm still not in love with 'test-execution'. Seeing it in the sidebar, it really gives no hints about what it does-

I'm now coming back around to test-target, test-host or test-runtime. Actually test-target looks quite sensible in the options sidebar-

I'm curious what resonates with you? bearing in mind I could also imagine this being used to spec a docker image, or set some pyodide/node options.

(apologies to go around the houses with this @freakboy3742 !)

I'm curious what resonates with you? bearing in mind I could also imagine this being used to spec a docker image, or set some pyodide/node options.

Of those options, test-runtime makes the most sense to me.

test-target sounds like "where do I want to run the tests?" rather than "how will I run this test on a target?" (and similar for test-host, with the added benefit of the confusion of whether the "host" is the architecture running the build, or the architecture you're building for).test-runtime sounds like the one that is most directly tied to "what runtime will I use for the tests?" - which is essentially what we're configuring, whether that's iOS, Android or Docker.

(apologies to go around the houses with this @freakboy3742 !)

No problems at all - it's an easy set of changes to make, and I'd rather go round a few times now and get it right than have to walk back a change in a couple of weeks/months.

test-runtime seems like a reasonable name to me as well.

Thanks both for your input! Okay, that feels like consensus, let's run with test-runtime :)

cibuildwheel/platforms/ios.py

.github/workflows/test.yml

docs/platforms.md

Co-authored-by: Malcolm Smith <[email protected]>

…the default args to the test runner.

docs/platforms.md

joerick

Thank you for your work on this! It looks good. A few comments-

joerick · 2025-10-25T13:52:49Z

docs/options.md

+### `test-execution-args` {: #test-execution-args toml env-var }
+
+> Define additional arguments that will be passed to the command that runs the tests.


I had considered test-runner-... too, but I didn't like it because to my mind 'pytest' is a test runner, and we're not talking about that. Ideally, the name would evoke more 'environment' or 'context' in its flavour, but nothing fit that didn't already have a meaning. So I landed on 'execution', which feels higher-level than runner(?).

docs/options.md

cibuildwheel/platforms/android.py

.github/workflows/test.yml

cibuildwheel/options.py

freakboy3742 · 2025-10-28T10:09:02Z

@joerick I've just pushed an update that includes:

Moving to test-execution = { args = [...] } as the syntax for configuration
Adding a default simulator on iOS when running under Github Actions on a macos-15 runner.
Links to the iOS and Android test runner documentation.

I've kept the explicit use of TEST_EXECUTION in the GitHub Actions CI configuration, as it's a viable "live test" of the new functionality.

joerick

Thanks for these latest changes - just a couple things...

joerick · 2025-10-29T11:57:18Z

cibuildwheel/platforms/ios.py

+                                test_execution_args = [
+                                    "--simulator",
+                                    "iPhone 16e,OS=18.5",
+                                    *test_execution_args,
+                                ]


It would be nice if the user could pass test-execution = "args: --simulator auto" to activate the select_simulator_device(platform) function from the test script. But I suppose that's a CPython change. Not essential, this is okay, it's probably a temporary issues anyway.

Yes - adding auto would require an upstream CPython change; and "auto" is already the default behavior for the iOS test runner.

This is only needed as a workaround for the current set of GitHub Actions weirdness - and the 2 month release cadence of CPython definitely doesn't lend itself to hard-coding defaults like this.

docs/options.md

Co-authored-by: Joe Rickerby <[email protected]>

mhsmith · 2025-10-30T09:03:15Z

docs/options.md

+    CIBW_EXECUTION_IOS: "args: --simulator='iPhone 16e,OS=18.5'"
+
+    # Run the Android tests on the minimum supported Android version.
+    CIBW_EXECUTION_ANDROID: "args: --managed minVersion"


Suggested change

CIBW_EXECUTION_IOS: "args: --simulator='iPhone 16e,OS=18.5'"

# Run the Android tests on the minimum supported Android version.

CIBW_EXECUTION_ANDROID: "args: --managed minVersion"

CIBW_TEST_EXECUTION_IOS: "args: --simulator='iPhone 16e,OS=18.5'"

# Run the Android tests on the minimum supported Android version.

CIBW_TEST_EXECUTION_ANDROID: "args: --managed minVersion"

freakboy3742 added 4 commits October 22, 2025 13:48

Add test-execution-args option.

3515485

Add usage of test-execution-args.

14fb8d1

Add CI configuration to use test-execution-args.

5d3e2ab

Document the test-execution-args setting.

8240dfe

freakboy3742 requested a review from mhsmith as a code owner October 22, 2025 07:14

freakboy3742 requested review from joerick and mhsmith and removed request for mhsmith October 22, 2025 07:16

mhsmith reviewed Oct 22, 2025

View reviewed changes

freakboy3742 and others added 2 commits October 23, 2025 08:18

Simplify code using or syntax instead of inline if.

d238fb9

Co-authored-by: Malcolm Smith <[email protected]>

Clarified some Android-specific terminology, and added details about …

080b33a

…the default args to the test runner.

freakboy3742 requested a review from mhsmith October 23, 2025 02:38

mhsmith reviewed Oct 23, 2025

View reviewed changes

docs/platforms.md Outdated Show resolved Hide resolved

joerick reviewed Oct 25, 2025

View reviewed changes

freakboy3742 added 4 commits October 27, 2025 14:47

Merge branch 'main' into test-execution-args

3a35df1

Switch to a dict-based test-execution configuration

5c0940c

Add tests for test-execution parsing.

e706370

Add all the files before pushing...

c33f4fe

Add note about default Android version for testbed.

9b24b80

joerick reviewed Oct 29, 2025

View reviewed changes

Improve description of test-execution setting.

4051400

Co-authored-by: Joe Rickerby <[email protected]>

freakboy3742 requested review from joerick and mhsmith October 29, 2025 22:39

mhsmith reviewed Oct 30, 2025

View reviewed changes

		### `test-execution-args` {: #test-execution-args toml env-var }

		> Define additional arguments that will be passed to the command that runs the tests.

Add configuration option for test executor arguments #2636

Are you sure you want to change the base?

Add configuration option for test executor arguments #2636

Conversation

freakboy3742 commented Oct 22, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhsmith Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joerick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

freakboy3742 commented Oct 28, 2025

Uh oh!

joerick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhsmith Oct 25, 2025 •

edited

Loading