Skip to content

Add configuration option for test executor arguments #2636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
+231 −8
Open

Add configuration option for test executor arguments #2636

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
3515485
Add test-execution-args option.
freakboy3742 Oct 22, 2025
14fb8d1
Add usage of test-execution-args.
freakboy3742 Oct 22, 2025
5d3e2ab
Add CI configuration to use test-execution-args.
freakboy3742 Oct 22, 2025
8240dfe
Document the test-execution-args setting.
freakboy3742 Oct 22, 2025
d238fb9
Simplify code using or syntax instead of inline if.
freakboy3742 Oct 23, 2025
080b33a
Clarified some Android-specific terminology, and added details about …
freakboy3742 Oct 23, 2025
3a35df1
Merge branch 'main' into test-execution-args
freakboy3742 Oct 27, 2025
5c0940c
Switch to a dict-based test-execution configuration
freakboy3742 Oct 28, 2025
e706370
Add tests for test-execution parsing.
freakboy3742 Oct 28, 2025
c33f4fe
Add all the files before pushing...
freakboy3742 Oct 28, 2025
9b24b80
Add note about default Android version for testbed.
freakboy3742 Oct 29, 2025
4051400
Improve description of test-execution setting.
freakboy3742 Oct 29, 2025
dcd0743
Switch to using test-runtime.
freakboy3742 Oct 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,16 @@ jobs:
- os: macos-15
python_version: '3.13'
test_select: ios
# Github Actions macOS-15 runner has disk performance issues; so it
# has problems using any simulator that isn't pre-warmed.
# Unfortunately, the simulator that it picks by default isn't
# pre-warmed. So - we need to explicitly select a simulator.
test_execution_args: '--simulator "iPhone 16e,OS=18.5"'
- os: macos-15-intel
python_version: '3.13'
test_select: android
# Exercise Android on a non-default simulator
test_execution_args: '--managed minVersion'
- os: macos-15
python_version: '3.13'
test_select: android
Expand Down Expand Up @@ -154,6 +161,7 @@ jobs:
CIBW_ARCHS_MACOS: x86_64 universal2 arm64
CIBW_BUILD_FRONTEND: ${{ matrix.test_select && 'build' || 'build[uv]' }}
CIBW_PLATFORM: ${{ matrix.test_select }}
CIBW_TEST_EXECUTION_ARGS: ${{ matrix.test_execution_args }}

- name: Run a sample build (GitHub Action, only)
uses: ./
Expand All @@ -179,6 +187,7 @@ jobs:
uses: ./
env:
CIBW_PLATFORM: ${{ matrix.test_select }}
CIBW_TEST_EXECUTION_ARGS: ${{ matrix.test_execution_args }}
with:
package-dir: sample_proj
output-dir: wheelhouse_config_file
Expand All @@ -202,6 +211,8 @@ jobs:
path: wheelhouse/*.whl

- name: Test cibuildwheel
env:
CIBW_TEST_EXECUTION_ARGS: ${{ matrix.test_execution_args }}
run: |
uv run --no-sync bin/run_tests.py --test-select=${{ matrix.test_select || 'native' }} ${{ (runner.os == 'Linux' && runner.arch == 'X64') && '--run-podman' || '' }}

Expand Down
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,8 @@ Usage

| | Linux | macOS | Windows | Linux ARM | macOS ARM | Windows ARM | Android | iOS |
|-----------------|-------|-------|---------|-----------|-----------|-------------|---------|-----|
| GitHub Actions | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅⁴ | ✅³⁵ |
| Azure Pipelines | ✅ | ✅ | ✅ | | ✅ | ✅² | ✅⁴ | ✅³⁵ |
| GitHub Actions | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅⁴ | ✅³ |
| Azure Pipelines | ✅ | ✅ | ✅ | | ✅ | ✅² | ✅⁴ | ✅³ |
| Travis CI | ✅ | | ✅ | ✅ | | | ✅⁴ | |
| CircleCI | ✅ | ✅ | | ✅ | ✅ | | ✅⁴ | ✅³ |
| Gitlab CI | ✅ | ✅ | ✅ | ✅¹ | ✅ | | ✅⁴ | ✅³ |
Expand All @@ -70,7 +70,6 @@ Usage
<sup>² [Uses cross-compilation](https://cibuildwheel.pypa.io/en/stable/faq/#windows-arm64). It is not possible to test `arm64` on this CI platform.</sup><br>
<sup>³ Requires a macOS runner; runs tests on the simulator for the runner's architecture. </sup><br>
<sup>⁴ Building for Android requires the runner to be Linux x86_64, macOS ARM64 or macOS x86_64. Testing has [additional requirements](https://cibuildwheel.pypa.io/en/stable/platforms/#android).</sup><br>
<sup>⁵ The `macos-15` and `macos-latest` images are [incompatible with cibuildwheel at this time](platforms/#ios-system-requirements)</sup><br> when building iOS wheels.

<!--intro-end-->

Expand Down Expand Up @@ -160,12 +159,13 @@ The following diagram summarises the steps that cibuildwheel takes on each platf
| | [`test-groups`](https://cibuildwheel.pypa.io/en/stable/options/#test-groups) | Specify test dependencies from your project's `dependency-groups` |
| | [`test-skip`](https://cibuildwheel.pypa.io/en/stable/options/#test-skip) | Skip running tests on some builds |
| | [`test-environment`](https://cibuildwheel.pypa.io/en/stable/options/#test-environment) | Set environment variables for the test environment |
| | [`test-execution-args`](https://cibuildwheel.pypa.io/en/stable/options/#test-execution-args) | Define additional arguments that will be passed to the command that runs the tests. |
| **Debugging** | [`debug-keep-container`](https://cibuildwheel.pypa.io/en/stable/options/#debug-keep-container) | Keep the container after running for debugging. |
| | [`debug-traceback`](https://cibuildwheel.pypa.io/en/stable/options/#debug-traceback) | Print full traceback when errors occur. |
| | [`build-verbosity`](https://cibuildwheel.pypa.io/en/stable/options/#build-verbosity) | Increase/decrease the output of the build |


<!--[[[end]]] (sum: FxE3nIgFiY) -->
<!--[[[end]]] (sum: gNUxwlNyTE) -->

These options can be specified in a pyproject.toml file, or as environment variables, see [configuration docs](https://cibuildwheel.pypa.io/en/latest/configuration/).

Expand Down
4 changes: 4 additions & 0 deletions bin/generate_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,9 @@
test-environment:
description: Set environment variables for the test environment
type: string_table
test-execution-args:
description: Additional arguments for the test runner to use when configuring the test environment
type: string_array
"""

schema = yaml.safe_load(starter)
Expand Down Expand Up @@ -304,6 +307,7 @@
test-sources: {"$ref": "#/$defs/inherit"}
test-requires: {"$ref": "#/$defs/inherit"}
test-environment: {"$ref": "#/$defs/inherit"}
test-execution-args: {"$ref": "#/$defs/inherit"}
"""
)

Expand Down
8 changes: 8 additions & 0 deletions cibuildwheel/options.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ class BuildOptions:
test_extras: str
test_groups: list[str]
test_environment: ParsedEnvironment
test_execution_args: list[str] | None
build_verbosity: int
build_frontend: BuildFrontendConfig
config_settings: str
Expand Down Expand Up @@ -761,6 +762,12 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
msg = f"Malformed environment option {test_environment_config!r}"
raise errors.ConfigurationError(msg) from e

test_execution_args = shlex.split(
self.reader.get(
"test-execution-args", option_format=ListFormat(sep=" ", quote=shlex.quote)
)
)

test_requires = self.reader.get(
"test-requires", option_format=ListFormat(sep=" ")
).split()
Expand Down Expand Up @@ -868,6 +875,7 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
test_command=test_command,
test_sources=test_sources,
test_environment=test_environment,
test_execution_args=test_execution_args,
test_requires=[*test_requires, *test_requirements_from_groups],
test_extras=test_extras,
test_groups=test_groups,
Expand Down
15 changes: 13 additions & 2 deletions cibuildwheel/platforms/android.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -635,17 +635,28 @@ def test_wheel(state: BuildState, wheel: Path) -> None:
)
raise errors.FatalError(msg)

# By default, run on a testbed managed emulator running the newest supported
# Android version. However, if the user specifies a --managed or --connected
# test execution argument, that argument takes precedence.
if state.options.test_execution_args and (
"--managed" in state.options.test_execution_args
or "--connected" in state.options.test_execution_args
):
emulator_args = []
else:
emulator_args = ["--managed", "maxVersion"]

# Run the test app.
call(
state.python_dir / "android.py",
"test",
"--managed",
"maxVersion",
"--site-packages",
site_packages_dir,
"--cwd",
cwd_dir,
*emulator_args,
*(["-v"] if state.options.build_verbosity > 0 else []),
*(state.options.test_execution_args or []),
*test_args,
env=state.build_env,
)
Expand Down
1 change: 1 addition & 0 deletions cibuildwheel/platforms/ios.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -658,6 +658,7 @@ def build(options: Options, tmp_path: Path) -> None:
testbed_path,
"run",
*(["--verbose"] if build_options.build_verbosity > 0 else []),
*(build_options.test_execution_args or []),
"--",
*final_command,
env=test_env,
Expand Down
39 changes: 39 additions & 0 deletions cibuildwheel/resources/cibuildwheel.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,6 +569,21 @@
],
"title": "CIBW_TEST_ENVIRONMENT"
},
"test-execution-args": {
"description": "Additional arguments for the test runner to use when configuring the test environment",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_EXECUTION_ARGS"
},
"overrides": {
"type": "array",
"description": "An overrides array",
Expand Down Expand Up @@ -638,6 +653,9 @@
},
"test-environment": {
"$ref": "#/$defs/inherit"
},
"test-execution-args": {
"$ref": "#/$defs/inherit"
}
}
},
Expand Down Expand Up @@ -748,6 +766,9 @@
},
"test-environment": {
"$ref": "#/properties/test-environment"
},
"test-execution-args": {
"$ref": "#/properties/test-execution-args"
}
}
}
Expand Down Expand Up @@ -876,6 +897,9 @@
},
"test-environment": {
"$ref": "#/properties/test-environment"
},
"test-execution-args": {
"$ref": "#/properties/test-execution-args"
}
}
},
Expand Down Expand Up @@ -936,6 +960,9 @@
},
"test-environment": {
"$ref": "#/properties/test-environment"
},
"test-execution-args": {
"$ref": "#/properties/test-execution-args"
}
}
},
Expand Down Expand Up @@ -1009,6 +1036,9 @@
},
"test-environment": {
"$ref": "#/properties/test-environment"
},
"test-execution-args": {
"$ref": "#/properties/test-execution-args"
}
}
},
Expand Down Expand Up @@ -1069,6 +1099,9 @@
},
"test-environment": {
"$ref": "#/properties/test-environment"
},
"test-execution-args": {
"$ref": "#/properties/test-execution-args"
}
}
},
Expand Down Expand Up @@ -1129,6 +1162,9 @@
},
"test-environment": {
"$ref": "#/properties/test-environment"
},
"test-execution-args": {
"$ref": "#/properties/test-execution-args"
}
}
},
Expand Down Expand Up @@ -1189,6 +1225,9 @@
},
"test-environment": {
"$ref": "#/properties/test-environment"
},
"test-execution-args": {
"$ref": "#/properties/test-execution-args"
}
}
}
Expand Down
1 change: 1 addition & 0 deletions cibuildwheel/resources/defaults.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ test-requires = []
test-extras = []
test-groups = []
test-environment = {}
test-execution-args = []

container-engine = "docker"

Expand Down
35 changes: 35 additions & 0 deletions docs/options.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1669,6 +1669,41 @@ Platform-specific environment variables are also available:<br/>
CIBW_TEST_ENVIRONMENT: PYTHONSAFEPATH=1
```

### `test-execution-args` {: #test-execution-args toml env-var }

> Define additional arguments that will be passed to the command that runs the tests.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test-execution-args still sounds like they might be arguments to the tests themselves. Would test-runner-args be a clearer name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I picked test-execution-args based on the discussion that lead to this PR; but test-runner-args would also work. The exact naming isn't a hill I'll die on, and it's easy to change; would be interested in hearing other opinions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had considered test-runner-... too, but I didn't like it because to my mind 'pytest' is a test runner, and we're not talking about that. Ideally, the name would evoke more 'environment' or 'context' in its flavour, but nothing fit that didn't already have a meaning. So I landed on 'execution', which feels higher-level than runner(?).

Copy link
Member

@mhsmith mhsmith Oct 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some other options:

  • "testbed" is the term we've used for both Android and iOS, but that wouldn't be appropriate for a Docker container, as mentioned in the linked discussion.
  • "container" is what Docker would call it, but that implies a certain level of isolation which we don't necessarily provide.
  • "launcher" seems like a fairly generic way of saying "the thing that starts (but doesn't "run") the test suite".

On the other hand, if this option could have additional sub-keys in the future as suggested below, then maybe a general name like "execution" is OK.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, if I'm completely honest, I'm still not in love with 'test-execution'. Seeing it in the sidebar, it really gives no hints about what it does-

image

I'm now coming back around to test-target, test-host or test-runtime. Actually test-target looks quite sensible in the options sidebar-

image

I'm curious what resonates with you? bearing in mind I could also imagine this being used to spec a docker image, or set some pyodide/node options.

(apologies to go around the houses with this @freakboy3742 !)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious what resonates with you? bearing in mind I could also imagine this being used to spec a docker image, or set some pyodide/node options.

Of those options, test-runtime makes the most sense to me.

test-target sounds like "where do I want to run the tests?" rather than "how will I run this test on a target?" (and similar for test-host, with the added benefit of the confusion of whether the "host" is the architecture running the build, or the architecture you're building for).test-runtime sounds like the one that is most directly tied to "what runtime will I use for the tests?" - which is essentially what we're configuring, whether that's iOS, Android or Docker.

(apologies to go around the houses with this @freakboy3742 !)

No problems at all - it's an easy set of changes to make, and I'd rather go round a few times now and get it right than have to walk back a change in a couple of weeks/months.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test-runtime seems like a reasonable name to me as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks both for your input! Okay, that feels like consensus, let's run with test-runtime :)


A list of arguments that will be used by the test runner when running tests. This is used by environments where the execution environment can be customized. For example, mobile platforms will use these arguments to control the execution of the testbed application that is used to run tests.

This option will be ignored on platforms that do not have a separate test runner.

Platform-specific environment variables are also available:<br/>
`CIBW_TEST_EXECUTION_ARGS_MACOS` | `CIBW_TEST_EXECUTION_ARGS_WINDOWS` | `CIBW_TEST_EXECUTION_ARGS_LINUX` | `CIBW_TEST_EXECUTION_ARGS_ANDROID` |`CIBW_TEST_EXECUTION_ARGS_IOS` | `CIBW_TEST_EXECUTION_ARGS_PYODIDE`

#### Examples

!!! tab examples "pyproject.toml"

```toml
[tool.cibuildwheel.ios]
# Run the tests on an iPhone 16e simulator running iOS 18.5.
test-execution-args = ["--simulator='iPhone 16e,OS=18.5'"]

[tool.cibuildwheel.android]
# Run the Android tests on the minimum supported Android version.
test-execution-args = ["--managed", "minVersion"]
```

!!! tab examples "Environment variables"

```yaml
# Run the tests on an iPhone 16e simulator running iOS 18.5.
CIBW_EXECUTION_ARGS_IOS: --simulator='iPhone 16e,OS=18.5'

# Run the Android tests on the minimum supported Android version.
CIBW_EXECUTION_ARGS_ANDROID: --managed minVersion
```


## Debugging

Expand Down
7 changes: 7 additions & 0 deletions docs/platforms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,8 @@ machine – for example, if you're building on an ARM64 machine, then you can te
ARM64 wheel. Wheels of other architectures can still be built, but testing will
automatically be skipped.

Any arguments specified using [`test-execution-args`](options.md#test-execution-args) will be passed as arguments to the Python script that starts the testbed project. cibuildwheel will automatically start the testbed project with `--site-packages` and `--cwd` arguments matching your test environment, as well as enabling verbose output with `-v` if [`build_verbosity`](options.md#build_verbosity) is enabled. The most common additional arguments to use will be `--managed minVersion` or `--managed maxVersion`, specifying the use of a managed Android emulator with the minimum or maximum supported Android version; or `--connected <serial>`, specifying the use of an existing booted Android emulator or device.

Running an emulator requires the build machine to either be bare-metal or support
nested virtualization. CI platforms known to meet this requirement are:

Expand Down Expand Up @@ -321,3 +323,8 @@ If tests have been configured, the test suite will be executed on the simulator
The iOS test environment can't support running shell scripts, so the [`test-command`](options.md#test-command) value must be specified as if it were a command line being passed to `python -m ...`.

The test process uses the same testbed used by CPython itself to run the CPython test suite. It is an Xcode project that has been configured to have a single Xcode "XCUnit" test - the result of which reports the success or failure of running `python -m <test-command>`.

Any arguments specified using [`test-execution-args`](options.md#test-execution-args) will be passed as arguments to the Python script that starts the testbed project. The testbed project will be started with `-v` enabling verbose output if [`build_verbosity`](options.md#build_verbosity) is enabled; the most common additional argument to use will be `--simulator`, which allows the specification of a specific device or iOS version for the test simulator. For example, `test_execution_args = ["--simulator", "iPhone 16e,OS=18.5"]` would specify the use of an iPhone 16e simulator running OS 18.5.

!!! note
The `macos-15` image on GitHub Actions and Azure has a [known performance issue](https://github.com/actions/runner-images/issues/12777) that can lead to iOS simulators failing to start if they have not been pre-warmed. At this time, the workaround is to explicitly specify a simulator that is pre-warmed by default on the image; the iPhone 16e simulator running iOS 18.5 is one such image. This workaround is only needed on the `macos-15`; the `macos-14` image is not affected.
Loading