Skip to content

Testing: Run all test SPM executable actions in a operation queue #9243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 15, 2025
Merged

Testing: Run all test SPM executable actions in a operation queue #9243

merged 2 commits into from
Oct 15, 2025

Conversation

kcieplak
Copy link
Contributor

@kcieplak kcieplak commented Oct 10, 2025 •
edited
Loading

Add an operation queue around execution of swiftPM process execution.

swift-testing based test will run all tests in parallel at the same time, causing each invocation to spawn a process which will then spawn many other subprocesses. This is compounded by tests that use arguments, as each combination will create another test that will run in parallel.

These leads to a process bomb consuming all CPU and causes massive I/O blocking.

Add a limit to how many swiftPM top level processes can run at the same time during testing. This is similar behaviour to XCtest based testing with the --num-workers option limiting the number of occurrent tests.

@kcieplak
[WIP] Experiment with adding a operation queue
5da76a3 
Add an operation queue around execution of swiftPM process
execution.

swift-testing based test will run all tests in parallel
at the same time, causing each invocation to spawn a process
which will then spawn many other subprocesses.

These leads to a process bomb and consumes all CPU and causes
massive I/O blocking.

Add a limit to how many swiftPM top level processes at the same
time.
@kcieplak kcieplak closed this Oct 10, 2025
@kcieplak kcieplak reopened this Oct 10, 2025
@kcieplak
Copy link
Contributor Author

@swift-ci test

@kcieplak
Copy link
Contributor Author

@swift-ci test windows

@kcieplak kcieplak marked this pull request as draft October 10, 2025 19:06
@kcieplak kcieplak changed the title [WIP] Experiment with adding a operation queue Testing: Run all SPM executable actions in a operation queue Oct 14, 2025
@kcieplak
Copy link
Contributor Author

kcieplak commented Oct 14, 2025 •
edited
Loading

Fixes #9245

Test times are roughly equivalent, locally the CPU usage and number of processes created during test are significantly
reduced.

| Build/Test Times

Host Total swift-testing w/Q Total w/Q swift-testing
Linux(self) 33m 25m 44m. 33m
MacOS(self) 2h 35m 1h 43m 2h 20m 1h 34m
Windows(self) 2h 5m 1h 27m 2h 13m 1h 36m 
pstree -p 18312
                   \---> [18312] swiftpm-testing-helper (kcieplak)
                      +---> [32907] git (kcieplak)
                      +---> [18807] swift-package (kcieplak)
                      +---> [18813] swift-package (kcieplak)
                      +---> [18868] swift-build (kcieplak)
                      +---> [18827] swift-build (kcieplak)
                      +---> [18875] swift-build (kcieplak)
                      +---> [18858] swift-build (kcieplak)
                      +---> [32600] git (kcieplak)
                      +---> [32612] git (kcieplak)
                      +---> [32865] git (kcieplak)
                      +---> [32901] git (kcieplak)
                      +---> [32908] git (kcieplak)
                      +---> [32184] git (kcieplak)
                      |  \---> [32744] Python (kcieplak)
                      +---> [32535] git (kcieplak)
                      |  \---> [32730] Python (kcieplak)
                      +---> [32599] git (kcieplak)
                      |  \---> [32786] Python (kcieplak)
                      +---> [32611] git (kcieplak)
                      |  \---> [32773] Python (kcieplak)
                      +---> [32868] git (kcieplak)
                      |  \---> [32896] Python (kcieplak)
                      +---> [32874] cp (kcieplak)
                      \---> [32904] git (kcieplak)

Versus without the Q

kcieplak@Mac swiftpm % pstree -p 19589
                   \---> [19589] swiftpm-testing-helper (kcieplak)
                      +---> [22608] swift-driver (kcieplak)
                      +---> [20131] cp (kcieplak)
                      +---> [20193] cp (kcieplak)
                      +---> [20249] cp (kcieplak)
                      +---> [20223] swift-build (kcieplak)
                      +---> [20173] swift-build (kcieplak)
                      +---> [20256] swift-build (kcieplak)
                      +---> [20161] swift-build (kcieplak)
                      +---> [20158] swift-build (kcieplak)
                      +---> [20214] swift-build (kcieplak)
                      +---> [20213] swift-build (kcieplak)
                      +---> [20165] swift-build (kcieplak)
                      +---> [20156] swift-build (kcieplak)
                      +---> [20186] swift-build (kcieplak)
                      +---> [20285] swift-build (kcieplak)
                      +---> [20255] swift-build (kcieplak)
                      +---> [20284] swift-build (kcieplak)
                      +---> [20474] swift-build (kcieplak)
                      +---> [20466] swift-build (kcieplak)
                      +---> [20053] swift-package (kcieplak)
                      +---> [20051] swift-package (kcieplak)
                      +---> [20056] swift-package (kcieplak)
                      +---> [20052] swift-package (kcieplak)
                      +---> [20172] swift-package (kcieplak)
                      +---> [20182] swift-package (kcieplak)
                      +---> [20905] cp (kcieplak)
                      +---> [20901] git (kcieplak)
                      +---> [20908] cp (kcieplak)
                      +---> [20915] cp (kcieplak)
                      +---> [20888] swift-build (kcieplak)
                      +---> [20128] swift-build (kcieplak)
                      +---> [20241] swift-build (kcieplak)
                      +---> [20163] swift-build (kcieplak)
                      +---> [20215] swift-build (kcieplak)
                      +---> [20277] swift-build (kcieplak)
                      +---> [21554] cp (kcieplak)
                      +---> [21678] git (kcieplak)
                      +---> [20889] swift-build (kcieplak)
                      +---> [20871] swift-build (kcieplak)
                      +---> [21521] swift-build (kcieplak)
                      +---> [21520] swift-build (kcieplak)
                      +---> [20473] swift-build (kcieplak)
                      +---> [20890] swift-build (kcieplak)
                      +---> [21689] git (kcieplak)
                      +---> [22321] cp (kcieplak)
                      +---> [21550] swift-build (kcieplak)
                      +---> [22430] git (kcieplak)
                      +---> [21583] swift-build (kcieplak)
                      +---> [20130] swift-build (kcieplak)
                      |  \---> [21464] clang-stat-cache (kcieplak)
                      +---> [20157] swift-build (kcieplak)
                      |  \---> [21167] swbuild (kcieplak)
                      |     \---> [21173] SWBBuildService (kcieplak)
                      |        \---> [22577] appintentsmetadataprocessor (kcieplak)
                      +---> [20162] swift-build (kcieplak)
                      |  \---> [21165] swbuild (kcieplak)
                      |     \---> [21176] SWBBuildService (kcieplak)
                      |        +---> [22601] swift-frontend (kcieplak)
                      |        +---> [22598] swift-frontend (kcieplak)
                      |        +---> [22599] swift-frontend (kcieplak)
                      |        \---> [22600] swift-frontend (kcieplak)
                      +---> [20164] swift-build (kcieplak)
                      |  \---> [21494] clang-stat-cache (kcieplak)
                      +---> [20245] swift-build (kcieplak)
                      |  \---> [21169] swbuild (kcieplak)
                      |     \---> [21174] SWBBuildService (kcieplak)
                      |        \---> [22504] clang (kcieplak)
                      |           \---> [22508] ld (kcieplak)
                      +---> [20263] swift-build (kcieplak)
                      |  \---> [21166] swbuild (kcieplak)
                      |     \---> [21175] SWBBuildService (kcieplak)
                      |        \---> [22363] dsymutil (kcieplak)
                      |           \---> [22610] lipo (kcieplak)
                      +---> [20270] swift-build (kcieplak)
                      |  \---> [21456] clang-stat-cache (kcieplak)
                      +---> [20292] swift-build (kcieplak)
                      |  \---> [21670] clang-stat-cache (kcieplak)
                      +---> [20300] swift-build (kcieplak)
                      |  \---> [21513] clang-stat-cache (kcieplak)
                      +---> [20455] swift-build (kcieplak)
                      |  \---> [21308] swbuild (kcieplak)
                      |     \---> [21310] SWBBuildService (kcieplak)
                      +---> [20483] swift-build (kcieplak)
                      |  \---> [21343] swbuild (kcieplak)
                      |     \---> [21344] SWBBuildService (kcieplak)
                      |        \---> [21516] clang-stat-cache (kcieplak)
                      +---> [20512] swift-build (kcieplak)
                      |  \---> [21697] clang-stat-cache (kcieplak)
                      +---> [20903] swift-build (kcieplak)
                      |  \---> [21455] swbuild (kcieplak)
                      |     \---> [21463] SWBBuildService (kcieplak)
                      |        \---> [21762] clang-stat-cache (kcieplak)
                      +---> [20945] swift-build (kcieplak)
                      |  \---> [21492] swbuild (kcieplak)
                      |     \---> [21499] SWBBuildService (kcieplak)
                      |        \---> [21777] clang-stat-cache (kcieplak)
                      +---> [21519] swift-build (kcieplak)
                      |  \---> [22511] clang-stat-cache (kcieplak)
                      +---> [21518] swift-build (kcieplak)
                      +---> [21546] swift-build (kcieplak)
                      |  \---> [22185] swbuild (kcieplak)
                      |     \---> [22203] SWBBuildService (kcieplak)
                      |        \---> [22446] clang-stat-cache (kcieplak)
                      +---> [21549] swift-build (kcieplak)
                      +---> [21570] swift-build (kcieplak)
                      |  \---> [22274] swbuild (kcieplak)
                      |     \---> [22276] SWBBuildService (kcieplak)
                      |        \---> [22517] clang-stat-cache (kcieplak)
                      +---> [21599] swift-build (kcieplak)
                      |  \---> [22491] swift-driver (kcieplak)
                      |     \---> [22514] swift-frontend (kcieplak)
                      +---> [21616] swift-build (kcieplak)
                      +---> [21629] swift-build (kcieplak)
                      |  \---> [22549] swift-driver (kcieplak)
                      +---> [21643] swift-build (kcieplak)
                      +---> [21656] swift-build (kcieplak)
                      |  \---> [22594] swift-driver (kcieplak)
                      +---> [21671] swift-build (kcieplak)
                      +---> [21686] swift-build (kcieplak)
                      |  \---> [22450] swbuild (kcieplak)
                      |     \---> [22458] SWBBuildService (kcieplak)
                      +---> [22140] swift-build (kcieplak)
                      |  \---> [22572] swift-driver (kcieplak)
                      |     \---> [22607] swift-frontend (kcieplak)
                      +---> [22154] swift-build (kcieplak)
                      |  \---> [22576] swift-driver (kcieplak)
                      |     \---> [22625] swift-frontend (kcieplak)
                      +---> [22164] swift-build (kcieplak)
                      |  \---> [22574] swift-driver (kcieplak)
                      +---> [22220] git (kcieplak)
                      |  \---> [22456] Python (kcieplak)
                      +---> [22228] swift-build (kcieplak)
                      |  \---> [22596] dummy-swiftc (kcieplak)
                      |     \---> [22602] swift-driver (kcieplak)
                      +---> [22229] swift-build (kcieplak)
                      |  \---> [22622] swift-driver (kcieplak)
                      +---> [22243] swift-build (kcieplak)
                      |  \---> [22618] swiftc (kcieplak)
                      +---> [22242] swift-build (kcieplak)
                      +---> [22257] swift-build (kcieplak)
                      |  \---> [22589] swift-help (kcieplak)
                      +---> [22265] swift-build (kcieplak)
                      |  \---> [22547] swift-driver (kcieplak)
                      |     \---> [22605] swift-frontend (kcieplak)
                      +---> [22289] swift-build (kcieplak)
                      |  \---> [22621] xcrun (kcieplak)
                      +---> [22299] swift-build (kcieplak)
                      |  \---> [22612] vtool (kcieplak)
                      +---> [22310] swift-build (kcieplak)
                      |  \---> [22604] swift-driver (kcieplak)
                      +---> [22313] swift-build (kcieplak)
                      |  \---> [22609] dummy-swiftc (kcieplak)
                      |     \---> [22615] swift-driver (kcieplak)
                      +---> [22348] swift-build (kcieplak)
                      |  \---> [22624] xcrun (kcieplak)
                      +---> [22388] swift-build (kcieplak)
                      +---> [22405] swift-build (kcieplak)
                      |  \---> [22617] xcrun (kcieplak)
                      +---> [22421] swift-build (kcieplak)
                      +---> [22564] swift-build (kcieplak)
                      \---> [22582] swift-build (kcieplak)
                      .... Goes on and on .....

@kcieplak kcieplak marked this pull request as ready for review October 14, 2025 17:10
owenv
owenv approved these changes Oct 14, 2025
View reviewed changes
Sources/_InternalTestSupport/SwiftPMProduct.swift Outdated

import enum TSCBasic.ProcessEnv

private let swiftPMExecutionQueue = AsyncOperationQueue(concurrentTasks: 6)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be ProcessInfo.processInfo.activeProcessorCount * x for some x instead, wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

swift-build where the AsyncOperationQueue was pulled from also uses 6, I started from there.
Maybe NCPUs * 0.5 The process fanout from invoking things like swift build is quite large.

@kcieplak kcieplak changed the title Testing: Run all SPM executable actions in a operation queue Testing: Run all test SPM executable actions in a operation queue Oct 14, 2025
@kcieplak
Updates based on PR review
e99a75c 
* Initialize the queue size to some fraction of number of CPUs.
* Add a comment on why we are running executions on a queue.
@kcieplak
Copy link
Contributor Author

@swift-ci test

@kcieplak
Copy link
Contributor Author

@swift-ci test windows

1 similar comment
@kcieplak
Copy link
Contributor Author

@swift-ci test windows

@kcieplak
Copy link
Contributor Author

@swift-ci test macos

@kcieplak kcieplak merged commit b16bb8f into swiftlang:main Oct 15, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@owenv owenv owenv approved these changes

@jakepetroules jakepetroules Awaiting requested review from jakepetroules jakepetroules is a code owner

@dschaefer2 dschaefer2 Awaiting requested review from dschaefer2 dschaefer2 is a code owner

@shawnhyam shawnhyam Awaiting requested review from shawnhyam shawnhyam is a code owner

@bripeticca bripeticca Awaiting requested review from bripeticca bripeticca is a code owner

@plemarquand plemarquand Awaiting requested review from plemarquand plemarquand is a code owner

@bkhouri bkhouri Awaiting requested review from bkhouri bkhouri is a code owner

@cmcgee1024 cmcgee1024 Awaiting requested review from cmcgee1024 cmcgee1024 is a code owner

@daveyc123 daveyc123 Awaiting requested review from daveyc123 daveyc123 is a code owner

@rconnell9 rconnell9 Awaiting requested review from rconnell9 rconnell9 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kcieplak @owenv