The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1190913	dotnet/runtime	Send to Helix	Log
1190883	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121265
1190870	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121263
1190858	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121058
2828993	dotnet-runtime	Send job to Helix (Windows)	Log
1190831	dotnet/arcade	XHarness Android Helix Testing (Windows)	Log	dotnet/arcade#16256
1190757	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120395
1190732	dotnet/runtime	Send tests to Helix (Windows)	Log
1190690	dotnet/runtime	Build product	Log	dotnet/runtime#119432
1190728	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121260
1190689	dotnet/runtime	Build product	Log	dotnet/runtime#119432
1190699	dotnet/aspnetcore	Run build.cmd helix target	Log
1190723	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121261
1190685	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121225
2828872	dotnet-aspnetcore	Run build.cmd helix target	Log
1190661	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121217
1190675	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51207
1190603	dotnet/runtime	Send tests to Helix (Windows)	Log
1190628	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121163
2828817	dotnet-runtime	Send job to Helix (Windows)	Log
1190619	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121144
1190606	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#119420
1190604	dotnet/runtime	Send to Helix	Log
1190585	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120895
1190569	dotnet/runtime	Send tests to Helix (Windows)	Log
2828752	dotnet-runtime	Send job to Helix (Windows)	Log
2828758	dotnet-sdk	🛡 Guardian: PoliCheck	Log
1190503	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121212
1190541	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121225
1190486	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121177
1190475	dotnet/runtime	Send tests to Helix (Windows)	Log
2828695	dotnet-roslyn	Run Unit Tests	Log
1190220	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120976
1190414	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121186
1190446	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121251
1190451	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#64202
1190441	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51512
2828480	dotnet-performance	Send job to Helix (Windows)	Log
1190384	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121192
1190367	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120949
1190371	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121249
1190425	dotnet/roslyn-sdk	CmdLine	Log	dotnet/roslyn-sdk#1232
2828595	dotnet-runtime	Send job to Helix (Windows)	Log
1190340	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121203
1190334	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121247
1190328	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#119203
2828593	dotnet-dotnet	Build	Log
2828580	dotnet-node	Validate Hosted Pool Information (1ES PT)	Log
2828579	dotnet-node	UsePythonVersion@0	Log
2828578	dotnet-node	Build and package	Log
1190290	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121246
1190131	dotnet/runtime	Send to Helix	Log
2828566	dotnet-arcade-validation	Install Preview MicroBuild plugin (Windows) (LinuxContainer)	Log
2828563	dotnet-arcade-validation	Download secrets: HelixProdKV	Log
1190260	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121245
2828507	dotnet-maui	Run Integration Tests - Blazor macOS	Log
2828510	dotnet-dotnet	SDL Binary Analysis Checks via output (1ES PT)	Log
1190234	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121174
1190206	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121202
1190130	dotnet/runtime	Send to Helix	Log
2828479	dotnet-aspnetcore	Run build.cmd helix target	Log
1190229	dotnet/aspnetcore	Run build.cmd helix target	Log
1190176	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120303
1190173	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121237
1190211	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121243
1190151	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121232
1190142	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121117
2828408	dotnet-runtime	Send job to Helix (Windows)	Log
1190129	dotnet/runtime	Send tests to Helix (Windows)	Log
1190102	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121230
1190125	dotnet/aspnetcore	Run build.cmd helix target	Log
1190101	dotnet/runtime	Send tests to Helix (Windows)	Log
1190095	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120887
1190060	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121192
1190086	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50468
2828315	dotnet-runtime	Send job to Helix (Windows)	Log
1190052	dotnet/aspnetcore	Run build.cmd helix target	Log
1190024	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121226
1190011	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121186
2828249	dotnet-runtime	Send job to Helix (Windows)	Log
1189978	dotnet/runtime	Send tests to Helix (Windows)	Log
1189903	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51207
1189979	dotnet/runtime	Send tests to Helix (Windows)	Log
2828085	dotnet-performance	Send job to Helix (Windows)	Log
1189996	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121225
1189970	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121218
2828232	dotnet-dotnet-monitor	Build	Log
1189982	dotnet/runtime	Send to Helix	Log
1189937	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#120834
1189884	dotnet/runtime	Send tests to Helix (Windows)	Log
2828092	dotnet-sdk	🛡 Generate SBoM Manifest	Log
1189875	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121130
2827986	dotnet-performance	Send job to Helix (Windows)	Log
2828150	dotnet-sdk	🟣 Run TestBuild Tests	Log
1189840	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#117788
1189844	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121221
1189867	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121223
1189855	dotnet/aspnetcore	Run build.cmd helix target	Log
2828143	dotnet-aspnetcore	Run build.cmd helix target	Log
2828140	dotnet-arcade-validation	Run Tests (Windows)	Log
Displaying 100 of 1395 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
56	477	1395

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez