Skip to content

Commit 7e70f3e

Browse files
trylekjkoritzinsky
trylek
and
jkoritzinsky
authored
Infra fixes for bugs uncovered by testing of the JIT/Methodical merged tests (#67393)
* Move copying of merged wrapper native components to copynative step * Move export of test exclusion file before ILLink changing CORE_ROOT * Fix file mode for test execution bash scripts * Add out-of-proc test folders to Helix work item payloads * Fix Helix publishing in the presence of out-of-process tests * Use a marker file triggered off of a project property to determine which files to skip running the Mono AOT compiler on (some tests fail at AOT time, so we need this to work around that limitation) * Pass runtimeVariant to the Core_Root construction for the test exclusion list creation * Don't write the output element when there's no test output. This causes issues with Helix test uploading. * Write xunit results the way xharness expects to read them for wasm so we correctly report failures and don't report a test harness failure. * Fix GeneratedTestRunner to build. * Fix copying native wrappers for merged test runners. Pre-emptively mark another test as RequiresProcessIsolation as I stumbled upon it while fixing this * Fix copying of native test components to merged wrapper outputs Due to Pri0 / Pri1 test grouping we may end up building the wrapper in a different group than its components. Make sure that we populate all native components of merged wrapper dependencies before we copy them over to the merged wrapper output folder. * Exclude wasm support files. Co-authored-by: Jeremy Koritzinsky <[email protected]>
1 parent 0dc0bee commit 7e70f3e

File tree

36 files changed

+144
-127
lines changed

36 files changed

+144
-127
lines changed

‎eng/pipelines/common/templates/runtimes/run-test-job.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -300,7 +300,7 @@ jobs:
300300
# Compose the Core_Root folder containing all artifacts needed for running
301301
# CoreCLR tests. This step also compiles the framework using Crossgen / Crossgen2
302302
# in ReadyToRun jobs.
303-
- script: $(Build.SourcesDirectory)/src/tests/build$(scriptExt) generatelayoutonly $(logRootNameArg)Layout $(runtimeFlavorArgs) $(crossgenArg) $(buildConfig) $(archType) $(crossArg) $(priorityArg) $(librariesOverrideArg)
303+
- script: $(Build.SourcesDirectory)/src/tests/build$(scriptExt) generatelayoutonly $(logRootNameArg)Layout $(runtimeFlavorArgs) $(crossgenArg) $(buildConfig) $(archType) $(crossArg) $(priorityArg) $(librariesOverrideArg) $(runtimeVariantArg)
304304
displayName: Generate CORE_ROOT
305305

306306
# Build a Mono LLVM AOT cross-compiler for non-amd64 targets (in this case, just arm64)

‎src/tests/Common/CLRTest.Execute.Bash.targets‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -306,8 +306,6 @@ $__Command msbuild $CORE_ROOT/wasm-test-runner/WasmTestRunner.proj /p:NetCoreApp
306306
<![CDATA[
307307
$(BashLinkerTestLaunchCmds)
308308
309-
export TestExclusionListPath=$CORE_ROOT/TestExclusionList.txt
310-
311309
_DebuggerArgsSeparator=
312310
if [[ "$_DebuggerFullPath" == *lldb* ]];
313311
then
@@ -549,6 +547,7 @@ $(BashCLRTestExitCodePrep)
549547
$(IlasmRoundTripBashScript)
550548
# Allow precommands to override the ExePath
551549
ExePath=$(InputAssemblyName)
550+
export TestExclusionListPath=$CORE_ROOT/TestExclusionList.txt
552551
# PreCommands
553552
$(BashCLRTestPreCommands)
554553
# Launch
@@ -561,15 +560,23 @@ $(BashCLRTestExitCodeCheck)
561560

562561
</PropertyGroup>
563562

563+
<PropertyGroup>
564+
<ExecutionBashScriptPath>$(OutputPath)/$(MSBuildProjectName).sh</ExecutionBashScriptPath>
565+
</PropertyGroup>
566+
564567
<!-- Write the file.
565568
Note: under the hood, this will rely on Environment.NewLine for line
566569
endings. This means that if the scripts are being generated on Windows,
567570
the line endings will need to be changed from CR-LF to Unix (LF) line
568-
endings before running the scripts on Unix platforms. -->
571+
endings before running the scripts on Unix platforms. In our current lab
572+
infra it shouldn't really matter as the execution scripts are regenerated
573+
in the 'test run' phase before sending the items to Helix i.o.w. at
574+
the point where we already know the exact targeting platform. -->
569575
<WriteLinesToFile
570-
File="$(OutputPath)\$(MSBuildProjectName).sh"
576+
File="$(ExecutionBashScriptPath)"
571577
Lines="$(_CLRTestExecutionScriptText)"
572578
Overwrite="true" />
579+
<Exec Command="chmod +x $(ExecutionBashScriptPath)" EchoOff="true" />
573580
</Target>
574581

575582
</Project>

‎src/tests/Common/CLRTest.Execute.Batch.targets‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -301,8 +301,6 @@ COPY /y %CORE_ROOT%\CoreShim.dll .
301301
$(BatchLinkerTestLaunchCmds)
302302
$(BatchCopyCoreShimLocalCmds)
303303
304-
set TestExclusionListPath=%CORE_ROOT%\TestExclusionList.txt
305-
306304
IF NOT "%CLRCustomTestLauncher%"=="" (
307305
set LAUNCHER=call %CLRCustomTestLauncher% %scriptPath%
308306
) ELSE (
@@ -449,6 +447,7 @@ $(IlasmRoundTripBatchScript)
449447
450448
REM Allow precommands to override the ExePath
451449
set ExePath=$(InputAssemblyName)
450+
set TestExclusionListPath=%CORE_ROOT%\TestExclusionList.txt
452451
453452
REM Precommands
454453
$(CLRTestBatchPreCommands)

‎src/tests/Common/Directory.Build.targets‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,4 +191,7 @@
191191

192192
</Target>
193193

194+
<!-- At this point Common test dependencies don't have any native components -->
195+
<Target Name="CopyAllNativeProjectReferenceBinaries" />
196+
194197
</Project>

‎src/tests/Common/XHarnessRunnerLibrary/GeneratedTestRunner.cs‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
using System.Collections.Generic;
33
using System.Diagnostics;
44
using System.IO;
5+
using System.Text;
56
using System.Threading.Tasks;
67
using Microsoft.DotNet.XHarness.Common;
78
using Microsoft.DotNet.XHarness.TestRunners.Common;
@@ -15,16 +16,20 @@ public sealed class GeneratedTestRunner : TestRunner
1516
TestFilter.ISearchClause? _filter;
1617
Func<TestFilter?, TestSummary> _runTestsCallback;
1718
HashSet<string> _testExclusionList;
19+
private readonly Boolean _writeBase64TestResults;
20+
1821
public GeneratedTestRunner(
1922
LogWriter logger,
2023
Func<TestFilter?, TestSummary> runTestsCallback,
2124
string assemblyName,
22-
HashSet<string> testExclusionList)
25+
HashSet<string> testExclusionList,
26+
bool writeBase64TestResults)
2327
:base(logger)
2428
{
2529
_assemblyName = assemblyName;
2630
_runTestsCallback = runTestsCallback;
2731
_testExclusionList = testExclusionList;
32+
_writeBase64TestResults = writeBase64TestResults;
2833
ResultsFileName = $"{_assemblyName}.testResults.xml";
2934
}
3035

@@ -53,7 +58,17 @@ public override string WriteResultsToFile(XmlResultJargon xmlResultJargon)
5358
public override void WriteResultsToFile(TextWriter writer, XmlResultJargon jargon)
5459
{
5560
Debug.Assert(jargon == XmlResultJargon.xUnit);
56-
writer.WriteLine(LastTestRun.GetTestResultOutput(_assemblyName));
61+
string lastTestResults = LastTestRun.GetTestResultOutput(_assemblyName);
62+
if (_writeBase64TestResults)
63+
{
64+
byte[] encodedBytes = Encoding.Unicode.GetBytes(lastTestResults);
65+
string base64Results = Convert.ToBase64String(encodedBytes);
66+
writer.WriteLine($"STARTRESULTXML {encodedBytes.Length} {base64Results} ENDRESULTXML");
67+
}
68+
else
69+
{
70+
writer.WriteLine(lastTestResults);
71+
}
5772
}
5873

5974
public override void SkipTests(IEnumerable<string> tests)

‎src/tests/Common/XHarnessRunnerLibrary/RunnerEntryPoint.cs‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,12 @@ public static async Task<int> RunTests(
3636
bool anyFailedTests = false;
3737
entryPoint.TestsCompleted += (o, e) => anyFailedTests = e.FailedTests > 0;
3838
await entryPoint.RunAsync();
39+
40+
if (OperatingSystem.IsBrowser())
41+
{
42+
// Browser expects all xharness processes to exit with 0, even in case of failure
43+
return 0;
44+
}
3945
return anyFailedTests ? 1 : 0;
4046
}
4147

@@ -65,7 +71,7 @@ public AppleEntryPoint(
6571
protected override bool IsXunit => true;
6672
protected override TestRunner GetTestRunner(LogWriter logWriter)
6773
{
68-
var runner = new GeneratedTestRunner(logWriter, _runTestsCallback, _assemblyName, _testExclusionList);
74+
var runner = new GeneratedTestRunner(logWriter, _runTestsCallback, _assemblyName, _testExclusionList, writeBase64TestResults: true);
6975
if (_methodNameToRun is not null)
7076
{
7177
runner.SkipMethod(_methodNameToRun, isExcluded: false);
@@ -103,7 +109,7 @@ public AndroidEntryPoint(
103109
protected override bool IsXunit => true;
104110
protected override TestRunner GetTestRunner(LogWriter logWriter)
105111
{
106-
var runner = new GeneratedTestRunner(logWriter, _runTestsCallback, _assemblyName, _testExclusionList);
112+
var runner = new GeneratedTestRunner(logWriter, _runTestsCallback, _assemblyName, _testExclusionList, writeBase64TestResults: false);
107113
if (_methodNameToRun is not null)
108114
{
109115
runner.SkipMethod(_methodNameToRun, isExcluded: false);
@@ -151,7 +157,7 @@ public WasmEntryPoint(
151157
protected override bool IsXunit => true;
152158
protected override TestRunner GetTestRunner(LogWriter logWriter)
153159
{
154-
var runner = new GeneratedTestRunner(logWriter, _runTestsCallback, _assemblyName, _testExclusionList);
160+
var runner = new GeneratedTestRunner(logWriter, _runTestsCallback, _assemblyName, _testExclusionList, writeBase64TestResults: true);
155161
if (_methodNameToRun is not null)
156162
{
157163
runner.SkipMethod(_methodNameToRun, isExcluded: false);

‎src/tests/Common/XUnitWrapperLibrary/TestSummary.cs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,17 +70,18 @@ public string GetTestResultOutput(string assemblyName)
7070
foreach (var test in _testResults)
7171
{
7272
resultsFile.Append($@"<test name=""{test.Name}"" type=""{test.ContainingTypeName}"" method=""{test.MethodName}"" time=""{test.Duration.TotalSeconds:F6}"" ");
73+
string outputElement = !string.IsNullOrWhiteSpace(test.Output) ? $"<output><![CDATA[{test.Output}]]></output>" : string.Empty;
7374
if (test.Exception is not null)
7475
{
75-
resultsFile.AppendLine($@"result=""Fail""><failure exception-type=""{test.Exception.GetType()}""><message><![CDATA[{test.Exception.Message}]]></message><stack-trace><![CDATA[{test.Exception.StackTrace}]]></stack-trace></failure><output><![CDATA[{test.Output}]]></output></test>");
76+
resultsFile.AppendLine($@"result=""Fail""><failure exception-type=""{test.Exception.GetType()}""><message><![CDATA[{test.Exception.Message}]]></message><stack-trace><![CDATA[{test.Exception.StackTrace}]]></stack-trace></failure>{outputElement}</test>");
7677
}
7778
else if (test.SkipReason is not null)
7879
{
7980
resultsFile.AppendLine($@"result=""Skip""><reason><![CDATA[{test.SkipReason}]]></reason></test>");
8081
}
8182
else
8283
{
83-
resultsFile.AppendLine($@" result=""Pass""><output><![CDATA[{test.Output}]]></output></test>");
84+
resultsFile.AppendLine($@" result=""Pass"">{outputElement}</test>");
8485
}
8586
}
8687

‎src/tests/Common/helixpublishwitharcade.proj‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -282,7 +282,8 @@
282282

283283
<Target Name="DiscoverMergedTestWrappers">
284284
<ItemGroup>
285-
<_MergedWrapperMarker Include="$(TestBinDir)**\*.MergedTestAssembly" />
285+
<!-- Exclude WASM support files. They can interfere with our discovery process and create extra work items that don't work. -->
286+
<_MergedWrapperMarker Include="$(TestBinDir)**\*.MergedTestAssembly" Exclude="$(TestBinDir)**\supportFiles\*.MergedTestAssembly" />
286287
</ItemGroup>
287288
</Target>
288289

@@ -294,14 +295,22 @@
294295
<_MergedWrapperRunScript Include="$([System.IO.Path]::ChangeExtension('%(_MergedWrapperMarker.Identity)', '.$(TestScriptExtension)'))" />
295296
</ItemGroup>
296297
<PropertyGroup>
297-
<_MergedWrapperDirectory>%(_MergedWrapperRunScript.RootDir)%(Directory)</_MergedWrapperDirectory>
298+
<_MergedWrapperDirectory>$([System.IO.Path]::GetDirectoryName('%(_MergedWrapperRunScript.Identity)'))</_MergedWrapperDirectory>
299+
<_MergedWrapperParentDirectory>$([System.IO.Path]::GetDirectoryName('$(_MergedWrapperDirectory)'))</_MergedWrapperParentDirectory>
298300
<_MergedWrapperName>%(_MergedWrapperRunScript.FileName)</_MergedWrapperName>
299301
<_MergedWrapperRunScriptRelative Condition="'%(_MergedWrapperRunScript.Identity)' != ''">$([System.IO.Path]::GetRelativePath($(TestBinDir), %(_MergedWrapperRunScript.FullPath)))</_MergedWrapperRunScriptRelative>
300302
</PropertyGroup>
303+
<ItemGroup>
304+
<_MergedWrapperOutOfProcessTestMarkers Include="$(_MergedWrapperParentDirectory)/**/*.OutOfProcessTest" />
305+
<_MergedWrapperOutOfProcessTestFiles
306+
Include="%(_MergedWrapperOutOfProcessTestMarkers.RootDir)%(_MergedWrapperOutOfProcessTestMarkers.Directory)/**"
307+
Condition="'@(_MergedWrapperOutOfProcessTestMarkers)' != ''" />
308+
</ItemGroup>
301309

302310
<ItemGroup>
303311
<_MergedPayloadGroups Include="$(_MergedWrapperName)" />
304-
<_MergedPayloadFiles Include="$(_MergedWrapperDirectory)**" />
312+
<_MergedPayloadFiles Include="$(_MergedWrapperDirectory)/**" />
313+
<_MergedPayloadFiles Include="@(_MergedWrapperOutOfProcessTestFiles)" />
305314
<_MergedPayloadFiles Update="@(_MergedPayloadFiles)">
306315
<!-- Never use [MSBuild]::MakeRelative here! We have some files containing Unicode characters in their %(FullPath) and
307316
MakeRelative function calls Escape function internally that replaces all the Unicode characters with %<xx>. -->
@@ -312,7 +321,7 @@
312321
<ItemGroup>
313322
<!-- Remove the managed pdbs from our payloads.
314323
This is for performance reasons to reduce our helix payload size -->
315-
<ReducedMergedPayloadFilesFinal Include="@(_MergedPayloadFiles)" Condition=" '%(Extension)' != '.pdb' " />
324+
<ReducedMergedPayloadFilesFinal Include="@(_MergedPayloadFiles)" Condition=" '%(Extension)' != '.pdb' and '%(Extension)' != '.OutOfProcessTest' " />
316325
</ItemGroup>
317326

318327
<Copy SourceFiles="@(ReducedMergedPayloadFilesFinal)" DestinationFiles="@(ReducedMergedPayloadFilesFinal->'$(MergedPayloadsRootDirectory)\$(_MergedWrapperName)\%(FileRelativeToPayloadsRootDirectory)')" />

‎src/tests/Directory.Build.targets‎

Lines changed: 42 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -114,21 +114,6 @@
114114
<SkipImportILTargets Condition="'$(CLRTestBuildAllTargets)' != '' And '$(CLRTestNeedTarget)' != '$(CLRTestBuildAllTargets)'">true</SkipImportILTargets>
115115
</PropertyGroup>
116116

117-
<Target Name="CopyMergedWrapperReferences"
118-
Condition="'$(IsMergedTestRunnerAssembly)' == 'true'"
119-
AfterTargets="Build"
120-
BeforeTargets="CopyNativeProjectBinaries">
121-
<ItemGroup>
122-
<MergedWrapperReferenceFolders Include="@(ProjectReference->'$([System.IO.Path]::GetFullPath('%(ProjectReference.Identity)/..', '$(OutDir)/..'))/%(ProjectReference.FileName)')" />
123-
<!-- For merged project wrappers, include native libraries in all project references -->
124-
<MergedWrapperReferenceFiles Include="%(MergedWrapperReferenceFolders.Identity)/*$(LibSuffix)" />
125-
</ItemGroup>
126-
<Copy SourceFiles="@(MergedWrapperReferenceFiles)"
127-
DestinationFiles="@(MergedWrapperReferenceFiles->'$(OutDir)/%(FileName)%(Extension)')"
128-
SkipUnchangedFiles="true"
129-
/>
130-
</Target>
131-
132117
<Target Name="CopyNativeProjectBinaries" Condition="'$(_CopyNativeProjectBinaries)' == 'true'">
133118
<ItemGroup Condition="'$(UseVisualStudioNativeBinariesLayout)' == 'true'">
134119
<NativeProjectBinaries Include="$(NativeProjectOutputFolder)\*.*" />
@@ -225,9 +210,26 @@
225210

226211
<MSBuild Projects="$(MSBuildProjectFile)" Targets="CopyNativeProjectBinaries" Properties="NativeProjectOutputFolder=%(NativeProjectOutputFoldersToCopy.Identity)" Condition="'@(NativeProjectReference)' != ''" />
227212

213+
<MSBuild Projects="@(ProjectReference)"
214+
Targets="CopyAllNativeProjectReferenceBinaries"
215+
Condition="'$(IsMergedTestRunnerAssembly)' == 'true'"
216+
BuildInParallel="true" />
228217
</Target>
229218

230-
<Target Name="CopyAllNativeProjectReferenceBinaries" DependsOnTargets="ResolveCMakeNativeProjectReference;ConsolidateNativeProjectReference" />
219+
<Target Name="CopyAllNativeProjectReferenceBinaries"
220+
DependsOnTargets="ResolveCMakeNativeProjectReference;ConsolidateNativeProjectReference">
221+
222+
<ItemGroup Condition="'$(IsMergedTestRunnerAssembly)' == 'true'">
223+
<MergedWrapperReferenceFolders Include="@(ProjectReference->'$([System.IO.Path]::GetFullPath('%(ProjectReference.Identity)/..', '$(OutDir)/..'))/%(ProjectReference.FileName)')" />
224+
<!-- For merged project wrappers, include native libraries in all project references -->
225+
<MergedWrapperReferenceFiles Include="%(MergedWrapperReferenceFolders.Identity)/*$(LibSuffix)" />
226+
</ItemGroup>
227+
<Copy SourceFiles="@(MergedWrapperReferenceFiles)"
228+
DestinationFiles="@(MergedWrapperReferenceFiles->'$(OutDir)/%(FileName)%(Extension)')"
229+
SkipUnchangedFiles="true"
230+
Condition="'@(MergedWrapperReferenceFiles)' != ''"
231+
/>
232+
</Target>
231233

232234
<!-- Build shell or command scripts whenever we copy native binaries -->
233235
<Import Project="$(MSBuildThisFileDirectory)Common\CLRTest.Execute.targets" />
@@ -259,20 +261,40 @@
259261
</ItemGroup>
260262
</Target>
261263

262-
<Target Name="AfterBuild">
264+
<Target Name="GenerateMarkerFiles" BeforeTargets="AssignTargetPaths">
265+
<ItemGroup>
266+
<Content Include="$(AssemblyName).reflect.xml" Condition="Exists('$(AssemblyName).reflect.xml')" CopyToOutputDirectory="PreserveNewest" />
267+
<MarkerFile Include="$(IntermediateOutputPath)\$(MSBuildProjectName).MergedTestAssembly" Condition="'$(IsMergedTestRunnerAssembly)' == 'true'" />
268+
<MarkerFile Include="$(IntermediateOutputPath)\$(AssemblyName).NoMonoAot" Condition="'$(MonoAotIncompatible)' == 'true'" />
269+
<Content Include="@(MarkerFile)" CopyToOutputDirectory="PreserveNewest" />
270+
</ItemGroup>
271+
263272
<Copy SourceFiles="$(AssemblyName).reflect.xml"
264273
DestinationFolder="$(OutputPath)"
265274
Condition="Exists('$(AssemblyName).reflect.xml')"/>
266275

267276
<WriteLinesToFile
268277
Condition="'$(IsMergedTestRunnerAssembly)' == 'true'"
269-
File="$(OutputPath)\$(MSBuildProjectName).MergedTestAssembly"
270-
Lines="MergedTestAssembly" />
278+
File="$(IntermediateOutputPath)\$(MSBuildProjectName).MergedTestAssembly"
279+
Lines="MergedTestAssembly"
280+
Overwrite="true"
281+
WriteOnlyWhenDifferent="true" />
282+
271283

284+
<WriteLinesToFile
285+
Condition="'$(MonoAotIncompatible)' == 'true'"
286+
File="$(IntermediateOutputPath)\$(AssemblyName).NoMonoAot"
287+
Lines="NoMonoAot"
288+
Overwrite="true"
289+
WriteOnlyWhenDifferent="true" />
290+
291+
<!-- We don't want the out-of-process test marker file to be included in referencing projects, so we don't include it as content. -->
272292
<WriteLinesToFile
273293
Condition="'$(RequiresProcessIsolation)' == 'true' and '$(BuildAsStandalone)' != 'true'"
274294
File="$(OutputPath)\$(MSBuildProjectName).OutOfProcessTest"
275-
Lines="OutOfProcessTest" />
295+
Lines="OutOfProcessTest"
296+
Overwrite="true"
297+
WriteOnlyWhenDifferent="true" />
276298
</Target>
277299

278300
<PropertyGroup>

‎src/tests/Exceptions/ForeignThread/ForeignThreadExceptions.csproj‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
<Project Sdk="Microsoft.NET.Sdk">
22
<PropertyGroup>
33
<OutputType>Exe</OutputType>
4+
<MonoAotIncompatible>true</MonoAotIncompatible>
45
</PropertyGroup>
56
<ItemGroup>
67
<Compile Include="ForeignThreadExceptions.cs" />

0 commit comments

Comments
 (0)