Skip to content

Add common arcade test targets #1704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Feb 21, 2019
Merged

Add common arcade test targets #1704

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2b1c5ee
Add common arcade test targets
ViktorHofer Jan 7, 2019
934eccc
Remove GenerateTestScript switch
ViktorHofer Jan 7, 2019
6fd6013
Right case Outerloop category name
ViktorHofer Jan 16, 2019
6e5985d
Minor cleanup
ViktorHofer Feb 18, 2019
33210a6
Fix perf test copying
ViktorHofer Feb 18, 2019
4b63058
Fine grained ArchiveTests option
ViktorHofer Feb 19, 2019
9bdcdbf
Time ArchiveTests after PrepareForRun
ViktorHofer Feb 20, 2019
5da266f
Change ArchiveTests prop to enum
ViktorHofer Feb 20, 2019
5d94a7a
Remove wrong GenerateRunScript i&o
ViktorHofer Feb 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/Microsoft.DotNet.CoreFxTesting/build/Microsoft.DotNet.CoreFxTesting.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,10 @@
<Import Condition="'$(BuildingUAPVertical)' == 'true'" Project="$(MSBuildThisFileDirectory)Resources.uap.targets" />

<!-- Performance test data upload to Benchview. -->
<Import Condition="'$(EnableBenchviewTarget)' == 'true' AND '$(Performance)' == 'true' AND '$(SkipTests)' != 'true'" Project="$([MSBuild]::NormalizePath('$(TestCoreDir)', 'performance', 'Benchview.targets'))" />
<Import Condition="'$(EnableBenchviewTarget)' == 'true'" Project="$([MSBuild]::NormalizePath('$(TestCoreDir)', 'performance', 'Benchview.targets'))" />

<!-- Full coverage report. -->
<Import Condition="'$(EnableFullCoverageReportTarget)' == 'true' AND '$(SkipCoverageReport)' != 'true' AND '$(Coverage)' == 'true' AND '$(SkipTests)' != 'true'" Project="$([MSBuild]::NormalizePath('$(TestCoreDir)', 'coverage', 'CoverageReport.targets'))" />
<Import Condition="'$(EnableFullCoverageReportTarget)' == 'true' AND '$(SkipCoverageReport)' != 'true' AND '$(Coverage)' == 'true'" Project="$([MSBuild]::NormalizePath('$(TestCoreDir)', 'coverage', 'CoverageReport.targets'))" />

<!-- Import the core testing infrastructure only for test projects. -->
<Import Condition="'$(IsTestProject)' == 'true' AND '$(IsTestSupportProject)' != 'true'" Project="$(TestCoreDir)Core.targets" />
Expand Down
1 change: 0 additions & 1 deletion src/Microsoft.DotNet.CoreFxTesting/build/core/Core.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
$(TestDependsOn);
GenerateRunScript;
RunTests;
ArchiveTestBuild;
</TestDependsOn>
</PropertyGroup>

Expand Down
24 changes: 9 additions & 15 deletions src/Microsoft.DotNet.CoreFxTesting/build/core/Core.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,10 @@
</Target>

<!-- Archive test binaries along with supporting files. -->
<Target Name="ArchiveTestBuild"
DependsOnTargets="GenerateRunScript"
Condition="'$(ArchiveTests)' == 'true'">
<Target Name="ArchiveTests"
Condition="'$(ArchiveTests)' == 'true' OR $(MSBuildProjectName.EndsWith('.$(ArchiveTests)'))"
AfterTargets="PrepareForRun"
DependsOnTargets="GenerateRunScript">

<Error Condition="'$(TestArchiveTestsDir)' == ''" Text="TestArchiveTestsDir property to archive the test folder must be set." />

Expand All @@ -82,6 +83,8 @@
</Target>

<Target Name="GenerateRunScript"
Inputs="unused"
Outputs="$(RunScriptOutputPath)"
Copy link
Member

@ericstj ericstj Feb 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why inputs outputs if you don't use them for incremental nor batching?

Copy link
Member Author

@ViktorHofer ViktorHofer Feb 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GenerateRunScript is part of the incremental build, at least that's what I believed.

Copy link
Member

@ericstj ericstj Feb 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant that if you have no inputs and only outputs I would expect that target to always run.

DependsOnTargets="$(GenerateRunScriptDependsOn)">

<PropertyGroup>
Expand Down Expand Up @@ -171,19 +174,15 @@
<Message Text="ValidateTestPlatform found TargetOS of [$(TargetOS)]." Importance="Low" />

<Message Condition="'%(UnsupportedPlatformsItems.Identity)' == '$(TargetOS)'"
Text="Skipping tests in $(AssemblyName) because it is not supported on $(TargetOS)" />
Text="Skipping tests in $(AssemblyName) because it is not supported on $(TargetOS)" />

<Message Condition="'$(ConfigurationErrorMsg)' != ''"
Text="Skipping tests in $(AssemblyName) because there is no configuration compatible with the current BuildConfiguration." />
Text="Skipping tests in $(AssemblyName) because there is no configuration compatible with the current BuildConfiguration." />

</Target>

<!--
SkipTests is a global property used in CI runs. Skips the test execution but
still generates the test execution script and archives the test dir for Helix consumption.
-->
<Target Name="RunTests"
Condition="'$(SkipTests)' != 'true' OR '$(TestDisabled)' == 'true'"
Copy link
Member

@ericstj ericstj Feb 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting that this was inverted before. No-one was using it?

Copy link
Member Author

@ViktorHofer ViktorHofer Feb 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I freaked out when I saw that condition 🤐

Copy link
Member Author

@ViktorHofer ViktorHofer Feb 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea, but will follow-up on that one.

Condition="'$(TestDisabled)' != 'true'"
DependsOnTargets="$(RunTestsDependsOn)"
Inputs="@(RunTestsInputs)"
Outputs="@(RunTestsOutputs)">
Expand Down Expand Up @@ -216,11 +215,6 @@

</Target>

<!-- Main test targets -->
<Target Name="Test" DependsOnTargets="$(TestDependsOn)" Condition="'$(DisableTestTargets)' != 'true'" />
<Target Name="BuildAndTest" DependsOnTargets="Build;Test" />
<Target Name="RebuildAndTest" DependsOnTargets="Rebuild;Test" />

<Import Condition="'$(BuildingUAPVertical)' == 'true'" Project="$(MSBuildThisFileDirectory)Core.uap.targets" />

<!--
Expand Down
11 changes: 5 additions & 6 deletions src/Microsoft.DotNet.CoreFxTesting/build/core/performance/Performance.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@
<Import Condition="'$(IsBenchmarkDotNetProject)' == 'true'" Project="$([MSBuild]::NormalizePath('$(MSBuildThisFileDirectory)', 'runner', 'BenchmarkDotNet.targets'))" />

<PropertyGroup>
<!-- Disable test targets if global property performance is set to false. -->
<DisableTestTargets Condition="'$(Performance)' == 'false'">true</DisableTestTargets>

<MeasurementPyCommand>$(PythonCommand) "$(BenchviewDir)measurement.py" xunit "$(RunId)-$(AssemblyName).xml" --better desc --drop-first-value --append -o "$(RepoRoot)measurement.json" || $(CliExitErrorCommand)</MeasurementPyCommand>
</PropertyGroup>

Expand All @@ -23,9 +20,6 @@

<Target Name="ValidatePerfConfigurations">

<Warning Condition="!$(ConfigurationGroup.ToLower().Contains('release'))"
Text="You are running performance tests in a configuration other than Release. Your results may be unreliable." />

<Error Condition="'$(PerformanceType)' != 'Diagnostic' AND '$(PerformanceType)' != 'Profile'"
Text="Invalid Performance Type value specified: $(PerformanceType)" />

Expand All @@ -48,4 +42,9 @@
</When>
</Choose>

<!-- Main test targets -->
<Target Name="PerformanceTest" DependsOnTargets="$(TestDependsOn)" Condition="'$(IsPerformanceTestProject)' == 'true'" />
<Target Name="BuildAndPerformanceTest" DependsOnTargets="Build;PerformanceTest" Condition="'$(IsPerformanceTestProject)' == 'true'" />
<Target Name="RebuildAndPerformanceTest" DependsOnTargets="Rebuild;PerformanceTest" Condition="'$(IsPerformanceTestProject)' == 'true'" />

</Project>
2 changes: 1 addition & 1 deletion src/Microsoft.DotNet.CoreFxTesting/build/core/performance/runner/BenchmarkDotNet.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
</PropertyGroup>

<!-- Helix dependencies -->
<ItemGroup Condition="'$(ArchiveTests)' == 'true' AND '$(IncludePerformanceTestFrameworkReferences)' == 'true'">
<ItemGroup Condition="'$(IncludePerformanceTestFrameworkReferences)' == 'true'">
<TestArchiveDependencies Include="$(RuntimePath)BenchmarkDotNet.dll" />
<TestArchiveDependencies Include="$(RuntimePath)BenchmarkDotNet.Diagnostics.Windows.dll" />
</ItemGroup>
Expand Down
6 changes: 3 additions & 3 deletions src/Microsoft.DotNet.CoreFxTesting/build/core/performance/runner/xunit-performance.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,13 @@
</ItemGroup>

<!-- Helix dependencies -->
<ItemGroup Condition="'$(ArchiveTests)' == 'true' AND '$(IncludePerformanceTestFrameworkReferences)' == 'true'">
<ItemGroup Condition="'$(IncludePerformanceTestFrameworkReferences)' == 'true'">
<TestArchiveDependencies Include="$(RuntimePath)Microsoft.DotNet.XUnitExtensions.dll" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.assert.dll" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.core.dll" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.abstractions.dll" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.performance.api" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.performance.core" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.performance.api.dll" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.performance.core.dll" />
</ItemGroup>

</Project>
2 changes: 1 addition & 1 deletion src/Microsoft.DotNet.CoreFxTesting/build/core/test/Test.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
</ItemGroup>

<!-- Helix dependencies -->
<ItemGroup Condition="'$(ArchiveTests)' == 'true' AND '$(IncludeTestFrameworkReferences)' == 'true'">
<ItemGroup Condition="'$(IncludeTestFrameworkReferences)' == 'true'">
<TestArchiveDependencies Include="$(RuntimePath)Microsoft.DotNet.XUnitExtensions.dll" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.assert.dll" />
<TestArchiveDependencies Include="$(RuntimePath)xunit.core.dll" />
Expand Down
11 changes: 10 additions & 1 deletion src/Microsoft.DotNet.CoreFxTesting/build/core/test/Test.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
<_withoutCategories Condition="'$(WithoutCategories)' != ''">;$(WithoutCategories.Trim(';'))</_withoutCategories>
<!-- Default non categories -->
<_withoutCategories Condition="!$(_withCategories.Contains('failing'))">$(_withoutCategories);failing</_withoutCategories>
<_withoutCategories Condition="'$(Outerloop)' != 'true'">$(_withoutCategories);Outerloop</_withoutCategories>
<_withoutCategories Condition="'$(OuterLoop)' != 'true'">$(_withoutCategories);OuterLoop</_withoutCategories>
<RunArguments>$(RunArguments)$(_withCategories.Replace(';', ' -trait category='))</RunArguments>
<RunArguments>$(RunArguments)$(_withoutCategories.Replace(';', ' -notrait category='))</RunArguments>

Expand Down Expand Up @@ -150,4 +150,13 @@

</Choose>

<!-- Main test targets -->
<Target Name="Test" DependsOnTargets="$(TestDependsOn)" Condition="'$(IsUnitTestProject)' == 'true'" />
<Target Name="BuildAndTest" DependsOnTargets="Build;Test" Condition="'$(IsUnitTestProject)' == 'true'" />
<Target Name="RebuildAndTest" DependsOnTargets="Rebuild;Test" Condition="'$(IsUnitTestProject)' == 'true'" />

<Target Name="IntegrationTest" DependsOnTargets="$(TestDependsOn)" Condition="'$(IsIntegrationTestProject)' == 'true'" />
<Target Name="BuildAndIntegrationTest" DependsOnTargets="Build;IntegrationTest" Condition="'$(IsIntegrationTestProject)' == 'true'" />
<Target Name="RebuildAndIntegrationTest" DependsOnTargets="Rebuild;IntegrationTest" Condition="'$(IsIntegrationTestProject)' == 'true'" />

</Project>