Add test targets to build script

[ViktorHofer]

Add common arcade test targets and align arguments to the arcade ones.

Requires dotnet/arcade#1704 (currently myget feed for CI)
Fixes https://github.com/dotnet/corefx/issues/15814
Fixes https://github.com/dotnet/corefx/issues/35084
Fixes https://github.com/dotnet/corefx/issues/34074
Fixes https://github.com/dotnet/corefx/issues/28620

Changes (in order of significance)

1. Actions can now be chained, i.e. .\build -restore -build -buildtests -test
2. Respect common arcade test options -test, -integrationTest and -performanceTest.
3. Change unix arguments to follow posix rules and introduce a -buildtests/--buildtests action/option. Also align existing arguments to the arcade ones.
4. Set WarningsAsErrors=1
   4.1. Fix a few targets in projects which hacked renaming or moving assets in the test path.
   4.2. Remove the Microsoft.DotNet.PlatformAbstractions dependency which caused warnings in netfx and allconfigurations leg + a circular dependency.
5. Combine building test projects and archiving them into one msbuild step (yml). Add a TestProjectFilter switch (values: Tests, IntegrationTests, PerformanceTests) and change the ArchiveTests value to additionally allow filtering which projects should be archived (values: Tests, IntegrationTests, PerformanceTests, Packages, All).
6. Add a help message to both Unix and Windows scripts which additionally call into the arcade script for a complete summary.

Command Line Experience

Windows command line:

C:\git\corefx>.\build.cmd -help
Default if no actions are passed in: -restore -build

CoreFX specific settings:
  -buildtests             Build test projects
  -framework              The target group assemblies are built for (short: -f)
  -os                     The operating system assemblies are built for
  -allconfigurations      Build packages for all build configurations
  -coverage               Collect code coverage when testing
  -outerloop              Include tests which are marked as OuterLoop
  -arch                   The architecture group

Common settings:
  -configuration <value>  Build configuration: 'Debug' or 'Release' (short: -c)
  -verbosity <value>      Msbuild verbosity: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic] (short: -v)
  -binaryLog              Output binary log (short: -bl)
  -help                   Print help and exit

Actions:
  -restore                Restore dependencies (short: -r)
  -build                  Build solution (short: -b)
  -rebuild                Rebuild solution
  -deploy                 Deploy built VSIXes
  -deployDeps             Deploy dependencies (e.g. VSIXes for integration tests)
  -test                   Run all unit tests in the solution
  -pack                   Package build outputs into NuGet packages and Willow components
  -integrationTest        Run all integration tests in the solution
  -performanceTest        Run all performance tests in the solution
  -sign                   Sign build outputs
  -publish                Publish artifacts (e.g. symbols)

Advanced settings:
  -projects <value>       Semi-colon delimited list of sln/proj's to build. Globbing is supported (*.sln)
  -ci                     Set when running on CI server
  -prepareMachine         Prepare machine for CI run
  -msbuildEngine <value>  Msbuild engine to use to run build ('dotnet', 'vs', or unspecified).

Command line arguments not listed above are passed thru to msbuild.
The above arguments can be shortened as much as to be unambiguous (e.g. -co for configuration, -t for test, etc.).

Unix command line:

vihofer@HOFER-BOOK:~/corefx$ ./build.sh --help
Default if no arguments are passed in: --restore --build

CoreFX specific settings:
  --buildtests             Build test projects
  --framework              The target group assemblies are built for (short: f)
  --os                     The operating system assemblies are built for
  --allconfigurations      Build packages for all build configurations
  --coverage               Collect code coverage when testing
  --outerloop              Include tests which are marked as OuterLoop
  --arch                   The architecture group

Common settings:
  --configuration <value>    Build configuration: 'Debug' or 'Release' (short: -c)
  --verbosity <value>        Msbuild verbosity: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic] (short: -v)
  --binaryLog                Create MSBuild binary log (short: -bl)

Actions:
  --restore                  Restore dependencies (short: -r)
  --build                    Build all projects (short: -b)
  --rebuild                  Rebuild all projects
  --test                     Run all unit tests (short: -t)
  --sign                     Sign build outputs
  --publish                  Publish artifacts (e.g. symbols)
  --pack                     Package build outputs into NuGet packages and Willow components
  --help                     Print help and exit (short: -h)

Advanced settings:
  --projects <value>       Project or solution file(s) to build
  --ci                     Set when running on CI server
  --prepareMachine         Prepare machine for CI run, clean up processes after build
  --nodeReuse <value>      Sets nodereuse msbuild parameter ('true' or 'false')
  --warnAsError <value>    Sets warnaserror msbuild parameter ('true' or 'false')

Command line arguments starting with '/p:' are passed through to MSBuild.

[ViktorHofer]

cc @safern @ericstj @wtgodbe

[safern]

How can I build the test vertical without building the source vertical and without running the tests with this change?

build.cmd -test /p:BuildTests=true /p:DisableTests=true?

[ViktorHofer]

How can I build the test vertical without building the source vertical and without running the tests with this change?

build -buildTests

[ViktorHofer]

Also the root scripts should never have been named build.cmd/sh as it provides many more entry points besides building.

[ViktorHofer]

@stephentoub blocked until Jenkins is removed.

[stephentoub]

blocked until Jenkins is removed.

When will that happen? Can we close this until then?

[ViktorHofer]

As soon as https://github.com/dotnet/corefx/issues/35121 is fixed, at least that's the current plan. I would prefer to keep this PR open for visibility until then. I expect this to go in this month.

[danmoseley]

Nice!

[sandreenko]

After this change build.cmd -Release /p:OSGroup=Windows_NT /p:ArchGroup=arm fails with:

MSBUILD : error MSB1001: Unknown switch.
Switch: -Release

without -Release build.cmd /p:OSGroup=Windows_NT /p:ArchGroup=arm it fails with:

F:\git\corefx\build.proj(25,26): error MSB4057: The target "GenerateConfigurationProps" does not exist in the project.

Build FAILED.

F:\git\corefx\build.proj(25,26): error MSB4057: The target "GenerateConfigurationProps" does not exist in the project.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:23.09
Build failed.

How should we run it now?

blocks green CoreCLR testing, cc @jashook , @dotnet/jit-contrib

[safern]

For windows: build -configuration Release -arch arm -os Windows_NT
For unix: build --configuration Release --arch arm --os <OS>

You can still use /p:OSGroup and /p:ArchGroup arguments, but we're motivating people to use the switches it looks cleaner 😄

[sandreenko]

Thank you @safern, why was it done differently for Unix and Windows? How do you suggest to write cross-platform scripts that uses your scripts?

[safern]

The reason why it was done differently in windows and unix is because we followed arcade convention which is that one... @jkotas opened: dotnet/arcade#2075 to track that difference. I guess you got away with it by using the short form, -c and leave OSGroup and ArchGroup as msbuild parameters.

[sandreenko]

The reason why it was done differently in windows and unix is because we followed arcade convention which is that one... @jkotas opened: dotnet/arcade#2075 to track that difference.

Thank you.

I guess you got away with it by using the short form, -c and leave OSGroup and ArchGroup as msbuild parameters.

Yes, I think it will work as a temporary solution until we get rid of the old system that has to build tests in our repo.