Enable using the Ninja generator instead of Visual Studio on Windows builds. (#41897)

* Move DacTableGen out of the CMake build and into tools-local.

* Attempt: usenmakefiles means ninja.

* Make changes to Windows build to support non-VS generators like ninja.

* Use CMake 3.16 PCHs.

* Remove explicit process count.

* Build and run DacTableGen via the .NET CLI we pull down for the build instead of via CMake.

* Update configurecompiler.cmake to specify language (needed for Ninja).

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Get Ninja build working again.

* Pass configuration to cmake configure step for Ninja build.

* Rename flag

* Fix configure step for cross-arch native build

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Support building via Ninja for coreclr via an MSBuild property

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Implement support for opening the CoreCLR solution in VS via the -vs flag.

* Fix setting generator to Ninja.

* Merge libraries gen-buildsys-win.bat with runtime gen-buildsys.cmd to share windows cmake configure infra.

* Enable building libraries native build with ninja.

* Centralize setting CMAKE_SYSTEM_VERSION for windows builds.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Update host build to use shared gen-buildsys.cmd

* Fix multi-config generator detection.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Exit with success when we build successfully.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Enable and fix up Ninja build for hosts.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Only link delayimp.lib on Windows

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Remove "overriding /W3 with /W1" warning in Ninja build of hosts by turning on /W3 and disabling any new warnings we hit.

* Reduce command line override warnings by adopting the MSVC_RUNTIME_LIBRARY abstraction feature in CMake 3.14+ (below the minimum requirement on Windows).

* Use the delayed expansion syntax to correctly pass in the config to cmake.

* Move all usages of MSVC runtime library selection to the CMake abstraction to reduce console spew about overridden flags.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Condition the rest of the compiler options correctly so they don't show up as invalid parameter warnings for assembly builds.

* Correctly build DIALib and DacTableGen as AnyCPU

* Fix WRITE_BARRIER_CHECK define

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Fix passing arch for MSBuild build.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Configure ASM compiler command line format in configurecompiler.cmake and use CMake's assembly compilation step to build ASM files for ARM/ARM64 instead of manually building them and adding the generated files as additional sources.

* Cleanup after fixing ARM/ARM64 support.

* Rename variable

* Condition PCH's for C++-only as needed to fix using them on non-Windows (where we compile C as C instead of as C++).

* Fix CoreCLR native test build on windows to point to new gen-buildsys location.

* Add missing compilation options in Ninja build that are implicitly defined in the MSBuild build.

* Disable analyzers for DacTableGen. Remove /MAP linker flag (that's for local testing).

* Upgrade to CMake 3.16.4

* Don't warn on unused variables in CMake (these usually vary on platform/config)

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Manually compile ARM and ARM64 asm files when using the VS generator since it doesn't support emitting MARMASM item types in projects (and as a result cannot compile ARM or ARM64 asm without assistance).

* Enable CMake policy in test build to use CMAKE_MSVC_RUNTIME_LIBRARY variable.

* Update initially disabled warnings in host.

* Add -ninja arg to root scripts to enable using Ninja.

* Enable using Ninja in CoreCLR CI.

* Try to fix passing the -ninja arg in yaml.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Add ninja to the Windows invocation

* Fix 2-phase cross targeting to do one phase at a time and initialize the VC++ environment for the correct build tools each time instead of interleaving them and relying on Visual Studio to handle the differing targets.

* Restore old /Ox optimization setting for Release.

* Fix typos, don't pass exception flags to C files.

* Remove unneeded pragma.

* Fix .pgd file installation.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Add mention of the `-ninja` flag in the Building CoreCLR documentation.

Signed-off-by: Jeremy Koritzinsky <[email protected]>

* Mention installation locations for Ninja and warn while building if CMake is older than 3.16.0.

* Update docs/workflow/requirements/windows-requirements.md

Author: jkoritzinsky

docs/workflow/building/coreclr/README.md

@@ -22,6 +22,13 @@ CoreCLR also supports a 'checked' build type which has asserts enabled like 'deb
 ./build.sh -subset clr -configuration checked
 ```
 
+If you want to use Ninja to drive the native build instead of Visual Studio MSBuild (on Windows) or Make (on non-Windows), you can pass the `-ninja` flag to the build script as follows:
+```
+./build.cmd -subset clr -ninja
+```
+
+We recommend using Ninja for building the project on Windows since it more efficiently uses the build machine's resources for the native runtime build in comparison to Visual Studio's MSBuild.
+
 To pass extra compiler/linker flags to the coreclr build, set the environment variables `EXTRA_CFLAGS`, `EXTRA_CXXFLAGS` and `EXTRA_LDFLAGS` as needed. Don't set `CFLAGS`/`CXXFLAGS`/`LDFLAGS` directly as that might lead to configure-time tests failing.
 
 This will produce outputs as follows:

docs/workflow/requirements/freebsd-requirements.md

@@ -57,11 +57,12 @@ Install the following packages:
 - lttng-ust
 - krb5
 - openssl (optional)
+- ninja (optional, enables building native code with ninja instead of make)
 
 The lines to install all the packages above using package manager.
 
 ```sh
-sudo pkg install --yes libunwind icu libinotify lttng-ust krb5 cmake autoconf automake openssl
+sudo pkg install --yes libunwind icu libinotify lttng-ust krb5 cmake autoconf automake openssl ninja
 ```
 
 Additionally, working dotnet cli with SDK is needed. On other platforms this would be downloaded automatically during build but it is not currently available for FreeBSD.

docs/workflow/requirements/linux-requirements.md

@@ -50,6 +50,7 @@ Install the following packages for the toolchain:
 - libkrb5-dev
 - libnuma-dev (optional, enables numa support)
 - zlib1g-dev
+- ninja (optional, enables building native code with ninja instead of make)
 
 The following dependencies are needed if Mono Runtime is enabled (default behavior):
 
@@ -61,7 +62,7 @@ The following dependencies are needed if Mono Runtime is enabled (default behavi
 sudo apt-get install -y cmake llvm-9 clang-9 autoconf automake \
 libtool build-essential python curl git lldb-6.0 liblldb-6.0-dev \
 libunwind8 libunwind8-dev gettext libicu-dev liblttng-ust-dev \
-libssl-dev libnuma-dev libkrb5-dev zlib1g-dev
+libssl-dev libnuma-dev libkrb5-dev zlib1g-dev ninja
 ```
 
 You now have all the required components.

docs/workflow/requirements/macos-requirements.md

@@ -28,8 +28,9 @@ Install the following packages:
 - openssl 1.1
 - pkg-config
 - python3
+- ninja (optional, enables building native code with ninja instead of make)
 
-You can install all the packages above using Homebrew by running this command in the repository root:
+You can install all the required packages above using Homebrew by running this command in the repository root:
 
 ```
 brew bundle --no-lock --file eng/Brewfile

docs/workflow/requirements/windows-requirements.md

@@ -42,7 +42,13 @@ The dotnet/runtime repository requires at least Visual Studio 2019 16.6.
 - Add its location (e.g. C:\Program Files (x86)\CMake\bin) to the PATH environment variable.
   The installation script has a check box to do this, but you can do it yourself after the fact following the instructions at [Adding to the Default PATH variable](#adding-to-the-default-path-variable).
 
-The dotnet/runtime repository requires at least CMake 3.15.5.
+The dotnet/runtime repository recommends using CMake 3.16.0 or newer, but works with CMake 3.15.5.
+
+## Ninja (optional)
+
+- Install Ninja in one of the two following ways
+  - [Download the executable](https://github.com/ninja-build/ninja/releases) and add its location to [the Default PATH variable](#adding-to-the-default-path-variable).
+  - [Install via a package manager](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages), which should automatically add it to the PATH environment variable.
 
 ## Python
 

eng/Subsets.props

@@ -86,6 +86,7 @@
   <ItemGroup>
     <!-- CoreClr -->
     <SubsetName Include="Clr" Description="The CoreCLR runtime, LinuxDac, CoreLib (+ native), tools and packages." />
+    <SubsetName Include="Clr.DacTools" Description="Managed tools that support building a runtime that supports debugging (such as DacTableGen)." />
     <SubsetName Include="Clr.Runtime" Description="The CoreCLR .NET runtime." />
     <SubsetName Include="Clr.PalTests" Description="The CoreCLR PAL tests." />
     <SubsetName Include="Clr.PalTestList" Description="Generate the list of the CoreCLR PAL tests. When using the command line, use Clr.PalTests instead." />
@@ -148,6 +149,12 @@
     <ProjectToBuild Include="$(CoreClrProjectRoot)src\System.Private.CoreLib\System.Private.CoreLib.csproj" Category="clr" />
   </ItemGroup>
 
+  <ItemGroup Condition="'$(TargetOS)' == 'Windows_NT' and ($(_subset.Contains('+clr.dactools+')) or $(_subset.Contains('+clr.runtime+')))">
+    <ProjectToBuild Include="$(CoreClrProjectRoot)src\ToolBox\SOS\DacTableGen\DacTableGen.csproj;
+                             $(CoreClrProjectRoot)src\ToolBox\SOS\DIALib\DIALib.ilproj"
+                    Category="clr"/>
+  </ItemGroup>
+
   <PropertyGroup Condition="$(_subset.Contains('+clr.runtime+'))">
     <ClrRuntimeBuildSubsets>$(ClrRuntimeBuildSubsets);ClrRuntimeSubset=true;ClrJitSubset=true</ClrRuntimeBuildSubsets>
   </PropertyGroup>

eng/build.ps1

@@ -16,6 +16,7 @@ Param(
   [ValidateSet("Debug","Release","Checked")][string][Alias('rc')]$runtimeConfiguration,
   [ValidateSet("Debug","Release")][string][Alias('lc')]$librariesConfiguration,
   [ValidateSet("CoreCLR","Mono")][string][Alias('rf')]$runtimeFlavor,
+  [switch]$ninja,
   [Parameter(ValueFromRemainingArguments=$true)][String[]]$properties
 )
 
@@ -74,6 +75,9 @@ function Get-Help() {
   Write-Host "  -testscope              Scope tests, allowed values: innerloop, outerloop, all."
   Write-Host ""
 
+  Write-Host "Native build settings:"
+  Write-Host "  -ninja                  Use Ninja instead of MSBuild to run the native build."
+
   Write-Host "Command-line arguments not listed above are passed through to MSBuild."
   Write-Host "The above arguments can be shortened as much as to be unambiguous."
   Write-Host "(Example: -con for configuration, -t for test, etc.)."
@@ -117,9 +121,28 @@ if ($subset -eq 'help') {
 }
 
 if ($vs) {
-  . $PSScriptRoot\common\tools.ps1
-
-  if (-Not (Test-Path $vs)) {
+  if ($vs -ieq "coreclr.sln") {
+    # If someone passes in coreclr.sln (case-insensitive),
+    # launch the generated CMake solution.
+    $archToOpen = $arch[0]
+    $configToOpen = $configuration[0]
+    if ($runtimeConfiguration) {
+      $configToOpen = $runtimeConfiguration
+    }
+    $vs = Split-Path $PSScriptRoot -Parent | Join-Path -ChildPath "artifacts\obj\coreclr" | Join-Path -ChildPath "Windows_NT.$archToOpen.$((Get-Culture).TextInfo.ToTitleCase($configToOpen))" | Join-Path -ChildPath "CoreCLR.sln"
+    if (-Not (Test-Path $vs)) {
+      $repoRoot = Split-Path $PSScriptRoot -Parent
+      Invoke-Expression "& `"$repoRoot/src/coreclr/build-runtime.cmd`" -configureonly -$archToOpen -$configToOpen"
+      if ($lastExitCode -ne 0) {
+        Write-Error "Failed to generate the CoreCLR solution file."
+        exit 1
+      }
+      if (-Not (Test-Path $vs)) {
+        Write-Error "Unable to find the CoreCLR solution file at $vs."
+      }
+    }
+  }
+  elseif (-Not (Test-Path $vs)) {
     $solution = $vs
 
     if ($runtimeFlavor -eq "Mono") {
@@ -153,6 +176,8 @@ if ($vs) {
       }
     }
   }
+  
+  . $PSScriptRoot\common\tools.ps1
 
   # This tells .NET Core to use the bootstrapped runtime
   $env:DOTNET_ROOT=InitializeDotNetCli -install:$true -createSdkLocationFile:$true
@@ -204,6 +229,7 @@ foreach ($argument in $PSBoundParameters.Keys)
     "allconfigurations"      { $arguments += " /p:BuildAllConfigurations=true" }
     "properties"             { $arguments += " " + $properties }
     "verbosity"              { $arguments += " -$argument " + $($PSBoundParameters[$argument]) }
+    "ninja"                  { $arguments += " /p:Ninja=$($PSBoundParameters[$argument])" }
     # configuration and arch can be specified multiple times, so they should be no-ops here
     "configuration"          {}
     "arch"                   {}

eng/build.sh

@@ -77,6 +77,7 @@ usage()
   echo "  --gccx.y                   Optional argument to build using gcc version x.y."
   echo "  --portablebuild            Optional argument: set to false to force a non-portable build."
   echo "  --keepnativesymbols        Optional argument: set to true to keep native symbols/debuginfo in generated binaries."
+  echo "  --ninja                    Optional argument: set to true to use Ninja instead of Make to run the native build."
   echo ""
 
   echo "Command line arguments starting with '/p:' are passed through to MSBuild."
@@ -417,6 +418,26 @@ while [[ $# > 0 ]]; do
       shift 2
       ;;
 
+
+      -ninja)
+      if [ -z ${2+x} ]; then
+        arguments="$arguments /p:Ninja=true"
+        shift 1
+      else
+        ninja="$(echo "$2" | awk '{print tolower($0)}')"
+        if [ "$ninja" = true ]; then
+          arguments="$arguments /p:Ninja=true"
+          shift 2
+        elif [ "$ninja" = false ]; then
+          arguments="$arguments /p:Ninja=false"
+          shift 2
+        else
+          arguments="$arguments /p:Ninja=true"
+          shift 1
+        fi
+      fi
+      ;;
+
       *)
       extraargs="$extraargs $1"
       shift 1