Decide on linker to use in runtime builds

[sbomer]

#84148 changed the linker that we use in runtime builds (including when running NativeAOT on our binaries like ILCompiler and crossgen2) to lld. This was motivated by a desire to move to cross-builds (mariner host OS, targeting old linux rootfs). Using lld simplifies the cross-compilation scenarios because it is a cross-linker, so doesn't require a separate bfd linker with appropriate target prefix. We have also been wanting to move away from binutils as a dependency. Note that it doesn't change the default linker for customer NativeAOT scenarios, which still use bfd.

There are a few concerns with this approach - see @am11's comments in #84493 (comment), #84493 (comment), #84148 (comment) for full context. In summary:

• NativeAOT support for lld was only added recently and has not been well tested.
• NativeAOT uses bfd by default in user scenarios, so we should be testing this in ci.
• lld introduces size regression compared to bfd. Numbers from Write linker script for lld to enable gc-sections #84493 (comment):

# dotnet new console (~2.2 K diff)
- 1496072
+ 1498192

# dotnet new api -aot (~41 K diff)
- 12743672
+ 12785208

@am11 I hope that's a fair summary - please add to this if I missed anything.

So I'm opening this to discuss the choice of linker when building our product. Options are:

• Use lld as we do with Build on CBL-mariner host with rootfs #84148.
• Add binutils linker (need separate cross-linker for each supported architecture) to the mariner build images. We will need to pick the right prefixed linker to use based on the target platform. This is not currently supported in NativeAot. (edit: NativeAot will detect and used the right prefixed linker if available).

My preference is to continue using lld, but add separate test legs that validate the customer scenarios using bfd. One challenge is that our NativeAOT tests usually run NativeAOT on the build machine, not on helix - so they are testing a cross-compilation scenario, not the customer scenario. This is simpler than getting all NativeAOT prerequisites onto the target machines to test native compilation. I think it would be worth adding separate tests for NativeAOT compilation that run in an environment that represents customer scenarios (for example, ubuntu with bfd).

I'm opening this issue to continue the discussion and hopefully reach agreement on the approach.

@am11 @janvorli @agocke @MichalStrehovsky @jkotas

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

#84148 changed the linker that we use in runtime builds (including when running NativeAOT on our binaries like ILCompiler and crossgen2) to lld. This was motivated by a desire to move to cross-builds (mariner host OS, targeting old linux rootfs). Using lld simplifies the cross-compilation scenarios because it is a cross-linker, so doesn't require a separate bfd linker with appropriate target prefix. We have also been wanting to move away from binutils as a dependency. Note that it doesn't change the default linker for customer NativeAOT scenarios, which still use bfd.

There are a few concerns with this approach - see @am11's comments in #84493 (comment), #84493 (comment), #84148 (comment) for full context. In summary:

• NativeAOT support for lld was only added recently and has not been well tested.
• NativeAOT uses bfd by default in user scenarios, so we should be testing this in ci.
• lld introduces size regression compared to bfd. Numbers from Write linker script for lld to enable gc-sections #84493 (comment):

# dotnet new console (~2.2 K diff)
- 1496072
+ 1498192

# dotnet new api -aot (~41 K diff)
- 12743672
+ 12785208

@am11 I hope that's a fair summary - please add to this if I missed anything.

So I'm opening this to discuss the choice of linker when building our product. Options are:

• Use lld as we do with Build on CBL-mariner host with rootfs #84148.
• Add binutils linker (need separate cross-linker for each supported architecture) to the mariner build images. We will need to pick the right prefixed linker to use based on the target platform. This is not currently supported in NativeAot.

My preference is to continue using lld, but add separate test legs that validate the customer scenarios using bfd. One challenge is that our NativeAOT tests usually run NativeAOT on the build machine, not on helix - so they are testing a cross-compilation scenario, not the customer scenario. This is simpler than getting all NativeAOT prerequisites onto the target machines to test native compilation. I think it would be worth adding separate tests for NativeAOT compilation that run in an environment that represents customer scenarios (for example, ubuntu with bfd).

I'm opening this issue to continue the discussion and hopefully reach agreement on the approach.

@am11 @janvorli @agocke @MichalStrehovsky @jkotas

Author:	sbomer
Assignees:	-
Labels:	untriaged, area-NativeAOT-coreclr
Milestone:	-

[agocke]

I thought there was another consideration: we need a relatively recent version of lld to produce good results. How many distros ship with a recent-enough lld for us to use it as the default?

[sbomer]

We need at least lld 13 to use @am11's fix that improves output size: #84493. But I am not suggesting to change the default for customers - that would stay bfd either way. We will be able to use @am11's fix in runtime builds as soon as we update to llvm 16 (which I'm working on).

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

#84148 changed the linker that we use in runtime builds (including when running NativeAOT on our binaries like ILCompiler and crossgen2) to lld. This was motivated by a desire to move to cross-builds (mariner host OS, targeting old linux rootfs). Using lld simplifies the cross-compilation scenarios because it is a cross-linker, so doesn't require a separate bfd linker with appropriate target prefix. We have also been wanting to move away from binutils as a dependency. Note that it doesn't change the default linker for customer NativeAOT scenarios, which still use bfd.

There are a few concerns with this approach - see @am11's comments in #84493 (comment), #84493 (comment), #84148 (comment) for full context. In summary:

• NativeAOT support for lld was only added recently and has not been well tested.
• NativeAOT uses bfd by default in user scenarios, so we should be testing this in ci.
• lld introduces size regression compared to bfd. Numbers from Write linker script for lld to enable gc-sections #84493 (comment):

# dotnet new console (~2.2 K diff)
- 1496072
+ 1498192

# dotnet new api -aot (~41 K diff)
- 12743672
+ 12785208

@am11 I hope that's a fair summary - please add to this if I missed anything.

So I'm opening this to discuss the choice of linker when building our product. Options are:

• Use lld as we do with Build on CBL-mariner host with rootfs #84148.
• Add binutils linker (need separate cross-linker for each supported architecture) to the mariner build images. We will need to pick the right prefixed linker to use based on the target platform. This is not currently supported in NativeAot.

My preference is to continue using lld, but add separate test legs that validate the customer scenarios using bfd. One challenge is that our NativeAOT tests usually run NativeAOT on the build machine, not on helix - so they are testing a cross-compilation scenario, not the customer scenario. This is simpler than getting all NativeAOT prerequisites onto the target machines to test native compilation. I think it would be worth adding separate tests for NativeAOT compilation that run in an environment that represents customer scenarios (for example, ubuntu with bfd).

I'm opening this issue to continue the discussion and hopefully reach agreement on the approach.

@am11 @janvorli @agocke @MichalStrehovsky @jkotas

Author:	sbomer
Assignees:	-
Labels:	area-Infrastructure, untriaged, area-NativeAOT-coreclr
Milestone:	-

[am11]

Note that it doesn't change the default linker for customer NativeAOT scenarios, which still use bfd.

We recommend clang compiler in docs, we build NativeAOT runtime with llvm-libunwind, we use llvm-based objwriter to create the linkable objects, so lets make lld as a default and document it as a recommended linker for PublishAot for linux. This will be aligned with other aspects and understandable. The upside is that we will be testing with the same component in CI which we recommend for users in docs.

So lets apply the patch, turn default to lld and cleanup LinkerFlavor=lld sprinkled in various places. That will hopefully resolve this matter satisfying everyone.

Alternatives are making less sense to me:

We have also been wanting to move away from binutils as a dependency.

Why is this? binutils is used since .NET Core 1.0, there is no issue with licensing in using binutils to build any software; proprietary or otherwise.

We will need to pick the right prefixed linker to use based on the target platform. This is not currently supported in NativeAot.

This is not correct. The "toolchain" does this job. If configured properly, we don't need to manually specify prefixes for cross-compilation.

How many distros ship with a recent-enough lld for us to use it as the default?

You will find it pretty hard to discover a distro which is not using bfd as their default linker, let alone the versioning concerns. bfd, aka GNU linker, has decades of more experience than other linkers, still actively maintained, choice of top distros and beats other linkers in binary-size benchmarks.

@am11 I hope that's a fair summary - please add to this if I missed anything.

• we can no longer build runtime repo with binutils alone.

i.e. LinkerFlavor=lld if target==linux constructs added are missing and Compiler!=gcc condition; we set Compiler at the start of build:

runtime/eng/build.sh

Line 432 in ed0f1c5

arguments="$arguments /p:Compiler=$compiler /p:CppCompilerAndLinker=$compiler"

(maps to ./build.sh -gcc)

[MichalStrehovsky]

I don't have a strong opinion on this. We're already in a situation where we can't test E2E the thing that will be produced by the toolchain because the version of ld is out of our control. The solution to that could be to just ship lld in the ILCompiler package. But that's a can of worms since constructing the command line to invoke the linker is per-distro bespoke magic and it's also the reason why we use clang as the "linker" instead of invoking ld directly.

Ubuntu 18 comes with lld that is ancient. Ubuntu 22 has a good one. I don't have a good sense of the landscape. If we tell people to use lld but the version we need is hard to obtain, that would be a problem.

I also don't like testing with a completely different linker codebase (bfd vs lld) than what we use on customer machines. We did run into linker issues in the past. At least using the same codebase (even though a different version) would keep things more sane.

[kg]

This has broken builds on Debian for me.

ILCompiler -> /home/kate/Projects/dotnet-runtime-wasm/artifacts/bin/coreclr/linux.x64.Release/ilc/ilc.dll
  Generating native code
clang : error : invalid linker name in argument '-fuse-ld=lld' [/home/kate/Projects/dotnet-runtime-wasm/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]
/home/kate/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/build/Microsoft.NETCore.Native.targets(335,5): error MSB3073: The command ""/usr/bin/clang-11" "/home/kate/Projects/dotnet-runtime-wasm/artifacts/obj/coreclr/ILCompiler/x64/Release/native/ilc.o" -o "/home/kate/Projects/dotnet-runtime-wasm/artifacts/bin/coreclr/linux.x64.Release/ilc/native/ilc" -fuse-ld=lld /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/sdk/libbootstrapper.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/sdk/libRuntime.ServerGC.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/sdk/libeventpipe-disabled.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/sdk/libstdc++compat.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/sdk/libnumasupportdynamic.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/framework/libSystem.Native.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/framework/libSystem.Globalization.Native.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/framework/libSystem.IO.Compression.Native.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/framework/libSystem.Net.Security.Native.a /home/kate/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.3.23174.8/framework/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lz -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--discard-all" exited with code 1. [/home/kate/Projects/dotnet-runtime-wasm/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]

I do have llvm, clang, and lld in my /usr/bin, but they have a version number on the end. I tried helping the build process out by using ln -s to put a 'lld' in my PATH pointing at the right one, but that didn't fix it. What do I do?

[am11]

@kg, this is the situation when we have different versions of clang and lld installed. e.g. if I have clang-16, clang-14 and lld-14, installed, then the following things happen:

1. ./build.sh via init-compiler will select clang-16 (because it is the latest version)
2. init-compiler will introspect if fuse-ld=lld and based on the result, will select the linker
3. then changes in Build on CBL-mariner host with rootfs #84148 will "force" fuse-ld=lld unconditionally, disregarding #2's introspection based linker selection, and fail the build because we didn't install lld-16

This is exactly how

• we can no longer build runtime repo with binutils alone.

community-driven scenarios are also affected..

Workaround is to apt install lld-<version of clang which build is picking up>

[kg]

Workaround is to apt install lld-<version of clang which build is picking up>

Thanks, that indeed worked. I did
/usr/bin/clang -v
which output Debian clang version 11.0.1-2, so sudo apt-get install lld-11 caused the build to succeed.

[tmds]

cc @omajid

[sbomer]

We have also been wanting to move away from binutils as a dependency.

Why is this?

I think mainly for the same reasoning as with lld lld - to simplify our official (cross) builds and avoid having to detect or install prefixed versions of objcopy and other tools. But I agree with your feedback that we should continue supporting binutils-based builds, at least for native builds.

We will need to pick the right prefixed linker to use based on the target platform. This is not currently supported in NativeAot.

This is not correct. The "toolchain" does this job. If configured properly, we don't need to manually specify prefixes for cross-compilation.

Thanks for the correction - fixed above. I got my wires crossed when I was thinking about the gcc support mentioned in #78559.

we can no longer build runtime repo with binutils alone.

Filed #84934 to track this.

Regarding runtime linker defaults: it sounds like we are all on board with using lld as the default for runtime builds, as long as it's optional.

Regarding customer NativeAot linker defaults: I don't have a strong opinion on what the default should be, but if we consider both of them to be supported, we should have testing in place for both, at least to cover basic scenarios, even if most of our testing uses the default.