Dotnet publish container regression in 8.0.400

[Vhab]

Describe the bug

Cross-posting a bug report here for extra visibility, as I suspect this can be affecting many users.

dotnet publish /t:PublishContainer throws a System.ArgumentNullException when using a specific hierarchy of ProjectReference.

To Reproduce

Code to reproduce:

https://github.com/Vhab/DotnetContainerPublish8-0-400-Repro/tree/main

Contains 3 projects.
Test depends on Test2 and Test3
Test3 depends on Test2

We try to publish Test

This hierarchy is sufficient to trigger the crash.

Exceptions (if any)

  Determining projects to restore...
  All projects are up-to-date for restore.
  Test2 -> D:\private-github\DotnetContainerPublish8-0-400-Repro\Test2\bin\Release\net8.0\Test2.dll
  Test3 -> D:\private-github\DotnetContainerPublish8-0-400-Repro\Test3\bin\Release\net8.0\Test3.dll
  Test -> D:\private-github\DotnetContainerPublish8-0-400-Repro\Test\bin\Release\net8.0\Test.dll
  Test -> D:\private-github\DotnetContainerPublish8-0-400-Repro\Test\bin\Release\net8.0\publish\
MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\RemcovanOosterhout\AppData\Local\Temp\MSBuildTemp\" and
will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
MSBUILD : error MSB4166: C:\Users\RemcovanOosterhout\AppData\Local\Temp\MSBuildTemp\MSBuild_pid-30864_86a9c20c09c04d99b1f1231a65f21341.failure.txt:
MSBUILD : error MSB4166: UNHANDLED EXCEPTIONS FROM PROCESS 30864:
MSBUILD : error MSB4166: =====================
MSBUILD : error MSB4166: 8/14/2024 10:10:31 AM
MSBUILD : error MSB4166: System.ArgumentNullException: Value cannot be null. (Parameter 'value')
MSBUILD : error MSB4166:    at System.ArgumentNullException.Throw(String paramName)
MSBUILD : error MSB4166:    at System.IO.BinaryWriter.Write(String value)
MSBUILD : error MSB4166:    at Microsoft.Build.Framework.TelemetryEventArgs.WriteToStream(BinaryWriter writer)
MSBUILD : error MSB4166:    at Microsoft.Build.Shared.LogMessagePacketBase.WriteToStream(ITranslator translator)
MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent
localPacketAvailable, AutoResetEvent localTerminatePacketPump)
MSBUILD : error MSB4166: ===================
MSBUILD : error MSB4166:
MSBUILD : error MSB4166:

Further technical details

.NET SDK version 8.0.400
Reproduced on both Windows 11 and Ubuntu 22.04.

[snailcatcher]

We have exactly the same problem in our build pipelines.
Explicitly using .NET 8.0.304 can be used as workaround, but is not a really good solution!

[snailcatcher]

I think this is the same issue as actions/runner-images#10437

[Vhab]

DOTNET_CLI_TELEMETRY_OPTOUT=1 appears to be a valid workaround.

This means the issue is localized to the telemetry code.

[baronfel]

This is indeed a hard crash in the MSBuild telemetry subsystem - we have a 401 hotfix SDK that we plan to release tomorrow to fix this regression.

Notably the CLI telemetry output did not seem to work consistently in our testing, so your mileage may vary.

[baronfel]

I'm going to treat this issue as the canonical tracking issue for this. Here's the details for the regression:

In 8.0.400, we added some telemetry (documented here) for container publish to help us direct investment in the tech. This telemetry is reported through MSBuild's built-in telemetry hooks, which for the dotnet CLI ends up forwarding telemetry from MSBuild through the CLI Telemetry's configuration (aka the environment variable mentioned above: DOTNET_CLI_TELEMETRY_OPTOUT).

There was a bug in the MSBuild engine's handling of this telemetry data, which we fixed in this PR, and this bug is the cause of the crash that's reported. We found this error during final testing, but too late to include it in the 400 release, and so began the process of making a hotfix immediately. This hotfix, 8.0.401, should be available in the next day or so.

The main reason we decided to do a hotfix for this is that the proposed workaround - setting DOTNET_CLI_TELEMETRY_OPTOUT to 1/true/other truthy values - did not consistently work in broader testing. This is because MSBuild is a multi-node system, and telemetry events generated on a worker node must be serialized and forwarded to the central node. This serialization is where the bug lies. If you can guarantee that MSBuild operates in a single-node mode (via /m:1 or similar) that may work as a workaround as well, but several high-impact use cases like azd deployment do not expose the configuration necessary to ensure this behavior.