[xUnit v3] Catastrophic failure: Test process did not return valid JSON (non-object) when using .NET 10 RC2

[eiriktsarpalis]

Opening duplicate issue to xunit/xunit#3413 since it appears to be impacting many repos (including first party repos) that are upgrading to .NET 10 RC2. I would appreciate if if the SDK team could take a look at this problem as well:

I opened this PR migrating my repo to the .NET 10 RC1 SDK. While everything appears to be working fine on my local dev machine, I'm getting the following error consistently when running tests in Windows github actions only:

[xUnit.net 00:00:00.25] PolyType.Roslyn.Tests: Catastrophic failure: System.InvalidOperationException: Test process did not return valid JSON (non-object).
No test is available in D:\a\PolyType\PolyType\tests\PolyType.Roslyn.Tests\bin\Debug\net10.0\PolyType.Roslyn.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
   at Xunit.v3.TestProcessLauncherAdapter.GetAssemblyInfo(ITestProcessLauncher launcher, XunitProjectAssembly projectAssembly)
   at Xunit.v3.TestProcessLauncherAdapter.GetAssemblyInfo(XunitProjectAssembly projectAssembly)
   at Xunit.Runner.v3.Xunit3..ctor(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.v3.Xunit3.ForDiscoveryAndExecution(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.XunitFrontController.Create(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)

The issue does not repro in my local Windows machine, nor does it repro in the macOS and Linux legs of the same build. It does not repro when attempting to run the same tests targeting non net10.0 targets.

I'm pretty much stumped at this point, I would appreciate any hints that might help me better debug the issue. Thanks.

[marcpopMSFT]

@dotnet/dotnet-testing-admin to prioritize looking into this and identifying any workarounds.

[tkapin]

@Youssef1313 @nohwnd - this seems to require immediate attention, please prioritize it.

[martincostello]

xunit/xunit#3413 (comment)

[halter73]

xunit/xunit#3413 (comment)

The takeaway here is that this is likely a dupe of #51265, right?

[martincostello]

Seems it - I guess that something is trimming away a dependency xunit needs and then it horribly crashes and burns before it can give any useful output.

[Youssef1313]

cc @nkolev92 @dsplaisted as well

[nohwnd]

The original issue with the crash seems to be the same as #50950, where covrun64.dll is not found. Which would explain why you can only see it when collecting coverage. In interactive session there is pop up, but I assume on CI there is none, and it will just directly continue to exiting the process.

Reproes locally when rc1 sdk is used, but also reproes when latest rtm is used.

So I am unsure if this is 100% the same bug, because you had trouble reproducing.

I am able to reproduce like this:

dotnet restore "$PSScriptRoot\tests\PolyType.Tests\PolyType.Tests.csproj"

dotnet build --no-restore "$PSScriptRoot\tests\PolyType.Tests\PolyType.Tests.csproj"

dotnet test --no-build "$PSScriptRoot\tests\PolyType.Tests\PolyType.Tests.csproj" --configuration Debug -p:ContinuousIntegrationBuild=true -warnAsError -warnNotAsError:NU1901,NU1902,NU1903,NU1904 --blame
-p:SkipTUnitTestRuns=true --results-directory testResults
--collect "Code Coverage;Format=cobertura" --logger "console;verbosity=normal"
--logger "trx" `
-tl:off -- RunConfiguration.CollectSourceInformation=true

But the other bug reproduces with just dotnet test --collect "Code Coverage;Format=cobertura" so that should be enough as well.

[nohwnd]

Workaround and more info on the bug in the linked issue: #50950 (comment)

[ericstj]

Double checking -- you were hitting this on .NETFramework target, correct?

Never mind, @eiriktsarpalis let me know offline there are two issues discussed here. The pruning one and #50950