[xUnit v3] Catastrophic failure: Test process did not return valid JSON (non-object) when targeting `net10.0`.

[eiriktsarpalis]

I opened this PR migrating my repo to the .NET 10 RC1 SDK. While everything appears to be working fine on my local dev machine, I'm getting the following error consistently when running tests in Windows github actions only:

[xUnit.net 00:00:00.25] PolyType.Roslyn.Tests: Catastrophic failure: System.InvalidOperationException: Test process did not return valid JSON (non-object).
No test is available in D:\a\PolyType\PolyType\tests\PolyType.Roslyn.Tests\bin\Debug\net10.0\PolyType.Roslyn.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
   at Xunit.v3.TestProcessLauncherAdapter.GetAssemblyInfo(ITestProcessLauncher launcher, XunitProjectAssembly projectAssembly)
   at Xunit.v3.TestProcessLauncherAdapter.GetAssemblyInfo(XunitProjectAssembly projectAssembly)
   at Xunit.Runner.v3.Xunit3..ctor(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.v3.Xunit3.ForDiscoveryAndExecution(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.XunitFrontController.Create(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)

The issue does not repro in my local Windows machine, nor does it repro in the macOS and Linux legs of the same build. It does not repro when attempting to run the same tests targeting non net10.0 targets.

I'm pretty much stumped at this point, I would appreciate any hints that might help me better debug the issue. Thanks.

[hamish-smith]

This may not be limited to net10.

I have started to hit the same exception since upgrading to XUnit3.

For my project, it is only happening on GitLab cloud shared macOS runners running against net9.0 (Linux and Windows have not hit the exception - for net9 or net472).

[xUnit.net 00:00:00.17] Tenduke.Core.Tests: Catastrophic failure: System.InvalidOperationException: Test process did not return valid JSON (non-object).
   at Xunit.v3.TestProcessLauncherAdapter.GetAssemblyInfo(ITestProcessLauncher launcher, XunitProjectAssembly projectAssembly)
   at Xunit.v3.TestProcessLauncherAdapter.GetAssemblyInfo(XunitProjectAssembly projectAssembly)
   at Xunit.Runner.v3.Xunit3..ctor(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.v3.Xunit3.ForDiscoveryAndExecution(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.XunitFrontController.Create(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)
No test is available in /Users/gitlab/builds/10Duke/core/dotnet-core/Tenduke.Core.Tests/bin/Debug/net9.0/Tenduke.Core.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Running the same tests on a macOS laptop has not reproduced the error so far.

[eiriktsarpalis]

@hamish-smith are you collecting codecov data in your test runs by any chance? I've been able to isolate the issue to that.

[hamish-smith]

@hamish-smith are you collecting codecov data in your test runs by any chance? I've been able to isolate the issue to that.

No, not for the specific failing pipeline job. It is using the JUnit format test result logger though:

dotnet test --framework NET9.0 --test-adapter-path:. --logger:"junit;LogFilePath=test-results-macos-net9.xml"

https://gitlab.com/10Duke/core/dotnet-core/-/blob/70a8dc4e7e44a1f0ce224b207e377a02c56ffb4f/.gitlab-ci.yml#L121

[hamish-smith]

@hamish-smith are you collecting codecov data in your test runs by any chance? I've been able to isolate the issue to that.

No, not for the specific failing pipeline job. It is using the JUnit format test result logger though:

dotnet test --framework NET9.0 --test-adapter-path:. --logger:"junit;LogFilePath=test-results-macos-net9.xml"

https://gitlab.com/10Duke/core/dotnet-core/-/blob/70a8dc4e7e44a1f0ce224b207e377a02c56ffb4f/.gitlab-ci.yml#L121

Perhaps there are multiple root causes here...

Removing the test logger has no effect for me. Error still occurs.

[eiriktsarpalis]

What's the smallest possible configuration that repros the issue for you? I.e. removing all possible dotnet test CLI parameters, dependencies, test suites, test-specific msbuild properties, etc. that are still causing the crash for you.

In my case, I was able to narrow down the issue to just a --collect "Code Coverage;Format=cobertura" argument, although I should say my issue reproes with Windows only.

[bradwilson]

Test process did not return valid JSON (non-object).

The fact that there is no logged output in this exception message says that the process exited without writing any standard output, which implies that either the process simply failed to launch, or something intercepted the launch and failed without writing anything to stdout.

We currently only read stdout, but I think perhaps adding stderr to these exceptions could be helpful in debugging scenarios like this.

[hamish-smith]

What's the smallest possible configuration that repros the issue for you? I.e. removing all possible dotnet test CLI parameters, dependencies, test suites, test-specific msbuild properties, etc. that are still causing the crash for you.

In my case, I was able to narrow down the issue to just a --collect "Code Coverage;Format=cobertura" argument, although I should say my issue reproes with Windows only.

Not sure about "minimum" reproducer.

I tried removing all arguments except the --framework (I am multi-targeting to include net472 so dotnet test on it's own only works on Windows) and still hit the same error.

It feels environmental (i.e. set up of the macOS VM that CI is running on) but XUnit version 2 runs tests fine on the same VMs.

[eiriktsarpalis]

Just started hitting the same issue in the MCP sdk: https://github.com/modelcontextprotocol/csharp-sdk/actions/runs/18488558154/job/52677377463?pr=862

[halter73]

Now that .NET 10.0.100-rc.2 is out, we're seeing a new "Catastrophic failure" in the MCP SDK when targeting net472 specifically.

https://github.com/modelcontextprotocol/csharp-sdk/actions/runs/18511181961/job/52752830677

When I run dotnet test -f net472 locally on the ModelContextProtocol.Tests project with the rc.2 SDK, I consistently see the following:

> dotnet test -f net472
    info NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  ModelContextProtocol.Core netstandard2.0 succeeded (3.5s) → H:\modelcontextprotocol\csharp-sdk\artifacts\bin\ModelContextProtocol.Core\netstandard2.0\ModelContextProtocol.Core.dll
  ModelContextProtocol netstandard2.0 succeeded (0.4s) → H:\modelcontextprotocol\csharp-sdk\artifacts\bin\ModelContextProtocol\netstandard2.0\ModelContextProtocol.dll
  TestServerWithHosting net472 win-x86 succeeded (0.6s) → H:\modelcontextprotocol\csharp-sdk\artifacts\bin\TestServerWithHosting\net472\TestServerWithHosting.exe
  ModelContextProtocol.TestServer net472 win-x86 succeeded (0.6s) → H:\modelcontextprotocol\csharp-sdk\artifacts\bin\ModelContextProtocol.TestServer\net472\TestServer.exe
  ModelContextProtocol.Tests net472 win-x86 succeeded (2.1s) → H:\modelcontextprotocol\csharp-sdk\artifacts\bin\ModelContextProtocol.Tests\net472\ModelContextProtocol.Tests.exe
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.4+50e68bbb8b (64-bit .NET Framework 4.8.9221.0)
Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'System.ValueTuple, Version=4.0.5.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) ---> System.IO.FileLoadException: Could not load file or assembly 'System.ValueTuple, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
   --- End of inner exception stack trace ---
   at Xunit.Runner.InProc.SystemConsole.ConsoleRunner.<EntryPoint>d__12.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Runner.InProc.SystemConsole.ConsoleRunner.EntryPoint(TextReader consoleReader, TextWriter consoleWriter)
   at Xunit.Runner.InProc.SystemConsole.ConsoleRunner.<Run>d__15.MoveNext() in /_/src/xunit.v3.runner.inproc.console/ConsoleRunner.cs:line 366
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ModelContextProtocol.Tests.AutoGenerated.XunitAutoGeneratedEntryPoint.Main(String[] args) in H:\modelcontextprotocol\csharp-sdk\artifacts\obj\ModelContextProtocol.Tests\net472\XunitAutoGeneratedEntryPoint.cs:line 13
[xUnit.net 00:00:02.69] ModelContextProtocol.Tests: Catastrophic failure: System.InvalidOperationException: Test process did not return valid JSON (non-object).
   at Xunit.v3.TestProcessLauncherAdapter.GetAssemblyInfo(ITestProcessLauncher launcher, XunitProjectAssembly projectAssembly)
   at Xunit.Runner.v3.Xunit3..ctor(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.v3.Xunit3.ForDiscoveryAndExecution(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.XunitFrontController.Create(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.VisualStudio.VsTestRunner.<RunTestsInAssembly>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Xunit.Runner.VisualStudio.VsTestRunner.<RunTestsInAssembly>d__33.MoveNext()
Results File: H:\modelcontextprotocol\csharp-sdk\artifacts\TestResults\ModelContextProtocol.Tests.net472.Windows_NT.trx
  ModelContextProtocol.Tests test net472 win-x86 failed with 1 error(s) and 1 warning(s) (3.7s)
    C:\Program Files\dotnet\sdk\10.0.100-rc.2.25502.107\Microsoft.TestPlatform.targets(48,5): warning No test is available in H:\modelcontextprotocol\csharp-sdk\artifacts\bin\ModelContextProtocol.Tests\net472\ModelContextProtocol.Tests.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
    C:\Program Files\dotnet\sdk\10.0.100-rc.2.25502.107\Microsoft.TestPlatform.targets(48,5): error MSB6006: "dotnet.exe" exited with code 1.

Test summary: total: 0, failed: 0, succeeded: 0, skipped: 0, duration: 3.7s
Build failed with 1 error(s) and 1 warning(s) in 12.0

Downgrading to 10.0.100-rc.1 fixes this issue.

[martincostello]

Just hit this too after updating one of my repos to 10 rc.2: martincostello/sqllocaldb#1321

[martincostello]

Could it be possible this is related to this issue? dotnet/sdk#51265

The only repo of my own I'm seeing this issue in is in a project that multi-targets with .NET Framework.

[eiriktsarpalis]

Do you have any indications that it might be related other than the fact that multi-targeting is involved? FWIW I was able to repro my crash without any multi-targeting.