CI has been failing since April

[jsturtevant]

Opening this issue to start tracking down why it is failing.

The last good run was https://github.com/bytecodealliance/wasmtime-dotnet/actions/runs/8808954841 on commit b2b0480

Started failing the next day with https://github.com/bytecodealliance/wasmtime-dotnet/actions/runs/8825026578 on commit b2b0480

Some dependency that isn't pinned in CI might be a culprit.

I am able to repo locally:

Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed : Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at Wasmtime.Value+Native.wasmtime_externref_new(IntPtr, IntPtr, Finalizer)
--------------------------------
   at Wasmtime.Value.FromValueBox(Wasmtime.Store, Wasmtime.ValueBox)
   at Wasmtime.ValueBox.ToValue(Wasmtime.Store, Wasmtime.ValueKind)
   at Wasmtime.Function.Invoke(System.ReadOnlySpan`1<Wasmtime.ValueBox>)
   at Wasmtime.Function.Invoke(Wasmtime.ValueBox[])
   at Wasmtime.Tests.ExternRefTests.<ItCollectsExternRefs>g__RunTest|18_0(Int32*)
   at Wasmtime.Tests.ExternRefTests.ItCollectsExternRefs()
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags)
   at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<InvokeTestMethodAsync>b__1()
   at Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<AggregateAsync>d__4 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<AggregateAsync>d__4 ByRef)
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<InvokeTestMethodAsync>b__0()
   at Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__9 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__9 ByRef)
   at Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>)
   at Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InvokeTestMethodAsync(System.Object)
   at Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(System.Object)
   at Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<RunAsync>b__47_0()
   at Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__10`1<System.Decimal> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__10`1<System.Decimal> ByRef)
   at Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator)
   at Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestAsync>d__4 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsy

Test Run Aborted.

[jsturtevant]

I can reproduce the failing test consistently:

   at Wasmtime.Tests.ExternRefTests.<ItCollectsExternRefs>g__RunTest|18_0(Int32*)
   at Wasmtime.Tests.ExternRefTests.ItCollectsExternRefs()

Also the one from CI is failing consistently locally too:

[xUnit.net 00:00:00.75]     Wasmtime.Tests.TableImportBindingTests.ItGrowsATable [FAIL]
  Error Message:
   System.EntryPointNotFoundException : Unable to find an entry point named 'wasmtime_val_delete' in DLL 'wasmtime'.
  Stack Trace:
     at Wasmtime.Value.Native.wasmtime_val_delete(Value& val)
   at Wasmtime.Value.Dispose() in D:\a\wasmtime-dotnet\wasmtime-dotnet\src\Value.cs:line 174
   at Wasmtime.Table..ctor(Store store, TableKind kind, Object initialValue, UInt[32](https://github.com/bytecodealliance/wasmtime-dotnet/actions/runs/8825026578/job/24228588501#step:8:33) initial, UInt32 maximum) in D:\a\wasmtime-dotnet\wasmtime-dotnet\src\Table.cs:line 84
   at Wasmtime.Tests.TableImportBindingTests.ItFailsToInstantiateWithTableLimitsMismatch() in D:\a\wasmtime-dotnet\wasmtime-dotnet\tests\TableImportBindingTests.cs:line 58

[jsturtevant]

I seems the results look random in CI because some tests hang, and when others fail it crashes rest of the test suite. Here is a list of failing tests:

[jsturtevant]

Both the passing and failing builds from above are running the same .net version:

dotnet-install: Extracting zip from https://dotnetcli.azureedge.net/dotnet/Sdk/8.0.204/dotnet-sdk-8.0.204-linux-x64.tar.gz

[jsturtevant]

The project downloads the wasmtime c api:

wasmtime-dotnet/src/Wasmtime.csproj

Lines 151 to 156 in b2b0480

	<DownloadFile
	Condition="('$(Packing)'=='true' Or '%(Wasmtime.Copy)'=='true') And !Exists('$(BaseIntermediateOutputPath)/$(ReleaseFileNameBase)-%(Wasmtime.Arch)-%(Wasmtime.OS)-c-api.%(Wasmtime.FileExtension)')"
	SourceUrl="$(ReleaseURLBase)/$(ReleaseFileNameBase)-%(Wasmtime.Arch)-%(Wasmtime.OS)-c-api.%(Wasmtime.FileExtension)"
	DestinationFolder="$(BaseIntermediateOutputPath)"
	SkipUnchangedFiles="true"
	/>

This changes regularly, and in ci the size of the download changed between the two runs. My current guess is that the Dev release is broken for this.

[jsturtevant]

Looks like Dev builds are being enabled even for pr runs:

wasmtime-dotnet/.github/workflows/main.yml

Lines 48 to 52 in b2b0480

	- name: Enable development builds for the main branch
	if: github.ref == 'refs/heads/main' || github.base_ref == 'main'
	shell: bash
	run: |
	echo "DevBuild=true" >> $GITHUB_ENV

Here is an example of a PR that should be using the matching c-library but is using DEV:

https://github.com/bytecodealliance/wasmtime-dotnet/pull/313/checks

Run dotnet build Wasmtime.sln -c debug --no-restore
  Tool 'dotnet-t4' (version '2.3.1') was restored. Available commands: t4
  
  Restore was successful.
  Downloading from "https://github.com/bytecodealliance/wasmtime/releases/download/dev/wasmtime-dev-x[8](https://github.com/bytecodealliance/wasmtime-dotnet/pull/313/checks#step:7:9)6_64-linux-c-api.tar.xz" to "/home/runner/work/wasmtime-dotnet/wasmtime-dotnet/src/obj/wasmtime-dev-x86_64-linux-c-api.tar.xz" (15,430,[9](https://github.com/bytecodealliance/wasmtime-dotnet/pull/313/checks#step:7:10)20 bytes).

[jsturtevant]

Switching the main build to pull the matching version locally (19.0.1) made many more of the tests pass, so might be on to something.

[kpreisser]

Hi,
AFAIK using dev builds of the Wasmtime C API when building from main or for PRs that target main is intentional, to detect any incompatibilities with upcoming Wasmtime C API releases.

There have been some changes in the C API recently, but wasmtime-dotnet hasn't yet been updated to it, which is why the build currently fails e.g. due to missing entry points (like wasmtime_val_delete) or due to changed function signatures (like wasmtime_externref_new),

For example, the following commits changed C API functions that wasmtime-dotnet currently uses, which have not yet been incorporated into wasmtime-dotnet:

• bytecodealliance/wasmtime@e55fa3c
• bytecodealliance/wasmtime@77405cc

However, it seems the project is currently missing a maintainer, see: #313 (comment)

[jsturtevant]

AFAIK using dev builds of the Wasmtime C API when building from main or for PRs that target main is intentional, to detect any incompatibilities with upcoming Wasmtime C API releases.

This makes sense to have an early signal. Currently it looks like it is running on the PR's too which is blocking any updates for simple things like https://github.com/bytecodealliance/wasmtime-dotnet/pull/313/files.

For example, the following commits changed C API functions that wasmtime-dotnet currently uses, which have not yet been incorporated into wasmtime-dotnet:

bytecodealliance/wasmtime@e55fa3c
bytecodealliance/wasmtime@77405cc

Thanks for the pointers! Do you know enough to be able to update the project for these fixes?

However, it seems the project is currently missing a maintainer, see: #313 (comment)

I can help on this front but new to the project. If you know enough to update to the API changes, I could help review and get it in. Otherwise I'll start digging in.

[martindevans]

@jsturtevant I've fixed the first of those two things you linked in #316.

[jsturtevant]

I dug into this a bit more, it was bothering me that I couldn't get the main build properly after making it only pull the dev nightly builds of wasmtime on main and not on the prs (#317). This should have fixed the PR's since we we are on the 19.0.1 and now pulling the 19.01 version of the c-api but it didn't.

So what looks to have happen is:

1. 19.0.1 is passing in Bump Wasmtime version to 19.0.1. #303. This passed because it wasn't against the main branch.

  Downloading from "https://github.com/bytecodealliance/wasmtime/releases/download/v19.0.1/wasmtime-v19.0.1-x86_64-windows-c-api.zip" to "D:\a\wasmtime-dotnet\wasmtime-dotnet\src\obj\wasmtime-v19.0.1-x86_64-windows-c-api.zip" (18,487,457 bytes).

2. 19.0.1 is failing in the PR Merge release-19.0.x to main #304. The are two different jobs that pass in that pr (the ones using 19.0.1 and the ones that run against main on dev api.

3. tests start passing again in Fix CI. #305 but this is using dev branch on april 16. Since the fixes where for the dev branch c-api (not 19.0.1 c-api) it means anything running against 19.0.1 will fail!

4. Since nightly CI run on dev we started failing again on April 24 becuase the API changed again: https://github.com/bytecodealliance/wasmtime/commits/main/crates/c-api?since=2024-04-24&until=2024-04-24

So we are in a strange state, the version in the code is 19.0.1 but we are closer to using the c-API of version 21.0 and the nightly dev version is way ahead.

We have a few options:

• go for latest version (22.0) and track down all the c-api changes between ~21 (which was around april 16th) to 22.0
• Try for 1.21 and track down the changes between april 16 and when the 21.0 branch was cut. Then go from 1.21 to 1.22 after that.