[NativeAOT] Can't call a function pointer with a ref return value

[kevingosse]

Description

In some cases with the System V ABI, when a "complex" type is returned, the caller is expected to allocate memory for the return value, and pass a pointer to that memory as a hidden first argument.

I'm trying to call a function that uses that convention, so I looked at how to instruct the runtime to do so. It looks like that may be the case when returning by ref:

runtime/src/coreclr/vm/callingconvention.h

Line 1784 in 0547344

flags |= RETURN_HAS_RET_BUFFER;

(note: I may be misinterpreting this code, since I haven't been able to try)

However, when trying to call an unmanaged pointer with a ref return value, the NativeAOT compilation fails.

Reproduction Steps

Create a new project, declare a function pointer with a ref return value, and try calling it:

public struct TestStruct
{
}

public static unsafe void CreateFunctionPointer(IntPtr address)
{
    var functionPointer = (delegate* unmanaged[Cdecl, MemberFunction]<IntPtr, ref TestStruct>)address;
    functionPointer(IntPtr.Zero);
}

Then try compiling it with NativeAOT:

dotnet publish /p:SelfContained=true -r win-x64 -c Release

Expected behavior

The call should work.

Actual behavior

An error is thrown during compilation:

  Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeNativeAOT
EXEC : error : VTable of type 'System.Runtime.InteropServices.MarshalDirectiveException' not computed by the IL scanner. You c
an work around by running the compilation with scanner disabled. [C:\Users\kevin\source\repos\TestNativeAot\TestNativeAot\Test
NativeAot.csproj]
  ILCompiler.ScannerFailedException: VTable of type 'System.Runtime.InteropServices.MarshalDirectiveException' not computed by
   the IL scanner. You can work around by running the compilation with scanner disabled.
     at ILCompiler.ILScanResults.ScannedVTableProvider.GetSlice(TypeDesc type) in /_/src/coreclr/tools/aot/ILCompiler.Compiler
  /Compiler/ILScanner.cs:line 282
     at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
     at ILCompiler.DependencyAnalysis.ConstructedEETypeNode.ComputeNonRelocationBasedDependencies(NodeFactory factory) in /_/s
  rc/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ConstructedEETypeNode.cs:line 64
     at ILCompiler.DependencyAnalysis.ObjectNode.GetStaticDependencies(NodeFactory factory) in /_/src/coreclr/tools/Common/Com
  piler/DependencyAnalysis/ObjectNode.cs:line 59
     at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependenciesImpl(DependencyNodeCore`1 node) in /_
  /src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/DependencyAnalyzer.cs:line 181
     at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependencies(DependencyNodeCore`1 node) in /_/src
  /coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/DependencyAnalyzer.cs:line 221
     at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ProcessMarkStack() in /_/src/coreclr/tools/aot/ILCompiler.
  DependencyAnalysisFramework/DependencyAnalyzer.cs:line 256
     at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes() in /_/src/coreclr/tools/aot/ILCompile
  r.DependencyAnalysisFramework/DependencyAnalyzer.cs:line 307
     at ILCompiler.RyuJitCompilation.CompileInternal(String outputFile, ObjectDumper dumper) in /_/src/coreclr/tools/aot/ILCom
  piler.RyuJit/Compiler/RyuJitCompilation.cs:line 88
     at ILCompiler.Compilation.ILCompiler.ICompilation.Compile(String outputFile, ObjectDumper dumper) in /_/src/coreclr/tools
  /aot/ILCompiler.Compiler/Compiler/Compilation.cs:line 526
     at ILCompiler.Program.Run(String[] args) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 898
     at ILCompiler.Program.Main(String[] args) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 1090

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[kevingosse]

I managed to make it work by adding System.Runtime.InteropServices.MarshalDirectiveException in a rd.xml file, but then it fails at runtime with:

Unhandled Exception: System.Runtime.InteropServices.MarshalDirectiveException: Method '[S.P.CompilerGenerated]Internal.CompilerGenerated..CalliMarshallingMethodThunk(native int,native int)' requires marshalling that is not yet supported by this compiler.

So that's my next question I guess 😄

[jkotas]

In some cases with the System V ABI, when a "complex" type is returned, the caller is expected to allocate memory for the return value, and pass a pointer to that memory as a hidden first argument. I'm trying to call a function that uses that convention

You should not need to manually apply ref on the return value in this case. delegate* unmanaged[Cdecl, MemberFunction]<IntPtr, TestStruct> should work just fine. Or did you run into problems with it?

[kevingosse]

@jkotas The unmanaged function comes from a C++ library that expects the caller to use a return buffer (the returned struct is in fact a C++ object with a non-trivial destructor, which is a special case in the System V ABI), so I'm trying to force the runtime to use this calling convention.

I'm not aware of an attribute that can be used to force that behavior, so I'm trying other things. I had good hope for the ref struct but I guess I'll need to use something else. I found a special case when the struct has no fields (https://cs.github.com/dotnet/runtime/blob/eb170a0523717094a921a38ac45c1c8e401ef8b6/src/coreclr/vm/methodtable.cpp#L2458) so I'll be using that for now but it's not great.

[EgorBo]

You can try this trick:

(depending on the size of your struct)

[kevingosse]

@EgorBo yeah that's what I ended up doing 👍

[teo-tsirpanis]

The main problem is that your unmanaged function pointer returns a managed reference. As the runtime error indicated, this requires marshalling which is expected to be handled by the LibraryImport source generator. I think that this feature is not supported by design; the diagnostics however can be much better. This is how your code should be:

public struct TestStruct
{
}

public static unsafe void CreateFunctionPointer(IntPtr address)
{
    var functionPointer = (delegate* unmanaged[Cdecl, MemberFunction]<IntPtr, TestStruct*>)address;
    functionPointer(IntPtr.Zero);
}

Source-generated marshalling of function pointer calls is tracked in #63590.

I also tested that the source generator does not support ref returns at all, causing error SYSLIB1052: The specified 'ref return' configuration for the return value of method 'Program.Test()' is not supported by source-generated P/Invokes. If the specified configuration is required, use a regularDllImport instead..