When compiling a delegate from a System.Linq.Expressions expression tree, if the tree is too complex, compilation fails when hoisted variables are used

[nike4613]

Description

When compiling a delegate from a System.Linq.Expressions expression tree, if the tree is too complex (and has hoisted variables), compilation fails at

runtime/src/libraries/System.Linq.Expressions/src/System/Linq/Expressions/Compiler/CompilerScope.cs

Lines 273 to 279 in 24fa00d

	//
	// If this is an unbound variable in the lambda, the error will be
	// thrown from VariableBinder. So an error here is generally caused
	// by an internal error, e.g. a scope was created but it bypassed
	// VariableBinder.
	//
	throw Error.UndefinedVariable(variable.Name, variable.Type, CurrentLambdaName);

If the comment is to be believed, it should never fail here.

I unfortunately cannot pinpoint what attribute specifically causes it, however I can give a few examples of the DebugView property of expression trees which do and do not work. These examples are, unfortunately, very long and repetitive even when it does work.

This, for example, fails with System.InvalidOperationException: 'variable 'context' of type 'Sentimentality.RPC.IRpcCallContext' referenced from scope 'Dispatch for DoThing', but it is not defined'.

This, however, compiles successfully and runs as expected.

Both of these expressions are built with the same generator code, only with a subtly different input type. The former has one more method being processed than the latter, and it does not seem to make a difference what the method is, only whether it is present. This is what leads me to believe that there is some "complexity" which the former exceeds while the latter does not which causes the issue to happen.

Until this is fixed, are there any simple workarounds that don't boil down to manually hoising variables?

Configuration

.NET 5.0.7 on Windows 10 x64, though I suspect that it is architecture independent.

Other information

A full stack trace for the failing example above:

   at System.Linq.Expressions.Compiler.CompilerScope.ResolveVariable(ParameterExpression variable, HoistedLocals hoistedLocals)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitParameterExpression(Expression expr)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitInstance(Expression instance, Type& type)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCall(Expression obj, MethodInfo method, IArgumentProvider methodCallExpr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCallExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitTryExpression(Expression expr)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitLambdaBody(CompilerScope parent, Boolean inlined, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitDelegateConstruction(LambdaExpression lambda)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitArguments(MethodBase method, IArgumentProvider args, Int32 skipParameters)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCall(MethodInfo mi, IArgumentProvider args, Type objectType, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCall(Expression obj, MethodInfo method, IArgumentProvider methodCallExpr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCallExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitTryExpression(Expression expr)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitConditionalExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitConditionalExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitSwitchCases(SwitchExpression node, Label[] labels, Boolean[] isGoto, Label default, Label end, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.TryEmitSwitchInstruction(SwitchExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitSwitchExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.TryEmitHashtableSwitch(SwitchExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitSwitchExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitLambdaBody(CompilerScope parent, Boolean inlined, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitLambdaBody()
   at System.Linq.Expressions.Compiler.LambdaCompiler.Compile(LambdaExpression lambda)
   at System.Linq.Expressions.Expression`1.Compile()

Tagging subscribers to this area: @cston
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When compiling a delegate from a System.Linq.Expressions expression tree, if the tree is too complex (and has hoisted variables), compilation fails at

runtime/src/libraries/System.Linq.Expressions/src/System/Linq/Expressions/Compiler/CompilerScope.cs

Lines 273 to 279 in 24fa00d

	//
	// If this is an unbound variable in the lambda, the error will be
	// thrown from VariableBinder. So an error here is generally caused
	// by an internal error, e.g. a scope was created but it bypassed
	// VariableBinder.
	//
	throw Error.UndefinedVariable(variable.Name, variable.Type, CurrentLambdaName);

If the comment is to be believed, it should never fail here.

I unfortunately cannot pinpoint what attribute specifically causes it, however I can give a few examples of the DebugView property of expression trees which do and do not work. These examples are, unfortunately, very long and repetitive even when it does work.

This, for example, fails with System.InvalidOperationException: 'variable 'context' of type 'Sentimentality.RPC.IRpcCallContext' referenced from scope 'Dispatch for DoThing', but it is not defined'.

This, however, compiles successfully and runs as expected.

Both of these expressions are built with the same generator code, only with a subtly different input type. The former has one more method being processed than the latter, and it does not seem to make a difference what the method is, only whether it is present. This is what leads me to believe that there is some "complexity" which the former exceeds while the latter does not which causes the issue to happen.

Until this is fixed, are there any simple workarounds that don't boil down to manually hoising variables?

Configuration

.NET 5.0.7 on Windows 10 x64, though I suspect that it is architecture independent.

Other information

A full stack trace for the failing example above:

   at System.Linq.Expressions.Compiler.CompilerScope.ResolveVariable(ParameterExpression variable, HoistedLocals hoistedLocals)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitParameterExpression(Expression expr)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitInstance(Expression instance, Type& type)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCall(Expression obj, MethodInfo method, IArgumentProvider methodCallExpr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCallExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitTryExpression(Expression expr)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitLambdaBody(CompilerScope parent, Boolean inlined, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitDelegateConstruction(LambdaExpression lambda)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitArguments(MethodBase method, IArgumentProvider args, Int32 skipParameters)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCall(MethodInfo mi, IArgumentProvider args, Type objectType, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCall(Expression obj, MethodInfo method, IArgumentProvider methodCallExpr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitMethodCallExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitTryExpression(Expression expr)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitConditionalExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitConditionalExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsVoid(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitSwitchCases(SwitchExpression node, Label[] labels, Boolean[] isGoto, Label default, Label end, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.TryEmitSwitchInstruction(SwitchExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitSwitchExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpressionAsType(Expression node, Type type, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Emit(BlockExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitBlockExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.TryEmitHashtableSwitch(SwitchExpression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitSwitchExpression(Expression expr, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitExpression(Expression node, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitLambdaBody(CompilerScope parent, Boolean inlined, CompilationFlags flags)
   at System.Linq.Expressions.Compiler.LambdaCompiler.EmitLambdaBody()
   at System.Linq.Expressions.Compiler.LambdaCompiler.Compile(LambdaExpression lambda)
   at System.Linq.Expressions.Expression`1.Compile()

Author:	nike4613
Assignees:	-
Labels:	area-System.Linq.Expressions, untriaged
Milestone:	-

[cston]

@nike4613, thanks for reporting this issue.

Unfortunately, there doesn't appear to be enough information here to reproduce the problem. If you wouldn't mind, please provide a simplified example that hits the InvalidOperationException.

[nike4613]

I can send the project that causes the error (and a test case that triggers it), but every attempt I have made to simplify it has caused the issue to dissapear.

[cston]

Thanks @nike4613.

It appears the issue is with Expressions with closures and string switch expressions. If the switch expression has 8 or more cases, then any hoisted variables declared outside of the switch expression are dropped when calculating the closures from lambdas within the switch cases.

The following example hits the exception and should be sufficient to investigate the fix:

var arg0 = Expression.Parameter(typeof(string), "value");

// f1 = (Func<int> f) => f();
var arg1 = Expression.Parameter(typeof(Func<int>), "f");
Expression<Func<Func<int>, int>> expr1 =
    Expression.Lambda<Func<Func<int>, int>>(
        body: Expression.Invoke(arg1),
        parameters: new[] { arg1 });

// f2 = () => value.Length;
Expression<Func<int>> expr2 =
    Expression.Lambda<Func<int>>(
        Expression.Property(arg0, typeof(string).GetProperty("Length")));

// f0 = (string value) => value switch { "1" => 1, ..., "7" => 7, _ => f1(f2) };
Expression<Func<string, int>> expr0 =
    Expression.Lambda<Func<string, int>>(
        body: Expression.Switch(
            switchValue: arg0,
            defaultBody: Expression.Invoke(expr1, expr2),
            cases: Enumerable.Range(1, 7).Select(index =>
                Expression.SwitchCase(Expression.Constant(index), Expression.Constant(index.ToString()))).ToArray()),
        parameters: new[] { arg0 });

Func<string, int> f0 = expr0.Compile();

[cston]

With the minimal repro above, the same exception is thrown with earlier versions of .NET Core (with .NET Core 2.0 for instance). With .NET Framework, the call to Expression.Compile() hangs.

[cston]

@nike4613, feel free to remove your repro. The example code above should be sufficient to investigate, thanks.