Skip to content

Conversation

@davidwrighton
Copy link
Member

The ExplicitLayoutValidator currently used an array of bytes one to one with the size of the layout of the class. However, we have some test cases for outrageously large explicit layout structures and classes which cause this approach to OOM in crossgen2.

The fix is to move to a data structure defined in terms of intervals. This is probably a bit slower, but it requires vastly less memory for pathological situations.

Fixes #55164
Fixes #53559

@davidwrighton davidwrighton requested a review from trylek August 3, 2021 01:49
@davidwrighton
Copy link
Member Author

@dotnet/crossgen-contrib

@MichalStrehovsky
Copy link
Member

TIL.

I suggested using a byte array when this was introduced because CoreCLR uses that and it was an easy to follow algorithm. Didn't think we allow such big classes.

[StructLayout(LayoutKind.Explicit, Size = Int32.MaxValue)]
struct Boo { }

[StructLayout(LayoutKind.Explicit)]
class Program
{
    [FieldOffset(0)]
    Boo b;

    static void Main() { }
}

This crashes 5.0 CoreCLR on x86 big time (somewhere in native-land that I didn't debug). Fun.

Copy link
Member

@trylek trylek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for making this more robust!

@davidwrighton davidwrighton merged commit 9d9aed6 into dotnet:main Aug 4, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Sep 3, 2021
@davidwrighton davidwrighton deleted the fix_ExplicitLayoutValidator_memoryAllocProblems branch April 13, 2023 18:49
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

3 participants