Fatal error. Internal CLR error. (0x80131506) in .Net 7

[ANahr]

Description

We are getting a fatal error with .Net 7:

[exec] Fatal error. Internal CLR error. (0x80131506)
[exec] at System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup(System.Object ByRef, Void*, System.Object)
[exec] at System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper(System.Object ByRef, Void*, System.Object)
[exec] at System.Collections.Generic.Stack`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Push(System.__Canon)
[exec] at DocumentFormat.OpenXml.OpenXmlElement+d__97.MoveNext()
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.WriteNamespaceAtributes(System.Xml.XmlWriter)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.WriteTo(System.Xml.XmlWriter)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.Save(System.IO.Stream)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.SaveToPart(DocumentFormat.OpenXml.Packaging.OpenXmlPart)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.Save()
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.SavePartContent(DocumentFormat.OpenXml.Packaging.OpenXmlPart)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.TrySavePartContent(DocumentFormat.OpenXml.Packaging.OpenXmlPart)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.SavePartContents(Boolean)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.Dispose(Boolean)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.Dispose()
...

Reproduction Steps

Unfortunatelly I cannot provide easy reproduction steps.
The error is reproduceable with a specific build of our software in combination with a specific data input (lots of GB of data).
It does not happen with other builds (although we only have very few builds with .Net 7 yet) nor with other input data.

Expected behavior

No error

Actual behavior

Fatal error. Internal CLR error. (0x80131506)

Regression?

Extremely likely an regression in .Net 7.
Never happend in .Net 6 or before.

Known Workarounds

None

Configuration

.Net 7.0.0.0 in Windows X64

Other information

No response

Tagging subscribers to this area: @dotnet/area-system-xml
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

We are getting a fatal error with .Net 7:

[exec] Fatal error. Internal CLR error. (0x80131506)
[exec] at System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup(System.Object ByRef, Void*, System.Object)
[exec] at System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper(System.Object ByRef, Void*, System.Object)
[exec] at System.Collections.Generic.Stack`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Push(System.__Canon)
[exec] at DocumentFormat.OpenXml.OpenXmlElement+d__97.MoveNext()
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.WriteNamespaceAtributes(System.Xml.XmlWriter)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.WriteTo(System.Xml.XmlWriter)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.Save(System.IO.Stream)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.SaveToPart(DocumentFormat.OpenXml.Packaging.OpenXmlPart)
[exec] at DocumentFormat.OpenXml.OpenXmlPartRootElement.Save()
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.SavePartContent(DocumentFormat.OpenXml.Packaging.OpenXmlPart)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.TrySavePartContent(DocumentFormat.OpenXml.Packaging.OpenXmlPart)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.SavePartContents(Boolean)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.Dispose(Boolean)
[exec] at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.Dispose()
...

Reproduction Steps

Unfortunatelly I cannot provide easy reproduction steps.
The error is reproduceable with a specific build of our software in combination with a specific data input (lots of GB of data).
It does not happen with other builds (although we only have very few builds with .Net 7 yet) nor with other input data.

Expected behavior

No error

Actual behavior

Fatal error. Internal CLR error. (0x80131506)

Regression?

Extremely likely an regression in .Net 7.
Never happend in .Net 6 or before.

Known Workarounds

None

Configuration

.Net 7.0.0.0 in Windows X64

Other information

No response

Author:	ANahr
Assignees:	-
Labels:	area-System.Xml
Milestone:	-

[EgorBo]

@ANahr could you collect and share a dump?

This issue has been marked needs-author-action and may be missing some important information.

[ANahr]

As we don't own the data unfortunatelly I'm not permitted to share a dump.
Is there anything else I can do to help with this issue?

Current status update: We have eight .Net 7 builds of our software and our test-suite now fails reproducable for 2 of those builds (both with the identical stacktrace but both at a different test)

[andrewb3010]

We're getting this Internal CLR error also when saving. Usually with larger workbooks (VS diagnostics will show 500-600MB process memory. Strangely the peak virtual memory shows as 2TB.) Smaller workbooks are ok.

Note if a GC.Collect() is done before saving instead a ExecutionEngineException occurs.
A GC.Collect() done after saving strangely raises null reference in the save method BEFORE the GC.Collect().

So GC.Collect() seems to obfuscate the root cause of the issue.

.NET 6.0 was fine. This fails with .NET 7.0 only (using OpenXML 2.16, 2.17.1 or the latest 2.18). Presumably memory related (since OpenXML has the known issue of memory bloating on the project page). Hope someone can shed some light on it?

[janvorli]

This sounds like a possible GC heap corruption. Could you please try to set the DOTNET_HeapVerify env var to 1 and execute your app under a debugger, ideally WinDbg? That will verify the GC heap integrity before and after every GC.

[andrewb3010]

Thanks for your response. I set the environment variable in Project Settings->Debug->Open debug launch profiles UI->Environment variables.

I'm using Visual Studio 2022 17.4.0 (debug not release) and am not sure how to run it under any other debugger (since have never really had to TBH.)

So I also checked 'Enable native code debugging' in the same 'debug launch profiles' UI above.

Below is a screenshot of the exception and call stack.

Please let me know if there's anything else I can do to troubleshoot (and thanks again for you response).

(FYI the WriteWorkSheet method in which there's the GC.Collect() just writes a worksheet out using a DataTable as the source data. Works fine with lower row counts - always has in .NET 6.0 regardless of row count. Since OpenXML is notoriously memory hungry I call GC.Collect as soon as the DataTable's no longer needed. So it seems the memory issue is creating an exception either when GC.Collect is called or when I comment GC.Collect out later when the OpenXML document is being saved - which is perhaps when the GC does it's own automatic collection - not sure.)

[andrewb3010]

Just tried this in the config (under the configuration/runtime elements not pasted here as editor removes XML for some reason):

  gcConcurrent enabled="false"

And it didn't help FYI.

Kinda hoping this issue may be spotted more (and hence troubleshooted more) as more people migrate to .NET 7.0 but as it stands this is somewhat of an obstacle (to migrating)? Shame because .NET 7.0 is soo much quicker.