May 9 21:11:41 server883 kernel: [218937.760135] .NET Server GC[1427719]: segfault at 8 ip 00007f7440ae835c sp 00007f74395bfdf0 error 6

[stefanschnabs]

Description

We are seeing segmentation faults as seen in the kernel logs:

May  9 21:11:41 server883 kernel: [218937.760135] .NET Server GC[1427719]: segfault at 8 ip 00007f7440ae835c sp 00007f74395bfdf0 error 6
May  9 21:11:41 server883 kernel: [218937.760141] .NET Server GC[1427720]: segfault at 4 ip 00007f7440940652 sp 00007f74395bb9c0 error 4
May  9 21:11:41 server883 kernel: [218937.760143]  in libcoreclr.so[7f74406a9000+4e2000]
May  9 21:11:41 server883 kernel: [218937.760149] Code: a2 91 0f 00 c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 ff 74 18 66 48 8d 3d b7 33 0c 00 66 66 48 e8 a7 0c 0a 00 48 8b 00 <f0> 83 48 08 40 48 8b 05 c8 33 0c 00 48 8b 38 48 85 ff 74 09 48 8b
May  9 21:11:41 server883 kernel: [218937.760152]  in libcoreclr.so[7f74406a9000+4e2000]
May  9 21:11:41 server883 kernel: [218937.760156] Code: c8 22 00 00 48 3b 8f c0 22 00 00 77 0e 4c 89 29 49 8b 8f c8 22 00 00 49 8b 45 00 48 83 c1 08 49 89 8f c8 22 00 00 48 83 e0 f8 <8b> 48 04 83 38 00 0f 88 d0 03 00 00 45 31 f6 49 01 ce 49 8b 97 b0

We have seen it on multiple servers now running dotnet 7, we have not identified the issue on any servers running dotnet 6

We have identified one server where this is happening quote often:

What addition information could help with your investigation? For example, turn on a extra reporting, or we may be able to privately share a mini dump. If so please specify what you need and the correct way to enable and share privately.

Reproduction Steps

Unfortunately we have no steps identified to repro, but as seen above we do have a server experiencing the problem often.

Expected behavior

No segfaults

Actual behavior

segfaults occurring causing the container to crash and restart

Regression?

We have not seen this occur in dotnet 6

Known Workarounds

N/A
Update May 17th 2023: Can workaround this issue using the dotnet 6 GC in dotnet 7 by setting ENV VAR COMPlus_GCName=libclrgc.so. Note: this can only work in .NET 7 until such a moment when libclrgc.so is removed from the distribution

Configuration

• mcr.microsoft.com/dotnet/sdk:7.0-jammy (7.0.203)
• Ubuntu 20.04.6 LTS
• x64

Other information

• We know there has been big changes in GC in dotnet 7 compared to dotnet 6 and have yet to pin the problem any other change as of yet

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

We are seeing segmentation faults as seen in the kernel logs:

May  9 21:11:41 server883 kernel: [218937.760135] .NET Server GC[1427719]: segfault at 8 ip 00007f7440ae835c sp 00007f74395bfdf0 error 6
May  9 21:11:41 server883 kernel: [218937.760141] .NET Server GC[1427720]: segfault at 4 ip 00007f7440940652 sp 00007f74395bb9c0 error 4
May  9 21:11:41 server883 kernel: [218937.760143]  in libcoreclr.so[7f74406a9000+4e2000]
May  9 21:11:41 server883 kernel: [218937.760149] Code: a2 91 0f 00 c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 ff 74 18 66 48 8d 3d b7 33 0c 00 66 66 48 e8 a7 0c 0a 00 48 8b 00 <f0> 83 48 08 40 48 8b 05 c8 33 0c 00 48 8b 38 48 85 ff 74 09 48 8b
May  9 21:11:41 server883 kernel: [218937.760152]  in libcoreclr.so[7f74406a9000+4e2000]
May  9 21:11:41 server883 kernel: [218937.760156] Code: c8 22 00 00 48 3b 8f c0 22 00 00 77 0e 4c 89 29 49 8b 8f c8 22 00 00 49 8b 45 00 48 83 c1 08 49 89 8f c8 22 00 00 48 83 e0 f8 <8b> 48 04 83 38 00 0f 88 d0 03 00 00 45 31 f6 49 01 ce 49 8b 97 b0

We have seen it on multiple servers now running dotnet 7, we have not identified the issue on any servers running dotnet 6

We have identified one server where this is happening quote often:

What addition information could help with your investigation? For example, turn on a extra reporting, or we may be able to privately share a mini dump. If so please specify what you need and the correct way to enable and share privately.

Reproduction Steps

Unfortunately we have no steps identified to repro, but as seen above we do have a server experiencing the problem often.

Expected behavior

No segfaults

Actual behavior

segfaults occurring causing the container to crash and restart

Regression?

We have not seen this occur in dotnet 6

Known Workarounds

N/A

Configuration

• mcr.microsoft.com/dotnet/sdk:7.0-jammy (7.0.203)
• Ubuntu 20.04.6 LTS
• x64

Other information

• We know there has been big changes in GC in dotnet 7 compared to dotnet 6 and have yet to pin the problem any other change as of yet

Author:	stefanschnabs
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[danmoseley]

Not on GC team, but they may ask you to try COMPlus_GCName=libclrgc.so. This will swap in the 6.0 GC - if it doesn't segfault, it's evidence the new GC is implicated.

[Maoni0]

yeah, if you could the env var that @danmoseley mentioned it would be great. if that doesn't repro, do you have a dump or at least a callstack that you could share with the default GC? and would it be possible to run with a private clrgc.dll if that would dramatically help with the investigation?

[stefanschnabs]

Hi @Maoni0,

• We have not been able to reproduce the issue by:

  • Reverting some instances back to dotnet 6
  • Using the dotnet 6 GC in dotnet 7 (COMPlus_GCName=libclrgc.so)

• We have been able to capture 2 native thread call stacks, hopefully this can help investigate

  • btallserver833.txt
  • btallserver1620.txt

• I am not familiar with what you mean by private clrgc.dll. A custom GC with addition debugging enabled? If the 2 stack traces above are not fruitful we can look at that next.

Let me know if you need something else,
Thanks for you help

[Maoni0]

the callstacks just say that GC detected a heap corruption but this is a very generic thing. is it possible to share a couple of dumps?

yes, by a private clrgc.dll (or in your case, libclrgc.so) I just meant we enable some instrumentation so it helps with debugging.

[stefanschnabs]

How can we send you dump(s) privately using a secure channel?

[stefanschnabs]

Have a couple other data points:

• We think a line in our code return new ABCClass() as T ?? new List<double>() as T; ended up with this stack
  bt: server883btallCast.txt

* thread #106, stop reason = signal SIGSEGV
  * frame #0: 0x00007f86bd43cc7f libc.so.6`wait4 + 95
    frame #1: 0x00007f86bd1bb143 libcoreclr.so`PROCCreateCrashDump(argv=<unavailable>, errorMessageBuffer=<unavailable>, cbErrorMessageBuffer=<unavailable>) at process.cpp:2486:22
    frame #2: 0x00007f86bd1bc287 libcoreclr.so`::PROCCreateCrashDumpIfEnabled(signal=<unavailable>, siginfo=<unavailable>) at process.cpp:2700:9
    frame #3: 0x00007f86bd1849ab libcoreclr.so`invoke_previous_action(action=<unavailable>, code=11, siginfo=0x00007f0e68687bf0, context=0x00007f0e68687ac0, signalRestarts=<unavailable>) at signal.cpp:416:5
    frame #4: 0x00007f86bd184067 libcoreclr.so`sigsegv_handler(code=11, siginfo=0x00007f0e68687bf0, context=0x00007f0e68687ac0) at signal.cpp:623:5
    frame #5: 0x00007f86bd8b2420 libpthread.so.0`__restore_rt
    frame #6: 0x00007fb2531fd817 libcoreclr.so`___lldb_unnamed_symbol2490$$libcoreclr.so + 39

00007E20B8FF70F0 00007fb253860c7f [HelperMethodFrame_1OBJ: 00007e20b8ff70f0]

System.Runtime.CompilerServices.CastHelpers.ChkCastAny(Void*, System.Object) [/_/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/CastHelpers.cs @ 374]

• verifyheap and dumpobj results

verifyheap                                                                                                                                                                                               
Heap      Segment       Object Failure                          
   4 7f0f48f5d698 7f103e404428 InvalidObjectReference           Object 7f103e404428 has a bad member at offset cee8: 7f106210cf58
   4 7f0f48fda068 7f12f3cf00a8 InvalidObjectReference           Object 7f12f3cf00a8 has a bad member at offset 16ccd8: 7f12d8ce5568
   8 7f0f48f6bec0 7f108f1a1348 InvalidMethodTable               Object 7f108f1a1348 has an invalid method table 7f8647dad500
   9 7f0f48f63070 7f105d891310 InvalidMethodTable               Object 7f105d891310 has an invalid method table 7f8647dad6e8
  13 7f0f48fad488 7f11fc906160 InvalidObjectReference           Object 7f11fc906160 has a bad member at offset 16ccd8: 7f12d8ce5568
  15 7f0f48f63d60 7f106210cf58 InvalidMethodTable               Object 7f106210cf58 has an invalid method table 7f8647daced8

101,216,252 objects verified, 6 errors.

> dumpobj 7f103e404428                                                                                                                                                                                     
Name:        System.Object[]
MethodTable: 00007f863e82a508
EEClass:     00007f863e82a480
Tracked Type: false
Size:        58640(0xe510) bytes
Array:       Rank 1, Number of elements 7327, Type CLASS
Fields:
None

> dumpobj 7f12f3cf00a8                                                                                                                                                                                     
Name:        Company.Product.ObjectModel.DTOParent.DTOChildren[]
MethodTable: 00007f864a75ea00
EEClass:     00007f863e82a480
Tracked Type: false
Size:        4194328(0x400018) bytes
Array:       Rank 1, Number of elements 524288, Type CLASS
Fields:
None

> dumpobj 7f11fc906160                                                                                                                                                                                     
Name:        Company.Product.ObjectModel.DTOParent.DTOChildren[]
MethodTable: 00007f864a75ea00
EEClass:     00007f863e82a480
Tracked Type: false
Size:        2097176(0x200018) bytes
Array:       Rank 1, Number of elements 262144, Type CLASS
Fields:
None

[mangod9]

hi @stefanschnabs, you could upload the dumps to a share and send details to my email (should be in my profile). Also since you seem to have been able to narrow down the issue to something around casting, is it possible to create a smaller repro?

@VSadov as fyi.

[VSadov]

Looks like corrupted heap issue. CastHelpers are some of the most called methods and thus tend to trip over GC holes more often. Seeing such crash in CastHelpers is typically (and sadly) not very helpful with figuring the root cause.

[jeffschwMSFT]

Our preference for receiving dumps is via the Visual Studio Developer Community. https://developercommunity.visualstudio.com/home

[stefanschnabs]

As an update, the dump had been uploaded using Secure File Exchange