Too much Gen0 memory usage on Region GC

[InCerryGit]

Description

I want to experience region GC on .NET6.0, I just tried this libclrgc.so in my test environment and .NET6.0 is able to use it. But it has a very high memory usage.

Use dotMemory analyze the dump, gen0 has 173 heaps.

Reproduction Steps

Create a .NET6 project, place the following *.so in the same directory as coreclr.so

https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

Configure environment variables:

DOTNET_GCName=libclrgc.so

Run the program。

Expected behavior

There shouldn't be so much memory usage。

Actual behavior

Memory usage is several times higher than SegmentGC

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

cc @Maoni0

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I want to experience region GC on .NET6.0, I just tried this libclrgc.so in my test environment and .NET6.0 is able to use it. But it has a very high memory usage.

Use dotMemory analyze the dump, gen0 has 173 heaps.

Reproduction Steps

Create a .NET6 project, place the following *.so in the same directory as coreclr.so

https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

Configure environment variables:

DOTNET_GCName=libclrgc.so

Run the program。

Expected behavior

There shouldn't be so much memory usage。

Actual behavior

Memory usage is several times higher than SegmentGC

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

cc @Maoni0

Author:	InCerryGit
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[InCerryGit]

NET7 has the same issue use the libclrgc.dll linked below.
https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

.NET Version:

Host:
  Version:      7.0.2
  Architecture: x64
  Commit:       d037e070eb

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

System version：

bash#: cat /etc/alpine-release
3.17.1

[mangod9]

@InCerryGit, could you please clarify your last comment about .NET 7? Regions are enabled by default in 7, so if you are using libclrgc.so in 7, then you are falling back to segments.

[InCerryGit]

@InCerryGit, could you please clarify your last comment about .NET 7? Regions are enabled by default in 7, so if you are using libclrgc.so in 7, then you are falling back to segments.

Sorry, I forgot to explain this, I used the libclrgc.dll linked below instead of the one in the .NET7 SDK. @mangod9

https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

[Maoni0]

the only difference between that libclrgc.dll and the .net 7.0 version you are using is

1. demotion fix - Fixing a bug that causes us to mistakenly demote a gen2 region to gen0 #82413
2. decommit fix - Decommit memory before returning to the region_allocator #80640
3. special_sweep_p fix only for workstation GC - Fix special sweep issue for workstation #81248
4. a premature OOM fix - [.NET 7] Fix spurious OOMs after upgrading to 7 #79158 (this was just because that was my baseline which had that change not included in 7.0.2)

it's doubtful any of these would cause dramatic changes in heap usage.

I also have some instrumentation which takes 22,528 bytes per heap and 3KiB for all heaps. each heap_segment filed is increased by 8 bytes. so that can't be contributing to such dramatic increase.

I have no idea what "173 heaps" means. I don't use dotMemory so I'm not familiar with its terminology.

if you have a top level GC trace when running with 7.0 and with the libclrgc build, we can take a look.

[Maoni0]

also just making sure, when you say .net 7, you mean you are using .net 7 with no COMPlus_GCName env var specified, correct?

[InCerryGit]

also just making sure, when you say .net 7, you mean you are using .net 7 with no COMPlus_GCName env var specified, correct?

Yes, the .NET7 mentioned above is the one without COMPlus_GCName set. I uploaded the .NET6 dmp to onedrive. @Maoni0
https://1drv.ms/u/s!AoKi8-KT2tEKj5VUDDcX7s3OqhuIfA?e=etBZ4Q

[Maoni0]

do you have a top level GC trace for with and without that libcoreclr? how to collect one is described in mem-doc: https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-collect-top-level-gc-metrics

I also don't have the pwd to your share. but dumps aren't a good way to diagnose this kind of issues. collecting a top level GC trace for both before and after is the best way to get started.

[InCerryGit]

do you have a top level GC trace for with and without that libcoreclr? how to collect one is described in mem-doc: https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-collect-top-level-gc-metrics

I also don't have the pwd to your share. but dumps aren't a good way to diagnose this kind of issues. collecting a top level GC trace for both before and after is the best way to get started.

I removed the pwd of dmp. is it possible for me to capture using the following command?

dotnet-trace collect -p 43 --profile gc-verbose

@Maoni0

[Maoni0]

please use the cmd from the link above -

dotnet trace collect -p <pid> -o <outputpath with .nettrace extension> --profile gc-collect --duration <in hh:mm:ss format>

don't use gc-verbose. it just adds overhead for no benefit. you don't have to do --duration if you are going to manually stop it.

[InCerryGit]

The trace file has been uploaded, the dump file is too big, I will upload it to the onedrive later。
@Maoni0
https://1drv.ms/u/s!AoKi8-KT2tEKj5VVkG3TOnnNAfi0mw?e=XP1aYV

NET7 without libclrgc.so nomaly

NET7 use libclrgc.so abnomaly
https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

[mangod9]

@InCerryGit wanted to understand why you are running with libclrgc here. With .NET 7, regions are enabled by default, and from your charts above it doesnt seem to show any memory regression?