Valgrind on Alpine Linux: Mismatched free() / delete / delete []

[am11]

According to valgrind, we still have mismatching issues on linux-musl-x64, which don't appear on linux-x64.

$ dotnet new console -n test1; cd test1
$ dotnet publish -c Release --use-current-runtime -p:PublishSingleFile=true --self-contained -o dist

$ dotnet-symbol dist/test1

$ valgrind --leak-check=full --show-leak-kinds=all dist/test1 >dump 2>&1

The dump contains stuff like:

==15289== Mismatched free() / delete / delete []
==15289==    at 0x48A4B0D: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==15289==    by 0x4E0DD8: ClrFree (src/coreclr/utilcode/clrhost_nodependencies.cpp:304)
==15289==    by 0x4E0DD8: operator delete(void*) (src/coreclr/utilcode/clrhost_nodependencies.cpp:398)
==15289==    by 0x1E15D9: deallocate (new_allocator.h:133)
==15289==    by 0x1E15D9: deallocate (alloc_traits.h:492)
==15289==    by 0x1E15D9: _M_destroy (basic_string.h:237)
==15289==    by 0x1E15D9: _M_dispose (basic_string.h:232)
==15289==    by 0x1E15D9: ~basic_string (basic_string.h:658)
==15289==    by 0x1E15D9: get_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (src/native/corehost/hostmisc/utils.cpp:152)
==15289==    by 0x1E76D3: exe_start(int, char const**) (src/native/corehost/corehost.cpp:128)
==15289==    by 0x1E7DFF: main (src/native/corehost/corehost.cpp:301)
==15289==  Address 0x4acdda0 is 0 bytes inside a block of size 23 alloc'd
==15289==    at 0x48A2E63: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==15289==    by 0x1E139D: _M_construct<char *> (basic_string.tcc:219)
==15289==    by 0x1E139D: _M_construct_aux<char *> (basic_string.h:247)
==15289==    by 0x1E139D: _M_construct<char *> (basic_string.h:266)
==15289==    by 0x1E139D: basic_string (basic_string.h:451)
==15289==    by 0x1E139D: get_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (src/native/corehost/hostmisc/utils.cpp:133)
==15289==    by 0x1E76D3: exe_start(int, char const**) (src/native/corehost/corehost.cpp:128)
==15289==    by 0x1E7DFF: main (src/native/corehost/corehost.cpp:301)

with .NET 6 (70ae3df).

Daily build of .NET 7 also has this problem but unfortunately preview and daily build symbols are not being published so we get ??? instead of the names in dump: #71735.

The mismatch issue is because on Alpine Linux build machine, it is picking up #else branch here (which is unconditionally calling free, and there is no tracking how the object was originally allocated; new/new[]/malloc ....):

runtime/src/coreclr/utilcode/clrhost_nodependencies.cpp

Lines 388 to 392 in d9d6e82

	#if defined(HAS_ADDRESS_SANITIZER) || defined(DACCESS_COMPILE)
	// use standard heap functions for address sanitizer
	#else
	void __cdecl
	operator delete(void *p) NOEXCEPT

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

According to valgrind, we still have mismatching issues on linux-musl-x64, which don't appear on linux-x64.

$ dotnet new console -n test1; cd test1
$ dotnet publish -c Release --use-current-runtime -p:PublishSingleFile=true --self-contained -o dist

$ dotnet-symbol dist/test1

$ valgrind --leak-check=full --show-leak-kinds=all dist/test1 >dump 2>&1

The dump contains stuff like:

==15289== Mismatched free() / delete / delete []
==15289==    at 0x48A4B0D: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==15289==    by 0x4E0DD8: ClrFree (src/coreclr/utilcode/clrhost_nodependencies.cpp:304)
==15289==    by 0x4E0DD8: operator delete(void*) (src/coreclr/utilcode/clrhost_nodependencies.cpp:398)
==15289==    by 0x1E15D9: deallocate (new_allocator.h:133)
==15289==    by 0x1E15D9: deallocate (alloc_traits.h:492)
==15289==    by 0x1E15D9: _M_destroy (basic_string.h:237)
==15289==    by 0x1E15D9: _M_dispose (basic_string.h:232)
==15289==    by 0x1E15D9: ~basic_string (basic_string.h:658)
==15289==    by 0x1E15D9: get_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (src/native/corehost/hostmisc/utils.cpp:152)
==15289==    by 0x1E76D3: exe_start(int, char const**) (src/native/corehost/corehost.cpp:128)
==15289==    by 0x1E7DFF: main (src/native/corehost/corehost.cpp:301)
==15289==  Address 0x4acdda0 is 0 bytes inside a block of size 23 alloc'd
==15289==    at 0x48A2E63: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==15289==    by 0x1E139D: _M_construct<char *> (basic_string.tcc:219)
==15289==    by 0x1E139D: _M_construct_aux<char *> (basic_string.h:247)
==15289==    by 0x1E139D: _M_construct<char *> (basic_string.h:266)
==15289==    by 0x1E139D: basic_string (basic_string.h:451)
==15289==    by 0x1E139D: get_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (src/native/corehost/hostmisc/utils.cpp:133)
==15289==    by 0x1E76D3: exe_start(int, char const**) (src/native/corehost/corehost.cpp:128)
==15289==    by 0x1E7DFF: main (src/native/corehost/corehost.cpp:301)

with .NET 6 (70ae3df).

Daily build of .NET 7 also has this problem but unfortunately preview and daily build symbols are not being published so we get ??? instead of the names in dump: #71735.

The mismatch issue is because on Alpine Linux build machine, it is picking up #else branch here (which is unconditionally calling free, and there is no tracking how the object was originally allocated; new/new[]/malloc ....):

runtime/src/coreclr/utilcode/clrhost_nodependencies.cpp

Lines 388 to 392 in d9d6e82

	#if defined(HAS_ADDRESS_SANITIZER) || defined(DACCESS_COMPILE)
	// use standard heap functions for address sanitizer
	#else
	void __cdecl
	operator delete(void *p) NOEXCEPT

Author:	am11
Assignees:	-
Labels:	area-Host
Milestone:	-

[am11]

cc @AaronRobinsonMSFT
do we still need this #if/#else or can we let everything use the standard heap?

[janvorli]

IIRC, the #if branch was meant to mitigate issues with the sanitizer that didn't like custom new / delete. The same branch should be taken on glibc linux builds as well.
This sounds like yet another case when STL code got inlined into runtime code.
Edit: actually, since this host relies on STL heavily, the problem is actually the opposite, the utilcode being pulled into the host compilation causing the problem.
I keep forgetting why we need to use utilcode in the host, I need to refresh my memory on that.

[janvorli]

Ah, this is the single file host, that explains why it has utilcode in it.

[janvorli]

Btw, the reason why we have our own implementation of new / delete using the ClrMalloc / ClrFree is that we want the new to throw our OutOfMemoryException that our runtime understands. See the discussion in the #55945 for more details. This is not needed by the host itself, but in the case of single file host, the runtime is statically linked into the host. C++ requires that there is just one global new and delete operator.

[am11]

@janvorli, thanks. Any ideas why would _M_construct not calling new(size_t) overload in utilcode:

runtime/src/coreclr/utilcode/clrhost_nodependencies.cpp

Lines 308 to 310 in d9d6e82

	void * __cdecl
	operator new(size_t n)
	{

but _M_destroy is calling delete(void*) in utilcode from the same context? Can we do something to align them?

[janvorli]

I would actually expect that with Valgrind, its new and delete operators would get called (from the /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so), as they are preloaded into the process.
The only idea I had is that the problematic delete operator call might have been completely inlined.
Would you mind trying to add __attribute__((noinline)) to the delete operator in clrhost_nodependencies.cpp and see if it helps?

[am11]

I applied this patch on main (27811e5):

--- a/src/coreclr/utilcode/clrhost_nodependencies.cpp
+++ b/src/coreclr/utilcode/clrhost_nodependencies.cpp
@@ -388,7 +388,7 @@ void * __cdecl operator new[](size_t n, const NoThrow&) NOEXCEPT
 #if defined(HAS_ADDRESS_SANITIZER) || defined(DACCESS_COMPILE)
 // use standard heap functions for address sanitizer
 #else
-void __cdecl
+void NOINLINE __cdecl
 operator delete(void *p) NOEXCEPT
 {
     STATIC_CONTRACT_NOTHROW;
@@ -400,7 +400,7 @@ operator delete(void *p) NOEXCEPT
     TRASH_LASTERROR;
 }
 
-void __cdecl
+void NOINLINE __cdecl
 operator delete[](void *p) NOEXCEPT
 {
     STATIC_CONTRACT_NOTHROW;

but it did not fix the problem:

==79033== Mismatched free() / delete / delete []
==79033==    at 0x48A4B0D: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==79033==    by 0x4C2898: ClrFree (src/coreclr/utilcode/clrhost_nodependencies.cpp:304)
==79033==    by 0x4C2898: operator delete(void*) (src/coreclr/utilcode/clrhost_nodependencies.cpp:398)
==79033==    by 0x1E06FA: deallocate (new_allocator.h:145)
==79033==    by 0x1E06FA: deallocate (alloc_traits.h:496)
==79033==    by 0x1E06FA: _M_destroy (basic_string.h:245)
==79033==    by 0x1E06FA: _M_dispose (basic_string.h:240)
==79033==    by 0x1E06FA: ~basic_string (basic_string.h:672)
==79033==    by 0x1E06FA: get_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (src/native/corehost/hostmisc/utils.cpp:161)
==79033==    by 0x1E6773: exe_start(int, char const**) (src/native/corehost/corehost.cpp:144)
==79033==    by 0x1E6E5F: main (src/native/corehost/corehost.cpp:319)
==79033==  Address 0x4af9ca0 is 0 bytes inside a block of size 18 alloc'd
==79033==    at 0x48A2E63: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==79033==    by 0x1E04BC: _M_construct<char *> (basic_string.tcc:219)
==79033==    by 0x1E04BC: _M_construct_aux<char *> (basic_string.h:255)
==79033==    by 0x1E04BC: _M_construct<char *> (basic_string.h:274)
==79033==    by 0x1E04BC: basic_string (basic_string.h:459)
==79033==    by 0x1E04BC: get_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (src/native/corehost/hostmisc/utils.cpp:142)
==79033==    by 0x1E6773: exe_start(int, char const**) (src/native/corehost/corehost.cpp:144)
==79033==    by 0x1E6E5F: main (src/native/corehost/corehost.cpp:319)

[janvorli]

but unfortunately preview and daily build symbols are not being published

The dotnet-symbol tool should be able to download the symbols even for these builds.

[am11]

It should, but apparently those symbols are not being published:

$ dotnet7 --version
7.0.100-preview.6.22314.18

$ dotnet7 publish -c Release --use-current-runtime -p:PublishSingleFile=true --self-contained -o dist
$ dotnet-symbol dist/test1

Downloading from http://msdl.microsoft.com/download/symbols/
ERROR: Not Found: singlefilehost.dbg - 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-3baf0d94146bef3f8125e46db0b7d4327b907ba3/_.debug'

I have opened #71735.