8368176: ASAN should not inhibit hs-err file generation

[tstuefe]

ASAN, when catching an error, will abort the process.

Two things control this:

1. the compiler option -fsanitize-recover=address (resp. -fno-sanitize-recover=address. This controls whether, once ASAN returns from its error report, the compiler-generated ASAN stubs will abort the process. This is by default set to -fno-sanitize-recover=address, so we won't recover.
2. The runtime option halt_on_error controls whether ASAN itself returns from its error handler or whether it aborts the process. This, by default, is set to 1, so by default ASAN aborts.

We "double abort" in the sense that two options are overlaid and both prevent the process from continuing.

I propose that we set, during build time for ASAN builds, the option -fsanitize-recover=address. Now, we can control whether to abort or not using the runtime setting halt_on_error=0. By default, we still will abort, since halt_on_error=1. So, the default behavior won't change. However, we can now at least decide to do it differently.

What would that give us?

By aborting right away, ASAN denies the JVM the option to catch the error and write an hs-err file. Of course, not every error that ASAN catches will result in a segfault or in an assertion. The JVM could lurch on for a bit before it stumbles. However, the chance for the JVM to stop on its own very soon after a memory corruption happens is pretty good. Then we get a hs-err file and a crash dump in close correlation to the error ASAN caught.

And even if there is no close relationship between the original ASAN error and the eventual segfault/assertion (think ASAN sees a double free, JVM continues, and after a while asserts somewhere else as a remote consequence of the error - the stacks in the hs-err file won't be related to the original error) - the hs-err file is shock-full of helpful information about running threads (see also JDK-8368124), memory mappings, JVM flags, etc. All of that would make it easier to understand the ASAN report.

And even if the JVM survives, one can still attach to the still living process and grab thread dumps, VM.info reports, heap dumps etc.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8368176: ASAN should not inhibit hs-err file generation (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27404/head:pull/27404
$ git checkout pull/27404

Update a local copy of the PR:
$ git checkout pull/27404
$ git pull https://git.openjdk.org/jdk.git pull/27404/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27404

View PR using the GUI difftool:
$ git pr show -t 27404

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27404.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@tstuefe The following label will be automatically applied to this pull request:

• build

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (7137bb1a)

[sendaoYan]

Does the -fsanitize-recover=address compiler options supported when TOOLCHAIN_TYPE == microsoft

[kimbarrett]

Looking at Visual Studio documentation and such, it looks like -fsanitize-recover isn't supported (yet).
https://developercommunity.visualstudio.com/t/add-fsanitize-recoveraddress-support-to-asan/1459414

[sendaoYan]

Hi, jdk support address sanitizer and leak sanitizer for now. It seems that this PR only concern address sanitizer, does the leak sanitizer also need this enhancement

[kimbarrett]

Hi, jdk support address sanitizer and leak sanitizer for now. It seems that this PR only concern address sanitizer, does the leak sanitizer also need this enhancement

gcc15.2 docs list the sanitizers that support recovery, and "leak" is not in
that list. Sanitizers that support it enable recovery by default, except for
"address", which is noted as still being experimental (which is presumably why
it's not enabled by default).
https://gcc.gnu.org/onlinedocs/gcc-15.2.0/gcc/Instrumentation-Options.html#index-fsanitize-recover
Once "address" is no longer experimental we can presumably remove the explicit
recovery enabling.

But maybe we should consider this PR premature, since `-fsanitize-recovery=address" is still experimental.

[kimbarrett]

Shouldn't there be a configure check for the availability of the -fsanitize-recovery=address option?