__sanitizer::StopTheWorld sometimes hangs

[evverx]

How to reproduce:

$ rpm -q clang compiler-rt libseccomp
clang-3.8.1-1.fc25.x86_64
compiler-rt-3.8.1-1.fc25.x86_64
libseccomp-2.3.1-1.fc25.x86_64

$ cat /proc/version
Linux version 4.9.10-200.fc25.x86_64 (mockbuild@bkernel01.phx2.fedoraproject.org) (gcc version 6.3.1 20161221 (Red Hat 6.3.1-1) (GCC) ) #1 SMP Wed Feb 15 23:28:59 UTC 2017

$ grep SECCOMP /boot/config-4.9.10-200.fc25.x86_64
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_SECCOMP=y

#include <seccomp.h>

int main(int argc, char *argv[]) {
        scmp_filter_ctx ctx;
        int r = -1;

        ctx = seccomp_init(SCMP_ACT_ALLOW);
        if (!ctx)
                goto out;

        r = seccomp_rule_add(ctx, SCMP_ACT_KILL, SCMP_SYS(ptrace), 0);
        if (r < 0)
                goto out;

        r = seccomp_load(ctx);
        if (r < 0)
                goto out;

out:
        seccomp_release(ctx);
        return -r;
}

$ clang -o hang-stop-the-world -fsanitize=address -lseccomp ./hang-stop-the-world.c

$ ./hang-stop-the-world &
[1] 30768

$ ps -C hang-stop-the-world
  PID TTY          TIME CMD
30768 pts/2    00:00:16 hang-stop-the-w
30769 pts/2    00:00:00 hang-stop-the-w <defunct>

$ sudo journalctl -b _TRANSPORT=audit -o cat | grep -i hang-stop
SECCOMP auid=1001 uid=1001 gid=1001 ses=4 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=30769 comm="hang-stop-the-w" exe="/home/vagrant/hang-stop-the-world" sig=31 arch=c000003e syscall=101 compat=0 ip=0x4cef73 code=0x0

$ pidstat 1 5
...
Average:      UID       PID    %usr %system  %guest    %CPU   CPU  Command
...
Average:     1001     30768    8.88   82.45    0.00   91.32     -  hang-stop-the-w

$ pstack 30768
#0  0x00007fec211ba2c7 in sched_yield () at ../sysdeps/unix/syscall-template.S:84
#1  0x00000000004dd105 in __sanitizer::StopTheWorld(void (*)(__sanitizer::SuspendedThreadsList const&, void*), void*) ()
#2  0x00000000004e28f0 in __lsan::DoStopTheWorldCallback(dl_phdr_info*, unsigned long, void*) ()
#3  0x00007fec21213ab4 in __GI___dl_iterate_phdr (callback=0x4e28e0 <__lsan::DoStopTheWorldCallback(dl_phdr_info*, unsigned long, void*)>, data=0x7ffe27214670) at dl-iteratephdr.c:76
#4  0x00000000004e2c8c in __lsan::DoStopTheWorld(void (*)(__sanitizer::SuspendedThreadsList const&, void*), void*) ()
#5  0x00000000004e1e5b in __lsan::CheckForLeaks() [clone .part.27] ()
#6  0x00000000004e2158 in __lsan::DoLeakCheck() ()
#7  0x00007fec21109410 in __run_exit_handlers (status=0, listp=0x7fec2148f5b8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83
#8  0x00007fec2110946a in __GI_exit (status=<optimized out>) at exit.c:105
#9  0x00007fec210ef408 in __libc_start_main (main=0x4e7be0 <main>, argc=1, argv=0x7ffe27214848, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe27214838) at ../csu/libc-start.c:323
#10 0x000000000041870a in _start ()

[dvyukov]

Isn't it expected result of killing random system calls?

[evverx]

@dvyukov , it's the simplest way to suddenly kill a helper thread. So I can reliably trigger this forever loop:

    while (atomic_load(&tracer_thread_argument.done, memory_order_relaxed) == 0)
      sched_yield();

The program is meaningless of course :-)

[dvyukov]

What behavior do you expect?

[evverx]

The program shouldn't hang

[dvyukov]

And do what?

[evverx]

Something like

log("helper thread suddenly disappeared")
_exit(1)

[dvyukov]

This may be slightly better failure mode... unless you also forbid exit_group system call.

[evverx]

Hm, do you mean:

r = seccomp_rule_add(ctx, SCMP_ACT_KILL, SCMP_SYS(exit_group), 0);

?

It produces

Bad system call
$ echo $?
159

There is no an infinity loop or anything else.

[dvyukov]

I mean making exit_group return ENOSYS. There may be something in libseccomp that prevents you from doing so, but it's generally possible with seccomp.

[evverx]

@dvyukov , I got it, thanks!

        r = seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(exit_group), 0);
        if (r < 0)
                goto out;

        r = seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(exit), 0);
        if (r < 0)
                goto out;

$ ./hang-stop-the-world
Segmentation fault (core dumped)

prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)  = 0
seccomp(SECCOMP_SET_MODE_STRICT, 1, NULL) = -1 EINVAL (Invalid argument)
seccomp(SECCOMP_SET_MODE_FILTER, 0, {len=10, filter=0x1c81f50}) = 0
exit_group(0)                           = -1 ENOSYS (Function not implemented)
exit(0)                                 = -1 ENOSYS (Function not implemented)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

Program received signal SIGSEGV, Segmentation fault.
__GI__exit (status=status@entry=0) at ../sysdeps/unix/sysv/linux/_exit.c:36
36            ABORT_INSTRUCTION;

(gdb) thread apply all bt

Thread 1 (process 31814):
#0  __GI__exit (status=status@entry=0) at ../sysdeps/unix/sysv/linux/_exit.c:36
#1  0x00007ffff780a3c3 in __run_exit_handlers (status=0, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:98
#2  0x00007ffff780a46a in __GI_exit (status=<optimized out>) at exit.c:105
#3  0x00007ffff77f0408 in __libc_start_main (main=0x400760 <main>, argc=1, argv=0x7fffffffe4e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4d8)
    at ../csu/libc-start.c:323
#4  0x000000000040068a in _start ()

I think that's OK :-)

[dvyukov]

Our general suggestion is to disable any sandboxes in sanitizer builds.

[evverx]

@dvyukov , sure.

I'm trying to understand why the helper thread's death is not a fatal error. Could you explain, please? Thanks!

[evverx]

By "fatal error" I mean something like

==785==ERROR: AddressSanitizer failed to allocate 0x1000 (4096) bytes of InternalMmapVectorNoCtor (error code: 38)
ERROR: Failed to mmap

[dvyukov]

I'm trying to understand why the helper thread's death is not a fatal error. Could you explain, please?

I guess nobody bothered to handle it. A patch is welcome.