benchmark-linux-try CQ flaky due to pkg/front_end/tool/fasta_perf.dart crash in VM

[sortie]

The benchmark-linux-try builder in the CQ has become flaky with a VM crash when running the pkg/front_end/tool/fasta_perf.dart benchmark on linux-ia32. This builder is not subject to deflaking and this situation currently disrupts the CQ runs for all Dart developers. Since the flake first appeared, it has hit 4 of 18 benchmark-linux-try tryjobs on multiple changelists.

Example crashes:

• https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8901521911311761152/+/steps/try_linux-ia32_benchmarking/0/stdout
• https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8901533162780692640/+/steps/try_linux-ia32_benchmarking/0/stdout
• https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8901539599402207312/+/steps/try_linux-ia32_benchmarking/0/stdout
• https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8901582727068385536/+/steps/try_linux-ia32_benchmarking/0/stdout

I can reproduce the issue locally on master (f546362) on linux-ia32 by running:

cat > hello.dart << EOF                                                                                                  
main() {
  print("Hello, World");
}
EOF
while out/ReleaseIA32/dart pkg/front_end/tool/fasta_perf.dart kernel_gen_e2e hello.dart; do true; done

After 75 iterations, I managed to hit the crash locally:

===== CRASH =====
si_signo=Segmentation fault(11), si_code=2, si_addr=0xf322ce06
version=2.6.0-edge.f546362691fe17b38298e5d9baa25c0c292f5448 (Mon Sep 23 16:35:05 2019 +0200) on "linux_ia32"
thread=92201, isolate=main(0x2a8c700)
  pc 0xf322ce06 fp 0xf603e210 Unknown symbol
...
  pc 0xf7880979 fp 0xf603eef4 Unknown symbol
  pc 0x01a3417b fp 0xf603ef68 dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&, unsigned int)
  pc 0x01a35fdd fp 0xf603efa8 dart::DartLibraryCalls::HandleMessage(dart::Object const&, dart::Instance const&)
  pc 0x01a63e20 fp 0xf603f0d8 dart::IsolateMessageHandler::HandleMessage(std::__2::unique_ptr<dart::Message, std::__2::default_delete<dart::Message> >)
  pc 0x01a88b45 fp 0xf603f128 dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool)
  pc 0x01a893b1 fp 0xf603f188 dart::MessageHandler::TaskCallback()
  pc 0x01a89b07 fp 0xf603f1a8 out/ReleaseIA32/dart+0x1689b07
  pc 0x01b7a751 fp 0xf603f1e8 dart::ThreadPool::Worker::Loop()
  pc 0x01b7a5f7 fp 0xf603f218 dart::ThreadPool::Worker::Main(unsigned int)
  pc 0x01b00e84 fp 0xf603f2d8 out/ReleaseIA32/dart+0x1700e84
  pc 0xf7bbc982 fp 0xf603f3a8 /lib/i386-linux-gnu/libpthread.so.0+0x6982
-- End of DumpStackTrace
Aborted

The issue has only manifested itself in 4 of 18 tryruns since it first appeared. If that ratio is to be believed, it might affect 22% of all CQ runs.

cc @mkustermann @askeksa-google

[sortie]

The crash hasn't happened for 104 try runs of benchmark-linux-try. Before that, over a duration of 138 commits since it first appeared, it happened 18 times (13%).

Perhaps a VM fix has landed that fixes the problem? Or maybe things just changed enough that it's not crashing anymore but the underlying problem isn't fixed. Since the problem isn't breaking the CQ anymore, I'm lowering the priority to P2, unless it appears again. If it doesn't reappear, I'll close this issue.

It would be good if someone could take a look at the crash and confirm that it is indeed fixed.

[mkustermann]

@aartbik If you have time, could you take a look? (I'm OOO atm)

[sortie]

Such a flake did just occur again at https://ci.chromium.org/p/dart/builders/try/benchmark-linux-try/23866

[aartbik]

This looks very similar to what we observed with the fuzzer, at least at first glance:

#38532

[aartbik]

CC @bkonyi @feli-citas @fizaaluthra

[aartbik]

I was able to reproduce the fuzzer bug (1 out of 100) so reopened that one (since it has a simpler reproducer).

[aartbik]

See #38532 for investigation.

[sortie]

This has begun happening again: #38971

Example: https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8899229532262907776/+/steps/try_linux-ia32_benchmarking/0/stdout