[CI][lldb] Flaky tests

[boomanaiden154]

We're seeing a significant number of CI flakes (3/~50 runs have failed).

1. https://lab.llvm.org/staging/#/builders/192/builds/1291
2. https://lab.llvm.org/staging/#/builders/192/builds/1295
3. https://lab.llvm.org/staging/#/builders/192/builds/1325

One shows the following failed test:

2025-08-21T06:51:18.635834216Z ********************
2025-08-21T06:51:18.635836396Z Failed Tests (1):
2025-08-21T06:51:18.635838606Z   lldb-unit :: DAP/./DAPTests/DisconnectRequestHandlerTest/DisconnectTriggersTerminateCommands

Two fail to resolve the following test:

2025-08-21T13:00:19.674647238Z ********************
2025-08-21T13:00:19.674650118Z Unresolved Tests (1):
2025-08-21T13:00:19.674653028Z   lldb-api :: functionalities/statusline/TestStatusline.py

This is on the premerge buildbot, but I assume these are showing up premerge as well given the configuration is almost exactly the same (same shell script running in the same container on the same cluster). This makes changes like d9d5090 a lot harder to justify given many more people now need to pay the price of flaky tests.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Aiden Grossman (boomanaiden154)

We're seeing a significant number of CI flakes (3/~50 runs have failed).

1. https://lab.llvm.org/staging/#/builders/192/builds/1291
2. https://lab.llvm.org/staging/#/builders/192/builds/1295
3. https://lab.llvm.org/staging/#/builders/192/builds/1325

One shows the following failed test:

2025-08-21T06:51:18.635834216Z ********************
2025-08-21T06:51:18.635836396Z Failed Tests (1):
2025-08-21T06:51:18.635838606Z   lldb-unit :: DAP/./DAPTests/DisconnectRequestHandlerTest/DisconnectTriggersTerminateCommands

Two fail to resolve the following test:

2025-08-21T13:00:19.674647238Z ********************
2025-08-21T13:00:19.674650118Z Unresolved Tests (1):
2025-08-21T13:00:19.674653028Z   lldb-api :: functionalities/statusline/TestStatusline.py

This is on the premerge buildbot, but I assume these are showing up premerge as well given the configuration is almost exactly the same (same shell script running in the same container on the same cluster). This makes changes like d9d5090 a lot harder to justify given many more people now need to pay the price of flaky tests.

[llvmbot]

@llvm/issue-subscribers-infrastructure

Author: Aiden Grossman (boomanaiden154)

We're seeing a significant number of CI flakes (3/~50 runs have failed).

1. https://lab.llvm.org/staging/#/builders/192/builds/1291
2. https://lab.llvm.org/staging/#/builders/192/builds/1295
3. https://lab.llvm.org/staging/#/builders/192/builds/1325

One shows the following failed test:

2025-08-21T06:51:18.635834216Z ********************
2025-08-21T06:51:18.635836396Z Failed Tests (1):
2025-08-21T06:51:18.635838606Z   lldb-unit :: DAP/./DAPTests/DisconnectRequestHandlerTest/DisconnectTriggersTerminateCommands

Two fail to resolve the following test:

2025-08-21T13:00:19.674647238Z ********************
2025-08-21T13:00:19.674650118Z Unresolved Tests (1):
2025-08-21T13:00:19.674653028Z   lldb-api :: functionalities/statusline/TestStatusline.py

This is on the premerge buildbot, but I assume these are showing up premerge as well given the configuration is almost exactly the same (same shell script running in the same container on the same cluster). This makes changes like d9d5090 a lot harder to justify given many more people now need to pay the price of flaky tests.

[Michael137]

Thanks for filing. FWIW, LLVM PRs already enabled LLDB tests pre-merge for a while. We should definitely deal with the extra noise though. Not sure what to do about the DAP flakiness. I've been noticing increasingly many of them on our macOS CI (@JDevlieghere @da-viper @ashgti might know why those are happening). Re. statusline, @JDevlieghere might know.

[JDevlieghere]

I can look into the statusline failure, I don't think I've ever seen that on our bots though.

For the DAP tests, I started a Discourse discussion a while ago and I think we've made some good progress but clearly we're not there yet.

[ashgti]

I have #147787 that I think should help improve the consistency of the debugger abstraction for tests.

The change makes it so we only apply state changes at specific times in the DebugCommunication class. Previously, it would mutate the state from a reader thread whenever messages came in and I think that caused some instability due to differences in performance between archs.

[DavidSpickett]

TestStatusline.py is flaky on Linaro's AArch64 Linux bot:
https://lab.llvm.org/buildbot/#/builders/59/builds/23337
https://lab.llvm.org/buildbot/#/builders/59/builds/23315
https://lab.llvm.org/buildbot/#/builders/59/builds/23271

Always the same part of the test:

======================================================================
ERROR: test_modulelist_deadlock (TestStatusline.TestStatusline)
   Regression test for a deadlock that occurs when the status line is enabled before connecting
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/functionalities/statusline/TestStatusline.py", line 199, in test_modulelist_deadlock
    self.expect(
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/lldbpexpect.py", line 95, in expect
    self.expect_prompt()
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/lldbpexpect.py", line 19, in expect_prompt
    self.child.expect_exact(self.PROMPT)
  File "/usr/local/lib/python3.10/dist-packages/pexpect/spawnbase.py", line 432, in expect_exact
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python3.10/dist-packages/pexpect/expect.py", line 181, in expect_loop
    return self.timeout(e)
  File "/usr/local/lib/python3.10/dist-packages/pexpect/expect.py", line 144, in timeout
    raise exc
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0xec0466807310>
command: /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/bin/lldb
args: ['/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/bin/lldb', '--no-lldbinit', '--no-use-colors', '-O', 'settings clear --all', '-O', 'settings set symbols.enable-external-lookup false', '-O', 'settings set target.inherit-tcc true', '-O', 'settings set target.disable-aslr false', '-O', 'settings set target.detach-on-error false', '-O', 'settings set target.auto-apply-fixits false', '-O', 'settings set plugin.process.gdb-remote.packet-timeout 60', '-O', 'settings set symbols.clang-modules-cache-path "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api"', '-O', 'settings set use-color false', '-O', 'settings set show-statusline false']
buffer (last 100 chars): b' \x1b[0m\x1b8'
before (last 100 chars): b'                                      \x1b[0m\x1b8'
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 2958969
child_fd: 6
closed: False
timeout: 60
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_string:
    0: b'(lldb) '
Config=aarch64-/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/bin/clang
----------------------------------------------------------------------

Same one seen in CI:

2025-08-28T08:24:13.4068448Z ======================================================================
2025-08-28T08:24:13.4068844Z ERROR: test_modulelist_deadlock (TestStatusline.TestStatusline.test_modulelist_deadlock)
2025-08-28T08:24:13.4069366Z    Regression test for a deadlock that occurs when the status line is enabled before connecting
2025-08-28T08:24:13.4069939Z ----------------------------------------------------------------------
2025-08-28T08:24:13.4070279Z Traceback (most recent call last):
2025-08-28T08:24:13.4070918Z   File "/home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/test/API/functionalities/statusline/TestStatusline.py", line 199, in test_modulelist_deadlock
2025-08-28T08:24:13.4071558Z     self.expect(
2025-08-28T08:24:13.4072041Z   File "/home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/lldbpexpect.py", line 94, in expect
2025-08-28T08:24:13.4072567Z     self.child.expect_exact(s)
2025-08-28T08:24:13.4072964Z   File "/home/gha/.local/lib/python3.12/site-packages/pexpect/spawnbase.py", line 432, in expect_exact
2025-08-28T08:24:13.4073374Z     return exp.expect_loop(timeout)
2025-08-28T08:24:13.4073607Z            ^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-28T08:24:13.4073984Z   File "/home/gha/.local/lib/python3.12/site-packages/pexpect/expect.py", line 181, in expect_loop
2025-08-28T08:24:13.4074372Z     return self.timeout(e)
2025-08-28T08:24:13.4074570Z            ^^^^^^^^^^^^^^^
2025-08-28T08:24:13.4074902Z   File "/home/gha/.local/lib/python3.12/site-packages/pexpect/expect.py", line 144, in timeout
2025-08-28T08:24:13.4075268Z     raise exc
2025-08-28T08:24:13.4075474Z pexpect.exceptions.TIMEOUT: Timeout exceeded.
2025-08-28T08:24:13.4075776Z <pexpect.pty_spawn.spawn object at 0x7fd65339b230>
2025-08-28T08:24:13.4076153Z command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/lldb
2025-08-28T08:24:13.4078513Z args: ['/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/lldb', '--no-lldbinit', '--no-use-colors', '-O', 'settings clear --all', '-O', 'settings set symbols.enable-external-lookup false', '-O', 'settings set target.inherit-tcc true', '-O', 'settings set target.disable-aslr false', '-O', 'settings set target.detach-on-error false', '-O', 'settings set target.auto-apply-fixits false', '-O', 'settings set plugin.process.gdb-remote.packet-timeout 60', '-O', 'settings set symbols.clang-modules-cache-path "/home/gha/actions-runner/_work/llvm-project/llvm-project/build/lldb-test-build.noindex/module-cache-lldb/lldb-api"', '-O', 'settings set use-color false', '-O', 'settings set show-statusline false']
2025-08-28T08:24:13.4080658Z buffer (last 100 chars): b'mote [127.0.0.1]:46707\r\n'
2025-08-28T08:24:13.4080985Z before (last 100 chars): b'gdb-remote [127.0.0.1]:46707\r\n'
2025-08-28T08:24:13.4081291Z after: <class 'pexpect.exceptions.TIMEOUT'>
2025-08-28T08:24:13.4081555Z match: None
2025-08-28T08:24:13.4081724Z match_index: None
2025-08-28T08:24:13.4081919Z exitstatus: None
2025-08-28T08:24:13.4082085Z flag_eof: False
2025-08-28T08:24:13.4082254Z pid: 712129
2025-08-28T08:24:13.4082433Z child_fd: 6
2025-08-28T08:24:13.4082582Z closed: False
2025-08-28T08:24:13.4082747Z timeout: 60
2025-08-28T08:24:13.4082930Z delimiter: <class 'pexpect.exceptions.EOF'>
2025-08-28T08:24:13.4083177Z logfile: None
2025-08-28T08:24:13.4083335Z logfile_read: None
2025-08-28T08:24:13.4083511Z logfile_send: None
2025-08-28T08:24:13.4083674Z maxread: 2000
2025-08-28T08:24:13.4083851Z ignorecase: False
2025-08-28T08:24:13.4084027Z searchwindowsize: None
2025-08-28T08:24:13.4084221Z delaybeforesend: 0.05
2025-08-28T08:24:13.4084413Z delayafterclose: 0.1
2025-08-28T08:24:13.4084597Z delayafterterminate: 0.1
2025-08-28T08:24:13.4084807Z searcher: searcher_string:
2025-08-28T08:24:13.4085013Z     0: b'a.out \xe2\x94\x82 signal SIGSTOP'
2025-08-28T08:24:13.4085374Z Config=x86_64-/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/clang
2025-08-28T08:24:13.4085772Z ----------------------------------------------------------------------