[Build 2025103005-4.3] openQA test fails in system_tests - hangs at dispvm cleanup, "logical volume in use" error

[marmarek]

Observation

openQA test in scenario qubesos-4.3-pull-requests-x86_64-system_tests_dispvm@64bit fails in
system_tests

Test fails with Logical volume qubes_dom0/vm-disp1249-root-snap in use error (on dispvm cleanup) and then hangs.

Test suite description

Reproducible

Fails since (at least) Build 2025102201-4.3

Expected result

Last good: 2025101620-4.3 (or more recent)

Further details

Always latest result in this scenario: latest

[ben-grande]

I am not sure but maybe it is unrelated to open PRs and may have been merged already.

This job hanged on qubesos-update on 2025-10-28:

• https://openqa.qubes-os.org/tests/157761#step/system_tests/27

Seeing the video shows that it hanged, unfortunately, no log was uploaded. I think that the domain-unpaused should not have happened at that stage.

[ben-grande]

qubesos-update:

• Succeeded: 2025102104-4.3: https://openqa.qubes-os.org/tests/156729

Pull requests:

• Failed: 2025102219-4.3: https://openqa.qubes-os.org/tests/157264#step/system_tests/27

Seems to be related to:

• Clean preloaded disposable remnants on next boot qubes-core-admin#742

OpenQA bot results did not post system_tests_dispvm. Most likely because the test hanged. Twice...., only has system_tests... that should have appeared in Upload failure...

[marmarek]

OpenQA bot results did not post system_tests_dispvm

Yes, it's neither included as normal failure nor upload failure :( not sure why, should be IMO considered upload failure.

[ben-grande]

Or related to:

• Redo netvm setup after unpause qubes-core-admin#722

Due to domain-unpaused becoming async.

[ben-grande]

As to why the openqa bot didn't print it correctly in the upload failure, it is because there are other failures than system_tests.... It was not added to the other sections

python3 utils/github_reporting.py --package-list utils/github_package_mapping.json --late                                st --build 2025102219-4.3 --flavor pull-requests --version 4.3 --compare-to-build 2025081011-4.3 --show-results-only --verbose --db-path /tmp/db > /tmp/report.old

# do some changes
user@disp8505:~/openqa-tests-qubesos$ git diff

diff --git a/utils/github_reporting.py b/utils/github_reporting.py
index d890bb7..924c49c 100644
--- a/utils/github_reporting.py
+++ b/utils/github_reporting.py
@@ -72,6 +72,11 @@ def format_results(results, jobs, reference_jobs=None, instability_analysis=None
             if all(f.fixed for f in results[k]):
                 continue
             if any(f.name != "system_tests" for f in results[k]):
+                print("--------------------")
+                for f in results[k]:
+                    print(f)
+                #    #if f.name == "system_tests":
+                #    #    print(f)
                 continue
             failed_tests_details += '* ' + str(k) + "\n"
             for fail in results[k]:

 python3 utils/github_reporting.py --package-list utils/github_package_mapping.json --late                                st --build 2025102219-4.3 --flavor pull-requests --version 4.3 --compare-to-build 2025081011-4.3 --show-results-only --verbose --db-path /tmp/db > /tmp/report.new

user@disp8505:~/openqa-tests-qubesos$ diff --color=auto /tmp/report.old /tmp/report.new     

0a1,81                                                                                                                                                  
> --------------------                                                                                                                                         
> qubesmanager_vmsettings: [unnamed test](https://openqa.qubes-os.org/tests/157185#step/qubesmanager_vmsettings/70) (unknown)                                  
> qubesmanager_vmsettings: [Failed](https://openqa.qubes-os.org/tests/157185#step/qubesmanager_vmsettings/71) (test died)                                      
>  `# Test died: no candidate needle with tag(s) 'vm-settings-devices-s...`                                                                                    
>                                                                                                                                       
> --------------------                                                                                                                                         
> system_tests: [Fail](https://openqa.qubes-os.org/tests/157193#step/system_tests/31) (unknown)                                                                
>  `Tests qubes.tests.extra failed (exit code 1), details reported sepa...`                                                                                    
>                                                                                                                                       
> system_tests: [Failed](https://openqa.qubes-os.org/tests/157193#step/system_tests/70) (test died)                                                            
>  `# Test died: Some tests failed at qubesos/tests/system_tests.pm lin...`                                                                                    
>                                                                                                                                       
> TC_00_QVCTest_whonix-workstation-17: [test_010_screenshare](https://openqa.qubes-os.org/tests/157193#step/TC_00_QVCTest_whoni                                x-workstation-17/1) (failure)                                                                                                                                  
>  `AssertionError: 1 != 0 : Timeout waiting for /dev/video0 in test-in...`                                                                                    
>                                                                                                                                       
> --------------------                                                                                                                                         
> gui_filecopy: [unnamed test](https://openqa.qubes-os.org/tests/157207#step/gui_filecopy/11) (unknown)                                                        
> gui_filecopy: [Failed](https://openqa.qubes-os.org/tests/157207#step/gui_filecopy/12) (test died)                                                            
>  `# Test died: no candidate needle with tag(s) 'files-test-file' matc...`                                                                                    
>
> --------------------                                                                                                                                         
> windows_install: [Failed](https://openqa.qubes-os.org/tests/157218#step/windows_install/31) (test died)                                                      
>  `# Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...`
>
> --------------------
> windows_clipboard_and_filecopy: [unnamed test](https://openqa.qubes-os.org/tests/157219#step/windows_clipboard_and_filecopy/69) (unknown)
> windows_clipboard_and_filecopy: [Failed](https://openqa.qubes-os.org/tests/157219#step/windows_clipboard_and_filecopy/70) (test died)
>  `# Test died: no candidate needle with tag(s) 'windows-Edge-address-...`
>
> --------------------
> windows_install: [wait_serial](https://openqa.qubes-os.org/tests/157220#step/windows_install/29) (wait serial expected)
>  `# wait_serial expected: qr/iDVvW-\d+-/...`
>
> windows_install: [Failed](https://openqa.qubes-os.org/tests/157220#step/windows_install/30) (test died + timed out)
>  `# Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...`
>
> --------------------
> qubesmanager_vmsettings: [unnamed test](https://openqa.qubes-os.org/tests/157231#step/qubesmanager_vmsettings/70) (unknown)
> qubesmanager_vmsettings: [Failed](https://openqa.qubes-os.org/tests/157231#step/qubesmanager_vmsettings/71) (test died)
>  `# Test died: no candidate needle with tag(s) 'vm-settings-devices-s...`
>
> --------------------
> system_tests: [wait_serial](https://openqa.qubes-os.org/tests/157264#step/system_tests/28) (wait serial expected)
>  `# wait_serial expected: qr/Tcmnz-\d+-/...`
>
> system_tests: [Failed](https://openqa.qubes-os.org/tests/157264#step/system_tests/29) (test died + timed out)
>  `# Test died: command 'testfunc qubes.tests.integ.dispvm' timed out ...`
>
> system_tests: [Fail](https://openqa.qubes-os.org/tests/149251#step/system_tests/27) (unknown)
>  `Tests qubes.tests.integ.dispvm failed (exit code 1), details report...`
>
> TC_20_DispVM_debian-13-xfce: [test_012_preload_low_mem](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/3) (failure)
>  `~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0`
>
> TC_20_DispVM_debian-13-xfce: [test_013_preload_gui](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/4) (error)
>  `raise KeyError(key)... KeyError: 'disp3723'`
>
> TC_20_DispVM_debian-13-xfce: [test_014_preload_nogui](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/5) (error + cleanup)
>  `raise TimeoutError from exc_val... TimeoutError`
>
> TC_20_DispVM_debian-13-xfce: [test_015_preload_race_more](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/6) (error + cleanup)
>  `raise KeyError(key)... KeyError: 'disp1187'`
>
> TC_20_DispVM_debian-13-xfce: [test_016_preload_race_less](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/7) (failure + cleanup)
>  `^^^^^^^^^^^^^^^^^^^^^^... AssertionError`
>
> TC_20_DispVM_debian-13-xfce: [test_017_preload_autostart](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/8) (error)
>  `raise KeyError(key)... KeyError: 'disp7317'`
>
> TC_20_DispVM_debian-13-xfce: [test_018_preload_global](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/9) (error)
>  `raise KeyError(key)... KeyError: 'disp8572'`
>
> TC_20_DispVM_debian-13-xfce: [test_019_preload_refresh](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_debian-13-xfce/10) (error)
>  `raise KeyError(key)... KeyError: 'disp6425'`
>
> TC_20_DispVM_fedora-42-xfce: [test_012_preload_low_mem](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_fedora-42-xfce/3) (failure)
>  `~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0`
>
> TC_20_DispVM_whonix-workstation-17: [test_012_preload_low_mem](https://openqa.qubes-os.org/tests/149251#step/TC_20_DispVM_whonix-workstation-17/3) (failure)
>  `~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0`

[ben-grande]

The culprit seems to be:

• Clean preloaded disposable remnants on next boot qubes-core-admin#742

Tests succeeds after reverting that commit.

commit 45f0c561392283d7ad231b03c66c722e66df9ae2 (HEAD)
Author: user <user@host>

    Revert "Clean preloaded disposable remnants on next boot"

    This reverts commit 3e842d714361054a3492f13d6589764202cea6a5.