Add tests for few upcoming changes #654

marmarek · 2025-02-13T15:52:05Z

See individual commit messages for details

codecov · 2025-02-13T16:31:25Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.73%. Comparing base (9a89893) to head (d75733d).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #654      +/-   ##
==========================================
- Coverage   69.75%   69.73%   -0.02%     
==========================================
  Files          58       58              
  Lines       12550    12550              
==========================================
- Hits         8754     8752       -2     
- Misses       3796     3798       +2

Flag	Coverage Δ
unittests	`69.73% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

qubesos-bot · 2025-02-13T23:20:45Z

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025030301-4.3&flavor=pull-requests

Test run included the following:

Assure Update window width does not break OpenQA qubes-desktop-linux-manager#248 (QubesOS/qubes-desktop-linux-manager@1af77ee)
Revert to red label while restoring on missing labels qubes-core-admin-client#335 (QubesOS/qubes-core-admin-client@c4940be)
Add tests for few upcoming changes #654 (d75733d)
tests: integration tests for boot mode #664 (fcca635)
Reproduce bug in qrexec tests #661 (87f187d)
Add Arch packaging qubes-core-admin-client#332 (QubesOS/qubes-core-admin-client@e22c531)
Mark VMs with prohibit-start feature in updater qubes-desktop-linux-manager#247 (QubesOS/qubes-desktop-linux-manager@0c8493d)
Move all binaries from /usr/sbin to /usr/bin qubes-core-qubesdb#68 (QubesOS/qubes-core-qubesdb@658ce33)
Add boot mode support #653 (975b945)
Add the feature to prohibit starting a qube qubes-manager#408 (QubesOS/qubes-manager@0fea5d5)
Update firmware packages from bookworm-backports on Debian 12 qubes-core-admin-linux#184 (QubesOS/qubes-core-admin-linux@078c9db)
Unify method to query available terminals qubes-core-agent-linux#556 (QubesOS/qubes-core-agent-linux@5e5fc52)
Allow equal sign and coma in feature requests #665 (60ed744)

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025021804-4.3&flavor=update

system_tests_basic_vm_qrexec_gui
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3b01c8...
system_tests_extra
- TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
  ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0
system_tests_qrexec
- TC_00_Qrexec_whonix-gateway-17: test_065_qrexec_exit_code_vm (failure)
  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^... AssertionError: b'125\n' != b'0\n'
system_tests_kde_gui_interactive
- gui_keyboard_layout: wait_serial (wait serial expected)
  # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...
- gui_keyboard_layout: Failed (test died)
  # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...
system_tests_basic_vm_qrexec_gui_zfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=4b3bbc...
system_tests_audio
- TC_20_AudioVM_Pulse_whonix-workstation-17: test_221_audio_rec_muted_pulseaudio (error + cleanup)
  raise TimeoutError from exc_val... TimeoutError
system_tests_basic_vm_qrexec_gui_btrfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=2094a4...
system_tests_basic_vm_qrexec_gui_ext4
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3d69b2...
system_tests_basic_vm_qrexec_gui_xfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=fddc82...
system_tests_basic_vm_qrexec_gui@hw7
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=1bd747...

Failed tests

11 failures

system_tests_basic_vm_qrexec_gui
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3b01c8...
system_tests_extra
- TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
  ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0
system_tests_qrexec
- TC_00_Qrexec_whonix-gateway-17: test_065_qrexec_exit_code_vm (failure)
  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^... AssertionError: b'125\n' != b'0\n'
system_tests_kde_gui_interactive
- gui_keyboard_layout: wait_serial (wait serial expected)
  # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...
- gui_keyboard_layout: Failed (test died)
  # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...
system_tests_basic_vm_qrexec_gui_zfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=4b3bbc...
system_tests_audio
- TC_20_AudioVM_Pulse_whonix-workstation-17: test_221_audio_rec_muted_pulseaudio (error + cleanup)
  raise TimeoutError from exc_val... TimeoutError
system_tests_basic_vm_qrexec_gui_btrfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=2094a4...
system_tests_basic_vm_qrexec_gui_ext4
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3d69b2...
system_tests_basic_vm_qrexec_gui_xfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=fddc82...
system_tests_basic_vm_qrexec_gui@hw7
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=1bd747...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/129058#dependencies

15 fixed

system_tests_suspend
- suspend: unnamed test (unknown)
- suspend: Failed (test died)
  # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
system_tests_qrexec
- TC_00_Qrexec_whonix-workstation-17: test_070_qrexec_vm_simultaneous_write (failure)
  exited with non-zero code 125; stderr: b"dd: error writing 'standa...
system_tests_kde_gui_interactive
- clipboard_and_web: unnamed test (unknown)
- clipboard_and_web: Failed (test died)
  # Test died: no candidate needle with tag(s) 'clipboard-paste-notif...
system_tests_audio
- TC_20_AudioVM_Pulse_whonix-workstation-17: test_223_audio_play_hvm (failure)
  AssertionError: too short audio, expected 10s, got 7.33528344671201...
- TC_20_AudioVM_PipeWire_whonix-workstation-17: test_226_audio_playback_pipewire (failure)
  AssertionError: too short audio, expected 10s, got 9.19027210884353...
- TC_20_AudioVM_PipeWire_whonix-workstation-17: test_251_audio_playback_audiovm_pipewire_late_start (failure)
  AssertionError: too short audio, expected 10s, got 9.22900226757369...
system_tests_whonix@hw7
- whonixcheck: fail (unknown)
  Whonixcheck for sys-whonix failed...
- whonixcheck: unnamed test (unknown)
system_tests_basic_vm_qrexec_gui_btrfs
- TC_03_QvmRevertTemplateChanges: test_000_revert_linux (error)
  subprocess.CalledProcessError: Command '['sha1sum', '/var/lib/qubes...
system_tests_suspend@hw1
- suspend: unnamed test (unknown)
- suspend: Failed (test died)
  # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
system_tests_whonix
- whonixcheck: fail (unknown)
  Whonixcheck for sys-whonix failed...
- whonixcheck: unnamed test (unknown)

Unstable tests

Performance Tests

Performance degradation:

31 performance degradations

debian-12-xfce_exec: 8.47 :small_red_triangle_up: ( previous job: 7.15, degradation: 118.39%)
debian-12-xfce_exec-data-simplex: 71.98 :small_red_triangle_up: ( previous job: 48.93, degradation: 147.11%)
debian-12-xfce_exec-data-duplex: 69.42 :small_red_triangle_up: ( previous job: 50.76, degradation: 136.76%)
debian-12-xfce_exec-data-duplex-root: 73.53 :small_red_triangle_up: ( previous job: 64.91, degradation: 113.28%)
debian-12-xfce_socket-data-duplex: 163.41 :small_red_triangle_up: ( previous job: 81.49, degradation: 200.54%)
fedora-41-xfce_exec-data-simplex: 79.01 :small_red_triangle_up: ( previous job: 49.65, degradation: 159.14%)
fedora-41-xfce_exec-data-duplex: 72.56 :small_red_triangle_up: ( previous job: 49.08, degradation: 147.83%)
fedora-41-xfce_exec-data-duplex-root: 101.24 :small_red_triangle_up: ( previous job: 81.65, degradation: 123.99%)
fedora-41-xfce_socket-data-duplex: 144.14 :small_red_triangle_up: ( previous job: 78.62, degradation: 183.33%)
whonix-gateway-17_exec-data-simplex: 75.85 :small_red_triangle_up: ( previous job: 48.76, degradation: 155.55%)
whonix-gateway-17_exec-data-duplex: 75.18 :small_red_triangle_up: ( previous job: 48.55, degradation: 154.85%)
whonix-gateway-17_exec-data-duplex-root: 102.93 :small_red_triangle_up: ( previous job: 70.13, degradation: 146.78%)
whonix-gateway-17_socket-data-duplex: 172.33 :small_red_triangle_up: ( previous job: 82.74, degradation: 208.28%)
whonix-workstation-17_exec-data-simplex: 70.48 :small_red_triangle_up: ( previous job: 47.01, degradation: 149.91%)
whonix-workstation-17_exec-data-duplex: 74.21 :small_red_triangle_up: ( previous job: 49.48, degradation: 149.99%)
whonix-workstation-17_exec-data-duplex-root: 101.23 :small_red_triangle_up: ( previous job: 79.93, degradation: 126.65%)
whonix-workstation-17_socket-data-duplex: 175.16 :small_red_triangle_up: ( previous job: 81.71, degradation: 214.36%)
dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 54344.00 :small_red_triangle_up: ( previous job: 276742.00, degradation: 19.64%)
dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 123293.00 :small_red_triangle_up: ( previous job: 423495.00, degradation: 29.11%)
dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 90898.00 :small_red_triangle_up: ( previous job: 185030.00, degradation: 49.13%)
dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 12512.00 :small_red_triangle_up: ( previous job: 100699.00, degradation: 12.43%)
dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 3952.00 :small_red_triangle_up: ( previous job: 8767.00, degradation: 45.08%)
fedora-41-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 305529.00 :small_red_triangle_up: ( previous job: 343795.00, degradation: 88.87%)
fedora-41-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 3126.00 :small_red_triangle_up: ( previous job: 3785.00, degradation: 82.59%)
fedora-41-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 342448.00 :small_red_triangle_up: ( previous job: 401907.00, degradation: 85.21%)
fedora-41-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 276159.00 :small_red_triangle_up: ( previous job: 357875.00, degradation: 77.17%)
fedora-41-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 32766.00 :small_red_triangle_up: ( previous job: 87999.00, degradation: 37.23%)
fedora-41-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 1974.00 :small_red_triangle_up: ( previous job: 3885.00, degradation: 50.81%)
fedora-41-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 831.00 :small_red_triangle_up: ( previous job: 1613.00, degradation: 51.52%)
fedora-41-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 52560.00 :small_red_triangle_up: ( previous job: 78093.00, degradation: 67.30%)
fedora-41-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1809.00 :small_red_triangle_up: ( previous job: 2693.00, degradation: 67.17%)

Remaining performance tests:

41 tests

debian-12-xfce_exec-root: 29.33 :small_red_triangle_up: ( previous job: 27.97, degradation: 104.88%)
debian-12-xfce_socket: 8.71 :small_red_triangle_up: ( previous job: 8.33, degradation: 104.63%)
debian-12-xfce_socket-root: 8.61 :small_red_triangle_up: ( previous job: 8.20, degradation: 104.96%)
fedora-41-xfce_exec: 9.20 :small_red_triangle_up: ( previous job: 9.13, degradation: 100.68%)
fedora-41-xfce_exec-root: 66.29 :small_red_triangle_up: ( previous job: 61.17, degradation: 108.37%)
fedora-41-xfce_socket: 8.57 🟢 ( previous job: 8.66, improvement: 98.99%)
fedora-41-xfce_socket-root: 8.20 🟢 ( previous job: 8.61, improvement: 95.30%)
whonix-gateway-17_exec: 7.01 🟢 ( previous job: 7.87, improvement: 89.08%)
whonix-gateway-17_exec-root: 38.99 :small_red_triangle_up: ( previous job: 38.36, degradation: 101.64%)
whonix-gateway-17_socket: 7.85 :small_red_triangle_up: ( previous job: 7.54, degradation: 104.02%)
whonix-gateway-17_socket-root: 8.32 :small_red_triangle_up: ( previous job: 8.27, degradation: 100.58%)
whonix-workstation-17_exec: 7.85 🟢 ( previous job: 8.23, improvement: 95.32%)
whonix-workstation-17_exec-root: 57.69 :small_red_triangle_up: ( previous job: 52.56, degradation: 109.75%)
whonix-workstation-17_socket: 8.84 :small_red_triangle_up: ( previous job: 8.21, degradation: 107.66%)
whonix-workstation-17_socket-root: 9.00 :small_red_triangle_up: ( previous job: 8.20, degradation: 109.81%)
dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 453536.00 :small_red_triangle_up: ( previous job: 486352.00, degradation: 93.25%)
dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 4119.00 :green_circle: ( previous job: 3277.00, improvement: 125.69%)
dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11846.00 :green_circle: ( previous job: 10163.00, improvement: 116.56%)
dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 342.00 :green_circle: ( previous job: 282.00, improvement: 121.28%)
dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 469792.00 :small_red_triangle_up: ( previous job: 475329.00, degradation: 98.84%)
dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 174268.00 :green_circle: ( previous job: 95209.00, improvement: 183.04%)
dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 433474.00 :green_circle: ( previous job: 433474.00, improvement: 100.00%)
dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 206843.00 :green_circle: ( previous job: 164133.00, improvement: 126.02%)
dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 102343.00 :green_circle: ( previous job: 99808.00, improvement: 102.54%)
dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 9448.00 :green_circle: ( previous job: 7053.00, improvement: 133.96%)
dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 5080.00 :green_circle: ( previous job: 3868.00, improvement: 131.33%)
fedora-41-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 377593.00 :small_red_triangle_up: ( previous job: 396586.00, degradation: 95.21%)
fedora-41-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 229648.00 :green_circle: ( previous job: 99783.00, improvement: 230.15%)
fedora-41-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 120771.00 :green_circle: ( previous job: 44770.00, improvement: 269.76%)
fedora-41-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 90394.00 :green_circle: ( previous job: 86742.00, improvement: 104.21%)
fedora-41-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8742.00 :green_circle: ( previous job: 8623.00, improvement: 101.38%)
fedora-41-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1407.00 :green_circle: ( previous job: 1126.00, improvement: 124.96%)
fedora-41-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 266271.00 :green_circle: ( previous job: 116848.00, improvement: 227.88%)
fedora-41-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 111419.00 :green_circle: ( previous job: 41375.00, improvement: 269.29%)
fedora-41-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8148.00 :small_red_triangle_up: ( previous job: 8744.00, degradation: 93.18%)
fedora-41-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 390531.00 :small_red_triangle_up: ( previous job: 392725.00, degradation: 99.44%)
fedora-41-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 144771.00 :green_circle: ( previous job: 139933.00, improvement: 103.46%)
fedora-41-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 302619.00 :green_circle: ( previous job: 294875.00, improvement: 102.63%)
fedora-41-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 87349.00 :green_circle: ( previous job: 71108.00, improvement: 122.84%)
fedora-41-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 3925.00 :small_red_triangle_up: ( previous job: 3959.00, degradation: 99.14%)
fedora-41-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8914.00 :green_circle: ( previous job: 8408.00, improvement: 106.02%)

Create a bunch of files/dirs and check if expected files remained and unexpected did not. QubesOS/qubes-issues#1006

Logging from testvm1 already checked for test failure, but not for testvm2.

marmarek added the openqa-pending label Feb 13, 2025

This was referenced Feb 13, 2025

vmupdate: load plugins in sorted order QubesOS/qubes-core-admin-linux#183

Merged

Add filter for qubes stored on inaccessible storage pools QubesOS/qubes-manager#399

Closed

marmarek force-pushed the tests20250213 branch 2 times, most recently from cd5c644 to 26d0080 Compare February 16, 2025 10:54

This was referenced Feb 16, 2025

Fix updating template switcher progress bar QubesOS/qubes-manager#404

Merged

Do not close FDs 0, 1, or 2 QubesOS/qubes-core-qrexec#186

Closed

Improved Icons and UX QubesOS/qubes-desktop-linux-manager#242

Merged

marmarek force-pushed the tests20250213 branch 2 times, most recently from a210046 to 6d51fa9 Compare February 20, 2025 11:59

marmarek force-pushed the tests20250213 branch from 6d51fa9 to 531c821 Compare February 21, 2025 03:47

marmarek force-pushed the tests20250213 branch from 531c821 to e3dfe65 Compare February 27, 2025 09:13

marmarek added 2 commits February 27, 2025 11:04

tests: add test for custom-persist feature

ce5ebaa

Create a bunch of files/dirs and check if expected files remained and unexpected did not. QubesOS/qubes-issues#1006

tests: log .xsession-errors from testvm2 only on failure

d75733d

Logging from testvm1 already checked for test failure, but not for testvm2.

marmarek force-pushed the tests20250213 branch from e3dfe65 to d75733d Compare February 27, 2025 10:04

marmarek merged commit 258f153 into QubesOS:main Mar 3, 2025
3 of 5 checks passed

marmarek deleted the tests20250213 branch March 16, 2025 01:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add tests for few upcoming changes #654

Add tests for few upcoming changes #654

Uh oh!

marmarek commented Feb 13, 2025

Uh oh!

codecov bot commented Feb 13, 2025 •

edited

Loading

Uh oh!

qubesos-bot commented Feb 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Add tests for few upcoming changes #654

Add tests for few upcoming changes #654

Uh oh!

Conversation

marmarek commented Feb 13, 2025

Uh oh!

codecov bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

qubesos-bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

OpenQA test summary

New failures, excluding unstable

Failed tests

Fixed failures

Unstable tests

Performance Tests

Performance degradation:

Remaining performance tests:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Feb 13, 2025 •

edited

Loading

qubesos-bot commented Feb 13, 2025 •

edited

Loading