Reduce unnecessary reboots in tests

[philipanda]

The problem you're addressing (if any)

Tests that are performed on multiple OSes generally perform a lot of unnecesary reboots. Instead of running all little checks on one OS and the moving to another, we are doing a little check on every supporteed OS before moving to another one.

Example: the DSP test suite. The DSP001, DSP002, DSP003 test on one OS can be done in one go. The total execution time would be very close to executing just one of them, because most of the time is wasted on rebooting the platform and the OS.

Describe the solution you'd like

The tests should not perform reboots if it is not necessary.
How to solve this is not straightforward.

Where is the value to a user, and who might that user be?

A significant decrease of tests run time, especially on regressions.

Describe alternatives you've considered

Ideas

1. The order of executing the tests has to be changed

• Just swap the test cases in test suites, so that all cases on one OS are next to each other. RF executes cases in
  the order they appear in the .robot file
• separate tests on different OSes to their own files
  • the test file name when running robot can include a wildcard character, which might make this a good solution
  • scripts/run.sh dasharo-compatibility/network*.robot works and runs network-boot-utilities, network-boot and
    network-speed in one run. Something like scrupts/run.sh dasharo-compatibility/*_201.robot could be used to run
    all tests on 201.
  • There's also the -t flag for robot which allows filtering by test ID
    • scripts/regression.sh -- -t "*.201*" -t "*.202*" -t "*.301" would run all the tests containging .201, .202 or .301 in
      their names allowing for selecting the environments manually
  • the test setup could also just contain a Skip if the OS, for which tests this file contains, is not set as supported in
    config variables

2. After the order is altered, we need to modify the tests to not reboot if unnecessary

• Keeping the state of the platform and making the Power On keyword work more like
  Ensure Power On.
  • many tests use Power On as a way to reboot the device. Such cases would need to be fixed. Maybe a
    separate keyword Power Off could be added and used to clearly state when the platform is actually
    rebooted, and when the Power On is used only to ensure it is On.
• If the tests on different OSes would be separated to their own files, they could be simplified, so that Powering
  On the platform only happens during test suite setup, not on test case basis.
• If tests on different OSes would stay in one file, a dummy test case could be added before OS change, that
  would perform the reboot and boot the OS used in the following tests.
  • definitely not a pretty solution, could be easy to implement, not so much to use later

3. What about running tests on all supported OSes

• Seperating the tests into files for every OS might require the tester to always choose them manually. Right now the
  supported OSes are stored in platform-configs.

[krystian-hebel]

One way of making them faster would be to gather all the required info in suite setup and just parse it in dedicated test, like I did in #559. This of course depends on exact tests, SMBIOS was easy to do in one go, but others may require multiple boots anyway.

separate tests on different OSes to their own files

This would probably be required if the solution I proposed were to be used.

the test setup could also just contain a Skip if the OS, for which tests this file contains, is not set as supported in
config variables

The problem with that approach would be that with each change to scope of tests in the release, we would have to update platform config. I like the -t *.201 idea better, or we can use tags for this.

[philipanda]

One way of making them faster would be to gather all the required info in suite setup and just parse it in dedicated test, like I did in #559. This of course depends on exact tests, SMBIOS was easy to do in one go, but others may require multiple boots anyway.

A way to achieve this would be to always run tests on only one environment, like one operating system. Switching between them automatically could be achieved by extending the scripts/run.sh script so that it would run the tests multiple times and change the ID of the OS that will be tested. The results could then be fused together using rebot. This way pretty much every test suite could be written in the way proposed by you in #559

[philipanda]

I'd like to emphasize how much of a difference reducing the wasteful reboots makes.
Here are some screenshots from Krystian's PR he linked in his comment:

Before:

After:

[philipanda]

This applies to the entire repository and every test we have.

It is not too burdensome for laptops because during Fedora testing some checks were added that remove unnecessary reboots when changing OSes.
It mainly affects other platforms.

Changes should rather be made test suite by test suite, not everywhere at once. It’s easy to divide the work by suites.

In my opinion, the best solution for test ordering would be to split tests into files depending on the OS, for example:

dasharo-compatibility/
 └── audio-subsystem/
     ├── 201_Ubuntu.robot
     ├── 202_Fedora.robot
     └── 301_Windows.robot

It allows for ordering the test files using prefixes: https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#execution-order

A potential problem could be tests like CapsuleUpdate, where part of the testing is done in firmware and the rest in the OS, e.g. CUP150.001 Capsule Update, followed by CUP170.201 Verifying UUID (Ubuntu), which must be executed after the first one. In such cases the test could just not be split into files per OS, or more than one file per OS could be made to allow for an order of execution we'd like. Example:

dasharo-compatibility/
 └── audio-subsystem/
     ├── 01_201_Ubuntu_part1.robot
     ├── 01_202_Fedora_part1.robot
     ├── 01_301_Windows_part1.robot
     ├── 02_some_common_setup.robot
     ├── 03_201_Ubuntu_part2.robot
     ├── 03_202_Fedora_part2.robot
     └── 03_301_Windows_part2.robot

or if there is no common setup, but we'd like to separate the tests and reduce reboots, we could still do something like:

dasharo-compatibility/
 └── audio-subsystem/
     ├── 201_Ubuntu_part1.robot
     ├── 201_Ubuntu_part2.robot
     ├── 202_Fedora_part1.robot
     ├── 202_Fedora_part2.robot
     ├── 301_Windows_part1.robot
     └── 301_Windows_part2.robot

These changes would work well with the concurrent testing modifications from #1130 where joining multiple test suites into one file caused the file to become too large to efficiently manage.

@wiktormowinski @matmacieje @macpijan what do you think about separating test suites into multiple files depending on the OS or other criteria?
I think I'm all for it as it does not interfere much with how we're running tests right now. -t parameter works the same as it filters test case names, running a whole module like dasharo-compatibility would work the same too.

[macpijan]

what do you think about separating test suites into multiple files depending on the OS or other criteria?

I am too not certain if just files or files/directories. But in general - strong yes from me for some form of separation of test cases for different OSes.

[wiktormowinski]

@philipanda I am all on board with the idea.
Furthermore, since we are on the point of overhauling osfv with #1130 there is no better time to make such changes like now.