dasharo-performance-concurrent: Add proof of concept for concurrent tests

[philipanda]

It's a proof of concept for running test cases concurrently, with common
execution steps. This approach, while might look complicated and
unconventional, can allow for devastating improvements in test suite
execution times.

By using the supported test cases dict, CONCURRENT test cases
the actual test scope can be determined, then all the tests can be
performed concurrently in any way we find reasonable.

Then the actual test cases can access suite variables created
by the CONCURRENT test cases to determine whether they PASS or not.

TODO before merging:

• remove debug config overrides
• update test_cases.json if needed

[philipanda]

Added the CPF001.201 test case to show the proof of concept for running tests using this scheme
measurements_log.html
No parallelization for now

[philipanda]

Added the CPF005.201 and CPT001.201 tests which measure the temperature and frequencies over some time in intervals. In this example they will effectively run in parallel while sharing single device and serial connection, which wouldn't be possible with simple pabot call

measurements_log.html

[philipanda]

Added the option for scripts/lib/robot.sh to add the test cases that would be run according to the given -t parameters, and the list of tags given with the -i parameter.

They are available as ${TEST_TAGS} and ${TEST_CASES} in RF allowing to skip the CANARY pseudo test cases and simplify this approach a lot.

Tested using a temp test case:

*** Settings ***

Default Tags    automated


*** Test Cases ***
TEST001.001 Test scanning args
    ${TEST_CASES}=    Evaluate    list(${TEST_CASES})
    ${TEST_TAGS}=    Evaluate    list(${TEST_TAGS})

    Log To Console    \ntags: ${TEST_TAGS}
    Log To Console    \ncases: ${TEST_CASES}
    FOR    ${case}    IN    @{TEST_CASES}
        Log To Console    \tcase: ${case}
    END

CPT001.201
    Pass Execution    1
CPF005.201
    Pass Execution    2
ABC000.000 automated test
    Pass Execution    3
ABC001.000 Not-automated case
    [Tags]    semiauto
    Pass Execution    4

./scripts/run.sh util/test.robot -- -t "*CPT001.201*" -t "*args" -t "ABC*" -i automated

==============================================================================
Test                                                                          
==============================================================================
TEST001.001 Test scanning args                                        ..
tags: ['automated']
.
cases: ['TEST001.001', 'CPT001.201', 'ABC000.000', 'ABC001.000']
.       case: TEST001.001
       case: CPT001.201
       case: ABC000.000
       case: ABC001.000
TEST001.001 Test scanning args                                        | PASS |
------------------------------------------------------------------------------
CPT001.201                                                            | PASS |
1
------------------------------------------------------------------------------
ABC000.000 automated test                                             | PASS |
3
------------------------------------------------------------------------------
Test                                                                  | PASS |
3 tests, 3 passed, 0 failed
==============================================================================

logs for a good measure:
test_log.html

The next step is to integrate this into the proof of concept, which should become much simpler

[philipanda]

With 6125b76 there is no real overhead for running tests in parallel.
There is only one quirk to remember:

• parallel test skip conditions must be configured in suite preparation, not in the test case

Other than that, there are a couple Keyword tools to check what tests should be run and decide which steps to perform in parallel gathering tasks. All the keywords and variables related to managing this were moved to a library file.

I don't think there is a smart way of simplifying places like the _PARALLEL_Background Measurements (no load) (Ubuntu) case much more. It must check and decide what operations to perform depending on the supported tests.

I'd like to ephasize, that this approach is only needed for tests which require synchronization.
The only real reason why the CPT and CPF test cases require it is because they can be run via serial.

SSH test cases, when they don't reboot, or in any other way affect the shared resource (device) can simply be run in parallel using rebot. Such test cases should carefully be separated, so that they can be run intentionally when the tester knows its's a good moment.

I am not sure how to keep a single definition of a test case and somehow treat it differently when it comes to parallelisation if it's performed via SSH. SSH test cases don't need the constructions suggested in this PR if they don't affect the device state, so they shouldn't be joined into large suites like the measurements.robot here. If one would intent to run them in parallel via serial, that would be inevitable though.

This approach would be also needed if there are a couple test cases in a suite, where at least one depends on any other, or if any single one reboots or in other way affects the device.

[philipanda]

Added the tests under load (reusing all gathering code) and stability tests (extending existing gathering code to run it in parallel with temp&freq measurements) to show how much effort does making parallel test cases actually take
measurements_log.html

[philipanda]

Added checking the power source: NA/Battery/AC/USB-PD
measurements_log.html

[wiktormowinski]

@philipanda fix pre-commit and rebase