add stress tests and benchmarks to Qubes automated tests

[adrelanos]

VMs and dom0 should be stress tested. Maybe using stress-ng or whatever else is suitable and/or other tools.

• https://wiki.ubuntu.com/Kernel/Reference/stress-ng
• https://packages.debian.org/search?keywords=stress-ng

Reason is that stress can lead to crashes and whatnot. Better noticed during testing than in stable release by user.

Also benchmarks would be useful. Phoronix test suite? That is because some changes might degrade performance. That is also better found out during testing. Maybe phoronix test suite? Others?

Would be useful to run benchmarks first. Then stress tests. Then again benchmarks. Then compare if benchmarks got sufficiently worse which is possible since there might be a memory leak or something.

References:

• https://openqa.qubes-os.org/
• https://www.qubes-os.org/doc/automated-tests/

[marmarek]

Our current automated tests already put a lot of stress on the system, like creating a lot of VMs, doing a lot of qrexec calls etc. And we do find issues that way, that are much harder to reproduce in normal usage (things happening at ~1% of VM start or such). So, I don't think we need specific improvements here right now. Also, things like (even not very severe) memory leaks are covered this way, because of so much activity that it pile up enough to cause out of memory errors (we specifically run tests with a limited RAM for this reason).

As for the benchmarks, that could be a good idea, but the current test setup is running in a nested virtualization (to have easy control over the environment), so the results will not be representative of a physical installation at all. Testing on a real hardware directly is hard to automate. We have some ideas - like using Intel AMT, but we're not there yet. Also, specific hardware may influence performance results a lot, so different test results will not be comparable...
Severe performance degradation is caught by timeouts. This is for example the case for HVM grub on R4.1 right now...

[DemiMarie]

Now that testing on real hardware is being done, I wonder if the Phoronix Benchmark Suite would be usable. At a minimum, it should be able to test our performance on various I/O and CPU bound workloads.