Xen-related performance problems

[DemiMarie]

How to file a helpful issue

Qubes OS release

R4.1

Brief summary

The Xen hypervisor has performance problems on certain compute-intensive workloads

Steps to reproduce

See @fepitre for details

Expected behavior

Same (or almost same) performance as bare hardware

Actual behavior

Less performance than bare hardware

[AlxHnr]

I can confirm that this problem is clearly noticeable. My Fedora AppVM with up to 8 GB of memory and all cores assigned to it, runs much slower than a native Fedora install on bare metal. Compiling code takes roughly twice a long. I first blamed the lack of hyperthreading, but passing smt=on sched-gran=core to xen makes no difference in my benchmarks (i7-3632qm):

Benchmark	Slowdown compared to native Fedora
sysbench cpu run	60%
sysbench memory run	1680%
Building some random C++ projects	58%

Watching 1080p@60fps YouTube videos on Qubes OS boils my laptop and doesn't feel like 60fps. Native Fedora handles this much better, despite using only CPU-based video decoders[1].

Startup latency of apps on Qubes OS is much worse. qvm-run personal echo "hello world" takes almost an entire second.

[1] I'm not 100% sure it's CPU-only. But according to htop it hits all my cores really hard, while still being smoother and much cooler than Qubes OS.

[DemiMarie]

Startup latency of apps on Qubes OS is much worse. qvm-run personal echo "hello world" takes almost an entire second.

This is definitely a problem and we’re working on it. It won’t ever be as fast as on bare hardware, but our goal is to go from qvm-run command to the VM spawning init in under 100ms. Runtime CPU performance should be within a few percent of bare silicon, so that it is not is definitely a bug.

I can confirm that this problem is clearly noticeable. My Fedora AppVM with up to 8 GB of memory and all cores assigned to it, runs much slower than a native Fedora install on bare metal. Compiling code takes roughly twice a long. I first blamed the lack of hyperthreading, but passing smt=on sched-gran=core to xen makes no difference in my benchmarks (i7-3632qm):

I suggest reverting this, as it is not security supported upstream. The fact that it did not help your benchmarks indicates that it is not likely to be the culprit.

Benchmark	Slowdown compared to native Fedora
sysbench cpu run	60%
sysbench memory run	1680%
Building some random C++ projects	58%

Yeah that’s not good. For clarification: if native Fedora takes time X to build C++ projects, does this mean Qubes OS takes (X / (1 - 0.58)) time? If you could post the raw benchmark data, that would be very helpful.

sysbench memory run is particularly concerning. Does turning off memory balancing for the qube help?

[crat0z]

I can't exactly reproduce this. More information on the workload is needed I think. In my test, I used Linpack Xtreme. On Qubes, I have SMT enabled, 12 vCPUs in my case and 12GB of RAM assigned to the VM (although linpack only uses about 2GB). The VM is PVH with memory balancing enabled. Everything else default for Xen and kernel-latest. Benchmark VM is the only VM running while testing. CPU frequency started at about 3.5GHz and went down to ~2.1GHz over duration of test. My result was 100 GFlops.

Then I started a fedora 36 iso. However, CPU frequency started at 3.9GHz and went down to about 2.4 or 2.5. My result was 112 GFlops.

Perhaps your CPU is not boosting correctly like mine does?

[AlxHnr]

sysbench memory run is particularly concerning. Does turning off memory balancing for the qube help?

I turned off memory balancing via Qubes Manager and assigned fixed 10 GB of memory to the Qube. No improvement 😕

For clarification: if native Fedora takes time X to build C++ projects, does this mean Qubes OS takes (X / (1 - 0.58)) time?

Yes! I don't have the raw benchmark data anymore, but I'm pretty sure it's the same problem that causes the sysbench deviations.

[AlxHnr]

The result of sysbench inside an AppVM heavily dependends on how many other AppVMs are running. Running only one single AppVM brings the results from sysbench cpu run pretty damn close to the values I get on native Fedora. I assume it's some form of scheduling issue. Running only one single AppVM also improves sysbench memory run results, but it's still way off compared to native Fedora.

I tried running the memory benchmark in dom0, where it is significantly faster. Here the results of sysbench memory run:

Environment	MiB/sec
Native Fedora	5290
dom0	4525
domU (only 1 AppVM)	401
domU (+7 other AppVMs)	267

I'm not sure if there is anything special about my system. My entire xen and kernel setup is pretty vanilla. Only deviation is qubes.enable_insecure_pv_passthrough because I don't have an IOMMU. Enabling/Disabling this flag makes no difference.

[DemiMarie]

@AlxHnr Can you try sysbench memory run in a PV VM (not dom0, sys-net, or sys-usb)?

[marmarek]

Is the domU a PVH (the default on qubes, if no PCI devices)? What CPU that is?

[AlxHnr]

Is the domU a PVH (the default on qubes, if no PCI devices)?

Yes. All my domU's are PVH (default), except sys-usb and sys-net.

@AlxHnr Can you try sysbench memory run in a PV VM (not dom0, sys-net, or sys-usb)?

VM Type	MiB/sec
PVH	243.81
HVM	216.34
PV	54.41

Giving the PV VM more cores and memory makes no difference. PV VMs are slow and laggy to the point of being unusable.

What CPU that is?

i7-3632qm. It supports VT-d, but my motherboard/BIOS/whatever does not.

I hope this problem is not specific to my setup. My goal here is to get to a point where others can reproduce these problems. I don't have much time and care less about temporary fixes for myself. I care more about achieving sane defaults that work for everybody.