[Xen-users] [Xen VGA Passthrough] DomU performance inversely proportional to VM memory size when VGA passthrough is enabled

[Andrej Komelj <akomelj@xxxxxxxxx>]

Hi,

I've noticed a peculiar behavior of Xen 4.4.x when enabling VGA
passthrough to DomU domain on my desktop computer - performance of the
DomU is inversely proportional to the size of the domain's memory. The
more memory the domain has, the slower the overall performance of the
domain gets. Degradation is considerable and noticeable, not only
theoretical or in a range of several percents.

The interesting thing though is: If VGA passthrough is disabled, the
domain performance remains constant, regardless of the size of memory
allocated to it.

Needless to say, performance without VGA passthrough is similar to
bare-metal system but performance with VGA passthrough starts
noticeably degrading when I assign more that 512 MB of memory to DomU.
With 8G assigned, it is practically unusable.

I've been using a simple gzip/gunzip test to confirm my findings: on
bare-metal system, it takes roughly 1 second to compress a big file,
on VGA-passthrough-enabled DomU it takes 5 seconds with 4 GB RAM and
15 seconds with 8 GB RAM. I ran additional benchmarking tools (cpu,
memory, graphics, overall) and they all confirmed that my system is
considerably slower when using passthrough and large amount of memory.

Performance drop is noticeable even when using purely text-mode DomU
with graphics drivers on the blacklist and disabled framebuffer in
DomU linux kernel.

Computer HW specs (cpuinfo & dmidecode in attachment) - HP Elite 7500 system:
- 8 core Intel i7-3770 CPU @ 3.40GHz
- Intel Z75 Express chipset
- integrated graphics Intel HD 4000
- 12 GB RAM
- VT-d supported and tested (been successfully running VGA passthrough
on 4.3.x with Windows 8.1/XP)

Dom0s:
- Arch Linux (Xen 4.4.0 & Xen 4.4.1 from AUR)
- Linux Mint 16 (Ubuntu 14.04 based - Xen 4.4.0)
- Ubuntu Linux 14.04 (Xen 4.4.0)

DomUs (PV-on-HVM):
- Linux Mint 16 (Ubuntu 14.04 based) - kernels 3.13.0, 3.14.1, 3.16.2, 3.17.0
- Arch Linux latest
- Debian Jessie (testing) - I've already wiped the logical volume so I
don't know what version of kernel was running

I've been playing with pretty much every setting in domain
configuration file to no avail:
- primary/secondary passthrough (gfx_passthru=0/1)
- shadow memory size
- xen_platform_pci
- hap, apic, acpi, ...
- pci_permissive, pci_power_mgmt, pci_msitranslate
- ...

The only thing that has any impact on the performance is:
- memory size - the smaller the better :-(
- PCI passthrough of graphics adapter - disabled (OK) or enabled (huge
performance hit)

Does anybody know what else I could try/check? The only similar post
on Xen-users I was able to find is:
http://xen.1045712.n5.nabble.com/Performance-degradation-with-pci-passthrough-amp-xen-4-4-td5724369.html

I've attached various debug/output files if anyone wants to deal with them:
- dmidecode output
- /proc/cpuinfo
- lspci -vv
- dom0 xen dmesg log
- dom0 kernel log
- domU kernel log
- domainU configuration file

Best regards,
Andrej

Attachment: cpuinfo.txt
Description: Text document

Attachment: dmidecode.txt
Description: Text document

Attachment: dom0_dmesg.txt
Description: Text document

Attachment: dom0_xen_dmesg.txt
Description: Text document

Attachment: domainU.txt
Description: Text document

Attachment: domU_dmesg.txt
Description: Text document

Attachment: lspci_vv.txt
Description: Text document

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users