[Xen-devel] [PATCH v2 00/10] Introduce AMD SVM AVIC

[Janakarajan Natarajan <Janakarajan.Natarajan@xxxxxxx>]

OVERVIEW
========
This patchset is the first of a two-part patch series to introduce
the AMD Advanced Virtual Interrupt Controller (AVIC) support.

The AVIC hardware virtualizes local APIC registers of each vCPU via
the virtual APIC (vAPIC) backing page. This allows the guest to access
certain APIC registers without the need for emulation of hardware
behaviour in the hypervisor. More information about AVIC can be found in

* AMD64 Architecture Programmers Manual Volume 2 - System Programming
  https://support.amd.com/TechDocs/24593.pdf

For SVM AVIC, we extend the existing SVM driver to:
* Check CPUID to detect AVIC support in the processor.
* Program new fields in VMCB to enable AVIC.
* Introduce new AVIC data structures and add code to manage them.
* Handle two new AVIC #VMEXITs.
* Add new interrupt injection code using vAPIC backing page
  instead of the existing V_IRQ, V_INTR_PRIO, V_INTR_VECTOR and
  V_IGN_TPR fields.

This patchset does not enable AVIC by default since it does not
yet support nested VMs. Until then, users can enable SVM AVIC by
specifying Xen parameter svm=avic.

Later, in Part 2, we will introduce the IOMMU AVIC support, which
provides speed-up for the PCI device passthrough use case by allowing
the IOMMU hardware to inject interrupts directly into the guest via
the vAPIC backing page.

OVERALL PERFORMANCE
===================
AVIC is available on AMD Family 15h models 6Xh (Carrizo) processors
and newer. An AMD Family 17h Epyc processor is used to collect the
performance data shown below.

Generally, SVM AVIC alone (w/o IOMMU AVIC) should provide overall
speed-up for HVM guest since it does not require #VMEXIT into the
hypervisor to emulate certain guest accesses to local APIC registers.

It should also improve performance when the hypervisor wants to
inject interrupts into a running vcpu. It can do this by setting the 
corresponding IRR bit in the vAPIC backing page and triggering the
AVIC_DOORBELL.

For sending IPI interrupts between running vcpus in a Linux guest,
Xen defaults to using event channels. However, in case of
non-paravirtualized guests, AVIC can also provide performance
improvements for sending IPIs.

BENCHMARK: HACKBENCH
====================
For measuring IPI performance used for scheduling workload, some
performance numbers are collected using hackbench.

    # hackbench -p -l 100000
    Running in process mode with 10 groups using 40 file descriptors
    each (== 400 tasks)
    Each sender will pass 100000 messages of 100 bytes
                          |      3 vcpus (sec)        |
    ---------------------------------------------------
    No AVIC w/ xen_nopv   |           517             |
    AVIC w/ xen_nopv      |           238             |
    No AVIC w/o xen_nopv  |           141             |
    AVIC w/o xen_nopv     |           135             |

Each benchmark test was averaged over 10 runs.

CURRENT UNTESTED USE_CASES
==========================
* Nested VM

Any feedback and comments are very much appreciated.

v1->v2:
* Remove use of MFN 0 as dummy page for AVIC as suggested by
  Jan and Andrew.
* Rename vlapic_read_aligned().
* Moved AVIC stats dump from new keyhandler to IRQ keyhandler.
* Changed avic_logical_id_entry from struct to union.
* Updated cover letter and patch descriptions.
* Miscellaneous fixes based on feedback.

Janakarajan Natarajan (2):
  x86/HVM: Rename vlapic_read_aligned() to vlapic_reg_read()
  x86/HVM: Make vlapic_reg_read/write() non-static

Suravee Suthikulpanit (8):
  x86/SVM: Modify VMCB fields to add AVIC support
  x86/HVM/SVM: Add AVIC initialization code
  x86/SVM: Add AVIC vmexit handlers
  x86/SVM: Add vcpu scheduling support for AVIC
  x86/SVM: Add interrupt management code via AVIC
  x86/HVM: Hook up miscellaneous AVIC functions
  x86/SVM: Introduce svm command line option
  x86/SVM: Append AMD AVIC related data to IRQ keyhandler 'i'

 docs/misc/xen-command-line.markdown |  16 +
 xen/arch/x86/hvm/svm/Makefile       |   1 +
 xen/arch/x86/hvm/svm/avic.c         | 614 ++++++++++++++++++++++++++++++++++++
 xen/arch/x86/hvm/svm/intr.c         |   4 +
 xen/arch/x86/hvm/svm/svm.c          |  76 ++++-
 xen/arch/x86/hvm/svm/vmcb.c         |   3 +
 xen/arch/x86/hvm/vlapic.c           |  28 +-
 xen/arch/x86/hvm/vmx/vmx.c          |   8 +-
 xen/arch/x86/irq.c                  |   2 +
 xen/include/asm-x86/hvm/hvm.h       |   4 +-
 xen/include/asm-x86/hvm/svm/avic.h  |  46 +++
 xen/include/asm-x86/hvm/svm/svm.h   |   2 +
 xen/include/asm-x86/hvm/svm/vmcb.h  |  53 +++-
 xen/include/asm-x86/hvm/vlapic.h    |   4 +
 xen/include/asm-x86/msr-index.h     |   1 +
 15 files changed, 828 insertions(+), 34 deletions(-)
 create mode 100644 xen/arch/x86/hvm/svm/avic.c
 create mode 100644 xen/include/asm-x86/hvm/svm/avic.h

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel