[Xen-devel] HVMlite ABI specification DRAFT B + implementation outline

[Roger Pau Monnà <roger.pau@xxxxxxxxxx>]

Hello,

I've Cced a bunch of people who have expressed interest in the HVMlite 
design/implementation, both from a Xen or OS point of view. If you 
would like to be removed, please say so and I will remove you in 
further iterations. The same applies if you want to be added to the Cc.

This is an initial draft on the HVMlite design and implementation. I've 
mixed certain aspects of the design with the implementation, because I 
think we are quite tied by the implementation possibilities in certain 
aspects, so not speaking about it would make the document incomplete. I 
might be wrong on that, so feel free to comment otherwise if you would 
prefer a different approach.

The document is still not complete. I'm of course not as knowledgeable 
as some people on the Cc, so please correct me if you think there are 
mistakes or simply impossible goals.

I think I've managed to integrate all the comments from DRAFT A. I still
haven't done a s/HVMlite/PVH/, but I plan to do so once the document is
finished and ready to go inside of the Xen tree.

Roger.

---
Xen HVMlite ABI
===============

Boot ABI
--------

Since the Xen entry point into the kernel can be different from the
native entry point, a `ELFNOTE` is used in order to tell the domain
builder how to load and jump into the kernel entry point:

    ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,          .long,  xen_start32)

The presence of the `XEN_ELFNOTE_PHYS32_ENTRY` note indicates that the
kernel supports the boot ABI described in this document.

The domain builder shall load the kernel into the guest memory space and
jump into the entry point defined at `XEN_ELFNOTE_PHYS32_ENTRY` with the
following machine state:

 * `ebx`: contains the physical memory address where the loader has placed
   the boot start info structure.

 * `cr0`: bit 0 (PE) must be set. All the other writeable bits are cleared.

 * `cr4`: all bits are cleared.

 * `cs`: must be a 32-bit read/execute code segment with a base of â0â
   and a limit of â0xFFFFFFFFâ. The selector value is unspecified.

 * `ds`, `es`: must be a 32-bit read/write data segment with a base of
   â0â and a limit of â0xFFFFFFFFâ. The selector values are all unspecified.

 * `tr`: must be a 32-bit TSS (active) with a base of '0' and a limit of '0x67'.

 * `eflags`: all user settable bits are clear.

All other processor registers and flag bits are unspecified. The OS is in
charge of setting up it's own stack, GDT and IDT.

The format of the boot start info structure is the following (pointed to
be %ebx):

NOTE: nothing will be loaded at physical address 0, so a 0 value in any of the
address fields should be treated as not present.

 0 +----------------+
   | magic          | Contains the magic value 0x336ec578
   |                | ("xEn3" with the 0x80 bit of the "E" set).
 4 +----------------+
   | flags          | SIF_xxx flags.
 8 +----------------+
   | cmdline_paddr  | Physical address of the command line,
   |                | a zero-terminated ASCII string.
12 +----------------+
   | nr_modules     | Number of modules passed to the kernel.
16 +----------------+
   | modlist_paddr  | Physical address of an array of modules
   |                | (layout of the structure below).
20 +----------------+

The layout of each entry in the module structure is the following:

 0 +----------------+
   | paddr          | Physical address of the module.
 4 +----------------+
   | size           | Size of the module in bytes.
 8 +----------------+
   | cmdline_paddr  | Physical address of the command line,
   |                | a zero-terminated ASCII string.
12 +----------------+
   | reserved       |
16 +----------------+

Other relevant information needed in order to boot a guest kernel
(console page address, xenstore event channel...) can be obtained
using HVMPARAMS, just like it's done on HVM guests.

The setup of the hypercall page is also performed in the same way
as HVM guests, using the hypervisor cpuid leaves and msr ranges.

Hardware description
--------------------

Hardware description can come from two different sources, just like on (PV)HVM
guests.

Description of PV devices will always come from xenbus, and in fact
xenbus is the only hardware description that is guaranteed to always be
provided to HVMlite guests.

Description of physical hardware devices will always come from ACPI, in the
absence of any physical hardware device no ACPI tables will be provided. The
presence of ACPI tables can be detected by finding the RSDP, just like on
bare metal.

Non-PV devices exposed to the guest
-----------------------------------

The initial idea was to simply don't provide any emulated devices to a HVMlite
guest as the default option. We have however identified certain situations
where emulated devices could be interesting, both from a performance and
ease of implementation point of view. The following list tries to encompass
the different identified scenarios:

 * 1. HVMlite with no emulated devices at all
   ------------------------------------------
   This is the current implementation inside of Xen, everything is disabled
   by default and the guest has access to the PV devices only. This is of
   course the most secure design because it has the smaller surface of attack.

 * 2. HVMlite with (or capable to) PCI-passthrough
   -----------------------------------------------
   The current model of PCI-passthrought in PV guests is complex and requires
   heavy modifications to the guest OS. Going forward we would like to remove
   this limitation, by providing an interface that's the same as found on bare
   metal. In order to do this, at least an emulated local APIC should be
   provided to guests, together with the access to a PCI-Root complex.
   As said in the 'Hardware description' section above, this will also require
   ACPI. So this proposed scenario will require the following elements that are
   not present in the minimal (or default) HVMlite implementation: ACPI, local
   APIC, IO APIC (optional) and PCI-Root complex.

 * 3. HVMlite hardware domain
   --------------------------
   The aim is that a HVMlite hardware domain is going to work exactly like a
   HVMlite domain with passed-through devices. This means that the domain will
   need access to the same set of emulated devices, and that some ACPI tables
   must be fixed in order to reflect the reality of the container the hardware
   domain is running on. The ACPI section contains more detailed information
   about which/how these tables are going to be fixed.

   Note that in this scenario the hardware domain will *always* have a local
   APIC and IO APIC, and that the usage of PHYSDEV operations and PIRQ event
   channels is going to be removed in favour of the bare metal mechanisms.

The default model for HVMlite guests is going to be to provide a local APIC
together with a minimal set of ACPI tables that accurately match the reality of
the container is guest is running on. An administrator should be able to change
the default setting using the following tunables that are part of the xl
toolstack:

 * lapic: default to true. Indicates whether a local APIC is provided.
 * ioapic: default to false. Indicates whether an IO APIC is provided
   (requires lapic set to true).
 * acpi: default to true. Indicates whether ACPI tables are provided.

ACPI
----

ACPI tables will be provided to the hardware domain or to unprivileged
domains. In the case of unprivileged guests ACPI tables are going to be
created by the toolstack and will only contain the set of devices available
to the guest, which will at least be the following: local APIC and
optionally an IO APIC and passed-through device(s). In order to provide this
information from ACPI the following tables are needed as a minimum: RSDT,
FADT, MADT and DSDT. If an administrator decides to not provide a local APIC,
the MADT table is not going to be provided to the guest OS.

The ACPI_FADT_NO_CMOS_RTC flag in the FADT boot_flags field is going to be used
to signal guests that there's no RTC device (the Xen PV wall clock should be
used instead). It is likely that this flag is not going to be set for the
hardware domain, since it should have access to the RTC present in the host
(if there's one). The ACPI_FADT_NO_VGA is also very likely to be set in the
same boot_flags FADT field for DomUs in order to signal that there's no VGA
adapter present.

Finally the ACPI_FADT_HW_REDUCED is going to be set in the FADT flags field
in order to signal that there are no legacy devices: i8259 PIC or i8254 PIT.
There's no intention to enable these devices, so it is expected that the
hardware-reduced FADT flag is always going to be set.

In the case of the hardware domain, Xen has traditionally passed-through the
native ACPI tables to the guest. This is something that of course we still
want to do, but in the case of HVMlite Xen will have to make sure that
the data passed in the ACPI tables to the hardware domain contain the accurate
hardware description. This means that at least certain tables will have to
be modified/mangled before being presented to the guest:

 * MADT: the number of local APIC entries need to be fixed to match the number
         of vCPUs available to the guest. The address of the IO APIC(s) also
         need to be fixed in order to match the emulated ones that we are going
         to provide.

 * DSDT: certain devices reported in the DSDT may not be available to the guest,
         but since the DSDT is a run-time generated table we cannot fix it. In
         order to cope with this, a STAO table will be provided that should
         be able to signal which devices are not available to the hardware
         domain. This is in line with the Xen/ACPI implementation for ARM.

 * MPST, PMTT, SBTT, SRAT and SLIT: won't be initially presented to the guest,
   until we get our act together on the vNUMA stuff.

NB: there are corner cases that I'm not sure how to solve properly. Currently
the hardware domain has some 'hacks' regarding ACPI and Xen. At least I'm aware
of the following:

 * 1. Reporting CPU PM info back to Xen: this comes from the DSDT table, and
   since this table is only available to the hardware domain it has to report
   the PM info back to Xen so that Xen can perform proper PM.
 * 2. Doing proper shutdown (S5) requires the usage of a hypercall, which is
   mixed with native ACPICA code in most OSes. This is awkward and requires
   the usage of hooks into ACPICA which we have not yet managed to upstream.
 * 3. Reporting the PCI devices it finds to the hypervisor: this is not very
   intrusive in general, so I'm not that pushed to remove it. It's generally
   easy in any OS to add some kind of hook that's executed every time a PCI
   device is discovered.
 * 4. Report PCI memory-mapped configuration areas to Xen: my opinion regarding
   this one is the same as (3), it's not really intrusive so I'm not very
   pushed to remove it.

I would ideally like to get rid of (2) in the list above, since I'm quite sure
we are never going to be able to merge the needed hooks into ACPICA. AFAICT Xen
should be able to parse the FADT table and find the address of the PM1a and
PM1b control registers and trap on access.

(1) is also quite nasty, but I don't see any possible way to get rid of it.

AP startup
----------

AP startup is performed using hypercalls. The following VCPU operations
are used in order to bring up secondary vCPUs:

 * VCPUOP_initialise is used to set the initial state of the vCPU. The
   argument passed to the hypercall must be of the type vcpu_hvm_context.
   See public/hvm/hvm_vcpu.h for the layout of the structure. Note that
   this hypercall allows starting the vCPU in several modes (16/32/64bits),
   regardless of the mode the BSP is currently running on.

 * VCPUOP_up is used to launch the vCPU once the initial state has been
   set using VCPUOP_initialise.

 * VCPUOP_down is used to bring down a vCPU.

 * VCPUOP_is_up is used to scan the number of available vCPUs.

Additionally, if a local APIC is available CPU bringup can also be performed
using the hardware native AP startup sequence (IPIs). In this case the
hypercall interface will still be provided, as a faster and more convenient
way of starting APs.

MMIO mapping
------------

For DomUs without any device passed-through no direct MMIO mappings will be
present in the physical memory map presented to the guest. For DomUs with
devices passed-though the toolstack will create direct MMIO mappings as
part of the domain build process, and thus no action will be required
from the DomU.

For the hardware domain initial direct MMIO mappings will be set for the
following regions:

NOTE: ranges are defined using memory addresses, not pages.

 * [0x0, 0xFFFFF]: the low 1MiB will be mapped into the physical guest
   memory map at the same position.

 * [0xF00000, 0xFFFFFF]: the ISA memory hole will be mapped 1:1 into the
   guest physical memory.

 * ACPI memory areas: regions with type E820_ACPI or E820_NVS will be mapped
   1:1 to the guest physical memory map. There are going to be exceptions if
   Xen has to modify the tables before presenting them to the guest.

 * PCI Express MMCFG: if Xen is able to identify any of these regions at boot
   time they will also be made available to the guest at the same position
   in it's physical memory map. It is possible that Xen will trap accesses to
   those regions, but a guest should be able to use the native configuration
   mechanism in order to interact with this configuration space. If the
   hardware domain reports the presence of any of those regions using the
   PHYSDEVOP_pci_mmcfg_reserved hypercall Xen will also all guest access to
   them.

 * PCI BARs: it's not possible for Xen to know the position of the BARs of
   the PCI devices without hardware domain interaction. In order to have
   the BARs of PCI devices properly mapped the hardware domain needs to
   call the PHYSDEVOP_pci_device_add hypercall, that will take care of setting
   up the BARs in the guest physical memory map using 1:1 MMIO mappings. This
   procedure will be transparent from guest point of view, and upon returning
   from the hypercall mappings must be already established.

Xen HVMlite implementation plan
===============================

This is of course not part of the ABI, but I guess it makes sense to add it
here in order to be able to more easily split the tasks required in order to
make the proposed implementation above a reality. I've tried to split
the tasks into smaller sub-tasks when possible.

DomU
----

 1. Initial HVMlite implementation based on a HVM guest: no emulated devices
    will be provided, interface exactly the same as a PVH guest except for the
    boot ABI.

 2. Provide ACPI tables to HVMlite guests: the initial set of provided tables
    will be: RSDT, FADT, MADT (iff local APIC is enabled).

 3. Enable the local APIC by default for HVMlite guests.

 4. Provide options to xl/libxl in order to allow admins to select the
    presence of a local APIC and IO APIC to HVMlite guests.

 5. Implement an emulated PCI Root Complex inside of Xen.

 6. Provide a DSDT table to HVMlite guests in order to signal the presence
    of PCI-passthrough devices.

IMHO, we should focus on (2) and (3) at the moment, and (4) is quite trivial
once those two are in place. (5) and (6) should be implemented once HVMlite
hardware domains are functional.

When implementing (2) it would be good to place the ACPI related code in a
place that's accessible from libxl, hvmloader and Xen itself, in order
to reduce code duplication. hvmloader already has most if not all the required
code in order to build the tables that are needed for HVMlite DomU.

Dom0
----

 1. Add a new Dom0 builder specific for HVM-like domains. PV domains have
    different requirements and sharing the same Dom0 domain builder only makes
    the code for both cases much harder to read and disentangle.

 2. Implement the code required in order to mangle/modify the ACPI tables
    provided to Dom0, so that it matches the reality of the container provided
    to Dom0.

 3. Allow HVM Dom0 to use PHYSDEVOP_pci_mmcfg_reserved and
    PHYSDEVOP_pci_device_add and make sure these hypercalls add the proper
    MMIO mappings.

 4. Do the necessary wiring so that interrupts from physical devices are
    received by Dom0 using the emulated interrupt controllers (local and IO
    APICs).

This plan is not as detailed as the DomU one, since the Dom0 work is not as
advanced as the DomU work, and is also tied to the DomU implementation. I
have an initial implementation for (1), and will continue working on it.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel