Re: [Xen-devel] [RFC v2] ARM VM System Specification

[Christoffer Dall <christoffer.dall@xxxxxxxxxx>]

Agree with what Stefano says, and I strongly recommend you reconsider
your position on FDT vs. ACPI for ARM VMs, but it's completely up to
you of course :)

-Christoffer

On Wed, Jun 11, 2014 at 10:50:04AM +0100, Stefano Stabellini wrote:
> Although not in the ARM VM System Specification, Xen is certainly going
> to support a fast boot path without UEFI firmware. I guess KVM will too.
> You'll have to rely on hypervisor specific mechanisms to achieve it.
> 
> In fact I wouldn't worry about UEFI when you need ACPI, that is not in
> this specification. Moreover the only hypervisor supporting virtio on
> ARM is KVM as far as I know.
> 
> So it looks like you are already tying yourself to one hypervisor (KVM)
> and an out-of-spec standard (ACPI). The ARM VM System Specification is
> for people that intend to produce cross-hypervisor images.
> 
> 
> On Tue, 10 Jun 2014, Claudio Fontana wrote:
> > Hello all,
> > 
> > I just wanted to share with you guys how we are using virtualization on 
> > ARM64 over here for the OSv project.
> > I don't know if that's something that could be useful for your 
> > specification effort.
> > 
> > In OSv, creating and starting a VM to some level means starting an 
> > application.
> > That is, OSv should be a very thin bare bones guest server OS, which also 
> > acts as a kind of run-time library for an application to run.
> > All the devices are assumed to be virtualized and heavily relying on virtio.
> > 
> > Therefore we see a higher need for quick VM launch than it might be for 
> > other use cases.
> > 
> > One aspect of this is that we currently start executing the image directly 
> > (no UEFI involved on the guest),
> > and in some cases we might not need a full fledged file system at all,
> > as the communication can happen via virtio channels.
> > 
> > We do have a need for ACPI for discovery of information like gic addresses, 
> > timers, interrupts... (no interest on device trees, really), and of PCI-E.
> > 
> > By skipping steps like UEFI, grub, firmware load, etc we strive to keep our 
> > application launch time low.
> > Is this going to create problems for us in the future if you start 
> > requiring every VM to boot using those instead?
> > 
> > Thank you for your comments,
> > 
> > Claudio
> > 
> > 
> > On 28.03.2014 19:45, Christoffer Dall wrote:
> > > ARM VM System Specification
> > > ===========================
> > > 
> > > Goal
> > > ----
> > > The goal of this spec is to allow suitably-built OS images to run on
> > > all ARM virtualization solutions, such as KVM or Xen.
> > > 
> > > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > > they aim to be hypervisor agnostic.
> > > 
> > > Note that simply adhering to the SBSA [2] is not a valid approach, for
> > > example because the SBSA mandates EL2, which will not be available for
> > > VMs.  Further, this spec also covers the aarch32 execution mode, not
> > > covered in the SBSA.
> > > 
> > > 
> > > Image format
> > > ------------
> > > The image format, as presented to the VM, needs to be well-defined in
> > > order for prepared disk images to be bootable across various
> > > virtualization implementations.
> > > 
> > > The raw disk format as presented to the VM must be partitioned with a
> > > GUID Partition Table (GPT).  The bootable software must be placed in the
> > > EFI System Partition (ESP), using the UEFI removable media path, and
> > > must be an EFI application complying to the UEFI Specification 2.4
> > > Revision A [6].
> > > 
> > > The ESP partition's GPT entry's partition type GUID must be
> > > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> > > 
> > > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > > state as specified in Section 3.3 (3.3 (Boot Option Variables Default Boot
> > > Behavior) and 3.4.1.1 (Removable Media Boot Behavior) in [6].
> > > 
> > > This ensures that tools for both Xen and KVM can load a binary UEFI
> > > firmware which can read and boot the EFI application in the disk image.
> > > 
> > > A typical scenario will be GRUB2 packaged as an EFI application, which
> > > mounts the system boot partition and boots Linux.
> > > 
> > > 
> > > Virtual Firmware
> > > ----------------
> > > The VM system must be UEFI compliant in order to be able to boot the EFI
> > > application in the ESP.  It is recommended that this is achieved by
> > > loading a UEFI binary as the first software executed by the VM, which
> > > then executes the EFI application.  The UEFI implementation should be
> > > compliant with UEFI Specification 2.4 Revision A [6] or later.
> > > 
> > > This document strongly recommends that the VM implementation supports
> > > persistent environment storage for virtual firmware implementation in
> > > order to ensure probable use cases such as adding additional disk images
> > > to a VM or running installers to perform upgrades.
> > > 
> > > This document strongly recommends that VM implementations implement
> > > persistent variable storage for their UEFI implementation.  Persistent
> > > variable storage shall be a property of a VM instance, but shall not be
> > > stored as part of a portable disk image.  Portable disk images shall
> > > conform to the UEFI removable disk requirements from the UEFI spec and
> > > cannot rely on on a pre-configured UEFI environment.
> > > 
> > > The binary UEFI firmware implementation should not be distributed as
> > > part of the VM image, but is specific to the VM implementation.
> > > 
> > > 
> > > Hardware Description
> > > --------------------
> > > The VM system must be UEFI compliant and therefore the UEFI system table
> > > will provide a means to access hardware description data.
> > > 
> > > The VM implementation must provide through its UEFI implementation:
> > > 
> > >   a complete FDT which describes the entire VM system and will boot
> > >   mainline kernels driven by device tree alone
> > > 
> > > For more information about the arm and arm64 boot conventions, see
> > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > Linux kernel source tree.
> > > 
> > > For more information about UEFI booting, see [4] and [5].
> > > 
> > > 
> > > VM Platform
> > > -----------
> > > The specification does not mandate any specific memory map.  The guest
> > > OS must be able to enumerate all processing elements, devices, and
> > > memory through HW description data (FDT) or a bus-specific
> > > mechanism such as PCI.
> > > 
> > > If aarch64 physical CPUs implement support for the aarch32 execution
> > > state in EL1 and EL0 execution, it is recommended that the VM
> > > implementation supports booting the VM at EL1 in both aarch32 and
> > > aarch64 execution states.
> > > 
> > > The virtual hardware platform must provide a number of mandatory
> > > peripherals:
> > > 
> > >   Serial console:  The platform should provide a console,
> > >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > > 
> > >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> > >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> > >   removes this limitation.
> > > 
> > >   The ARM virtual timer and counter should be available to the VM as
> > >   per the ARM Generic Timers specification in the ARM ARM [1].
> > > 
> > > It is strongly recommended that the VM implementation provides a
> > > hotpluggable bus to support hotplug of at least block and network
> > > devices.  Suitable buses include a virtual PCIe bus and the Xen PV bus.
> > > 
> > > For the VM image to be compliant with this spec, the following applies
> > > for the guest OS in the VM image:
> > > 
> > >   The guest OS must include support for pl011 UART, virtio-console, and
> > >   the Xen PV console.
> > > 
> > >   The guest OS must include support for GICv2 and any available newer
> > >   version of the GIC architecture to maintain compatibility with older
> > >   VM implementations.
> > > 
> > >   It is strongly recommended to include support for all available
> > >   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
> > >   drivers in the guest OS kernel or initial ramdisk.
> > > 
> > > 
> > > Other common peripherals for block devices, networking, and more can
> > > (and typically will) be provided, but OS software written and compiled
> > > to run on ARM VMs cannot make any assumptions about which variations
> > > of these should exist or which implementation they use (e.g. VirtIO or
> > > Xen PV).  See "Hardware Description" above.
> > > 
> > > 
> > > Changes from previous versions of this RFC
> > > ------------------------------------------
> > > Changes v1-v2:
> > >  - Clearly specify that the guest must support the pl011,
> > >    virtio-console, and Xen PV console.  (Note that it was discussed to
> > >    mandate a pl011 during Linaro Connect Asia 2014, but that was under the
> > >    impression that the SBSA specification was an output-only no-irq
> > >    serial port, which is not the case.  The only two benefits from
> > >    mandating a specific serial type was to handle "console=ttyAMA0"
> > >    kernel command line parameters and earlyprintk; Grant Likely has
> > >    submitted patches to avoid the need for "console=" parameters, and
> > >    Rob Herring has submitted patches for paravirtualized earlyprintk
> > >    consoles.)
> > >  - Reference EFI specification for bootable paths.
> > >  - Remove discussion on ACPI boot methods and do not suggest ACPI
> > >    support in VMs.
> > >  - Add specification about UEFI persistent variable storage and
> > >    portability.
> > > 
> > > 
> > > References
> > > ----------
> > > [1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
> > > http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> > > 
> > > [2]: ARM Server Base System Architecture
> > > http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html
> > > 
> > > [3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
> > > http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> > > 
> > > [4]: http://www.secretlab.ca/archives/27
> > > 
> > > [5]: 
> > > https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt
> > > 
> > > [6]: UEFI Specification 2.4 Revision A
> > > http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf
> > > _______________________________________________
> > > kvmarm mailing list
> > > kvmarm@xxxxxxxxxxxxxxxxxxxxx
> > > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
> > > 
> > 
> > 
> > -- 
> > Claudio Fontana
> > Server Virtualization Architect
> > Huawei Technologies Duesseldorf GmbH
> > Riesstraße 25 - 80992 München
> > 
> > office: +49 89 158834 4135
> > mobile: +49 15253060158
> > 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel