[Minios-devel] Summary for discussions of ARM Unikraft supports

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Simon,

I have done a summary for the discussions we had done these days.
Here is the new version, if I missed anything please remind me : )

Regards,
Wei Chen

Proposal:

My first target is to enable Unikraft on ARM64+Kvm, so this proposal would
focus on ARM64+Kvm. But the goal of ARM support is to enable Unikraft on
ARM32/ARM64 based hypervisors (ARM32/64 Kvm, ARM64 Xen and etc). So we have
to consider to keep multi-arch/multi-plat framework.

Because the QEMU is the de-facto standard deployment for KVM, we should start
with this. In order to reach out to as many KVM users as we can and make the
usage easy for them. We would lower the priority for other platforms, e.g.,
kvm-tool, or ukvm.

1. First target platform
   AArch64 QEMU+KVM, Machine is "virt" with 64-bit ARMv8 CPU:
   qemu-system-aarch64 -machine virt -cpu cortex-a53/cortex-a57

2. Modify the folders for multi-architectures
    2.1. Restructure the unikraft/arch to:
             unikraft----arch----arm----arm32
                         |      |-----arm64
                         |
                             |-----x86----x86
                                    |-----x86_64

         Above folders contains architecture specified Makefile, Config,
         Compiler flags and some code. Any code wants to be placed in these
         folders must follow:
         1) This code is exposed to the user with an interface in
            include/uk/arch/*
         2) This code works with all platforms (including linuxu which is
            special). For instance, you should not add code that uses
            privileged instruction that could not be executed in Linux
            userspace. If there is a different implementation needed, it is
            a hint that this functionality need to be moved to the platform
            API (include/uk/plat/*).

    2.2. Introduce a new variable "FAMILY" in Makefile:
         Before:$(UK_BASE)/arch/$(ARCH)/Makefile.uk.
         After:$(UK_BASE)/arch/$(FAMILY)/$(ARCH)/Makefile.uk

    2.3. Restructure the unikraft/include/uk/arch:
             unikraft/include/uk/arch----arm----arm32
                               |      |-----arm64
                               |
                                   |-----x86----x86
                                          |-----x86_64

    2.4. Restructure the unikraft/plat/:
        2.4.1. Add arm64 kvm platform code to unikraft/plat/kvm/arm, and use
               Makefile to select objects for correct architecutre:

               ifeq ($(ARCH_X86_64),y)
                  LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/entry64.S
                  LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/cpu_x86_64.c
               else ifeq ($(ARCH_ARM_64),y)
                  LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry64.S
                  LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm64.c
               else ifeq ($(ARCH_ARM_64),y)
                  LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry.S
                  LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm.c
               endif

        2.4.2. Add "common" folder to unikraft/plat/:
               1) plat/common/arm/* <-- code that is shared among multiple ARM
                  platform libs (probably includes bare essential drivers like
                  interrupt controllers and timers for scheduling).

                  plat/common/x86/* <-- same for x86 platform libs

               2) plat/common/drivers/* <-- device and bus drivers that are
                  going to be built as individual libraries (e.g., NIC, block
                  device drivers). We will reuse the Linux's drivers structure
                  for Unikraft.

                  If we have this, each of the platform Config.uk's would list
                  only a subset of drivers that they can work with (e.g.,
                  pcifront on the Xen platform lib only).

                  We need global switches for each driver that can enable by
                  one or multiple platforms. A new menu item (either in the
                  root or platform structure) should appear that lists only
                  enabled drivers and allows us to configure each of them
                  individually.

                  The platform's Linker.uk would then need to include the
                  depending and compiled driver library objects to the final
                  linking.

               3) plat/common/drivers/include/* <-- Include folder for driver
                  APIs that depend on each other (for example: PCI bus so that
                  e1000 works with pcifront but also linuxu's VFIO-based pci
                  bus).

        2.4.3. New hypervisors like kvmtool, ukvm will be considered as new
               platforms (e.g., plat/kvmtool/arm).

3. Bootloader
   Because of the BIOS, x86 is using multiboot to load kernel on Linux-KVM
   QEMU. But on ARM platforms, we can skip the EFI and boot from the Virtual
   Machine's RAM base address. So we can place _libkvmplat_entry to the CPU's
   reset entry by link script.

   On ARM64 platform, the RAM base address of machine "virt" is 0x40000000, so
   we can use a similar link script:

         plat/kvm/arm/link64.ld:
         ENTRY(_libkvmplat_entry)
         SECTIONS {
             . = 0x40000000;

             /* Code */
             _stext = .;

             .text :
             {
                 *(.text)
                 *(.text.*)
             }

             _etext = .;
             ...
         }

4. Support single CPU for the first version.

5. Support multiple threads.
    5.1. Implement GIC interrupt controller drivers. If we doesn't specify the
         GIC version in QEMU's parameter, default GIC will be detected by
         kvm_arm_vgic_probe. Most ARM hosts are using GICv2, GICv3 and GICv4,
         and QEMU will provide GICv2 and GICv3 emulators.

         For best compatibility, we have to implement gicv2 and gicv3 drivers
         without MSI/MSI-X support. This means we don't need to implement
         gicv2m, gicv3-its for Unikraft at this time.

    5.2. Implement ARMv8 virtual timer driver. I will contact Costin for what
         is required from the Unikraft's scheduler API.

6. Setup a 1:1 mapping pagetable for Physical memory and Virtual memory.
    6.1. Configure MMU
    6.2. Create page tables with 1GB or 2MB block

7. Support device tree.
   Using fdt APIs to get the resource information of devices instead of 
hardcode.

8. Foreseeable drivers:
    1) Bare essential drivers:
           GICv2, GICv3, ARMv8 virtual timer, PL011 UART.

    2) Bus drivers:
           virtio-mmio
           Generic ECAM PCI host controller (optional)

       Currently, we can attach all devices to virtio-mmio bus. So the PCI bus
       would not be mandatory for ARM.

    3) Block device drivers:
            virtio-block

    4) Network device drivers:
            virtio-net

    5) PSCI driver:
       QEMU machine "virt" will provide PSCI 0.2 emulation. So need to
       implement a PSCI interface to shutdown machine.
_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel