Re: [Minios-devel] Summary for discussions of ARM Unikraft supports

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Simon,

> -----Original Message-----
> From: Simon Kuenzer [mailto:simon.kuenzer@xxxxxxxxx]
> Sent: 2018年2月9日 18:49
> To: Wei Chen <Wei.Chen@xxxxxxx>; Costin Lupu <costin.lup@xxxxxxxxx>
> Cc: Felipe Huici <Felipe.Huici@xxxxxxxxx>; Kaly Xin <Kaly.Xin@xxxxxxx>; Shijie
> Huang <Shijie.Huang@xxxxxxx>; Florian Schmidt <Florian.Schmidt@xxxxxxxxx>; nd
> <nd@xxxxxxx>; minios-devel@xxxxxxxxxxxxx
> Subject: Re: Summary for discussions of ARM Unikraft supports
> 
> Hi Wei,
> 
> thanks again for wrapping up your proposal. See my comments inline.
> 
> On 09.02.2018 09:07, Wei Chen wrote:
> > Hi Simon,
> >
> > I have done a summary for the discussions we had done these days.
> > Here is the new version, if I missed anything please remind me : )
> >
> > Regards,
> > Wei Chen
> >
> > Proposal:
> >
> > My first target is to enable Unikraft on ARM64+Kvm, so this proposal would
> > focus on ARM64+Kvm. But the goal of ARM support is to enable Unikraft on
> > ARM32/ARM64 based hypervisors (ARM32/64 Kvm, ARM64 Xen and etc). So we have
> > to consider to keep multi-arch/multi-plat framework.
> >
> > Because the QEMU is the de-facto standard deployment for KVM, we should
> start
> > with this. In order to reach out to as many KVM users as we can and make the
> > usage easy for them. We would lower the priority for other platforms, e.g.,
> > kvm-tool, or ukvm.
> >
> > 1. First target platform
> >     AArch64 QEMU+KVM, Machine is "virt" with 64-bit ARMv8 CPU:
> >     qemu-system-aarch64 -machine virt -cpu cortex-a53/cortex-a57
> >
> > 2. Modify the folders for multi-architectures
> >      2.1. Restructure the unikraft/arch to:
> >                  unikraft----arch----arm----arm32
> >                           |      |-----arm64
> >                           |
> >                                  |-----x86----x86
> >                                         |-----x86_64
> >
> >           Above folders contains architecture specified Makefile, Config,
> >           Compiler flags and some code. Any code wants to be placed in these
> >           folders must follow:
> >           1) This code is exposed to the user with an interface in
> >              include/uk/arch/*
> >           2) This code works with all platforms (including linuxu which is
> >              special). For instance, you should not add code that uses
> >              privileged instruction that could not be executed in Linux
> >              userspace. If there is a different implementation needed, it is
> >              a hint that this functionality need to be moved to the platform
> >              API (include/uk/plat/*).
> 
> This is fine. Please provide first a patch series that modifies the
> underlying build system before you introduce new code. Your patches
> would need to adopt the changes to existing x86 code. I recommend to use
> the staging branch as base (since we introduce new features there).
> 

Ok, I will send this patch series first.

> >
> >      2.2. Introduce a new variable "FAMILY" in Makefile:
> >           Before:$(UK_BASE)/arch/$(ARCH)/Makefile.uk.
> >           After:$(UK_BASE)/arch/$(FAMILY)/$(ARCH)/Makefile.uk
> >
> 
> Internally, Unikraft is using UK_ARCH (ARCH is read from environment
> variable or given as parameter to the build). So, you should use
> UK_FAMILY if required. However, I think we should not expect the user
> needed to set or select FAMILY in a menu or by setting the variable
> externally. I think we can retrieve UK_FAMILY directly from ARCH / UK_ARCH.
> Because of this I think we will keep Compiler.uk, Makefile.uk, and
> Config.uk directly under the FAMILY directory.

Yes, I also think so.

> 
> >      2.3. Restructure the unikraft/include/uk/arch:
> >                  unikraft/include/uk/arch----arm----arm32
> >                                    |      |-----arm64
> >                                 |
> >                                        |-----x86----x86
> >                                               |-----x86_64
> This is fine. Notice that arch/*.h has wrappers that include
> automatically the actual architecture files depending on your current
> build. I think for your initial version, you do not need to add new
> function prototypes because we need to implement it on x86 then, too.
> One exception might be some endian conversions; I guess your ARM64 port
> is also using little endian, right? If your arch and your plat is
> implementing what is provided in include/, our libraries should "just
> work" on ARM64 (hopefully ;-))
> 

Yes, that's true : )

> >
> >      2.4. Restructure the unikraft/plat/:
> >          2.4.1. Add arm64 kvm platform code to unikraft/plat/kvm/arm, and
> use
> >                 Makefile to select objects for correct architecutre:
> >
> >                 ifeq ($(ARCH_X86_64),y)
> >                    LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/entry64.S
> >                    LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/cpu_x86_64.c
> >                 else ifeq ($(ARCH_ARM_64),y)
> >                    LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry64.S
> >                    LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm64.c
> >                 else ifeq ($(ARCH_ARM_64),y)
> >                    LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry.S
> >                    LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm.c
> >                 endif
> >
> 
> This is fine.
> 
> >          2.4.2. Add "common" folder to unikraft/plat/:
> >                 1) plat/common/arm/* <-- code that is shared among multiple
> ARM
> >                    platform libs (probably includes bare essential drivers
> like
> >                    interrupt controllers and timers for scheduling).
> >
> >                    plat/common/x86/* <-- same for x86 platform libs
> 
> This is fine too. I am going to introduce a patch that basically defines
> 
>       UK_PLAT_BASE := $(UK_BASE)/plat
>       UK_PLAT_COMMON_BASE := $(UK_PLAT_BASE)/common
> 
> in plat/Makefile.uk . This should simplify your Makefile writing.
> 
> >
> >                 2) plat/common/drivers/* <-- device and bus drivers that are
> >                    going to be built as individual libraries (e.g., NIC,
> block
> >                    device drivers). We will reuse the Linux's drivers
> structure
> >                    for Unikraft.
> >
> 
> Maybe lets do plat/drivers/* instead (sorry for being inconsistent). We
> are going to treat these drivers as a new library type and it might be
> less confusing when they are not under common. They work different than
> code that is going to be placed under plat/common/*
> 

Yes, this change would be better than "plat/common/drivers"

> >                    If we have this, each of the platform Config.uk's would
> list
> >                    only a subset of drivers that they can work with (e.g.,
> >                    pcifront on the Xen platform lib only).
> >
> >                    We need global switches for each driver that can enable
> by
> >                    one or multiple platforms. A new menu item (either in the
> >                    root or platform structure) should appear that lists only
> >                    enabled drivers and allows us to configure each of them
> >                    individually.
> >
> >                    The platform's Linker.uk would then need to include the
> >                    depending and compiled driver library objects to the
> final
> >                    linking.
> >
> >                 3) plat/common/drivers/include/* <-- Include folder for
> driver
> >                    APIs that depend on each other (for example: PCI bus so
> that
> >                    e1000 works with pcifront but also linuxu's VFIO-based
> pci
> >                    bus).
> 
> Right. I may publish in a few weeks an example for a PCI bus and
> virtio-net driver which might make this more clear.
> 
> >
> >          2.4.3. New hypervisors like kvmtool, ukvm will be considered as new
> >                 platforms (e.g., plat/kvmtool/arm).
> >
> > 3. Bootloader
> >     Because of the BIOS, x86 is using multiboot to load kernel on Linux-KVM
> >     QEMU. But on ARM platforms, we can skip the EFI and boot from the
> Virtual
> >     Machine's RAM base address. So we can place _libkvmplat_entry to the
> CPU's
> >     reset entry by link script.
> >
> >     On ARM64 platform, the RAM base address of machine "virt" is 0x40000000,
> so
> >     we can use a similar link script:
> >
> >           plat/kvm/arm/link64.ld:
> >           ENTRY(_libkvmplat_entry)
> >           SECTIONS {
> >               . = 0x40000000;
> >
> >               /* Code */
> >               _stext = .;
> >
> >               .text :
> >               {
> >                   *(.text)
> >                   *(.text.*)
> >               }
> >
> >               _etext = .;
> >               ...
> >           }
> 
> Sounds good.
> 
> >
> > 4. Support single CPU for the first version.
> >
> > 5. Support multiple threads.
> >      5.1. Implement GIC interrupt controller drivers. If we doesn't specify
> the
> >           GIC version in QEMU's parameter, default GIC will be detected by
> >           kvm_arm_vgic_probe. Most ARM hosts are using GICv2, GICv3 and
> GICv4,
> >           and QEMU will provide GICv2 and GICv3 emulators.
> >
> >           For best compatibility, we have to implement gicv2 and gicv3
> drivers
> >           without MSI/MSI-X support. This means we don't need to implement
> >           gicv2m, gicv3-its for Unikraft at this time.
> >
> >      5.2. Implement ARMv8 virtual timer driver. I will contact Costin for
> what
> >           is required from the Unikraft's scheduler API.
> >
> > 6. Setup a 1:1 mapping pagetable for Physical memory and Virtual memory.
> >      6.1. Configure MMU
> >      6.2. Create page tables with 1GB or 2MB block
> 
> What is the default page size on ARM64 (what uses Linux)? Is it also 4k?

Yes, Linux is using 4K as default page size.

> If yes, I think it is a nice feature to merge as many pages together to
> have less entries in the page tables (e.g., use 1GB and 2MB's as much as
> possible and fill the rest with 4K ones). However, I still prefer having
> 4KB as smallest available since we are going to have Unikernels which
> are less than 2MB in total size.
> This smallest unit should be set as __PAGE_SIZE in include/uk/arch/*
> If it makes implementing easier, start with 4KB only.
> 

Oh, I had forgotten that use case. Ok, I will start with 4KB only.

> >
> > 7. Support device tree.
> >     Using fdt APIs to get the resource information of devices instead of
> hardcode.
> >
> > 8. Foreseeable drivers:
> >      1) Bare essential drivers:
> >             GICv2, GICv3, ARMv8 virtual timer, PL011 UART.
> >
> >      2) Bus drivers:
> >             virtio-mmio
> >             Generic ECAM PCI host controller (optional)
> >
> >         Currently, we can attach all devices to virtio-mmio bus. So the PCI
> bus
> >         would not be mandatory for ARM.
> >
> >      3) Block device drivers:
> >              virtio-block
> >
> >      4) Network device drivers:
> >              virtio-net
> >
> >      5) PSCI driver:
> >         QEMU machine "virt" will provide PSCI 0.2 emulation. So need to
> >         implement a PSCI interface to shutdown machine.
> >
> 
> Good!
> Thanks a lot for your work.
> 
> Simon
_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel