[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Minios-devel] Summary for discussions of ARM Unikraft supports
Hi Wei,
thanks again for wrapping up your proposal. See my comments inline.
On 09.02.2018 09:07, Wei Chen wrote:
Hi Simon,
I have done a summary for the discussions we had done these days.
Here is the new version, if I missed anything please remind me : )
Regards,
Wei Chen
Proposal:
My first target is to enable Unikraft on ARM64+Kvm, so this proposal would
focus on ARM64+Kvm. But the goal of ARM support is to enable Unikraft on
ARM32/ARM64 based hypervisors (ARM32/64 Kvm, ARM64 Xen and etc). So we have
to consider to keep multi-arch/multi-plat framework.
Because the QEMU is the de-facto standard deployment for KVM, we should start
with this. In order to reach out to as many KVM users as we can and make the
usage easy for them. We would lower the priority for other platforms, e.g.,
kvm-tool, or ukvm.
1. First target platform
AArch64 QEMU+KVM, Machine is "virt" with 64-bit ARMv8 CPU:
qemu-system-aarch64 -machine virt -cpu cortex-a53/cortex-a57
2. Modify the folders for multi-architectures
2.1. Restructure the unikraft/arch to:
unikraft----arch----arm----arm32
| |-----arm64
|
|-----x86----x86
|-----x86_64
Above folders contains architecture specified Makefile, Config,
Compiler flags and some code. Any code wants to be placed in these
folders must follow:
1) This code is exposed to the user with an interface in
include/uk/arch/*
2) This code works with all platforms (including linuxu which is
special). For instance, you should not add code that uses
privileged instruction that could not be executed in Linux
userspace. If there is a different implementation needed, it is
a hint that this functionality need to be moved to the platform
API (include/uk/plat/*).
This is fine. Please provide first a patch series that modifies the
underlying build system before you introduce new code. Your patches
would need to adopt the changes to existing x86 code. I recommend to use
the staging branch as base (since we introduce new features there).
2.2. Introduce a new variable "FAMILY" in Makefile:
Before:$(UK_BASE)/arch/$(ARCH)/Makefile.uk.
After:$(UK_BASE)/arch/$(FAMILY)/$(ARCH)/Makefile.uk
Internally, Unikraft is using UK_ARCH (ARCH is read from environment
variable or given as parameter to the build). So, you should use
UK_FAMILY if required. However, I think we should not expect the user
needed to set or select FAMILY in a menu or by setting the variable
externally. I think we can retrieve UK_FAMILY directly from ARCH / UK_ARCH.
Because of this I think we will keep Compiler.uk, Makefile.uk, and
Config.uk directly under the FAMILY directory.
2.3. Restructure the unikraft/include/uk/arch:
unikraft/include/uk/arch----arm----arm32
| |-----arm64
|
|-----x86----x86
|-----x86_64
This is fine. Notice that arch/*.h has wrappers that include
automatically the actual architecture files depending on your current
build. I think for your initial version, you do not need to add new
function prototypes because we need to implement it on x86 then, too.
One exception might be some endian conversions; I guess your ARM64 port
is also using little endian, right? If your arch and your plat is
implementing what is provided in include/, our libraries should "just
work" on ARM64 (hopefully ;-))
2.4. Restructure the unikraft/plat/:
2.4.1. Add arm64 kvm platform code to unikraft/plat/kvm/arm, and use
Makefile to select objects for correct architecutre:
ifeq ($(ARCH_X86_64),y)
LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/entry64.S
LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/cpu_x86_64.c
else ifeq ($(ARCH_ARM_64),y)
LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry64.S
LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm64.c
else ifeq ($(ARCH_ARM_64),y)
LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry.S
LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm.c
endif
This is fine.
2.4.2. Add "common" folder to unikraft/plat/:
1) plat/common/arm/* <-- code that is shared among multiple ARM
platform libs (probably includes bare essential drivers like
interrupt controllers and timers for scheduling).
plat/common/x86/* <-- same for x86 platform libs
This is fine too. I am going to introduce a patch that basically defines
UK_PLAT_BASE := $(UK_BASE)/plat
UK_PLAT_COMMON_BASE := $(UK_PLAT_BASE)/common
in plat/Makefile.uk . This should simplify your Makefile writing.
2) plat/common/drivers/* <-- device and bus drivers that are
going to be built as individual libraries (e.g., NIC, block
device drivers). We will reuse the Linux's drivers structure
for Unikraft.
Maybe lets do plat/drivers/* instead (sorry for being inconsistent). We
are going to treat these drivers as a new library type and it might be
less confusing when they are not under common. They work different than
code that is going to be placed under plat/common/*
If we have this, each of the platform Config.uk's would list
only a subset of drivers that they can work with (e.g.,
pcifront on the Xen platform lib only).
We need global switches for each driver that can enable by
one or multiple platforms. A new menu item (either in the
root or platform structure) should appear that lists only
enabled drivers and allows us to configure each of them
individually.
The platform's Linker.uk would then need to include the
depending and compiled driver library objects to the final
linking.
3) plat/common/drivers/include/* <-- Include folder for driver
APIs that depend on each other (for example: PCI bus so that
e1000 works with pcifront but also linuxu's VFIO-based pci
bus).
Right. I may publish in a few weeks an example for a PCI bus and
virtio-net driver which might make this more clear.
2.4.3. New hypervisors like kvmtool, ukvm will be considered as new
platforms (e.g., plat/kvmtool/arm).
3. Bootloader
Because of the BIOS, x86 is using multiboot to load kernel on Linux-KVM
QEMU. But on ARM platforms, we can skip the EFI and boot from the Virtual
Machine's RAM base address. So we can place _libkvmplat_entry to the CPU's
reset entry by link script.
On ARM64 platform, the RAM base address of machine "virt" is 0x40000000, so
we can use a similar link script:
plat/kvm/arm/link64.ld:
ENTRY(_libkvmplat_entry)
SECTIONS {
. = 0x40000000;
/* Code */
_stext = .;
.text :
{
*(.text)
*(.text.*)
}
_etext = .;
...
}
Sounds good.
4. Support single CPU for the first version.
5. Support multiple threads.
5.1. Implement GIC interrupt controller drivers. If we doesn't specify the
GIC version in QEMU's parameter, default GIC will be detected by
kvm_arm_vgic_probe. Most ARM hosts are using GICv2, GICv3 and GICv4,
and QEMU will provide GICv2 and GICv3 emulators.
For best compatibility, we have to implement gicv2 and gicv3 drivers
without MSI/MSI-X support. This means we don't need to implement
gicv2m, gicv3-its for Unikraft at this time.
5.2. Implement ARMv8 virtual timer driver. I will contact Costin for what
is required from the Unikraft's scheduler API.
6. Setup a 1:1 mapping pagetable for Physical memory and Virtual memory.
6.1. Configure MMU
6.2. Create page tables with 1GB or 2MB block
What is the default page size on ARM64 (what uses Linux)? Is it also 4k?
If yes, I think it is a nice feature to merge as many pages together to
have less entries in the page tables (e.g., use 1GB and 2MB's as much as
possible and fill the rest with 4K ones). However, I still prefer having
4KB as smallest available since we are going to have Unikernels which
are less than 2MB in total size.
This smallest unit should be set as __PAGE_SIZE in include/uk/arch/*
If it makes implementing easier, start with 4KB only.
7. Support device tree.
Using fdt APIs to get the resource information of devices instead of
hardcode.
8. Foreseeable drivers:
1) Bare essential drivers:
GICv2, GICv3, ARMv8 virtual timer, PL011 UART.
2) Bus drivers:
virtio-mmio
Generic ECAM PCI host controller (optional)
Currently, we can attach all devices to virtio-mmio bus. So the PCI bus
would not be mandatory for ARM.
3) Block device drivers:
virtio-block
4) Network device drivers:
virtio-net
5) PSCI driver:
QEMU machine "virt" will provide PSCI 0.2 emulation. So need to
implement a PSCI interface to shutdown machine.
Good!
Thanks a lot for your work.
Simon
_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel
|
