Xen project Mailing List

[Xen-devel] [RFC] Add support for Xen ARM guest on FreeBSD

Hello, During the last couple of months I have been working on porting FreeBSD on Xen on ARM. It's my first big project on FreeBSD and I'm not sure if I have cc the right MLs and people. If not, could somebody point me to the right email address? With this patch series I'm able to boot FreeBSD with a single VCPU on every 32 bits platform supported by Xen. The patch series is divided in 5 parts: * #1-#12: Clean up and bug fix with some options * #13-#17: Update the Xen interface to Xen 4.4 and fix compilation * #18-#32: Update Xen code to support ARM guest * #33-#38: Update ARM code to support Xen guest * #39-#40: Add platform code to support ARM guest I have pushed a branch on my git based on Royger's pvh work version 10: git clone git://xenbits.xen.org/people/julieng/freebsd.git -b xen-arm http://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm If needed, I can send the patch one by one the different mailing list. This new support brings 2 open questions (for both Xen and FreeBSD community). When a new guest is created, the toolstack will generated a device tree which will contains: - The amount of memory - The description of the platform (gic, timer, hypervisor node) - PSCI node for SMP bringup Until now, Xen on ARM supported only Linux-based OS. When the support of device tree was added in Xen for guest, we chose to use Linux device tree bindings (gic, timer,...). It seems that FreeBSD choose a different way to implement device tree: - strictly respect ePAR (for interrupt-parent property) - gic bindings different (only 1 interrupt cell) I would like to come with a common device tree specification (bindings, ...) across every operating system. I know the Linux community is working on having a device tree bindings out of the kernel tree. Does FreeBSD community plan to work with Linux community for this purpose? As specified early, the device tree can vary accross Xen version and user input (eg the memory). I have noticed few places (mainly the timer) where the IRQs number are harcoded. In the long-term, it would be nice to see FreeBSD booting out-of-box (eg the device tree is directly provided by the board firmware). I plan to add support for Device Tree loading via Linux Boot ABI, it the way that Xen use to boot a new guest. The second question is related to memory attribute for page table. The early page table setup by FreeBSD are using Write-Through memory attribute which result to a likely random (not every time the same place) crash before the real page table are initialized. Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today, I have no idea if it's a requirement from Xen or a bug (either in Xen or FreeBSD). The code is taking its first faltering steps, therefore the TODO is quite big: - Try the code on x86 architecture. I did lots of rework for the event channels code and didn't even try to compile - Clean up event channel code - Clean up xen control code - Add support for Device Tree loading via Linux Boot ABI - Fixing crashing on userspace. Could be related to the patch series "arm SMP on Cortex-A15" - Add guest SMP support - DOM0 support? Any help, comments, questions are welcomed. Sincerely yours, ============= Instruction to test FreeBSD on Xen on ARM =========== Xen upstream tree doesn't fully support ELF loading for ARM. You will need to recompile the tools with the following 2 patches: - https://patches.linaro.org/22228/ - https://patches.linaro.org/22227/ To compile and boot Xen on your board, you can refer to the wiki page: http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions The instruction to compile FreeBSD for Xen on ARM: $ truncate -s 512 xenvm.img $ sudo mdconfig -f xenvm.img -u0 $ sudo newfs /dev/md0 $ sudo mount /dev/md0 /mnt $ sudo make TARGET_ARCH=armv6 kernel-toolchain $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel $ sudo make TARGET_ARCH=armv6 buildworld $ sudo make TARGET_ARCH=armv6 DESTDIR=/mnt installworld distribution $ echo "/dev/xbd0 / ufs rw 1 1" > /mnt/etc/fstab $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure") $ sudo umount /mnt $ sudo mdconfig -d u0 Then you can copy the rootfs and the kernel to DOM 0 on your board. To boot the a FreeBSD your will required the following configuration file $ cat freebsd.xl kernel="kernel" memory=64 name="freebsd" vcpus=1 autoballon="off" disk=[ 'phy:/dev/loop0,xvda,w' ] $ losetup /dev/loop0 xenvm.img $ xl create freebsd.xl $ xl console freebsd Julien Grall (40): xen/netfront: Don't need to include machine/intr_machdep.h xen/blkfront: Don't need to include machine/intr_machdep.h xen/control: Remove include machine/intr_machdep.h xen: Define xen_intr_handle_upcall in common headers xen: Remove duplicate features.h header in i386 arch xen/hypercall: Allow HYPERVISOR_console_write to take a const string xen/timer: Make xen timer optional xen/console: Fix build when DDB is enabled xen/console: clean up identify callback xen/xenstore: xs_probe should return BUS_PROBE_NOWILDCARD xen/control: xctlr_probe shoud return BUS_PROBE_NOWILDCARD xen/hypervisor: Be sure to set __XEN_INTERFACE_VERSION__ xen/interface: Update interface to Xen 4.4 headers xen: Bump __XEN_INTERFACE_VERSION__ to 4.3 xen/baloon: Fix compilation with Xen 4.4 headers xen/netfront: Fix compilation with Xen 4.4 headers xen/gnttab: Fix compilation with Xen 4.4 headers xen/ballon: Use correct type for frame list xen/netback: Fix printf format for xen_pfn_t xen/netfront: Use the correct type for rx_pfn_array xen/gnttab: Use the right type for the frames xen/gnttab: Export resume_frames xen/gnttab: Add a guard for xenpci functions xen/netfront: Define PTE flags per architecture xen/control: Implement suspend has panic for ARM xen: Introduce xen_pmap xen: move x86/xen/xenpv.c in dev/xen/xenpv.c xen/xenpv: Only add isa for x86 architecture xen/xenpv: Protect xenpci code by DEV_XENPCI xen: move xen_intr.c to common code xen/evtchn: Rework the event channels handler to make it generic xen/console: handle console for HVM arm/timer: Handle timer with no clock-frequency property arm/timer: WORKAROUND The timer should support DT... arm: Detect new revision for cortex A15 arm: add ELFNOTE macro arm: Implement disable_intr arm: Use Write-Back as memory attribute for early page table dts: Add xenvm.dts arm: Add xenhvm platform sys/amd64/conf/GENERIC | 4 +- sys/amd64/include/apicvar.h | 1 - sys/amd64/include/xen/hypercall.h | 2 +- sys/amd64/include/xen/xen-os.h | 2 + sys/amd64/include/xen/xenpmap.h | 2 +- sys/arm/arm/cpufunc.c | 2 + sys/arm/arm/generic_timer.c | 12 +- sys/arm/arm/locore.S | 4 +- sys/arm/conf/XENHVM | 97 +++ sys/arm/include/armreg.h | 2 + sys/arm/include/asmacros.h | 26 + sys/arm/include/cpufunc.h | 6 + sys/arm/include/xen/hypercall.h | 50 ++ sys/arm/include/xen/synch_bitops.h | 52 ++ sys/arm/include/xen/xen-os.h | 29 + sys/arm/include/xen/xenfunc.h | 4 + sys/arm/include/xen/xenvar.h | 14 + sys/arm/xenhvm/elf.S | 12 + sys/arm/xenhvm/files.xenhvm | 18 + sys/arm/xenhvm/hypercall.S | 107 +++ sys/arm/xenhvm/xen-dt.c | 148 ++++ sys/arm/xenhvm/xenhvm_machdep.c | 132 ++++ sys/boot/fdt/dts/xenvm.dts | 77 ++ sys/conf/files | 4 +- sys/conf/options | 3 + sys/conf/options.arm | 1 + sys/dev/xen/balloon/balloon.c | 6 +- sys/dev/xen/blkfront/blkfront.c | 1 - sys/dev/xen/console/console.c | 28 +- sys/dev/xen/console/xencons_ring.c | 41 +- sys/dev/xen/control/control.c | 19 +- sys/dev/xen/netback/netback.c | 6 +- sys/dev/xen/netfront/netfront.c | 15 +- sys/dev/xen/xenpci/xenpci.c | 2 +- sys/dev/xen/xenpv.c | 133 ++++ sys/i386/conf/GENERIC | 4 +- sys/i386/conf/XEN | 1 + sys/i386/include/apicvar.h | 1 - sys/i386/include/xen/features.h | 22 - sys/i386/include/xen/hypercall.h | 2 +- sys/i386/include/xen/xen-os.h | 2 + sys/i386/include/xen/xenvar.h | 2 +- sys/i386/xen/xen_machdep.c | 2 +- sys/x86/xen/xen_intr.c | 1241 ------------------------------- sys/x86/xen/xenpv.c | 128 ---- sys/xen/gnttab.c | 23 +- sys/xen/hypervisor.h | 3 +- sys/xen/interface/arch-arm.h | 259 +++++-- sys/xen/interface/arch-arm/hvm/save.h | 2 +- sys/xen/interface/arch-x86/hvm/save.h | 25 +- sys/xen/interface/arch-x86/xen-mca.h | 2 +- sys/xen/interface/arch-x86/xen-x86_32.h | 2 +- sys/xen/interface/arch-x86/xen-x86_64.h | 2 +- sys/xen/interface/arch-x86/xen.h | 31 +- sys/xen/interface/callback.h | 4 +- sys/xen/interface/dom0_ops.h | 2 +- sys/xen/interface/domctl.h | 65 +- sys/xen/interface/elfnote.h | 10 +- sys/xen/interface/event_channel.h | 5 +- sys/xen/interface/features.h | 16 +- sys/xen/interface/gcov.h | 115 +++ sys/xen/interface/grant_table.h | 8 +- sys/xen/interface/hvm/hvm_xs_strings.h | 80 ++ sys/xen/interface/hvm/ioreq.h | 20 +- sys/xen/interface/hvm/params.h | 14 +- sys/xen/interface/hvm/pvdrivers.h | 47 ++ sys/xen/interface/hvm/save.h | 4 +- sys/xen/interface/io/blkif.h | 104 ++- sys/xen/interface/io/console.h | 2 +- sys/xen/interface/io/fbif.h | 2 +- sys/xen/interface/io/kbdif.h | 2 +- sys/xen/interface/io/netif.h | 47 +- sys/xen/interface/io/pciif.h | 3 +- sys/xen/interface/io/protocols.h | 5 +- sys/xen/interface/io/tpmif.h | 68 +- sys/xen/interface/io/usbif.h | 1 - sys/xen/interface/io/vscsiif.h | 42 +- sys/xen/interface/io/xenbus.h | 20 +- sys/xen/interface/io/xs_wire.h | 7 +- sys/xen/interface/kexec.h | 7 +- sys/xen/interface/mem_event.h | 4 +- sys/xen/interface/memory.h | 80 +- sys/xen/interface/nmi.h | 7 +- sys/xen/interface/physdev.h | 33 +- sys/xen/interface/platform.h | 27 +- sys/xen/interface/sched.h | 87 ++- sys/xen/interface/sysctl.h | 47 +- sys/xen/interface/tmem.h | 2 +- sys/xen/interface/trace.h | 93 ++- sys/xen/interface/vcpu.h | 2 +- sys/xen/interface/version.h | 6 +- sys/xen/interface/xen-compat.h | 2 +- sys/xen/interface/xen.h | 128 +++- sys/xen/interface/xenoprof.h | 2 +- sys/xen/interface/xsm/flask_op.h | 8 + sys/xen/xen-os.h | 2 +- sys/xen/xen_intr.c | 1072 ++++++++++++++++++++++++++ sys/xen/xen_intr.h | 8 +- sys/xen/xenstore/xenstore.c | 4 +- 99 files changed, 3339 insertions(+), 1791 deletions(-) create mode 100644 sys/arm/conf/XENHVM create mode 100644 sys/arm/include/xen/hypercall.h create mode 100644 sys/arm/include/xen/synch_bitops.h create mode 100644 sys/arm/include/xen/xen-os.h create mode 100644 sys/arm/include/xen/xenfunc.h create mode 100644 sys/arm/include/xen/xenvar.h create mode 100644 sys/arm/xenhvm/elf.S create mode 100644 sys/arm/xenhvm/files.xenhvm create mode 100644 sys/arm/xenhvm/hypercall.S create mode 100644 sys/arm/xenhvm/xen-dt.c create mode 100644 sys/arm/xenhvm/xenhvm_machdep.c create mode 100644 sys/boot/fdt/dts/xenvm.dts create mode 100644 sys/dev/xen/xenpv.c delete mode 100644 sys/i386/include/xen/features.h delete mode 100644 sys/x86/xen/xen_intr.c delete mode 100644 sys/x86/xen/xenpv.c create mode 100644 sys/xen/interface/gcov.h create mode 100644 sys/xen/interface/hvm/hvm_xs_strings.h create mode 100644 sys/xen/interface/hvm/pvdrivers.h create mode 100644 sys/xen/xen_intr.c -- 1.8.0 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.