[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit


  • To: Mario Marietto <marietto2008@xxxxxxxxx>
  • From: Henry Wang <Henry.Wang@xxxxxxx>
  • Date: Mon, 27 Nov 2023 10:28:13 +0000
  • Accept-language: zh-CN, en-US
  • Arc-authentication-results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com])
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none
  • Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZDDgmvC3H7XGbddnRDRljG9XE/z5a7o6VdO5cApr62k=; b=he9vTyAqxwGTVhbkSBX7qswNf9m4vH69TQRfi3o3VIi633ROhd2C5Sh6/UOoIu7CFQWd9m9tzrip9TFBPzA2vrb1c/yONPbTr9E5gSl6J1uxBOIPbKwcMTYfxXvc8STSKj8YwsFhA8drl4EcPF+GgeCpHugXNfrYbL3bgYV/LdLzlojR8JLMsk394uKRiXO/BLWQEgFPIVlVZz3M8muc5knEg93E08TqbXVIZBMPzv52ea8sFnYe5ULQHi5DkTnRT1Irw8egrQeKvpcN/qvReDq3hMQvjqkCu0tVQw8sCcPS106kYO4zzXDt+ONobVJB4CTSGQ8hz9o9JRnpIKCW0Q==
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZDDgmvC3H7XGbddnRDRljG9XE/z5a7o6VdO5cApr62k=; b=OXJKIHcyv8Q/GQCptuD0f02NIAd4AOXnx9sOlDPeclszAOvF87QEGaYAaJwTU+B2oxPkl9uI2igZtmWUFJm/BzrGVzPV2J3iv7nID77jyztpgCjpUf6kIoBipE8pX1y6NCZKUgtuv+G+S8yG7YvHQDK1t1oPxC/c5EmDGCQrdUhHYW+NZJkMd4+yTU+JJla5gL32pQ0EGueMaFPZdWlQGG67ZSDVtXtbjDONCDbtl4ZC6yTeOyKttQep5LBCFzZs3ZAN60c1TVN4qS4VQAagJWj40DGaPBW8DnDg7IpdRhxAE7Fi3iY8V3LewsrdtoBcQFGb36QG/xp8kfGc5cNO+A==
  • Arc-seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=MDSAmbO0IALIddBK2mER5ueBx+6Xr++6cBrpMpHv/mpa/vwoMENG4c4jtf0LMscdjSR6E6PWJLV9q+6sVRTr+LuzvyoMQHtBLY9ifcfAly1a0KX0uyC58nlzF1V1M4Bx56nv/8KED9xLjIgQBKbLB2i0BheSLCjznyNU/kXXKbwQ9swpHcp78Q5CPU4NqqHgvXjowlPdphdizwFwduFghiqoqIEzl1zW5vXnE7/dHCG+dC0RoIfE28lyZhxQrZuLW/N0vIBTLSpHI41aFt1v/t/MkP6By9n9AwRUoXiIwSaNYZj0Fm1/dOTh1ug8OApWts6wi5pTHnCVW7jILymW1w==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cm2kxOj1QH/dKO8xAuS4514kCAHbtK22oYjq8m1PmS4lY6T/SDBIOd4ht7ApRVIINFebkaoS1/rzjphhZrbmEdKJPG4dt0aLGjjr3+nLCEuHWnIDc59F4zxnlU4Y79yjewipDsoK3gPqeCcS2p2zs+Y9ksygpBYHmRx0sVBgJsKgAoCKG5nGj9ccgAnzgltuYybNtZoTHera2juSIGsadaeoQaWXWn25Q66tj+iWGk7ArHWD3fBojetWv4oqWK0i9dgHvuPDu+ql9lwI3XGTNYJlOmmNqTTXg+OKfIO/R7VX4Nkqvb3SHvMY7hxCg6uzXhTgvBtSD8ey5wfaoCH93A==
  • Authentication-results-original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Cc: Demi Marie Obenour <demi@xxxxxxxxxxxxxxxxxxxxxx>, Anthony PERARD <anthony.perard@xxxxxxxxxx>, Roger Pau Monné <roger.pau@xxxxxxxxxx>, Julien Grall <julien@xxxxxxx>, Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, Michal Orzel <michal.orzel@xxxxxxx>
  • Delivery-date: Mon, 27 Nov 2023 10:35:51 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Nodisclaimer: true
  • Original-authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Thread-index: AQHaIRqC1LNxkfiXQEOwpgfnTEw2i7CN9pcA
  • Thread-topic: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

+(xen-devel and Arm maintainers, including Julien)

> On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@xxxxxxxxx> wrote:
> 
> Hello.
> We have just virtualized Debian 12 on our arm (32 bit) Chromebook model 
> xe303c12 . As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 
> 12. It works great. But our goal is different. We want to virtualize FreeBSD 
> as domU. Can we have a working Xen PV network driver for a FreeBSD arm guest 
> ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I 
> would like to know if Julien's work was accepted upstream by FreeBSD, in 
> which case FreeBSD as a Xen guest on arm should work if we enable the Xen PV 
> drivers in the FreeBSD on arm kernel. If Julien's work was not accepted 
> upstream by FreeBSD, we will have to find his patches and apply them 
> ourselves to the FreeBSD on arm kernel.
> We found these slides :
> 
> https://events.static.linuxfound.org/sites/events/files/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf
> 
> Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what we 
> want to find.
> It looks like when that slide presentation was written, there were some 
> limitations on FreeBSD Xen guests. For example, for our debian bookworm 
> guest, I am using vcpus = '2' to match the number of real cpus on our 
> Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD 
> guest, so I will need to change that vcpus = '1' in the FreeBSD guest config 
> unless support for 2 or more vcpus was added later, which is possible because 
> that slide presentation is 9 years old.
> Here is where I would expect to find the XENHVM FreeBSD on arm kernel config 
> file:
> 
> https://cgit.freebsd.org/src/tree/sys/arm/conf
> 
> But it is not there unless I am not understanding something correctly. For 
> now, unfortunately conclude that the support for Xen on arm that Julien Grall 
> mentioned in that slide presentation 9 years ago was never added to the 
> official FreeBSD source code. I am searching the web now to see if the 
> patches that Julien Grall wrote are still posted somewhere online. If we 
> cannot find them, we can ask here and on the xen-users mailing list. Julien 
> regularly reads that list and responds to questions about Xen on arm, so I 
> think he will tell us how to find the patches if we cannot find them online.
> According to this page from the FreeBSD wiki:
> 
> https://wiki.freebsd.org/Xen
> 
> I think FreeBSD only supports Xen on x86, not arm. So this is going to be a 
> bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien 
> Grall has some patches that made it work in the past !
> I found a slightly newer slide presentation by Julien here:
> 
> https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd
> 
> It is about the same, but it mentions the GENERIC FreeBSD kernel supports Xen 
> on arm64, but still says we need the XENHVM FreeBSD config for Xen on arm 32 
> bit, which I haven't found online yet.
> Please,take a look at this output of the linux kernel that can boot on Xen, 
> and the FreeBSD kernel that cannot :
> 
> % file zImage-6.1.59-stb-xen-cbe+
> zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage 
> (little-endian)
> 
> % file FREEBSD-XENVIRT 
> FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), 
> dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not 
> stripped
> 
> 
> The FreeBSD kernel that won't boot is in ELF format but the Linux kernel that 
> does boot is in zImage format. This is the error we get :
> # xl create freebsd.cfg 
> Parsing config from freebsd.cfg xc: error: panic: xg_dom_elfloader.c:63: 
> xc_dom_guest_type: image not capable of booting inside a HV
> M container: Invalid kernel 
> libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image failed 
> libxl: error: libxl_create.c:1640:domcreate_rebuild_done: Domain 4:cannot 
> (re-)build domain: -3 
> libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 4:Non-existant 
> domain 
> libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 4:Unable to 
> destroy guest 
> libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 4:Destruction of 
> domain failed
> 
> We are spending time reading the docs on xenbits.xenproject.org, and 
> according to those docs Xen on arm only knows how to boot a kernel in the 
> zImage format, so the FreeBSD kernel is in a format that modern Xen 
> incorrectly detects as an x86 kernel.
> I also watched Julien Grall's 30 minute video presentation of his work to 
> boot FreeBSD/arm on Xen at FOSDEM 2014 here :
> 
> https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/
> 
> In that video, and in other places, Julien mentions that the boot ABI for 
> FreeBSD/arm on Xen was not yet developed and he was getting occasional 
> crashes and needed to investigate the problem. He mentioned the zImage ABI 
> that Linux uses, but pointed out FreeBSD does not use that format, and back 
> then it was an open question which format to use to boot FreeBSD/arm on Xen. 
> Unfortunately, nine years later, the only supported format is still the 
> zImage format that Linux uses.
> It looks like Julien's work back then was using an ELF binary to boot 
> FreeBSD/arm on Xen instead of the supported zImage format that Linux uses and 
> the modern Xen toolstack exits with an error when trying to boot the FreeBSD 
> ELF formatted binary that Julien's patch creates. So the best solution would 
> be to try to port the rules to build a FreeBSD kernel in the zImage format 
> instead of the ELF format. I have been studying the Makefiles in Linux to see 
> how Linux builds the Linux arm kernel in the zImage format, but it is not 
> trivial to understand.
> -- 
> Mario.




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.