Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

Hi Mario,

I suggest to do one step at a time and first to get it fully working
using "xl" to create the FreeBSD guest. And afterwords to try to get
libvirt to work. Both xl and libvirt call the same library underneath
(libxl) to create guests, but xl is tested a lot more with Xen on ARM.

I am not going to comment on how to create a FreeBSD xenvm.img as I
don't have any knowledge on FreeBSD.



On Wed, 29 Nov 2023, Mario Marietto wrote:
> Hello to everyone.
> 
> I tried to use xen as a hypervisor instead of kvm + libvirt + virt-manager to 
> boot FreeBSD on my ARM Chromebook where I have installed
> Devuan 5,since Stefano said : 
> 
> "That might work. libvirt + virt-manager with the xen accelerator might work 
> on the ARM Chromebook. That's because as far as I know Xen
> integration in libvirt is done via linking to libxl directly and libxl is 
> supported and working on ARM" 
> 
> Unfortunately something is not working properly. What I did has been to 
> reboot the machine in xen,enable libvirtd & and virtlogd & and
> virt-manager &,but this is what happened :
> 
> Traceback (most recent call last):
>   File "/usr/lib/xen-4.17/bin/pygrub", line 884, in <module>
>     raise RuntimeError("Unable to find partition containing kernel")
> RuntimeError: Unable to find partition containing kernel
> 
> I think it does not recognize the FreeBSD file system structure and its 
> kernel. Libvirt seems to have been programmed to boot Linux,not
> FreeBSD. 
> 
> In Fact,I did the counterproof and it seems to be like this :
> 
> 
> According with this post :
> 
> https://blog.roberthallam.org/2020/05/solving-unable-to-find-partition-containing-kernel/
> 
> I've created a file called menu.lst inside the boot directory of the image 
> file called "debian.img",adding the following content inside :
> 
> default 0
> timeout 10
> title Debian
> root (hd0,1)
> kernel /boot/vmlinux-6.1.59-stb-xen-cbe+ root=/dev/xvda
> initrd /boot/initrd.img-6.1.59-stb-xen-cbe+
> 
> and I tried again to boot the image using virt-manager. It gave this error 
> again :
> 
> root@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebook/FreeBSD-guestOS/linux-xen/debian2/boot#
>  2023-11-29 15:21:09.266+0000: 2467: error :
> libxlDomainStartPerform:1256 : internal error: libxenlight failed to create 
> new domain 'debian12'
> 
> but giving a look inside the log file and I found this interesting situation ;
> 
> Using <class 'grub.GrubConf.GrubConfigFile'> to parse /boot/grub/menu.lst
> (B )0 [1;24r [m [?7h [?1h = [H [J [?1h = [1B    pyGRUB  version 0.6
> [1B [0m 
> lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
> [1B [0m x [0;7m Debian 12                                                     
>         [m [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m 
> mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj 
> [1B [70D [0m Use the ↑ and ↓ keys to select which
> entry is highlighted. [1B [58DPress enter to boot the selected OS, 'e' to 
> edit the [1B [52Dcommands before booting, 'a' to modify the
> kernel arguments [1B [59Dbefore booting, or 'c' for a command line. [12A [26C 
> [17B [68DWill boot selected entry in 10 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  9 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  8 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  7 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  6 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  5 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  4 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  3 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  2 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  1 seconds [?1l > [24;1H
> [?1l >
> 
> so,it seems that it tried to boot,but for an unknown reason,it still gives 
> the error.
>  
> Anyway : My xen setup is not broken anymore ; Using Libvirt Linux seems to be 
> able to boot,FreeBSD does not. Using the "raw" method of
> booting FreeBSD as domU could be another story that I will try soon.
> 
> But before trying to compile the correct freebsd kernel that's recognized by 
> xen using the Elliott Michell code,I need to understand well
> what's the procedure that will work. So below you can read what I will try to 
> do :
> 
> $ truncate -s 100G xenvm.img
> $ mdconfig -f xenvm.img -u 0 
> $ newfs /dev/md0 
> $ mount /dev/md0 /mnt
> $ git clone https://gitlab.com/ehem/freebsd-src.git
> $ cd freebsd-src
> $ make  -DNO_MODULES   KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 
> DESTDIR=/build buildkernel
> $ echo "/dev/xbd0       /       ufs     rw      1 
> 1" > /mnt/etc/fstab  
> 
> $ nano /build/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on 
> secure")
> $ umount /build
> $ mdconfig -d -u 0
> 
> Do you see errors ? some missing ?
> ---> I've only ever tried arm64, but since arm32 didn't appear to need much 
> to be operational I tried to
> make it possible.  In theory it /should/ work on arm32, but I've never tried 
> it.  What was missing was 
> I had never added it to the configuration and one link was needed. Updated 
> "submit" branch has a tiny 
> adjustment.
> 
> I didn't understand how to apply the patch that's inside the "submit" branch. 
> 
> 
> On Wed, Nov 29, 2023 at 12:05 AM Elliott Mitchell <ehem+freebsd@xxxxxxx> 
> wrote:
>       On Tue, Nov 28, 2023 at 04:10:50PM +0100, Roger Pau Monné wrote:
>       > On Tue, Nov 28, 2023 at 03:09:14PM +0100, Mario Marietto wrote:
>       > > For booting a FreeBSD kernel as a guest OS on XEN,should we install 
> xen
>       > > 4.18 from source ?
> 
>       > I don't think so, I'm not aware of the FreeBSD port requiring a
>       > specific version of Xen.  I do think the work is limited to aarch64
>       > however, so there's no support in sight for arm32 FreeBSD guests as
>       > far as I'm aware.
> 
>       I've only ever tried arm64, but since arm32 didn't appear to need much
>       to have operational I tried to make it possible.  In theory it /should/
>       work on arm32, but I've never tried it.  What was missing was I had 
> never
>       added it to the configuration and one link was needed.  Updated "submit"
>       branch has the tiny adjustment.
> 
>       (the only difference is the hypercall wrappers, register naming and 
> where
>       the op code goes, very simple compatibility)
> 
> 
>       On Tue, Nov 28, 2023 at 02:45:40PM +0100, Roger Pau Monné wrote:
>       > On Mon, Nov 27, 2023 at 03:04:30PM -0800, Elliott Mitchell wrote:
>       > > BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
>       > > "royger" branch?
>       >
>       > I've pushed a bunch of those, there are still some, I've made comments
>       > on the branch.
>       >
>       > I think there isn't much left after the swept I've done.
>       >
>       > If you can rebase and reply to the comments I will take a look at
>       > what's remaining.
> 
>       Done.  I'm unsure you'll like the xs_attach_children() approach.  Thing
>       is that really is appropriate given the situation.  #2 is the urgent one
>       as that is the handy approach to the hypercall declarations.
> 
> 
>       --
>       (\___(\___(\______          --=> 8-) EHM <=--          
> ______/)___/)___/)
>        \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
>         \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
>       8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 
> 8714\_|_/___/5445
> 
> 
> 
> 
> --
> Mario.
> 
>