Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx>]

On 11/27/2023 7:45 AM, Mario Marietto wrote:
> @Chuck Zmudzinski <mailto:brchuckz@xxxxxxxxxxxx> : Stay tuned. They want to 
> help us. The xen developers are great. Very good support for us. I'm sure 
> that you can give a good contribution to understand what's our problem and 
> how to implement a fix with the help of all those good guys.
> 
> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@xxxxxxxxxx 
> <mailto:roger.pau@xxxxxxxxxx>> wrote:
> 
>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
>     > +(xen-devel and Arm maintainers, including Julien)
>     >
>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@xxxxxxxxx 
> <mailto:marietto2008@xxxxxxxxx>>
>     > > wrote:
>     > >
>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>     > > different. We want to virtualize FreeBSD as domU. Can we have a
>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>     > > would like to know if Julien's work was accepted upstream by
>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>     > > Julien's work was not accepted upstream by FreeBSD, we will have
>     > > to find his patches and apply them ourselves to the FreeBSD on arm
>     > > kernel.
> 
>     I've added Elliot on Cc as he is working on upstreaming the patches to
>     FreeBSD.  He will be able to provide a better update than myself.
> 
>     Regards, Roger.

I have been collaborating with Mario, and I can explain what we have done so 
far :

We are using Julien's patch set against an old development version of FreeBSD 11
from 2014-12-03 :

https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2

We successfully built the XENVIRT kernel and FreeBSD world, and created the
FreeBSD rootfs according to Julien's instructions here :

https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html

There were some adjustments to the instructions :

To build the kernel, we used :

$ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel

instead of

$ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel

The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
some time trying to convert it to the zImage format without realizing the
build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
So when booting with the 'kernel.bin' file instead, it actually boots :

user@devuan-bunsen ~ % sudo xl create freebsd.cfg
Parsing config from freebsd.cfg
user@devuan-bunsen ~ % sudo xl li
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   768     2     r-----    1439.4
freebsd                                      1  1152     1     r-----       3.0
user@devuan-bunsen ~ %

However, the guest is still not working correctly :

1. Attaching the console with the -c option at creation or with
   'xl console freebsd' results in no output to the console.

2. The timestamp on the virtual disk image file shows that the filesystem
   was at best mounted read-only, if it was mounted at all by the guest
   FreeBSD kernel.

3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
   the guest, you need to do 'xl destroy freebsd'.

However, I think we can get the console to work and the rootfs to mount because 
I
just realized I forgot to do the steps from Julien's instructions of editing the
/etc/fstab and /etc/ttys files in the FreeBSD rootfs :

$ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
$ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")

I will add those and see if the console and disk are working.

But it would be much better if we could have this working with FreeBSD 13 / 14
instead of the old FreeBSD 11. Also, Julien's patch set only supports one vcpu
and it would be great to get the smp support for FreeBSD also.

Thanks,

Chuck

> 
> 
> 
> -- 
> Mario.