Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx>]

On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> On 11/27/2023 7:45 AM, Mario Marietto wrote:
>> @Chuck Zmudzinski <mailto:brchuckz@xxxxxxxxxxxx> : Stay tuned. They want to 
>> help us. The xen developers are great. Very good support for us. I'm sure 
>> that you can give a good contribution to understand what's our problem and 
>> how to implement a fix with the help of all those good guys.
>> 
>> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@xxxxxxxxxx 
>> <mailto:roger.pau@xxxxxxxxxx>> wrote:
>> 
>>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
>>     > +(xen-devel and Arm maintainers, including Julien)
>>     >
>>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@xxxxxxxxx 
>> <mailto:marietto2008@xxxxxxxxx>>
>>     > > wrote:
>>     > >
>>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>>     > > different. We want to virtualize FreeBSD as domU. Can we have a
>>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>>     > > would like to know if Julien's work was accepted upstream by
>>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>>     > > Julien's work was not accepted upstream by FreeBSD, we will have
>>     > > to find his patches and apply them ourselves to the FreeBSD on arm
>>     > > kernel.
>> 
>>     I've added Elliot on Cc as he is working on upstreaming the patches to
>>     FreeBSD.  He will be able to provide a better update than myself.
>> 
>>     Regards, Roger.
> 
> I have been collaborating with Mario, and I can explain what we have done so 
> far :
> 
> We are using Julien's patch set against an old development version of FreeBSD 
> 11
> from 2014-12-03 :
> 
> https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> 
> We successfully built the XENVIRT kernel and FreeBSD world, and created the
> FreeBSD rootfs according to Julien's instructions here :
> 
> https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> 
> There were some adjustments to the instructions :
> 
> To build the kernel, we used :
> 
> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> 
> instead of
> 
> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> 
> The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> some time trying to convert it to the zImage format without realizing the
> build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage 
> format.
> So when booting with the 'kernel.bin' file instead, it actually boots :
> 
> user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> Parsing config from freebsd.cfg
> user@devuan-bunsen ~ % sudo xl li
> Name                                        ID   Mem VCPUs    State   Time(s)
> Domain-0                                     0   768     2     r-----    
> 1439.4
> freebsd                                      1  1152     1     r-----       
> 3.0
> user@devuan-bunsen ~ %
> 
> However, the guest is still not working correctly :
> 
> 1. Attaching the console with the -c option at creation or with
>    'xl console freebsd' results in no output to the console.
> 
> 2. The timestamp on the virtual disk image file shows that the filesystem
>    was at best mounted read-only, if it was mounted at all by the guest
>    FreeBSD kernel.
> 
> 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
>    the guest, you need to do 'xl destroy freebsd'.
> 
> However, I think we can get the console to work and the rootfs to mount 
> because I
> just realized I forgot to do the steps from Julien's instructions of editing 
> the
> /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> 
> $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
> $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on 
> secure")
> 
> I will add those and see if the console and disk are working.

Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
/etc/fstab did not make the console or disk work. Still no output on the
xen console from the guest kernel, and the timestamp on the rootfs image
file did not change so it did not mount read-write.

We could use some advice for troubleshooting this. Now, we are blind because
we are not getting any xen console output But I am pleased we were able to
demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
stable Xen 4.17 packages.

Cheers

> 
> But it would be much better if we could have this working with FreeBSD 13 / 14
> instead of the old FreeBSD 11. Also, Julien's patch set only supports one vcpu
> and it would be great to get the smp support for FreeBSD also.
> 
> Thanks,
> 
> Chuck
> 
>> 
>> 
>> 
>> -- 
>> Mario.
>