Re: [Xen-devel] [RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall <julien.grall@xxxxxxxxxx>]

On 01/15/2014 01:26 AM, Warner Losh wrote:
> 
> On Jan 14, 2014, at 2:01 PM, Julien Grall wrote:
>> This new support brings 2 open questions (for both Xen and FreeBSD 
>> community).
>> When a new guest is created, the toolstack will generated a device tree which
>> will contains:
>>      - The amount of memory
>>      - The description of the platform (gic, timer, hypervisor node)
>>      - PSCI node for SMP bringup
>>
>> Until now, Xen on ARM supported only Linux-based OS. When the support of
>> device tree was added in Xen for guest, we chose to use Linux device tree
>> bindings (gic, timer,...). It seems that FreeBSD choose a different way to
>> implement device tree:
>>      - strictly respect ePAR (for interrupt-parent property)
>>      - gic bindings different (only 1 interrupt cell)
>>
>> I would like to come with a common device tree specification (bindings, ...)
>> across every operating system. I know the Linux community is working on 
>> having
>> a device tree bindings out of the kernel tree. Does FreeBSD community plan to
>> work with Linux community for this purpose?
> 
> We generally try to follow the common definitions for the FDT stuff.
> There are a few cases where we either lack the feature set of Linux, or
> where the Linux folks are moving quickly and changing the underlying
definitions
> where we wait for the standards to mature before we implement. In some
cases, where
> maturity hasn't happened, or where the bindings are overly Linux
centric (which in
> theory they aren't supposed to be, but sometimes wind up that way).
where we've not
> implemented things.

As I understand main bindings (gic, timer) are set in stone and won't
change. Ian Campbell has a repository with all the ARM bindings here:
http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;
a=tree;f=Bindings/arm

To compare the difference between the DT provided by Xen, and the one
correctly parsed by FreeBSD
        - Xen: http://xenbits.xen.org/people/julieng/xenvm-4.2.dts
        - FreeBSD: http://xenbits.xen.org/people/julieng/xenvm-bsd.dts

From Xen side:
        - Every device should move under a simple-bus. I think it's harmless
for Linux side.
        - How about hypervisor node? IHMO this node should also live under the
simple-bus.

From FreeBSD side:
        - GIC and Timer bindings needs to be handled correctly (see below the
problem for the GIC)
        - Less stricter about interrupt-parent property. Eg, look at the
grant-parent if there is no property interrupt-parent
        - If the hypervisor doesn't live under simple-bus, the interrupt/memory
retrieving should be moved from simple-bus to nexus?

Before the revision r260282 (I have added Nathan on cc), I was able to
use the Linux GIC bindings (see
http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=blob;f=Bindings/arm/gic.txt)
without any issue.

It seems the fdt bus, now consider the number of interrupt cells is
always 1.

>> As specified early, the device tree can vary accross Xen version and user 
>> input
>> (eg the memory). I have noticed few places (mainly the timer) where the IRQs
>> number are harcoded.
> 
> These cases should be viewed as bugs in FreeBSD, I believe. One of the goals
> that has wide support, at leas tin theory, is that we can boot with an
unaltered
> Linux FDT. This goal is some time in the future.

The timer interrupt used by FreeBSD doesn't match the one used by Xen
(secure vs non-secure interrupt). This should be easily resolved by
retrieving the interrupt from the device tree.

Is there a particular reason to use the secure interrupt for the timer?

> 
>> The second question is related to memory attribute for page table. The early
>> page table setup by FreeBSD are using Write-Through memory attribute which
>> result to a likely random (not every time the same place) crash before the
>> real page table are initialized.
>> Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even 
>> today,
>> I have no idea if it's a requirement from Xen or a bug (either in Xen or
>> FreeBSD).
> 
> There were some problems with pages being setup improperly for the mutexes
> to operate properly. Have you confirmed this on a recent version of
FreeBSD?

I have checkout the freeBSD branch last week.

I'm not sure to understand the relation with mutexes. The symptoms I
have is mostly data/prefetch abort receive to FreeBSD. Could it be related?

>> The code is taking its first faltering steps, therefore the TODO is quite 
>> big:
>>      - Try the code on x86 architecture. I did lots of rework for the event
>>      channels code and didn't even try to compile
>>      - Clean up event channel code
>>      - Clean up xen control code
>>      - Add support for Device Tree loading via Linux Boot ABI
>>      - Fixing crashing on userspace. Could be related to the patch
>>      series "arm SMP on Cortex-A15"
>>      - Add guest SMP support
>>      - DOM0 support?
>>
>> Any help, comments, questions are welcomed.
> 
> I think this is great! We'd love to work with you to make this happen.
> 
> Warner
> 
>> Sincerely yours,
>>
>> ============= Instruction to test FreeBSD on Xen on ARM ===========
> [trimmed]
> 


-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel