Re: [Xen-devel] [RFC] Add support for Xen ARM guest on FreeBSD

[Nathan Whitehorn <nwhitehorn@xxxxxxxxxxx>]

On 01/18/14 21:08, Warner Losh wrote:
> On Jan 18, 2014, at 8:01 PM, Nathan Whitehorn wrote:
>
>> On 01/18/14 20:44, Warner Losh wrote:
>>> On Jan 18, 2014, at 4:41 PM, Julien Grall wrote:
>>>
>>>> Hello Nathan,
>>>>
>>>> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
>>>>> On 01/16/14 18:36, Julien Grall wrote:
>>>>>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>>>>> As I understand, only the simple bus code (see simplebus_attach) is
>>>>>> translating the interrupts in the device on a resource.
>>>>>> So if you have a node directly attached to the root node with
>>>>>> interrupts and MMIO, the driver won't be able to retrieve and
>>>>>> translate the interrupts via bus_alloc_resources.
>>>>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.
>>>> I have digged into the code to find the reason of my issue. FreeBSD is 
>>>> receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ.
>>>>
>>>> This is because the GIC is not yet initialized but FreeBSD asks to unmask 
>>>> the IRQ (sys/arm/arm/gic.c:306).
>>>>
>>>> With this problem, all device nodes that are before the GIC in the device 
>>>> tree can't have interrupts. For instance this simple device will segfault 
>>>> on FreeBSD:
>>>>
>>>> / {
>>>>
>>>> mybus {
>>>>    compatible = "simple-bus";
>>>>
>>>>    mynode {
>>>>       interrupt-parent = &gic;
>>>>       interrupts = <...>;
>>>>    };
>>>>
>>>>    gic: gic@xxxx {
>>>>       interrupt-controller;
>>>>    }
>>>> };
>>>> };
>>>>
>>>> The node "mynode" will have to move after the GIC to be able to work 
>>>> correctly.
>>> This stems from a difference in enumeration between FreeBSD and Linux. 
>>> FreeBSD enumerates the devices in DTB order, while Linux does a partial 
>>> ordering based on dependencies.
>>>
>>> Warner
>> Enumerating in some other order doesn't necessarily help: since the
>> interrupt and bus trees are independent, circular dependencies can
>> happen. This is not a hypothetical: on most powermacs, the main
>> interrupt controller is a functional unit on a PCI device -- a PCI
>> device whose other units have interrupt lines that eventually connect
>> back to itself. There is no way to fix that with ordering. So I think we
>> still need to defer interrupt setup. It's not that bad -- PPC already
>> does this to handle the powermac case.
> I guess I've looked at simpler cases where interrupts and GPIO pins need to 
> be up before anything can work on Atmel...  We kinda fake it now, but there's 
> some ordering issues that are solved in this way. But I've not finished the 
> work on bringing Atmel into the FDT world yet. Deferred setup isn't always an 
> option, but I'll keep that in mind in case I hit that case...
>
> The other way to cope is to use the multi-pass enumeration stuff that John 
> Baldwin put into the tree a while ago. In that case, you could enumerate 
> bridges, interrupt controllers, gpio providers, etc first, and then do  a 
> second pass that catches the rest of the devices and the interrupt processing 
> for the first pass devices. This is a variation on the deferred enumeration 
> stuff you are talking about, so that might be a better, more general solution 
> to these sorts of problems.
>
> Warner
>
>

I'm still not sure that helps. Take the powermac case. The PCI parent
bridge assigns and routes interrupts as part of its probe stage. One
device (called mac-io) is the one that actually has the interrupt
controller, along with other things like ATA and sound. It has several
interrupts to support e.g. ATA. How can the PCI bus attachment sensibly
attach the macio device? If it delays attaching it until the interrupt
controller registers, the bus probing will deadlock, since the interrupt
controller is itself a child of macio! If it attaches it immediately, it
will not be able to route the interrupts of the macio device. It's a
catch-22.

The only solution we were able to come up with when this situation arose
was to treat the bus and interrupt trees as entirely separate things,
built independently, which allows the kernel to break the loop. GPIOs
can have similar problems. The basic issue is that newbus ultimately
assumes that the system topology can be described by a single tree.
Interconnections between the branches -- especially if it isn't even a
DAG -- fundamentally break the model and multipass doesn't necessarily help.
-Nathan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel