Re: [Xen-devel] [PATCH v2 00/21] xen/arm: Add support for non-pci passthrough

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Wed, 2014-09-10 at 11:22 +0200, Christoffer Dall wrote:
> On Tue, Sep 9, 2014 at 9:34 PM, Julien Grall <julien.grall@xxxxxxxxxx> wrote:
> > (Adding Christoffer)
> >
> > Hi Ian,
> >
> > On 09/09/14 07:34, Ian Campbell wrote:
> >>
> >> On Thu, 2014-07-31 at 16:00 +0100, Julien Grall wrote:
> >>>
> >>>      - Only common device properties (interrupts, regs) are written to
> >>>      the guest device tree. Device that needs other properties may not
> >>> work.
> >>
> >>
> >> So I've glanced through the later (more toolstack oriented) bits from
> >> towards the end but I think there's a question of the target users which
> >> needs thinking about before I can have a sensible opinion on those.
> >>
> >> As I see it the main purpose of this series is to get the underlying
> >> plumbing in place (wiring up iommus, routing IRQs etc) to support guests
> >> with passthrough devices, for embedded folks to use and to provide a
> >> basis for eventual PCI passthrough functionality. I really want to see
> >> this stuff in 4.5
> >>
> >> What I'm concerned about is the toolstack side. TBH I'm not very keen on
> >> the thing with exposing very DT specific stuff like compatible strings
> >> down from the hypervisor via domctls.
> >> It's not really clear how best to expose this functionality, I have a
> >> feeling that this series either goes too far or not far enough and ends
> >> up not really satisfying anyone.
> >
> >
> > I don't see many other solutions to get the compatible strings. There is no
> > easy way to get the property from DOM0, unless we introduce a new driver in
> > Linux.
> >
> 
> The toolstack you are using to create your guest must necessarily know
> which guest it is creating, including device properties of a device a
> user wishes to assign  It can know this because it's hardcoded,
> included in some config files, or supplied directly by the user.  I
> think you really want to decouple the hardware description method for
> Dom0 from retrieving resource description about your device. Can't you
> simply reference the device to Linux through its sysfs handle and use
> your Xen-passthrough-layer-hypercall-magic (which I know nothing
> about) to have Dom0 tell Xen to map/route the relevant resources?

By Xen-passthrough-layer-hypercall-magic do you mean the thing which
lets the userspace toolstack make hypercalls (which is called "privcmd"
FWIW) or are you talking about some specific passtrhough related kernel
driver (like VFIO? which has no Xen equivalent right now)

If you mean the former then I think Julien's code already does this --
it makes hypercalls telling Xen to map certain MMIO regions to guests.
What's in question is where the inputs to those hypercalls came from.

> >> My suspicion is that regular folks won't really be using passthrough
> >> until it is via PCI and that in the meantime this functionality is only
> >> going to be used by e.g. people building embedded system and superkeen
> >> early adopters both of whom know what they are doing and can tolerate
> >> some hacks etc to get things working (and I think that's fine, it's
> >> still a worthwhile set of things to get into 4.5 and those folks are
> >> worth supporting).
> >>
> >> I'm also worried that we may be committing ourselves to a libxl API
> >> already without really working through all the issues (e.g. other
> >> properties).
> >>
> >> Given that I wonder if we wouldn't be better off for 4.5 supporting
> >> something much simpler at the toolstack level, namely allowing users to
> >> use iomem= and irq= in their domain config to map platform devices
> >> through (already works with your series today?)
> >
> >
> > This would need a bit a plumbing for irq part to allow the user choosing the
> > VIRQ (like Arianna did for MMIO range).
> >
> 
> My Xen knowledge is limited here.  Is the iomem= and irq= commands
> given to your Dom0 toolstack, Dom0, or Xen itself?

They are things that the user can write into the guest cfg file,
containing lists of the relevant resources. e.g. to pass IRQ 42 to the
guest: irqs = [ 42 ]

The toolstack parses that and makes hypercalls while building a guest to
tell Xen to map those regions through.

>   How does a user
> know which physical address in Xen's physical address space needs to
> be remapped based on the hardware description language for Dom0?

This is an interesting question. The short answer for platform device
type things is "they just know" (they presumably have datasheets etc).
But I think the underlying question here is whether they are given in PA
space or dom0 IPA space, right?

The way that this works on x86 is that dom0 sees the real underlying
MMIO addresses in e.g. PCI BARs and /proc/iomem etc. So the hypercalls
to map MMIO regions to guests all take real physical addresses and
nothing has to worry about IPA vs PA issues.

On ARM things are potentially a bit more complex because dom0 is running
with second stage paging. However we always map the MMIO regions 1:1 for
dom0, and I think we will always have to do that (the "1:1 workaround"
refers to RAM regions only). So I think we can continue to treat these
things as proper physical addresses and don't need to introduce variants
of the hypercalls which work in terms of IPAs (or you could argue that
they already do and the translation is currently a nop).

> It still seems to me that you need to abstract the simple concept of a
> device passthrough vector (device handle, associated MMIO regions,
> associated IRQs).  That would extend more nicely to PCI/ACPI as well.
> Am I missing something?

I don't think so.

For PCI the toolstack would read sysfs to get at the BAR info which
tells us the real physical addresses and it would then program those
using the existing mechanisms.

> >> and perhaps a back door
> >>
> >> to allow the injection a blob of DT into the guest's DT to describe
> >> them. i.e. enough to actually get stuff done but not pretending to be
> >> too finely integrated.
> >
> 
> can we not treat the problem of how to describe hardware to the guest
> independently?

Yes, that's essentially what I am proposing, get the basics working via
iomem= and irq= now and leave the tricky problem of describing platform
hardware until later.

>   For this matter, I still think you need some way to
> retrieve the canonical information specific to your running instance
> (mmio regions, irqs) and then you need to be able to create a DT/ACPI
> tables as you want.

I think might be a future nice to have but m specifically advocating
that we not do it for 4.5. For 4.5 I think it will be acceptable to give
the user the parts which they need to put something together, which is:
iomem=/irq= (existing Xen interfaces), the dtc/iasl compiler and some
way to add a predefined blob to the guests generated firmware tables.

Trying to invent up an interface like you suggest simply isn't going to
happen in time for 4.5 (the freeze of which was going to be today but
has been slipped by a couple of weeks). And regardless of that at least
iomem= and irq= are things which we should support anyway, since are
existing Xen interfaces.

> > I would be fine with this solution for Xen 4.5. I will give a look to see
> > what could be done.
> >
> >> Then we can revisit the "proper" toolstack side for 4.6. Otherwise I
> >> fear that by the time we get the toolstack side sorted out to our
> >> satisfaction the basic functionality (which seems to be largely done)
> >> will have missed 4.5.
> >
> >
> > Depending of the use cases, your solution based on "iomem", "irq" and DT
> > blob might be enough for device platform passthrough.
> >
> 
> Hmmm, I don't really know what the Xen development policy is, but I am
> generally not very much for creating a system solution on an OS level
> that addresses a very limited use case without any sort of generic
> abstractions...

In the case of passthrough we prove both low level ways of configuring
things and higher level abstractions built on top of them. e.g.
iomem=/irq= vs pci= in a guest cfg file. iomem=/irq= let you assign
arbitrary platform resources (subject to permissions etc) to a guest and
in theory you could assign a PCI device manually using them. pci= lets
you just say "assign this device please" and the toolstack sorts out the
rest, in reality this is the one people use for PCI passthrough (for the
obvious reason). On x86 iomme=/irq= get used for things like passing
through serial ports to guests.

The problem we are having here is that we have no existing higher level
abstraction for platform devices (because on x86 everyone just knows the
address of the std serial ports ;-)). We can consider adding one in the
future, but we should still be providing iomem=/irq= anyway and that is
something which can realistically be done for 4.5.

So I think we should leave the question of what a higher level
abstraction for passing through platform devices aside for now since it
is a distraction from getting something useful in for 4.5, which is on a
very tight timescale right now.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel