Re: [Xen-devel] [Draft D] Xen on ARM vITS Handling

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Fri, 2015-06-05 at 11:37 +0530, Vijay Kilari wrote:
> > ## Virtual LPI interrupt injection
> >
> > A physical interrupt which is routed to a guest vCPU has the
> > `_IRQ_GUEST` flag set in the `irq_desc` status mask. Such interrupts
> > have an associated instance of `struct irq_guest` which contains the
> > target `struct domain` pointer and virtual interrupt number.
> >
> > In Xen a virtual interrupt (either arising from a physical interrupt
> > or completely virtual) is ultimately injected to a VCPU using the
> > `vgic_vcpu_inject_irq` function, or `vgic_vcpu_inject_spi`.
> >
> > This mechanism will likely need updating to handle the injection of
> > virtual LPIs. In particular rather than `GICD_ITARGERRn` or
> > `GICD_IROUTERn` routing of LPIs is performed via the ITS collections
> > mechanism. This is discussed below (In _vITS_:_Virtual LPI injection_).
> >
> 
> For 4.6 scope, we can inject LPIs always to VCPU0?

If that isn't going to be too upsetting for guest OSes we might be able
to get away with it. I can't remember if we did this for SPIs in the
early days also.


> > During assignment all LPIs associated with the device will be routed
> > to the guest (i.e. `route_irq_to_guest` will be called for each LPI in
> > the `struct its_device.events` array).
> 
>  Is there any hypercall where in we can assign & route LPI's to the guest?

There's:
        /*
         * Dom0 should use these two to announce MMIO resources assigned to
         * MSI-X capable devices won't (prepare) or may (release) change.
         */
        #define PHYSDEVOP_prepare_msix          30
        #define PHYSDEVOP_release_msix          31
        struct physdev_pci_device {
            /* IN */
            uint16_t seg;
            uint8_t bus;
            uint8_t devfn;
        };
which sort of is?

But, the paragraph above is proposing to do the routing on _device_
assignment, e.g. on XEN_DOMCTL_assign_device route all LPIs which we
have prellocated to the device to the designated domain.

> > ## vITS to pITS mapping
> >
> > A physical system may have multiple physical ITSs.
> >
> > With the simplified vits command model presented here only a single
> > `vits` is required.
> >
> > In the future a more complex arrangement may be desired. Since the
> > choice of model is internal to the hypervisor/tools and is
> > communicated to the guest via firmware tables we are not tied to this
> > model as an ABI if we decide to change.
> 
> Do you mean that for 4.6 1:N model is followed? why not
> 1:1 vITS to pITS mapping?

In the model here there isn't really a mapping from vITS to pITS at all,
that was one of the simplifications I tried to make, since it avoids all
the stuff about scheduling etc.

> > Since everything is based upon IPA (guest addresses) a malicious guest
> > can only reference its own RAM here.
> 
>    Here device table memory allocated by guest is used to lookup for the 
> device.
> Why can't we avoid using the guest memory all together and just only emulate
> read and write of GITS_BASER0.

Emulate how? All as GITS_BASERn.Valid == 0 for all?

> Let Xen manage device information using its_device structure based on
> MAPD command.

Between C and D I had a version which was like this, and used the
per-domain device map to lookup the its_device.

The problem with it is that the INT command can be given a DeviceID
which doesn't really exist (in order to generate completion interrupts
etc).

But a non-existent DeviceID won't have a struct its device to go with
it, hence there was no way to translate the Event into an LPI.

> am I missing any spec where in it specifies format of Device table
> managed by ITS hw which software can read/update directly?

AIUI the software is never supposed to read/update the table directly,
irrespective of whether it is technically able to. Accessing those bits
directly would be a bug and have UNPREDICTABLE results. The whole
GITS_BASERn stuff is just a mechanism to avoid the h/w having to
provision RAM specifically for the ITS by allowing it to take over
portions of the real RAM.

I think you are correct that in principal a hardware designer would be
allowed to throw some RAM into the ITS h/w block itself and not request
anything via GITS_BASERn, as I say I had an interim version which did
this (using the per domain device map rather than RAM in the ITS h/w
block), but this didn't handle the INT on a non-existing/invented device
case.

I decided to handle that case by moving the data structure into guest
RAM (using the mechanisms provided to do so), which I considered to have
the benefit of being reasonably simple, but it's not the only possible
answer.

As an alternative I suppose we could e.g. insert dummy struct its_device
entries into the per-domain virtual deviceid mapping table in response
to `MPAD` requests for guest invented devices. Then we would need a
mechanism to stop a guest from populating all 2^32 entries in the tree
(which would use a lot of Xen guest RAM) and ideally we would want to
avoid allocating dummy struct its_device during emulation.

Perhaps each guest could have a small pool of "invented its_devices"
allocated at start of day (small == say, 4) which could be inserted into
the device table on MAPD, once they are all used further MAPD commands
for invented devices would be ignored (this assumes that most guests
will only want one such device for completion purposes), which also
solves the problem of filling the tree completely (or does it: a
sequence of MAPD commands setting and clearing all 2^32 devices might
cause us to populate the whole tree).

Do you have any other ideas? I was trying to keep things simple and
using a guest hosted table seemed to achieve that.

> > Again since this is IPA based a malicious guest can only point things
> > to its own ram.
> 
> Same as above comment.  Xen can manage vlpi<=>plpi mapping
> for each device. This info can also be used in enabling/disableing
> of LPI (LPI configuration table update). The size of this vlpi can
> be equal to number of vectors supported by the device.

I think the vpli is chosen by the guest, isn't it? So the mapping from
vlpi->plpi is under guest control, and not restricted to the number of
vectors provided by the device. I suppose we could maintain another tree
data structure, which might.

> > ## Command Queue Virtualisation
> >
> > The command translation/emulation in this design has been arranged to
> > be as cheap as possible (e.g. in many cases the actions are NOPs),
> > avoiding previous concerns about the length of time which an emulated
> > write to a `CWRITER` register may block the vcpu.
> >
> > The vits will simply track its reader and writer pointers. On write
> > to `CWRITER` it will immediately and synchronously process all
> > commands in the queue and update its state accordingly.
> 
>  you mean for 4.6 we don't use any completion INT or softirq approach?

I've deliberately left the operation of the pits driver out of scope for
this iteration of the spec. So far as the vits is concerned it makes
irq_enable (or whatever) calls to the core Xen interrupt handling code,
but it doesn't then care about how it is achieved, the physical driver
owns the physical its (or multiple of them) and is free to implement
things in whichever way it deems best, in isolation from any particular
vits design.

This was one of the major simplifications made in this iteration, since
all the vits->pits scheduling etc stuff was causing .

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel