[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Re: [RFC, PATCH 1/24] i386 Vmi documentation

Chris Wright wrote:
Yup.  Just noting that API without clear users is the type of thing that
is regularly rejected from Linux.

Yes. It is becoming clear from feedback from you and Andi that there are things in the API that are unnecessary for Linux. But keep in mind, they may be necessary for other operating systems. I think we should probably drop the Linux changes to issue things like RDTSC and such via VMI call wrappers. It does simplify the Linux interface.

But I still think they should be part of the spec - an optional part of the spec, that need not be implemented by Linux or even by the hypervisor. If some vendor or kernel combination finds that they are a performance concern, as they readily could become, they can drop in the functionality when and if they need it. No reason to complicate things on either end, but also no reason to purposely add asymmetry to the spec just because the current set of calls is sufficient for the currently known fast paths.

Many of these will look the same on x86-64, but the API is not
64-bit clean so has to be duplicated.
Yes, register pressure forces the PAE API to be slightly different from the long mode API. But long mode has different register calling conventions anyway, so it is not a big deal. The important thing is, once the MMU mess is sorted out, the same interface can be used from C code for both platforms, and the details about which lock primitives are used can be hidden. The cost of which lock primitives to use differs on 32-bit and 64-bit platforms, across vendor, and the style of the hypervisor implementation (direct / writable / shadowed page tables).

My mistake, it makes perfect sense from ABI point of view.

Is this the batching, multi-call analog?
Yes. This interface needs to be documented in a much better fashion. But the idea is that VMI calls are mapped into Xen multicalls by allowing deferred completion of certain classes of operations. That same mode of deferred operation is used to batch PTE updates in our implementation (although Xen uses writable page tables now, this used to provide the same support facility in Xen as well). To complement this, there is an explicit flush - and it turns out this maps very nicely, getting rid of a lot of the XenoLinux changes around mmu_context.h.

Are these valid differences?  Or did I misunderstand the batching

1) can't use stack based args, so have to allocate each data structure,
which could conceivably fail unless it's some fixed buffer.

We use a fixed buffer that is private to our VMI layer. It's a per-cpu packing struct for hypercalls. Dynamically allocating from the kernel inside the interface layer is a really great way to get into a whole lot of trouble.

2) complicates the rom implementation slightly where implementation of
each deferrable part of the API needs to have switch (am I deferred or
not) to then build the batch, or make direct hypercall.

This is an overhead that is easily absorbed by the gain. The direct hypercalls are mostly either always direct, or always queued. The page table updates already have conditional logic to do the right thing, and Xen doesn't require the queueing of these anymore anyways. And the flush happens at an explicit point. The best approach can still be fine tuned. You could have separate VMI calls for queued vs. non-queued operation. But that greatly bloats the interface and doesn't make sense for everything. I believe the best solution is to annotate this in the VMI call itself. Consider the VMI call number, not as an integer, but as an identifier tuple. Perhaps I'm going overboard here. Perhaps not.

| family   | call number | reserved  | annotation |

Now, you have multiple families of calls -

0x00 legacy
0x01 CPU
0x02 Segmentation
0x03 MMU
0xFF reserved for experimentation

And each family has children:

0x03 MMU:
  0x00  SetPTE
  0x01  SetLongPTE
  0x02  FlushTLB

Now, lets say I add a new feature, and I don't want to redefine part of the interface. Lets say that feature is queuing of hypercalls. I have this private, annotation field as part of the identifier for each hypercall - in effect, really just the hypercall number.

And I don't want to break binary compatibility of the interface. So what I do is I define a new annotation that is specific to the affected calls.
  0x00  SetPTE
       0x00 - no annotation
       0x01 - may be queued !

Now, the hypercall isn't any different. Hypervisors which are unware of the annotation treat it no differently. But hypervisors that support PTE queuing recognize it as a hint and use it appropriately.

Queuing is a common enough optimization that it might even make sense to have a bit set aside in the call ID for it. Having this type of static annotation allows you to get rid of the dynamic concerns you have.

The really nice thing about defining your interface this way is you have a hierarchy of different classes of the interface, with the ability to add new classes, new calls within a class, and new annotations (upgrades, if you will) or those calls. And it provides a natural way to query for supported families of support - do you support a virtual event channel? Should I do some extra work to give you MMU hints or not? And you can add extra, optional functionality on to existing call sites. Something vary useful, if you say, realize that you want to add a hint field to one of your calls without breaking the old interface or forcing another vendor into complicating the ir hypervisor. Which is most of what paravirtualization is anyway. Extra, optionally used hints about how things are being used that allow the hypervisor implementation to avoid making costly assumptions to ensure correctness under unknown constraints.

Is this worth threshing out more? I think so, since it does provide a nice value proposition as well as overcoming the rather clumsy top level versioning scheme.

Thanks again for your feedback,


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.