Re: [Xen-devel] Request for input: Extended event channel support

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

On 27 Mar 2013, at 21:53, David Vrabel <dvrabel@xxxxxxxxxx> wrote:
> On 27/03/2013 19:36, Anil Madhavapeddy wrote:
>> On 27 Mar 2013, at 11:23, George Dunlap <dunlapg@xxxxxxxxx> wrote:
>>> 
>>> The FIFO solution makes event delivery a matter of adding items to a
>>> highly structured linked list.  The number of event channels for the
>>> interface design has a theoretical maximum of 2^28; the current
>>> implementation is limimited at 2^17, which is over 100,000.  The
>>> number is the same for both 32-bit and 64-bit kernels.
>> 
>> Is there any reason for such a low default?  If I'm not mistaken,
>> every guest needs at least 2 event channels (console, xenstore) and
>> probably has two more for a net and disk device.
> 
> 131,072 seemed high enough to me but I'd forgotten about the Mirage use
> case.
> 
> This can be trivially raised to 2^19 (524,288).  Beyond that, the
> implementation becomes slightly more complex as the pointers to the
> event array pages no longer fit in a single page.

Makes sense.

>> With stub-domains in the mix, we could easily imagine running 25,000
>> VMs with a couple of megabytes of RAM each using Mirage (which can
>> boot very low memory guests without too much trouble).
> 
> Having said that, with 25,000 VMs it would seem sensible to disaggregate
> things like the console and xenstore (in addition to the network and
> block backends). Thus reducing the need for event channels for any
> single domain.


Yeah indeed; this should be pretty easy to do and let the existing
2^17 be enough for a long time too.  We'd need to think a bit about a
distributed xenstored to avoid having one hotspot servicing so many
VMs.

One nice thing about the OCaml xenstored is that it should be possible
to make an explicitly distributed implementation of the protocol.
The data-structure is already based around immutable trees, so it's a
matter of figuring out where to put the consensus logic (probably around
/local/domain/*).

-anil