Re: [Xen-devel] [PATCH v8 for-4.12 17/17] docs, argo: add design document for Argo

[Christopher Clark <christopher.w.clark@xxxxxxxxx>]

On Wed, Feb 6, 2019 at 2:31 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>
> On Wed, Feb 06, 2019 at 12:55:08AM -0800, Christopher Clark wrote:
> > Document provides a brief introduction to the Argo interdomain
> > communication mechanism and a detailed description of the granular
> > locking used within the Argo implementation.
> >
> > Signed-off-by: Christopher Clark <christopher.clark6@xxxxxxxxxxxxxx>
>
> Reviewed-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
>
> I think the document is fine, and can be expanded as more people
> interact with argos.

Thanks.

>
> > ---
> >  docs/designs/argo.pandoc | 448 
> > +++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 448 insertions(+)
> >  create mode 100644 docs/designs/argo.pandoc
> >
> > diff --git a/docs/designs/argo.pandoc b/docs/designs/argo.pandoc
> > new file mode 100644
> > index 0000000..2ce253b
> > --- /dev/null
> > +++ b/docs/designs/argo.pandoc
> > @@ -0,0 +1,448 @@
> > +# Argo
> > +
> > +## Introduction
> > +
> > +Argo is an interdomain communication mechanism. It provides Xen hypervisor
> > +primitives to transmit data between VMs, by performing data copies into
> > +receive memory rings registered by domains. It does not require memory
> > +sharing between VMs and does not use the grant tables or Xenstore.
> > +
> > +Argo has requirements for performance isolation between domains, to prevent
> > +negative performance impact from malicious or disruptive activity of other
> > +domains, or even other VCPUs of the same domain operating other rings.
> > +
> > +## Hypervisor-Mediated data eXchange (HMX)
> > +
> > +This term references inter-VM communication protocols that have this
> > +key architectural point: The hypervisor is responsible for performing the
> > +write of data into the guest-accessible memory buffer, in the manner
> > +according to the agreed transfer protocol. This structure ensures that
> > +there is strength to the transport mechanism, because the transmitting side
> > +of the communication is the hypervisor, which can be trusted by the 
> > receiver,
> > +and the buffer is isolated from access by any other potential sources
> > +outside the receiver.
> > +
> > +The receiver can trust that the hypervisor will:
> > +
> > +- Provide a protocol implementation adhering to hardware synchronization
> > +requirements for concurrent access to system memory by communicating
> > +components
> > +- Deliver data only from an approved source, enforcing policy for Mandatory
> > +Access Control.
> > +- Indicate the correct sender of the data.
> > +- Transmit only the intended data, adhering to the access protocol of the 
> > data
> > +structure in the buffer. If the memory region is being used as a ring, 
> > then:
> > +    - Data writes will only occur within the ring region that is indicated 
> > as
> > +    available for incoming data by the ring indexes.
> > +    - The indicated length of data written will exactly match the length of
> > +    data actually written.
> > +    - The write for each piece of data will occur only once.
> > +    - Data will be written sequentially in the order that it is sent.
> > +- Issue notification of data delivered correctly.
> > +
> > +This structure allows for augmentation by the hypervisor to identify the
> > +sending entity within the source VM, and then provide the receiver with
> > +assured context information about the data source. This enables the 
> > receiver
> > +to make decisions based on fine-grained knowledge of the source of the 
> > data.
> > +
> > +This structure is also of strong interest for nested virtualization:
> > +transport via the hypervisor can enable construction of efficient
> > +communications between VMs at different levels of nesting.
>
> This AFAICT also requires some kind of cooperation between
> hypervisors?

Yes, that is my understanding with the current interface. We've proposed
exploring Xen support for L0 and L1 KCONFIG'd configurations:
https://lists.xen.org/archives/html/xen-devel/2019-02/msg00811.html


> It might be good to expand why hypervisor mediated is better than shared
> memory in the nested virtualization case, since you mention this
> explicitly but don't give any details.

Yes, this doc will be expanded. In the meantime, Bromium's PSEC video covers
shared memory vs. copy-based primitives:
https://lists.xen.org/archives/html/xen-devel/2019-01/msg00946.html

My PSEC video talks about mandatory access control enforcement by Xen:
https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen


> > +## FAQ / Other Considerations
> > +
> > +### Why not have a single per-domain lock?
> > +
> > +Due to performance isolation / DoS avoidance: if there is a single 
> > per-domain
> > +lock, acquiring this lock will stall operations on other active rings 
> > owned by
> > +the domain. A malicious domain can loop registering and unregistering 
> > rings,
> > +without any consent by the targetted domain, which would experience 
> > decreased
> > +throughput due to the contention on the single per-domain lock.
>
> I'm not sure I see how this is prevented, there's still a
> rings_L2_rwlock that AFAICT needs to be write-locked when adding a
> ring,

Yes, ring registration results in a domain write-locking it's own
rings_L2_rwlock while that is done.

> and that would prevent any other accesses to the list of rings,
> thus stalling operations?

Yes, for the domain itself; so a malicious domain registering and
unregistering rings will affect the performance of its _own_ rings.
Ring (un/)registration is used for connection setup/teardown rather than
data transmission, so in common use is appreciably less frequent than
the send/notify data path operations which use a read lock instead,
allowing for concurrent use of the internal data structures.

> I guess they key point here (and what alleviates the issue listed
> above) is using multiple locks allows each one to be locked for
> smaller periods of time, thus reducing the DoS.

Yes, granular locks should be more available and reduce DoS potential.

XSM policy can constrain which other domains a given domain is able to
register rings to communicate with, and this further limits a domain's
ability to interact with the locks that protect another domain's data
structures.

Christopher

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel