Re: [Xen-devel] RFC: XenSock brainstorming

[Stefano Stabellini <stefano@xxxxxxxxxxx>]

Although discussing the goals is fun, feedback on the design of the
protocol is particularly welcome.

On Thu, 23 Jun 2016, Stefano Stabellini wrote:
> Now that Xen 4.7 is out of the door, any more feedback on this?
> 
> On Mon, 6 Jun 2016, Stefano Stabellini wrote:
> > Hi all,
> > 
> > a couple of months ago I started working on a new PV protocol for
> > virtualizing syscalls. I named it XenSock, as its main purpose is to
> > allow the implementation of the POSIX socket API in a domain other than
> > the one of the caller. It allows connect, accept, recvmsg, sendmsg, etc
> > to be implemented directly in Dom0. In a way this is conceptually
> > similar to virtio-9pfs, but for sockets rather than filesystem APIs.
> > See this diagram as reference:
> > 
> > https://docs.google.com/presentation/d/1z4AICTY2ejAjZ-Ul15GTL3i_wcmhKQJA7tcXwhI3dys/edit?usp=sharing
> > 
> > The frontends and backends could live either in userspace or kernel
> > space, with different trade-offs. My current prototype is based on Linux
> > kernel drivers but it would be nice to have userspace drivers too.
> > Discussing where the drivers could be implemented it's beyond the scope
> > of this email.
> > 
> > 
> > # Goals
> > 
> > The goal of the protocol is to provide networking capabilities to any
> > guests, with the following added benefits:
> > 
> > * guest networking should work out of the box with VPNs, wireless
> >   networks and any other complex network configurations in Dom0
> > 
> > * guest services should listen on ports bound directly to Dom0 IP
> >   addresses, fitting naturally in a Docker based workflow, where guests
> >   are Docker containers
> > 
> > * Dom0 should have full visibility on the guest behavior and should be
> >   able to perform inexpensive filtering and manipulation of guest calls
> > 
> > * XenSock should provide excellent performance. Unoptimized early code
> >   reaches 22 Gbit/sec TCP single stream and scales to 60 Gbit/sec with 3
> >   streams.
> > 
> > 
> > # Status
> > 
> > I would like to get feedback on the high level architecture, the data
> > path and the ring formats.
> > 
> > Beware that protocol and drivers are in their very early days. I don't
> > have all the information to write a design document yet. The ABI is
> > neither complete nor stable.
> > 
> > The code is not ready for xen-devel yet, but I would be happy to push a
> > git branch if somebody is interested in contributing to the project.
> > 
> > 
> > # Design and limitations
> > 
> > The frontend connects to the backend following the traditional xenstore
> > based exchange of information.
> > 
> > Frontend and backend setup an event channel and shared ring. The ring is
> > used by the frontend to forward socket API calls to the backend. I am
> > referring to this ring as command ring. This is an example of the ring
> > format:
> > 
> > #define XENSOCK_CONNECT        0
> > #define XENSOCK_RELEASE        3
> > #define XENSOCK_BIND           4
> > #define XENSOCK_LISTEN         5
> > #define XENSOCK_ACCEPT         6
> > #define XENSOCK_POLL           7
> > 
> > struct xen_xensock_request {
> >     uint32_t id;     /* private to guest, echoed in response */
> >     uint32_t cmd;    /* command to execute */
> >     uint64_t sockid; /* id of the socket */
> >     union {
> >             struct xen_xensock_connect {
> >                     uint8_t addr[28];
> >                     uint32_t len;
> >                     uint32_t flags;
> >                     grant_ref_t ref[XENSOCK_DATARING_PAGES];
> >                     uint32_t evtchn;
> >             } connect;
> >             struct xen_xensock_bind {
> >                     uint8_t addr[28]; /* ipv6 ready */
> >                     uint32_t len;
> >             } bind;
> >             struct xen_xensock_accept {
> >                     grant_ref_t ref[XENSOCK_DATARING_PAGES];
> >                     uint32_t evtchn;
> >                     uint64_t sockid;
> >             } accept;
> >     } u;
> > };
> > 
> > struct xen_xensock_response {
> >     uint32_t id;
> >     uint32_t cmd;
> >     uint64_t sockid;
> >     int32_t ret;
> > };
> > 
> > DEFINE_RING_TYPES(xen_xensock, struct xen_xensock_request,
> >               struct xen_xensock_response);
> > 
> > 
> > Connect and accept lead to the creation of new active sockets. Today
> > each active socket has its own event channel and ring for sending and
> > receiving data. Data rings have the following format:
> > 
> > #define XENSOCK_DATARING_ORDER 2
> > #define XENSOCK_DATARING_PAGES (1 << XENSOCK_DATARING_ORDER)
> > #define XENSOCK_DATARING_SIZE (XENSOCK_DATARING_PAGES << PAGE_SHIFT)
> > 
> > typedef uint32_t XENSOCK_RING_IDX;
> > 
> > struct xensock_ring_intf {
> >     char in[XENSOCK_DATARING_SIZE/4];
> >     char out[XENSOCK_DATARING_SIZE/2];
> >     XENSOCK_RING_IDX in_cons, in_prod;
> >     XENSOCK_RING_IDX out_cons, out_prod;
> >     int32_t in_error, out_error;
> > };
> > 
> > The ring works like the Xen console ring (see
> > xen/include/public/io/console.h). Data is copied to/from the ring by
> > both frontend and backend. in_error, out_error are used to report
> > errors. This simple design works well, but it requires at least 1 page
> > per active socket. To get good performance (~20 Gbit/sec single stream),
> > we need buffers of at least 64K, so actually we are looking at about 64
> > pages per ring (order 6).
> > 
> > I am currently investigating the usage of AVX2 to perform the data copy.
> > 
> > 
> > # Brainstorming
> > 
> > Are 64 pages per active socket a reasonable amount in the context of
> > modern OS level networking? I believe that regular Linux tcp sockets
> > allocate something in that order of magnitude.
> > 
> > If that's too much, I spent some time thinking about ways to reduce it.
> > Some ideas follow.
> > 
> > 
> > We could split up send and receive into two different data structures. I
> > am thinking of introducing a single ring for all active sockets with
> > variable size messages for sending data. Something like the following:
> > 
> > struct xensock_ring_entry {
> >     uint64_t sockid; /* identifies a socket */
> >     uint32_t len;    /* length of data to follow */
> >     uint8_t data[];  /* variable length data */
> > };
> >  
> > One ring would be dedicated to holding xensock_ring_entry structures,
> > one after another in a classic circular fashion. Two indexes, out_cons
> > and out_prod, would still be used the same way the are used in the
> > console ring, but I would place them on a separate page for clarity:
> > 
> > struct xensock_ring_intf {
> >     XENSOCK_RING_IDX out_cons, out_prod;
> > };
> > 
> > The frontend, that is the producer, writes a new struct
> > xensock_ring_entry to the ring, careful not to exceed the remaining free
> > bytes available. Then it increments out_prod by the written amount. The
> > backend, that is the consumer, reads the new struct xensock_ring_entry,
> > reading as much data as specified by "len". Then it increments out_cons
> > by the size of the struct xensock_ring_entry read.
> > 
> > I think this could work. Theoretically we could do the same thing for
> > receive: a separate single ring shared by all active sockets. We could
> > even reuse struct xensock_ring_entry.
> > 
> > 
> > However I have doubts that this model could work well for receive. When
> > sending data, all sockets on the frontend side copy buffers onto this
> > single ring. If there is no room, the frontend returns ENOBUFS. The
> > backend picks up the data from the ring and calls sendmsg, which can
> > also return ENOBUFS. In that case we don't increment out_cons, leaving
> > the data on the ring. The backend will try again in the near future.
> > Error messages would have to go on a separate data structure which I
> > haven't finalized yet.
> > 
> > When receiving from a socket, the backend copies data to the ring as
> > soon as data is available, perhaps before the frontend requests the
> > data. Buffers are copied to the ring not necessarily in the order that
> > the frontend might want to read them. Thus the frontend would have to
> > copy them out of the common ring into private per-socket dynamic buffers
> > just to free the ring as soon as possible and consume the next
> > xensock_ring_entry. It doesn't look very advantageous in terms of memory
> > consumption and performance.
> > 
> > Alternatively, the frontend would have to leave the data on the ring if
> > the application didn't ask for it yet. In that case the frontend could
> > look ahead without incrementing the in_cons pointer. It would have to
> > keep track of which entries have been consumed and which entries have
> > not been consumed. Only when the ring is full, the frontend would have
> > no other choice but to copy the data out of the ring into temporary
> > buffers. I am not sure how well this could work in practice.
> > 
> > As a compromise, we could use a single shared ring for sending data, and
> > 1 ring per active socket to receive data. This would cut the per-socket
> > memory consumption in half (maybe to a quarter, moving out the indexes
> > from the shared data ring into a separate page) and might be an
> > acceptable trade-off.
> > 
> > Any feedback or ideas?
> > 
> > 
> > Many thanks,
> > 
> > Stefano
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel