Re: [Xen-devel] myrinet dma

[Ian Pratt <Ian.Pratt@xxxxxxxxxxxx>]

> There are some folks here interested in using Myrinet with Xen.
> Applications using Myrinet link against a library that registers memory
> and DMAs directly to it. The library loads the memory addresses onto the
> card which is setting off some alarms and we're attempting to think
> about exactly how XenoLinux guests will handle this situation.

Presumably it's not just a library, and there's some trusted OS
component that pins the physical pages and registers their bus
addresses with the Myrinet NIC?

I guess the library sets up a mmap of some of the NICs control
registers, then provides a set of library functions to do
open/close, send/receive/RDMA etc and provide block-until-receive
functionality.
 
> What memory issues might arise using this in a guest domain with
> privileged drivers, in non privileged domains (with one privileged
> domain actually using the libraries), and with multiple privileged
> domains simultaneously?  Is it possible?

Interesting.

Assuming I'm right about there being a trusted OS component that
deals with the creation/deletion of memory apertures, you'd want
exactly one of these, running in e.g. domain 0. You'd then need
to create a way of virtualising this functionality to other
domains, so their OS component doesn't talk to the card directly
but talks to the controlling domain that will then interact with
Xen's mmu to check the pages actually belong to the domain, and
then pin them and register them with the NIC.

This is going to require a little coding, but shouldn't be too
hard. It's quite an interesting problem, so I'd be happy to help
with the design. It's something we'll have to do for inifiniband
anyhow. 

> I would think in the case that the one privileged domain bridging to
> other, non-privileged domains that full speed transfers would be
> impossible if there is any copying necessary.

I presume it's also possible to use the Myrinet card as a plain
ehernet/ip interface with a suitable kernel driver (yes, I know
this sucks)?

You could use this to enable the privileged domain to at least
provide network connectivity to other domains using the normal
netback/netfront drivers. (the privileged domain could also use
the normal library directly).  This would be zero-copy into the
domain, but the normal OS stack would usually end up copying
things into the application socket buffer.

> But would several privileged guests using the libraries be able to
> coexist?  Would there be swapping and memory pinning issues?  The card
> returns a port, one of several (I believe six).  Would it be possible
> for six guests to each have access to one of these channels?

Only six ports? That's a bit lame. I'd like to see the memory
mapped communication extended right down into user-space
applications in multiple domains, but six doesn't give a whole
lot of flexibility...

Ian


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel