[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [RFC v3]Proposal to allow setting up shared memory areas between VMs from xl config file
2017-07-20 2:47 GMT+08:00 Stefano Stabellini <sstabellini@xxxxxxxxxx>: > On Wed, 19 Jul 2017, Zhongze Liu wrote: >> ==================================================== >> 1. Motivation and Description >> ==================================================== >> Virtual machines use grant table hypercalls to setup a share page for >> inter-VMs communications. These hypercalls are used by all PV >> protocols today. However, very simple guests, such as baremetal >> applications, might not have the infrastructure to handle the grant table. >> This project is about setting up several shared memory areas for inter-VMs >> communications directly from the VM config file. >> So that the guest kernel doesn't have to have grant table support (in the >> embedded space, this is not unusual) to be able to communicate with >> other guests. >> >> ==================================================== >> 2. Implementation Plan: >> ==================================================== >> >> ====================================== >> 2.1 Introduce a new VM config option in xl: >> ====================================== >> >> 2.1.1 Design Goals >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> The shared areas should be shareable among several (>=2) VMs, so every shared >> physical memory area is assigned to a set of VMs. Therefore, a “token” or >> “identifier” should be used here to uniquely identify a backing memory area. >> A string no longer than 128 bytes is used here to serve the purpose. >> >> The backing area would be taken from one domain, which we will regard >> as the "master domain", and this domain should be created prior to any >> other "slave domain"s. Again, we have to use some kind of tag to tell who >> is the "master domain". >> >> And the ability to specify the permissions and cacheability (and shareability >> for arm HVM's) of the pages to be shared should be also given to the user. >> >> 2.2.2 Syntax and Behavior >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> The following example illustrates the syntax of the proposed config entry: >> >> In xl config file of vm1: >> >> static_shm = [ 'id=ID1, begin=0x100000, end=0x200000, role=master, >> arm_shareattr=inner, arm_inner_cacheattr=wb, >> arm_outer_cacheattr=wb, x86_cacheattr=wb, prot=ro', >> >> 'id=ID2, begin=0x300000, end=0x400000, role=master, >> arm_shareattr=inner, arm_inner_cacheattr=wb, >> arm_outer_cacheattr=wb, x86_cacheattr=wb, prot=rw' ] > > Probably not a good idea to mix x86 and arm attributes in the example :-) > Just make a couple of examples instead. > > >> In xl config file of vm2: >> >> static_shm = [ 'id=ID1, begin=0x500000, end=0x600000, role=slave, >> prot=ro' ] >> >> In xl config file of vm3: >> >> static_shm = [ 'id=ID2, begin=0x700000, end=0x800000, role=slave, >> prot=ro' ] >> >> where: >> @id can be any string that matches the regexp "[^ >> \t\n,]+" >> and no logner than 128 characters >> @begin/end can be decimals or hexidemicals of the form >> "0x20000". >> @role can only be 'master' or 'slave' >> @prot can be 'n', 'r', 'ro', 'w', 'wo', 'x', 'xo', 'rw', >> 'rx', >> 'wx' or 'rwx'. Default is 'rw'. >> @arm_shareattr can be 'inner' our 'outter', this will be ignored and >> a warning will be printed out to the screen if it >> is specified in an x86 HVM config file. >> Default is 'inner' >> @arm_outer_cacheattr can be 'uc', 'wt', 'wb', 'bufferable' or 'wa', this >> will >> be ignored and a warning will be printed out to the >> screen if it is specified in an x86 HVM config file. >> Default is 'inner' >> @arm_inner_cacheattr can be 'uc', 'wt', 'wb', 'bufferable' or 'wa'. >> Default >> is 'wb'. > > I don't think we need both @arm_outer_cacheattr and > @arm_inner_cacheattr: a single @arm_cacheattr should suffice. > > Also, we need to explain what each of these values mean. Instead, I > would only say that today we only support write-back: > > @arm_cacheattr Only 'wb' (write-back) is supported today. > > In the code I would check that arm_cacheattr is either missing, or set > to 'wb'. Throw an error in all other cases. > > >> @x86_cacheattr can be 'uc', 'wc', 'wt', 'wp', 'wb' or 'suc'. Default >> is 'wb'. > > Also here, I would write: > > @x86_cacheattr Only 'wb' (write-back) is supported today. > > Like you wrote later, begin and end addresses need to be multiple of 4K. > > >> Besides, the sizes of the areas specified by @begin and @end in the slave >> domain's config file should be smaller than the corresponding sizes specified >> in its master's domain. And overlapping backing memory areas are allowed. >> >> In the example above. A memory area ID1 will be shared between vm1 and vm2. >> This area will be taken from vm1 and mapped into vm2's stage-2 page table. >> The parameter "prot=ro" means that this memory area are offered with >> read-only >> permission. vm1 can access this area using 0x100000~0x200000, and vm2 using >> 0x500000~0x600000. >> >> Likewise, a memory area ID2 will be shared between vm1 and vm3 with read and >> write permissions. vm1 is the master and vm2 the slave. vm1 can access the >> area using 0x300000~0x400000 and vm3 using 0x700000~0x800000. >> >> For the arm_* and x86_* cache attributes and shareability attributes, the >> behavior is briefly described below: >> >> + The the permission flags (i.e. ro/wo/rw etc.): >> - If specified in the master domains' config, they describe the largest >> set >> of permissions that are granted to the shared memory area, which means >> if >> master says 'rw' in its own config file, then the slaves can only say >> 'r' >> or 'w' or 'rw', but not 'x'. >> - If specified in the slave domains' config, they describe the stage-2 >> page >> permissions that would be used when we map the shared pages into the >> slave >> But this doesn't make any restrictions on how the slave domains are >> going >> to manipulate the related stage-1 page tables (and we can't). >> + The cacheability flags and shareability flags: >> These are valid only if they are specified in the master domain's config >> files. They also control the stage-2 page attributes of the shared >> memory. >> >> Note that the "master" role in vm1 for both ID1 and ID2 indicates that vm1 >> should be created prior to both vm2 and vm3, for they both rely on the pages >> backed by vm1. If one tries to create vm2 or vm3 prior to vm1, she will get >> an error. And in vm1's config file, the "prot=ro" parameter of ID1 indicates >> that if one tries to share this page with vm1 with, say, "rw" permission, >> she will get an error, too. >> >> ====================================== >> 2.2 Store the mem-sharing information in xenstore >> ====================================== >> For we don't have some persistent storage for xl to store the information >> of the shared memory areas, we have to find some way to keep it between xl >> launches. And xenstore is a good place to do this. The information for one >> shared area should include the ID, master's domid, address range, >> memory attributes and information of the slaves etc. >> A current plan is to place the information under /local/shared_mem/ID. >> Still take the above config files as an example: >> >> Suppose we are running under x86 (and thus the arm_* attributes will be >> ignored, >> if we instantiate vm1, vm2 and vm3, one after another, “xenstore ls -f” >> should >> output something like this: >> >> After VM1 was instantiated, the output of “xenstore ls -f” >> will be something like this: >> >> /local/shared_mem/ID1/master = domid_of_vm1 >> /local/shared_mem/ID1/begin = 0x100 >> /local/shared_mem/ID1/end = 0x200 >> /local/shared_mem/ID1/permissions = "r" >> /local/shared_mem/ID1/x86_cacheattr = "wb" >> /local/shared_mem/ID1/slaves = "" >> >> /local/shared_mem/ID2/master = domid_of_vm1 >> /local/shared_mem/ID2/begin = 0x300 >> /local/shared_mem/ID2/end = 0x400 >> /local/shared_mem/ID2/permissions = "rw" >> /local/shared_mem/ID1/x86_cacheattr = "wb" >> /local/shared_mem/ID2/slaves = "" >> >> After VM2 was instantiated, the following new lines will appear: >> >> /local/shared_mem/ID1/slaves/domid_of_vm2/begin = 0x500 >> /local/shared_mem/ID1/slaves/domid_of_vm2/end = 0x600 >> /local/shared_mem/ID1/slaves/domid_of_vm2/permissions = "r" >> >> After VM2 was instantiated, the following new lines will appear: >> >> /local/shared_mem/ID2/slaves/domid_of_vm3/gmfn_begin = 0x700 >> /local/shared_mem/ID2/slaves/domid_of_vm3/gmfn_end = 0x800 >> /local/shared_mem/ID2/slaves/domid_of_vm3/permissions = "rw" >> >> >> When we encounter an id IDx during "xl create": > > I take that you mean that a static_shm option was given to xl create, > right? > > >> + If it’s not under /local/shared_mem: >> + If the the corresponding entry has "role=master", create the >> corresponding entries for IDx in xenstore >> + If there isn't a "master" tag, say error. >> >> + If it’s found under /local/shared_mem: >> + If the corresponding entry has a "master" tag, say error >> + If there isn't a "master" tag, map the pages to the newly >> created domain, and add the current domain and necessary information >> under /local/shared_mem/IDx/slaves. >> >> Locks should be used to make sure that the creation of these entries are >> atomic. >> >> ====================================== >> 2.3 mapping the memory areas >> ====================================== >> Handle the newly added config option in tools/{xl, libxl} and utilize >> toos/libxc to do the actual memory mapping. Specifically, we will use >> xc_domain_add_to_physmap_batch with XENMAPSPACE_gmfn_foreign to >> do the actual mapping. > > What are we going to use to set the permissions RO or RW? I should have added the permissions into the 'unfortunately' list below. > > >> Unfortunately, we don't have the suitable API to change the catcheability >> and shareability attributes of the shared memory pages in the stage-2 >> page table. So these attributes are currently marked as "not implemented", >> and xl should print an error if any of these attributes are set to their >> non-default values (See 2.2.2 Syntax and Behavior). >> >> They will be implemented when a suitable API becomes available. >> >> ====================================== >> 2.4 error handling >> ====================================== >> Add code to handle various errors: Invalid address, invalid permissions, >> wrong >> order of vm creation, mismatched length of memory area etc. >> >> ==================================================== >> 3. Expected Outcomes/Goals: >> ==================================================== >> A new VM config option in xl will be introduced, allowing users to setup >> several shared memory areas for inter-VMs communications. >> This should work on both x86 and ARM. >> >> ==================================================== >> 3. Future Directions: >> ==================================================== >> Implement the prot, x86_* and arm_* memory attribute options. >> >> Set up a notification channel between domains who are communicating through >> shared memory regions, this allows one vm to signal her friends when data is >> available in the shared memory or when the data in the shared memory is >> consumed. The channel could be built upon PPI or SGI. >> >> >> [See also: >> https://wiki.xenproject.org/wiki/Outreach_Program_Projects#Share_a_page_in_memory_from_the_VM_config_file] >> >> >> Cheers, >> >> Zhongze Liu >> Cheers, Zhongze Liu _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx https://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |