Xen project Mailing List

[Xen-devel] [RFC v3]Proposal to allow setting up shared memory areas between VMs from xl config file

==================================================== 1. Motivation and Description ==================================================== Virtual machines use grant table hypercalls to setup a share page for inter-VMs communications. These hypercalls are used by all PV protocols today. However, very simple guests, such as baremetal applications, might not have the infrastructure to handle the grant table. This project is about setting up several shared memory areas for inter-VMs communications directly from the VM config file. So that the guest kernel doesn't have to have grant table support (in the embedded space, this is not unusual) to be able to communicate with other guests. ==================================================== 2. Implementation Plan: ==================================================== ====================================== 2.1 Introduce a new VM config option in xl: ====================================== 2.1.1 Design Goals ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The shared areas should be shareable among several (>=2) VMs, so every shared physical memory area is assigned to a set of VMs. Therefore, a “token” or “identifier” should be used here to uniquely identify a backing memory area. A string no longer than 128 bytes is used here to serve the purpose. The backing area would be taken from one domain, which we will regard as the "master domain", and this domain should be created prior to any other "slave domain"s. Again, we have to use some kind of tag to tell who is the "master domain". And the ability to specify the permissions and cacheability (and shareability for arm HVM's) of the pages to be shared should be also given to the user. 2.2.2 Syntax and Behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following example illustrates the syntax of the proposed config entry: In xl config file of vm1: static_shm = [ 'id=ID1, begin=0x100000, end=0x200000, role=master, arm_shareattr=inner, arm_inner_cacheattr=wb, arm_outer_cacheattr=wb, x86_cacheattr=wb, prot=ro', 'id=ID2, begin=0x300000, end=0x400000, role=master, arm_shareattr=inner, arm_inner_cacheattr=wb, arm_outer_cacheattr=wb, x86_cacheattr=wb, prot=rw' ] In xl config file of vm2: static_shm = [ 'id=ID1, begin=0x500000, end=0x600000, role=slave, prot=ro' ] In xl config file of vm3: static_shm = [ 'id=ID2, begin=0x700000, end=0x800000, role=slave, prot=ro' ] where: @id can be any string that matches the regexp "[^ \t\n,]+" and no logner than 128 characters @begin/end can be decimals or hexidemicals of the form "0x20000". @role can only be 'master' or 'slave' @prot can be 'n', 'r', 'ro', 'w', 'wo', 'x', 'xo', 'rw', 'rx', 'wx' or 'rwx'. Default is 'rw'. @arm_shareattr can be 'inner' our 'outter', this will be ignored and a warning will be printed out to the screen if it is specified in an x86 HVM config file. Default is 'inner' @arm_outer_cacheattr can be 'uc', 'wt', 'wb', 'bufferable' or 'wa', this will be ignored and a warning will be printed out to the screen if it is specified in an x86 HVM config file. Default is 'inner' @arm_inner_cacheattr can be 'uc', 'wt', 'wb', 'bufferable' or 'wa'. Default is 'wb'. @x86_cacheattr can be 'uc', 'wc', 'wt', 'wp', 'wb' or 'suc'. Default is 'wb'. Besides, the sizes of the areas specified by @begin and @end in the slave domain's config file should be smaller than the corresponding sizes specified in its master's domain. And overlapping backing memory areas are allowed. In the example above. A memory area ID1 will be shared between vm1 and vm2. This area will be taken from vm1 and mapped into vm2's stage-2 page table. The parameter "prot=ro" means that this memory area are offered with read-only permission. vm1 can access this area using 0x100000~0x200000, and vm2 using 0x500000~0x600000. Likewise, a memory area ID2 will be shared between vm1 and vm3 with read and write permissions. vm1 is the master and vm2 the slave. vm1 can access the area using 0x300000~0x400000 and vm3 using 0x700000~0x800000. For the arm_* and x86_* cache attributes and shareability attributes, the behavior is briefly described below: + The the permission flags (i.e. ro/wo/rw etc.): - If specified in the master domains' config, they describe the largest set of permissions that are granted to the shared memory area, which means if master says 'rw' in its own config file, then the slaves can only say 'r' or 'w' or 'rw', but not 'x'. - If specified in the slave domains' config, they describe the stage-2 page permissions that would be used when we map the shared pages into the slave But this doesn't make any restrictions on how the slave domains are going to manipulate the related stage-1 page tables (and we can't). + The cacheability flags and shareability flags: These are valid only if they are specified in the master domain's config files. They also control the stage-2 page attributes of the shared memory. Note that the "master" role in vm1 for both ID1 and ID2 indicates that vm1 should be created prior to both vm2 and vm3, for they both rely on the pages backed by vm1. If one tries to create vm2 or vm3 prior to vm1, she will get an error. And in vm1's config file, the "prot=ro" parameter of ID1 indicates that if one tries to share this page with vm1 with, say, "rw" permission, she will get an error, too. ====================================== 2.2 Store the mem-sharing information in xenstore ====================================== For we don't have some persistent storage for xl to store the information of the shared memory areas, we have to find some way to keep it between xl launches. And xenstore is a good place to do this. The information for one shared area should include the ID, master's domid, address range, memory attributes and information of the slaves etc. A current plan is to place the information under /local/shared_mem/ID. Still take the above config files as an example: Suppose we are running under x86 (and thus the arm_* attributes will be ignored, if we instantiate vm1, vm2 and vm3, one after another, “xenstore ls -f” should output something like this: After VM1 was instantiated, the output of “xenstore ls -f” will be something like this: /local/shared_mem/ID1/master = domid_of_vm1 /local/shared_mem/ID1/begin = 0x100 /local/shared_mem/ID1/end = 0x200 /local/shared_mem/ID1/permissions = "r" /local/shared_mem/ID1/x86_cacheattr = "wb" /local/shared_mem/ID1/slaves = "" /local/shared_mem/ID2/master = domid_of_vm1 /local/shared_mem/ID2/begin = 0x300 /local/shared_mem/ID2/end = 0x400 /local/shared_mem/ID2/permissions = "rw" /local/shared_mem/ID1/x86_cacheattr = "wb" /local/shared_mem/ID2/slaves = "" After VM2 was instantiated, the following new lines will appear: /local/shared_mem/ID1/slaves/domid_of_vm2/begin = 0x500 /local/shared_mem/ID1/slaves/domid_of_vm2/end = 0x600 /local/shared_mem/ID1/slaves/domid_of_vm2/permissions = "r" After VM2 was instantiated, the following new lines will appear: /local/shared_mem/ID2/slaves/domid_of_vm3/gmfn_begin = 0x700 /local/shared_mem/ID2/slaves/domid_of_vm3/gmfn_end = 0x800 /local/shared_mem/ID2/slaves/domid_of_vm3/permissions = "rw" When we encounter an id IDx during "xl create": + If it’s not under /local/shared_mem: + If the the corresponding entry has "role=master", create the corresponding entries for IDx in xenstore + If there isn't a "master" tag, say error. + If it’s found under /local/shared_mem: + If the corresponding entry has a "master" tag, say error + If there isn't a "master" tag, map the pages to the newly created domain, and add the current domain and necessary information under /local/shared_mem/IDx/slaves. Locks should be used to make sure that the creation of these entries are atomic. ====================================== 2.3 mapping the memory areas ====================================== Handle the newly added config option in tools/{xl, libxl} and utilize toos/libxc to do the actual memory mapping. Specifically, we will use xc_domain_add_to_physmap_batch with XENMAPSPACE_gmfn_foreign to do the actual mapping. Unfortunately, we don't have the suitable API to change the catcheability and shareability attributes of the shared memory pages in the stage-2 page table. So these attributes are currently marked as "not implemented", and xl should print an error if any of these attributes are set to their non-default values (See 2.2.2 Syntax and Behavior). They will be implemented when a suitable API becomes available. ====================================== 2.4 error handling ====================================== Add code to handle various errors: Invalid address, invalid permissions, wrong order of vm creation, mismatched length of memory area etc. ==================================================== 3. Expected Outcomes/Goals: ==================================================== A new VM config option in xl will be introduced, allowing users to setup several shared memory areas for inter-VMs communications. This should work on both x86 and ARM. ==================================================== 3. Future Directions: ==================================================== Implement the prot, x86_* and arm_* memory attribute options. Set up a notification channel between domains who are communicating through shared memory regions, this allows one vm to signal her friends when data is available in the shared memory or when the data in the shared memory is consumed. The channel could be built upon PPI or SGI. [See also: https://wiki.xenproject.org/wiki/Outreach_Program_Projects#Share_a_page_in_memory_from_the_VM_config_file] Cheers, Zhongze Liu _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx https://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.