Re: [Xen-devel] [RFC v3]Proposal to allow setting up shared memory areas between VMs from xl config file

[Zhongze Liu <blackskygg@xxxxxxxxx>]

Hi Stefano,

I missed some of your comments in the last reply.  adding responses to them.

2017-07-20 2:47 GMT+08:00 Stefano Stabellini <sstabellini@xxxxxxxxxx>:
> On Wed, 19 Jul 2017, Zhongze Liu wrote:
>> ====================================================
>> 1. Motivation and Description
>> ====================================================
>> Virtual machines use grant table hypercalls to setup a share page for
>> inter-VMs communications. These hypercalls are used by all PV
>> protocols today. However, very simple guests, such as baremetal
>> applications, might not have the infrastructure to handle the grant table.
>> This project is about setting up several shared memory areas for inter-VMs
>> communications directly from the VM config file.
>> So that the guest kernel doesn't have to have grant table support (in the
>> embedded space, this is not unusual) to be able to communicate with
>> other guests.
>>
>> ====================================================
>> 2. Implementation Plan:
>> ====================================================
>>
>> ======================================
>> 2.1 Introduce a new VM config option in xl:
>> ======================================
>>
>> 2.1.1 Design Goals
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> The shared areas should be shareable among several (>=2) VMs, so every shared
>> physical memory area is assigned to a set of VMs. Therefore, a “token” or
>> “identifier” should be used here to uniquely identify a backing memory area.
>> A string no longer than 128 bytes is used here to serve the purpose.
>>
>> The backing area would be taken from one domain, which we will regard
>> as the "master domain", and this domain should be created prior to any
>> other "slave domain"s. Again, we have to use some kind of tag to tell who
>> is the "master domain".
>>
>> And the ability to specify the permissions and cacheability (and shareability
>> for arm HVM's) of the pages to be shared should be also given to the user.
>>
>> 2.2.2 Syntax and Behavior
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> The following example illustrates the syntax of the proposed config entry:
>>
>> In xl config file of vm1:
>>
>>    static_shm = [ 'id=ID1, begin=0x100000, end=0x200000, role=master,
>>                    arm_shareattr=inner, arm_inner_cacheattr=wb,
>>                    arm_outer_cacheattr=wb, x86_cacheattr=wb, prot=ro',
>>
>>                    'id=ID2, begin=0x300000, end=0x400000, role=master,
>>                    arm_shareattr=inner, arm_inner_cacheattr=wb,
>>                    arm_outer_cacheattr=wb, x86_cacheattr=wb, prot=rw' ]
>
> Probably not a good idea to mix x86 and arm attributes in the example :-)
> Just make a couple of examples instead.

OK. I'll separate this into two examples.

>
>
>> In xl config file of vm2:
>>
>>     static_shm = [ 'id=ID1, begin=0x500000, end=0x600000, role=slave, 
>> prot=ro' ]
>>
>> In xl config file of vm3:
>>
>>     static_shm = [ 'id=ID2, begin=0x700000, end=0x800000, role=slave, 
>> prot=ro' ]
>>
>> where:
>>   @id                   can be any string that matches the regexp "[^ 
>> \t\n,]+"
>>                         and no logner than 128 characters
>>   @begin/end            can be decimals or hexidemicals of the form 
>> "0x20000".
>>   @role                 can only be 'master' or 'slave'
>>   @prot                 can be 'n', 'r', 'ro', 'w', 'wo', 'x', 'xo', 'rw', 
>> 'rx',
>>                         'wx' or 'rwx'. Default is 'rw'.
>>   @arm_shareattr        can be 'inner' our 'outter', this will be ignored and
>>                         a warning will be printed out to the screen if it
>>                         is specified in an x86 HVM config file.
>>                         Default is 'inner'
>>   @arm_outer_cacheattr  can be 'uc', 'wt', 'wb', 'bufferable' or 'wa', this 
>> will
>>                         be ignored and a warning will be printed out to the
>>                         screen if it is specified in an x86 HVM config file.
>>                         Default is 'inner'
>>   @arm_inner_cacheattr  can be 'uc', 'wt', 'wb', 'bufferable' or 'wa'. 
>> Default
>>                         is 'wb'.
>
> I don't think we need both @arm_outer_cacheattr and
> @arm_inner_cacheattr: a single @arm_cacheattr should suffice.
>
> Also, we need to explain what each of these values mean. Instead, I
> would only say that today we only support write-back:
>
> @arm_cacheattr  Only 'wb' (write-back) is supported today.
>
> In the code I would check that arm_cacheattr is either missing, or set
> to 'wb'. Throw an error in all other cases.

I'm not sure whether I should first list out all the flags that are *supposed
to be* accepted and then mark some of the flags unavailable or just simply
list only the flags that are currently available.

>
>
>>   @x86_cacheattr        can be 'uc', 'wc', 'wt', 'wp', 'wb' or 'suc'. Default
>>                         is 'wb'.
>
> Also here, I would write:
>
> @x86_cacheattr  Only 'wb' (write-back) is supported today.
>
> Like you wrote later, begin and end addresses need to be multiple of 4K.
>
>
>> Besides, the sizes of the areas specified by @begin and @end in the slave
>> domain's config file should be smaller than the corresponding sizes specified
>> in its master's domain. And overlapping backing memory areas are allowed.
>>
>> In the example above. A memory area ID1 will be shared between vm1 and vm2.
>> This area will be taken from vm1 and mapped into vm2's stage-2 page table.
>> The parameter "prot=ro" means that this memory area are offered with 
>> read-only
>> permission. vm1 can access this area using 0x100000~0x200000, and vm2 using
>> 0x500000~0x600000.
>>
>> Likewise, a memory area ID2 will be shared between vm1 and vm3 with read and
>> write permissions. vm1 is the master and vm2 the slave. vm1 can access the
>> area using 0x300000~0x400000 and vm3 using 0x700000~0x800000.
>>
>> For the arm_* and x86_* cache attributes and shareability attributes, the
>> behavior is briefly described below:
>>
>>   + The the permission flags (i.e. ro/wo/rw etc.):
>>     - If specified in the master domains' config, they describe the largest 
>> set
>>       of permissions that are granted to the shared memory area, which means 
>> if
>>       master says 'rw' in its own config file, then the slaves can only say 
>> 'r'
>>       or 'w' or 'rw', but not 'x'.
>>     - If specified in the slave domains' config, they describe the stage-2 
>> page
>>       permissions that would be used when we map the shared pages into the 
>> slave
>>       But this doesn't make any restrictions on how the slave domains are 
>> going
>>       to manipulate the related stage-1 page tables (and we can't).
>>   + The cacheability flags and shareability flags:
>>     These are valid only if they are specified in the master domain's config
>>     files. They also control the stage-2 page attributes of the shared 
>> memory.
>>
>> Note that the "master" role in vm1 for both ID1 and ID2 indicates that vm1
>> should be created prior to both vm2 and vm3, for they both rely on the pages
>> backed by vm1. If one tries to create vm2 or vm3 prior to vm1, she will get
>> an error. And in vm1's config file, the "prot=ro" parameter of ID1 indicates
>> that if one tries to share this page with vm1 with, say, "rw" permission,
>> she will get an error, too.
>>
>> ======================================
>> 2.2 Store the mem-sharing information in xenstore
>> ======================================
>> For we don't have some persistent storage for xl to store the information
>> of the shared memory areas, we have to find some way to keep it between xl
>> launches. And xenstore is a good place to do this. The information for one
>> shared area should include the ID, master's domid, address range,
>> memory attributes and information of the slaves etc.
>> A current plan is to place the information under /local/shared_mem/ID.
>> Still take the above config files as an example:
>>
>> Suppose we are running under x86 (and thus the arm_* attributes will be 
>> ignored,
>> if we instantiate vm1, vm2 and vm3, one after another, “xenstore ls -f” 
>> should
>> output something like this:
>>
>> After VM1 was instantiated, the output of “xenstore ls -f”
>> will be something like this:
>>
>>     /local/shared_mem/ID1/master = domid_of_vm1
>>     /local/shared_mem/ID1/begin = 0x100
>>     /local/shared_mem/ID1/end = 0x200
>>     /local/shared_mem/ID1/permissions = "r"
>>     /local/shared_mem/ID1/x86_cacheattr = "wb"
>>     /local/shared_mem/ID1/slaves = ""
>>
>>     /local/shared_mem/ID2/master = domid_of_vm1
>>     /local/shared_mem/ID2/begin = 0x300
>>     /local/shared_mem/ID2/end = 0x400
>>     /local/shared_mem/ID2/permissions = "rw"
>>     /local/shared_mem/ID1/x86_cacheattr = "wb"
>>     /local/shared_mem/ID2/slaves = ""
>>
>> After VM2 was instantiated, the following new lines will appear:
>>
>>     /local/shared_mem/ID1/slaves/domid_of_vm2/begin = 0x500
>>     /local/shared_mem/ID1/slaves/domid_of_vm2/end = 0x600
>>     /local/shared_mem/ID1/slaves/domid_of_vm2/permissions = "r"
>>
>> After VM2 was instantiated, the following new lines will appear:
>>
>>     /local/shared_mem/ID2/slaves/domid_of_vm3/gmfn_begin = 0x700
>>     /local/shared_mem/ID2/slaves/domid_of_vm3/gmfn_end = 0x800
>>     /local/shared_mem/ID2/slaves/domid_of_vm3/permissions = "rw"
>>
>>
>> When we encounter an id IDx during "xl create":
>
> I take that you mean that a static_shm option was given to xl create,
> right?

Yes. I'll make the expression more precise.

>
>
>>   + If it’s not under /local/shared_mem:
>>     + If the the corresponding entry has "role=master", create the
>>       corresponding entries for IDx in xenstore
>>     + If there isn't a "master" tag, say error.
>>
>>   + If it’s found under /local/shared_mem:
>>     + If the corresponding entry has a "master" tag, say error
>>     + If there isn't a "master" tag, map the pages to the newly
>>       created domain, and add the current domain and necessary information
>>       under /local/shared_mem/IDx/slaves.
>>
>> Locks should be used to make sure that the creation of these entries are
>> atomic.
>>
>> ======================================
>> 2.3 mapping the memory areas
>> ======================================
>> Handle the newly added config option in tools/{xl, libxl} and utilize
>> toos/libxc to do the actual memory mapping. Specifically, we will use
>> xc_domain_add_to_physmap_batch with XENMAPSPACE_gmfn_foreign to
>> do the actual mapping.
>
> What are we going to use to set the permissions RO or RW?
>
>
>> Unfortunately, we don't have the suitable API to change the catcheability
>> and shareability attributes of the shared memory pages in the stage-2
>> page table. So these attributes are currently marked as "not implemented",
>> and xl should print an error if any of these attributes are set to their
>> non-default values (See 2.2.2 Syntax and Behavior).
>>
>> They will be implemented when a suitable API becomes available.
>>
>> ======================================
>> 2.4 error handling
>> ======================================
>> Add code to handle various errors: Invalid address, invalid permissions, 
>> wrong
>> order of vm creation, mismatched length of memory area etc.
>>
>> ====================================================
>> 3. Expected Outcomes/Goals:
>> ====================================================
>> A new VM config option in xl will be introduced, allowing users to setup
>> several shared memory areas for inter-VMs communications.
>> This should work on both x86 and ARM.
>>
>> ====================================================
>> 3. Future Directions:
>> ====================================================
>> Implement the prot, x86_* and arm_* memory attribute options.
>>
>> Set up a notification channel between domains who are communicating through
>> shared memory regions, this allows one vm to signal her friends when data is
>> available in the shared memory or when the data in the shared memory is
>> consumed. The channel could be built upon PPI or SGI.
>>
>>
>> [See also:
>> https://wiki.xenproject.org/wiki/Outreach_Program_Projects#Share_a_page_in_memory_from_the_VM_config_file]
>>
>>
>> Cheers,
>>
>> Zhongze Liu
>>

Cheers,

Zhongze Liu

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel