Re: [Xen-devel] [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu <xumengpanda@xxxxxxxxx>]

2015-02-23 10:57 GMT-05:00 Wei Liu <wei.liu2@xxxxxxxxxx>:
> Not an expert on scheduler, so bear with me if my questions / comments
> are stupid.

This is more about the toolstack (xl, libxl, libxc), so you are
definitely the expert. :-P

>
> On Sat, Feb 21, 2015 at 10:51:59AM -0500, Meng Xu wrote:
>> Hi all,
>>
>> This is for Xen 4.6: Enabling XL to set the parameters of each
>> individual VCPU of a domain for RTDS scheduler.
>>
>> [Goal]
>> Right now, xl sched-rtds tool can only set the VCPUs of a domain to
>> the same parameter although the scheduler did support VCPUs with
>> different parameters. This work is to provide the âxl sched-rtdsâ tool
>> the ability to configure the VCPUs of a domain with different
>> parameters.
>>
>> After this has been implemented, an example of the functionality of XL
>> will look like this:
>>
>> #xl sched-rtds
>> Cpupool Pool-0: sched=RTDS
>> Name                                ID VCPU Period Budget
>> Domain-0                             0    0  10000  10000
>> Domain-0                             0    1  20000  20000
>> Domain-0                             0    2  30000  30000
>> Domain-0                             0    3  10000  10000
>> litmus1                              1    0  10000   4000
>> litmus1                              1    1  10000   4000
>> //comment: VCPU is the index of the VCPUs in the domain. The
>> parameters of VCPUs in the same domain are different, while they are
>> the same in Xen 4.5.
>>
>> //set the parameters of the vcpu 1 of domain litmus1:
>> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000
>>
>> //domain litmus1's vcpu 1's parameters are changed, display each VCPU's
>> parameters separately:
>> # xl sched-rtds -d litmus1
>> Name                                ID VCPU Period Budget
>> litmus1                              1    0  10000   4000
>> litmus1                              1    1  20000  10000
>>
>> [Design sketch]
>> We submitted the patch along with the RTDS scheduler before last year.
>> The thread can be found at
>> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
>>
>> In order to implement the get function, which return the parameters of
>> all VCPUs of the same domain, we can bounce an array that has all
>> parameters of all VCPUs of the domain from hypervisor to userspace and
>> output it. (This part didn't cause much buzz. :-) )
>> The other choice is using a hypercall to get the information of each
>> VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
>> get all information of this domain. This choice causes more overhead
>> since the number of hypercalls is linear to the number of VCPUs in a
>> domain. The first choice only needs one hypercall for a domain to get
>> all needed information.
>>
>> In order to implement the set function, which set the parameters of
>> one specific VCPU of a domain, we have two different approaches to
>> implement it:
>> Choice 1): When users set the parameters of k^th VCPUs of a domain,
>> which has x VCPUs in total, the toolstack passes "only" the modified
>> VCPUsâ information from userspace to hypervisor via hypercall;  So
>
> A new hypercall or existing hypercall?

We only need to change the current hypercall
(XEN_DOMCTL_SCHEDOP_setinfo) implementation:


>
>> this implementation does "not" need an array to bounce between
>> userspace and hypervisor. (The implementation in lilbxc was at
>> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
>> Choice 2): When users set the parameters of k^th VCPUs of a domain,
>> which has x VCPUs in total, the toolstack will create an array with
>> length x and set the information (period, budget) of the k^th element
>> in the array. The other elements in the array are marked as 0. After
>> the toolstack bounce the array into the hypervisor, it will only set
>> the parameters of the k^th VCPU.
>>
>
> One thing to consider is when you have hundreds of cpus the overhead
> might be too big? What's the amount of data you need to pass down to
> hypervisor?

Suppose a domain have k VCPUs, each VCPU has info: period (uint32_t),
budget(uint32_t), and index(uint32_t); So each hypercall will pass k *
(3*8Bytes) to the hypervisor.

>
>> My questions are:
>> 1) Should we allow users to set the parameters of "several" VCPUs with
>> one command:
>> For example, should we allow users to set the parameters of VCPU 1 and
>> 2 of domain litmus1 with one xl sched-rtds command?
>> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
>> IMO, we'd better not allow them to do this because 1) setting the
>> parameters of VCPUs of a domain should not be very frequent; 2) the
>> command will become quite long when users want to set the parameters
>> of many VCPUs of a domain.
>>
>> 2) Which design choice should we choose to implement the set function,
>> Choice 1 or Choice 2? or do you have better suggestion?
>>
>
> I have some general questions. Here in this email you only mention xl. I
> think in fact you mean xl + libxl? We surely want to enable other
> toolstacks to control specific parameters of schedulers as well.

Yes. I need to change xl + libxl + libxc + rt_dom_cntl() in hypervisor. :-)

I'm not sure if other schedulers need such functionality right now,
because the credit2 scheduler account for the credit based on each
domain instead of each VCPU. But if the scheduler will consider the
vcpu-level credit/budget accounting, this could be helpful.

>
> And I presume in the future other schedulers might also want to have
> some per-vcpu tuning interface. Is it possible to abstract out common
> interface in hypervisor and libxl to do that? Even if we can't sensible
> come up with a hypervisor interface to do that, we might still want to
> do that inside libxl. I.e. to provide some kind of common infrastructure
> that individual scheduler can hook in.

Good point! This should be possible. I think maybe we can do this step
by step: First have a concrete implementation for the RTDS scheduler,
and then we can extend it to a more general interface.

>
> This is all vague idea. If my idea is stupid feel free to disagree.

This is very useful and insightful idea, actually! :-)

Thank you very much!

best,

Meng

>
> Wei.
>
>> Thank you very much!
>>
>> best,
>>
>> Meng
>> -----------
>> Meng Xu
>> PhD Student in Computer and Information Science
>> University of Pennsylvania



-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel