Re: [Xen-devel] [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Wei Liu <wei.liu2@xxxxxxxxxx>]

Not an expert on scheduler, so bear with me if my questions / comments
are stupid.

On Sat, Feb 21, 2015 at 10:51:59AM -0500, Meng Xu wrote:
> Hi all,
> 
> This is for Xen 4.6: Enabling XL to set the parameters of each
> individual VCPU of a domain for RTDS scheduler.
> 
> [Goal]
> Right now, xl sched-rtds tool can only set the VCPUs of a domain to
> the same parameter although the scheduler did support VCPUs with
> different parameters. This work is to provide the âxl sched-rtdsâ tool
> the ability to configure the VCPUs of a domain with different
> parameters.
> 
> After this has been implemented, an example of the functionality of XL
> will look like this:
> 
> #xl sched-rtds
> Cpupool Pool-0: sched=RTDS
> Name                                ID VCPU Period Budget
> Domain-0                             0    0  10000  10000
> Domain-0                             0    1  20000  20000
> Domain-0                             0    2  30000  30000
> Domain-0                             0    3  10000  10000
> litmus1                              1    0  10000   4000
> litmus1                              1    1  10000   4000
> //comment: VCPU is the index of the VCPUs in the domain. The
> parameters of VCPUs in the same domain are different, while they are
> the same in Xen 4.5.
> 
> //set the parameters of the vcpu 1 of domain litmus1:
> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000
> 
> //domain litmus1's vcpu 1's parameters are changed, display each VCPU's
> parameters separately:
> # xl sched-rtds -d litmus1
> Name                                ID VCPU Period Budget
> litmus1                              1    0  10000   4000
> litmus1                              1    1  20000  10000
> 
> [Design sketch]
> We submitted the patch along with the RTDS scheduler before last year.
> The thread can be found at
> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
> 
> In order to implement the get function, which return the parameters of
> all VCPUs of the same domain, we can bounce an array that has all
> parameters of all VCPUs of the domain from hypervisor to userspace and
> output it. (This part didn't cause much buzz. :-) )
> The other choice is using a hypercall to get the information of each
> VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
> get all information of this domain. This choice causes more overhead
> since the number of hypercalls is linear to the number of VCPUs in a
> domain. The first choice only needs one hypercall for a domain to get
> all needed information.
> 
> In order to implement the set function, which set the parameters of
> one specific VCPU of a domain, we have two different approaches to
> implement it:
> Choice 1): When users set the parameters of k^th VCPUs of a domain,
> which has x VCPUs in total, the toolstack passes "only" the modified
> VCPUsâ information from userspace to hypervisor via hypercall;  So

A new hypercall or existing hypercall?

> this implementation does "not" need an array to bounce between
> userspace and hypervisor. (The implementation in lilbxc was at
> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
> Choice 2): When users set the parameters of k^th VCPUs of a domain,
> which has x VCPUs in total, the toolstack will create an array with
> length x and set the information (period, budget) of the k^th element
> in the array. The other elements in the array are marked as 0. After
> the toolstack bounce the array into the hypervisor, it will only set
> the parameters of the k^th VCPU.
> 

One thing to consider is when you have hundreds of cpus the overhead
might be too big? What's the amount of data you need to pass down to
hypervisor?

> My questions are:
> 1) Should we allow users to set the parameters of "several" VCPUs with
> one command:
> For example, should we allow users to set the parameters of VCPU 1 and
> 2 of domain litmus1 with one xl sched-rtds command?
> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
> IMO, we'd better not allow them to do this because 1) setting the
> parameters of VCPUs of a domain should not be very frequent; 2) the
> command will become quite long when users want to set the parameters
> of many VCPUs of a domain.
> 
> 2) Which design choice should we choose to implement the set function,
> Choice 1 or Choice 2? or do you have better suggestion?
> 

I have some general questions. Here in this email you only mention xl. I
think in fact you mean xl + libxl? We surely want to enable other
toolstacks to control specific parameters of schedulers as well.

And I presume in the future other schedulers might also want to have
some per-vcpu tuning interface. Is it possible to abstract out common
interface in hypervisor and libxl to do that? Even if we can't sensible
come up with a hypervisor interface to do that, we might still want to
do that inside libxl. I.e. to provide some kind of common infrastructure
that individual scheduler can hook in.

This is all vague idea. If my idea is stupid feel free to disagree.

Wei.

> Thank you very much!
> 
> best,
> 
> Meng
> -----------
> Meng Xu
> PhD Student in Computer and Information Science
> University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel