Re: [Xen-devel] [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu <xumengpanda@xxxxxxxxx>]

2015-02-23 12:27 GMT-05:00 Dario Faggioli <dario.faggioli@xxxxxxxxxx>:
>> 2015-02-21 10:51 GMT-05:00 Meng Xu <xumengpanda@xxxxxxxxx>:
>> > Hi all,
>> >
> Hey Meng... First of all, thanks for keeping up working on this, we really 
> need it!

Since we promised, we will definitely keep our words. :-)
My plan is to have this done soon, since globallogic guys is also
waiting for it for some experiments. :-)

>
>> > This is for Xen 4.6: Enabling XL to set the parameters of each
>> > individual VCPU of a domain for RTDS scheduler.
>> >
> Right.
>
>> > [Goal]
>> > Right now, xl sched-rtds tool can only set the VCPUs of a domain to
>> > the same parameter although the scheduler did support VCPUs with
>> > different parameters. This work is to provide the âxl sched-rtdsâ tool
>> > the ability to configure the VCPUs of a domain with different
>> > parameters.
>> >
> Yes, for RTDS, we need this, much more than for the other available
> schedulers we have in Xen.
>
> The main reason, IMO, is that I see it quite common for someone to want
> to pin tasks to (v)cpus inside a real-time enabled VM, and if they do
> this, they expect to be able to specify the scheduling parameters for
> each one of the vcpus (or for a group of vcpus), depending on what tasks
> inside the guest they are pinning where.

Right! Especially when you want to have zero-schedule latency to
respond an interrupt. You can just have full capacity VCPU pinned to a
cpu and direct the interrupt to that VCPU.

>
>> > [Design sketch]
>> > We submitted the patch along with the RTDS scheduler before last year.
>> > The thread can be found at
>> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
>> >
>> > In order to implement the get function, which return the parameters of
>> > all VCPUs of the same domain, we can bounce an array that has all
>> > parameters of all VCPUs of the domain from hypervisor to userspace and
>> > output it. (This part didn't cause much buzz. :-) )
>> > The other choice is using a hypercall to get the information of each
>> > VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
>> > get all information of this domain. This choice causes more overhead
>> > since the number of hypercalls is linear to the number of VCPUs in a
>> > domain. The first choice only needs one hypercall for a domain to get
>> > all needed information.
>> >
> ISTR that at some point we agreed on the fact that, even if getting and
> setting vcpu params won't happen in hot paths (neither from the tools
> nor from the hypervisor point of view), batching hypercall is always
> good, an we should always batch as far as we can.

Yes.

>
> That of course does not mean that we the API, at all levels, need to
> always deal with all vcpus. As you know very well, there is the hcall
> interface, where we said we want to batch; then there is the libxc
> interface, which can well diverge, but it's usually better map quite
> closely to the hcall one; and then there is libxl, on top of libxl, and
> things can be significantly different here.
>
> So, I think your design proposal/code should explicitly state what the
> plan is at each level.

I agree.
In my mind, my plan is to have all levels batching the request, from
xl to libxl to libxc to rt_dom_cntl() in sched_rt.c.

>
>> > In order to implement the set function, which set the parameters of
>> > one specific VCPU of a domain, we have two different approaches to
>> > implement it:
>> > Choice 1): When users set the parameters of k^th VCPUs of a domain,
>> > which has x VCPUs in total, the toolstack passes "only" the modified
>> > VCPUsâ information from userspace to hypervisor via hypercall;  So
>> > this implementation does "not" need an array to bounce between
>> > userspace and hypervisor. (The implementation in lilbxc was at
>> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
>> > Choice 2): When users set the parameters of k^th VCPUs of a domain,
>> > which has x VCPUs in total, the toolstack will create an array with
>> > length x and set the information (period, budget) of the k^th element
>> > in the array. The other elements in the array are marked as 0. After
>> > the toolstack bounce the array into the hypervisor, it will only set
>> > the parameters of the k^th VCPU.
>> >
> Get and set interface should match, at each level, so we should decide
> once and for both of them, bearing in mind all the pros and the cons of
> each solution together, not split the two decisions.

I see. Then using batch seems a better choice to me.

>
>> > My questions are:
>> > 1) Should we allow users to set the parameters of "several" VCPUs with
>> > one command:
>> > For example, should we allow users to set the parameters of VCPU 1 and
>> > 2 of domain litmus1 with one xl sched-rtds command?
>> > # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
>> > IMO, we'd better not allow them to do this because 1) setting the
>> > parameters of VCPUs of a domain should not be very frequent; 2) the
>> > command will become quite long when users want to set the parameters
>> > of many VCPUs of a domain.
>> >
> I actually see some value in supporting such syntax. It's not something
> critical, though, I think. For sure it doesn't have much to do with the
> decision on how the interface should look like at the various level.
>
> This means you can just go ahead and send code implementing only one '-v
> xxx -p yy y -b zzz', and we can extend it later, adding the support for
> the list.
>
>> > 2) Which design choice should we choose to implement the set function,
>> > Choice 1 or Choice 2? or do you have better suggestion?
>> >
> If we batch for get, I think we should batch for set. As I said, I
> personally see some value in an interface allowing setting all the
> parameters of all the vcpus at the same time. In fact, this is probably
> better, even as far as real-time theory is concerned: as I'm sure you
> know better than me, parameter changing can be tricky for the algorithm,
> so it's probably better to encourage the user to change the params all
> together, synchronously, rather than having to deal with a stream of
> adjustments scattered in time.
>
> So, my opinion, choice 2).

OK. Agree! :-P
I think this is also the reason why libxl and xl should use batch as
well, for consistence?

>
> If I can add something, the big challenge here is how to make the
> interfaces good and homogeneous enough with what we already have in
> place for other schedulers. I really suggest you to address this too in
> this design conversation and/or in RFC/code.

Actually, there is a basic question:
Should we back support the existing xl tool for RTDS? Right now, we
assume all VCPU of the same domain have the same parameter and the
toolstack (xl, libxl, libxc and rt_dom_cntl()) are implemented based
on such assumption.
If we have implemented the vcpu-level parameter control, it will be a
superset of the current 'xl sched-rtds'. Back-supporting the current
xl tool for RTDS scheduler (only) will just make the logic a bit more
complex.
I'm not sure if back-supporting for the experimental feature is a must?
If it's not a must, I tend to not back-support it for simplicity.
(since less code ==> less possible bugs :-) )
What do you think?

Thank you very much!

Best regards,

Meng

-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel