Re: [Xen-devel] libxl/xl memory paging/sharing/ballooning/etc proposal.

["Andres Lagar-Cavilla" <andres@xxxxxxxxxxxxxxxx>]

> On Thu, 2012-03-15 at 14:12 +0000, Andres Lagar-Cavilla wrote:
> [...]
>> > libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb,
>> relative)
>> >
>> >         Read/write whichever XS path controls the xenpaging daemon's
>> >         target. (Olaf?). I think this path should not be
>> >         under /local/domain since it should not be guest visible
>> >         (probably it already isn't).
>> It is /local/domain/X/memory/target-tot_pages. I agree it should not be
>> guest visible, and we're still in time to change it.
>>
>> /libxl/X/memory-policy/paging-target? Is there a non-libxl
>> non-/local/domain path xenpaging could look into, should it be launched
>> autonomously? Olaf, do you care about xenpaging in non-libxl
>> environments?
>
> /xenpaging/N? or /tools/xenpaging/N?
>
> For comparison Roger's hotplug patches seem to be using /hotplug/...
>
>> >         It is an error to call this for a PV guest (ERROR_??) for a
>> >         guest which does not have paging enabled
>> >         (ERROR_PAGING_DISABLED?).
>> >
>> > libxl_domain_(enable|disable)_paging(ctx, domid)
>> >
>> >         Idempotently starts or stops the xenpaging daemon for a
>> domain.
>> >
>>
>> Paging disable takes time. It has to swap back in everything it has
>> paged
>> out. So it falls under the (really) long-lasting asynchronous operation
>> framework.
>
> Good point, thanks.
>
>>
>> > libxl_domain_(enable|disable)_sharing(ctx, domid)
>> >
>> >         Does what it says on the tin.
>
> Also long running?

Enable is simply setting a flag. Disable is not a problem either: the
hypervisor will silently take care of unshares.

However, if there is an ENOMEM in an unshare, someone has to be listening
or the domain will crash. It could be that the xl daemon is tasked with
this for the lifetime of the domain, once sharing has been enabled at
least once. I don't think it's something to decide upon right now.

Andres

>
>> >
>> > actor behaviour
>> > ---------------
>> >
>> > An actor should use the libxl event system to monitor for
>> > DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any event
>> for
>> > which actor != their name.
>> >
>> > On receiving such an event the actor should apply its policy and
>> > configuration in order (try and) cause the domain to use "target"
>> amount
>> > of actual RAM. This may involve calling the libxl low level
>> interfaces,
>> > including enabled paging and/or sharing as it desires.
>> >
>> > It is an actor specific policy whether it guarantees to meet the
>> target
>> > (e.g. a balloon only actor would not be able to make that guarantee)
>> >
>> > An actor might be either be a single daemon with host wide knowledge
>> or
>> > a daemon per domain (or anything in between). It is expected in the
>> > multiple daemon case that all the daemons will either co-ordinate as
>> > necessary or implement a policy where coordination is not necessary.
>> >
>> > Normally the same actor would be used for all domains but we do not
>> rule
>> > out the possibility for non-homogeneous configurations (but it is up
>> to
>> > the user to make sure the actors don't have conflicting, potentially
>> > catastrophic, interactions...)
>> >
>> > XXX I think we need a DOMAIN_INTRODUCE event to allow for a system
>> wide
>> > actor. This seems like a generally useful event anyhow.
>>
>> System-wide actors (squeezed) and other actors (xenpaging) already place
>> a
>> watch on xenstore's @introduceDomain (or whatever it's called). Since
>> this
>> is not their only use of libxenstore, there is little gain to be made of
>> a
>> libxl-equivalent event. Referring only to this context.
>
> What other use of xenstore do they make? Depending what those are it's
> possible that it may fall under the remit of libxl to provide such
> interfaces.
>
> It should be possible for an actor to be only implemented using libxl,
> which I think requires us to provide such an event. This doesn't
> preclude some actors from using facilities from other places too though.
>
> [...]
>> > "xl" actor behaviour
>> > --------------------
>> >
>> > Each xl daemon instance will listen for DOMAIN_MEMORY_TARGET_CHANGED
>> > events for the domain which it is managing. When the event is
>> triggered
>> > (and actor == "xl") then xl will:
>> >
>> >         if actor != "xl": return
>> >
>> >         call libxl_domain_set_balloon_target(ctx, domid, target,
>> >         enforcing)
>> >
>> >         if domain == pv: return # cannot do more for a PV guest
>> >
>> >         XXX if not enforcing: return ??? (not sure about this, we
>> don't
>> >         actually expose the ability to set enforcing in the xl
>> >         interface)
>> >
>> >         XXX if actor-params:paging!=true return ??? I think we should
>> do
>> >         this to enable memory_policy_params="paging=true|false" based
>> >         control of paging.
>> >
>> >         delay for a configurable number of seconds (configurable via
>> >         actor-params:delay=N, default 5s?) XXX does libxl provide a
>> >         guest accessible timeout event? Would be usefully convenient.
>>
>> Do you want the guest to try to stop ballooning after the timeout? Just
>> write /local/domain/X/memory/target to the current value of
>> xcinfo.nr_pages.
>
> Could do, I don't mind.
>
> Leaving it alone means that a guest which becomes co-operative "too
> late" will eventually reduce the amount of paging it is subjected to to
> 0. Doing as you suggest stops the guest waking up every $PERIOD to try
> ballooning a bit more.
>
> Could be an actor-param, although it would be better if xl could just Do
> The Right Thing.
>
>> >         XXX libxl_domain_paging_enable here or on domain build /
>> policy
>> >         set? libxl_domain_paging_enable is idempotent so calling it
>> here
>> >         is always safe?
>>
>> You can enable the pager on build, unless the actor settings preclude
>> it.
>> Then all you need to do to get the pager to work is write its xenstore
>> key.
>
> My thinking was that if you boot "unpaged" you might want it to be
> enabled when you call mem-set. Likewise if you mem-set such that no
> paging is required you might like the daemon to go away. I think the
> former is more important than the latter. We don't really want to run
> xenpaging for every HVM domain even when it isn't used, whereas it is
> fine if it persists after it has been used.
>
>>
>> But generally speaking, the pager might have been disabled ex
>> xl-machina,
>> so you really want to be able to test_and_set the pager here.
>>
>> Olaf, does xenpaging have standard daemonize behavior of locking a
>> /var/foo/pid file?
>
> If it is running this should be indicated in the same XS path as we
> choose above?
>
>> >         call libxl_domain_set_paging_target(ctx, domid, target)
>> >
>> > The "xl" actor explicitly does not deal with page sharing. It is
>> assumed
>> > that page sharing requires system wide knowledge and policy which is
>> out
>> > of xl's scope and hence a different actor would be needed.
>> >
>> > xl low level interface
>> > ----------------------
>> >
>> > All of the following setters functions should error out unless actor
>> ==
>> > "xl-manual". The getter could return the current value regardless.
>> >
>> > xl mem-balloon-(set|get)-target
>> > xl mem-paging-(set|get)-target
>> >
>> >         Set or get the relevant targets by calling the equivalent
>> libxl
>> >         function.
>> >
>> >         Enables or disables paging as necessary, i.e. target == 0 =>
>> >         disable
>> >
>> > NB if you call "xl mem-set" when actor == "xl-manual" then it won't do
>> > anything since nothing is reacting as that actor.
>> >
>> > XXX We could alternatively have xl also implement an "xl-manual" actor
>> > which immediately sets both ballooning and paging targets to the given
>> > value. Probably a good idea?
>>
>> xl-manual makes sense, but setting targets immediately (e.g.
>> automatically) is an oxymoron :)
>>
>> >
>> > XXX Maybe "xl" (the default) should be called "xl-auto" ?
>>
>> Or xl-manual called "manual"
>
> I think namespacing the actors makes sense myself, there's no harm?
>
> Perhaps <actor>-<policy> would be a good guideline for naming the thing
> we've been calling the actor?
>
>> > If you "xl create -e" (or -F) then you won't get a monitoring
>> > daemon and therefore you won't get any actor implementing the memory
>> > policy, just like you won't get automatic reboot control. I think this
>> > is expected and reasonable.
>>
>> Ok. There actors that conceivably do not depend on libxl events
>> (squeezed,
>> it can watch on its own memory-policy/target). On one hand they should
>> not
>> depend on the xl daemon.
>
> Squeezed should eventually be implemented in terms of the mechanisms
> discussed here IMHO. This would remove the duplication of mechanism,
> leaving only the policy. Of course if the author of an actor wants to
> use other mechanisms then fine.
>
> Also bear in mind that the toolstack might not be xl, i.e. xapi should
> eventually be ported to use the highlevel libxl interface and would
> likely choose to set "squeezed" as its default actor.
>
>>  On the other hand, they would duplicate
>> functionality that the xl daemon would have to provide anyways, for
>> monitoring these memory-policy keys.
>
> If actor="squeezed" then, even if it is the toolstack, xl will not be
> providing any mechanism, since it's own actor will not do anything when
> actor=="squeezed".
>
> Ian.
>
>
>



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel