xen summit 2023 design session: Using only stable interfaces for (o)xenstored

[Edwin Torok <edwin.torok@xxxxxxxxx>]

Background: 
https://design-sessions.xenproject.org/uid/discussion/disc_MEXyUIXV6clI8n1kgzQ9/view

Possible solutions:
Andrew Cooper: use get domain info list instead of querying domains one-by-one 
to reduce number of hypercalls, even though Alejandro's patch makes the 
get_domainfo lookups O(1).
To avoid a hypercall a shared read-only memory page between Xen and Dom0, 
containing a bitmap of domain existence and shutdown state.
We have 32k domains max and this fits into a small number of 4K pages.
Hypervisor can update it using atomic set/clear-bit, Dom0 will memcpy and then 
check for differences
Bernhardt/Andrew: discussion about efficiency and find first bit set, XOR. A 
*lot* more efficient than using hypercalls anyway
Edwin: do I have to implement that for various architectures?
Andrew Cooper: already implemented for all arches. Some architectures can only 
do atomic updates on natural register width (RISC-V), but that is already 
handled
Edwin: how would the stable interface look like? Memory layout definition in 
the header and ..?
Andrew Cooper: use acquire_resource with a new resource type, already have 
stable xen libs for resource mapping

Christopher Clarke/Daniel Smith (virtually from chatroom): alternatives to 
bitmap, how is it indexed?
Andrew Coooper: indexed by domid, we only have 32k
Marcus Granado: what if we want to increase that?
Andrew Coooper: 16-bit domid is baked into so many ABIs that if we want to 
increase that then this API won't be the blocker. No need to worry about more 
than 32k domids.
Christopher Clark: how about using an Argo ring
Andrew Cooper: Xen must NOT block when delivering the notification and 
notification delivery cannot be lossy, cannot afford to drop events: 
toolstacks/xenstored critically rely on this information.
Christopher Clark: bip buffers: 
https://www.codeproject.com/Articles/3479/The-Bip-Buffer-The-Circular-Buffer-with-a-Twist
 as possible solution to avoid running out of space
Andrew Cooper: bitmap also avoids having to take the domctl lock and reduce 
background idle CPU usage
Edwin: on that topic: we should also fix xcp-rrdd so it doesn't shell out to 
xenpm every 5s to collect statistics...
Andrew Cooper: should be possible to use libxenctrl directly for that
Edwin Torok: there are other interfacs that would need to be stabilized (see 
xenopsd), but lets start small and fix just this one for now, which will result 
in an immediate improvement for (o)xenstored since this was the final blocker 
to getting off stable interfaces (there is one other usage, but that already 
has a patch posted/solution)

Edwin Torok: how about release cadence? Doesn't necessarily have to go into 
immediately next Xen release, but don't know in which phase the release is in 
right now
Andrew Cooper: probably about 6 weeks time to get into next release