Re: Xen Rust VirtIO demos work breakdown for Project Stratos

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 24/09/2021 17:02, Alex Bennée wrote:
> 1.1 Upstream an "official" rust crate for Xen ([STR-52])
> ────────────────────────────────────────────────────────
>
>   To start with we will want an upstream location for future work to be
>   based upon. The intention is the crate is independent of the version
>   of Xen it runs on (above the baseline version chosen). This will
>   entail:
>
>   • ☐ agreeing with upstream the name/location for the source

Probably github/xen-project/rust-bindings unless anyone has a better
suggestion.

We almost certainly want a companion repository configured as a
hello-world example using the bindings and (cross-)compiled for each
backend target.

>   • ☐ documenting the rules for the "stable" hypercall ABI

Easy.  There shall be no use of unstable interfaces at all.

This is the *only* way to avoid making the bindings dependent on the
version of the hypervisor, and will be a major improvement in the Xen
ecosystem.

Any unstable hypercall wanting to be used shall be stabilised in Xen
first, which has been vehemently agreed to at multiple dev summits in
the past, and will be a useful way of guiding the stabilisation effort.

>   • ☐ establish an internal interface to elide between ioctl mediated
>     and direct hypercalls
>   • ☐ ensure the crate is multi-arch and has feature parity for arm64
>
>   As such we expect the implementation to be standalone, i.e. not
>   wrapping the existing Xen libraries for mediation. There should be a
>   close (1-to-1) mapping between the interfaces in the crate and the
>   eventual hypercall made to the hypervisor.
>
>   Estimate: 4w (elapsed likely longer due to discussion)
>
>
> [STR-52] 
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinaro.atlassian.net%2Fbrowse%2FSTR-52&amp;data=04%7C01%7CAndrew.Cooper3%40citrix.com%7C973087248f404018a36008d97f78a0ea%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C637680978363070576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=0k6ES9IXCvTlqP8jTiFeuTN51GlMngxaepWvQC6hMAg%3D&amp;reserved=0>
>
>
> 1.2 Basic Hypervisor Interactions hypercalls ([STR-53])
> ───────────────────────────────────────────────────────
>
>   These are the bare minimum hypercalls implemented as both ioctl and
>   direct calls. These allow for a very basic binary to:
>
>   • ☐ console_io - output IO via the Xen console
>   • ☐ domctl stub - basic stub for domain control (different API?)
>   • ☐ sysctl stub - basic stub for system control (different API?)
>
>   The idea would be this provides enough hypercall interface to query
>   the list of domains and output their status via the xen console. There
>   is an open question about if the domctl and sysctl hypercalls are way
>   to go.

console_io probably wants implementing as a backend to println!() or the
log module, because users of the crate won't want change how they
printf()/etc depending on the target.

That said, console_io hypercalls only do anything for unprivleged VMs in
debug builds of the hypervisor.  This is fine for development, and less
fine in production, so logging ought to use the PV console instead (with
room for future expansion to an Argo transport).

domctl/sysctl are unstable interfaces.  I don't think they'll be
necessary for a basic virtio backend, and they will be the most
complicated hypercalls to stabilise.

>
>   Estimate: 6w
>
>
> [STR-53] 
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinaro.atlassian.net%2Fbrowse%2FSTR-53&amp;data=04%7C01%7CAndrew.Cooper3%40citrix.com%7C973087248f404018a36008d97f78a0ea%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C637680978363070576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=0mgMPS4ikzD%2F8fc%2BcqVjtRuSKzMu%2Ba8XgOs4hOC9pY4%3D&amp;reserved=0>
>
>
> 1.3 [#10] Access to XenStore service ([STR-54])
> ───────────────────────────────────────────────
>
>   This is a shared configuration storage space accessed via either Unix
>   sockets (on dom0) or via the Xenbus. This is used to access
>   configuration information for the domain.
>
>   Is this needed for a backend though? Can everything just be passed
>   direct on the command line?

Currently, if you want a stubdom and you want to instruct it to shut
down cleanly, it needs xenstore.  Any stubdom which wants disk or
network needs xenstore too.

xenbus (the transport) does need to split between ioctl()'s and raw
hypercalls.  xenstore (the protocol) could be in the xen crate, or a
separate one as it is a piece of higher level functionality.

However, we should pay attention to non-xenstore usecases and not paint
ourselves into a corner.  Some security usecases would prefer not to use
shared memory, and e.g. might consider using an Argo transport instead
of the traditional grant-shared page.

>
>   Estimate: 4w
>
>
> [STR-54] 
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinaro.atlassian.net%2Fbrowse%2FSTR-54&amp;data=04%7C01%7CAndrew.Cooper3%40citrix.com%7C973087248f404018a36008d97f78a0ea%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C637680978363070576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=mURxxo7vQwfTkR4cX5yN5l7kPav2gXhluOhm2%2BSBAIs%3D&amp;reserved=0>
>
>
> 1.4 VirtIO support hypercalls ([STR-55])
> ────────────────────────────────────────
>
>   These are the hypercalls that need to be implemented to support a
>   VirtIO backend. This includes the ability to map another guests memory
>   into the current domains address space, register to receive IOREQ
>   events when the guest knocks at the doorbell and inject kicks into the
>   guest. The hypercalls we need to support would be:
>
>   • ☐ dmop - device model ops (*_ioreq_server, setirq, nr_vpus)
>   • ☐ foreignmemory - map and unmap guest memory

also evtchn, which you need for ioreq notifications.

>   The DMOP space is larger than what we need for an IOREQ backend so
>   I've based it just on what arch/arm/dm.c exports which is the subset
>   introduced for EPAM's virtio work.

One thing we will want to be is careful with the interface.  The current
DMOPs are a mess of units (particularly frames vs addresses, which will
need to change in Xen in due course) as well as range
inclusivity/exclusivity.

>
>   Estimate: 12w
>
>
> [STR-55] 
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinaro.atlassian.net%2Fbrowse%2FSTR-55&amp;data=04%7C01%7CAndrew.Cooper3%40citrix.com%7C973087248f404018a36008d97f78a0ea%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C637680978363070576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=tczcGl6YWGRwnBooBwodDX2BHLKu5tJG3%2FTe%2Fpqf%2B9w%3D&amp;reserved=0>
>
>
> 2 Xen Hypervisor Support for Stratos ([STR-56])
> ═══════════════════════════════════════════════
>
>   These tasks include tasks needed to support the various different
>   deployments of Stratos components in Xen.
>
>
> [STR-56] 
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinaro.atlassian.net%2Fbrowse%2FSTR-56&amp;data=04%7C01%7CAndrew.Cooper3%40citrix.com%7C973087248f404018a36008d97f78a0ea%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C637680978363070576%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=31lFqxkbZtyNdQcyVI1d8M53l4JqVhni1s1aowrtoXg%3D&amp;reserved=0>
>
> 2.1 Stable ABI for foreignmemory mapping to non-dom0 ([STR-57])
> ───────────────────────────────────────────────────────────────
>
>   Currently the foreign memory mapping support only works for dom0 due
>   to reference counting issues. If we are to support backends running in
>   their own domains this will need to get fixed.

Oh.  It appears as if some of this was completed in
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=4922caf1de5a08d3eefb4058de1b7f0122c8f76f

~Andrew