Re: Is it time to start implementing Xen bindings for rust-vmm?

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 14/09/2021 15:44, Alex Bennée wrote:
> Andrew Cooper <andrew.cooper3@xxxxxxxxxx> writes:
>
>> On 13/09/2021 13:44, Alex Bennée wrote:
>>> Hi,
>>>
>>> As we consider the next cycle for Project Stratos I would like to make
>>> some more progress on hypervisor agnosticism for our virtio backends.
>>> While we have implemented a number of virtio vhost-user backends using C
>>> we've rapidly switched to using rust-vmm based ones for virtio-i2c,
>>> virtio-rng and virtio-gpio. Given the interest in Rust for implementing
>>> backends does it make sense to do some enabling work in rust-vmm to
>>> support Xen?
>>>
>>> There are two chunks of work I can think of:
>>>
>>>   1. Enough of libxl/hypervisor interface to implement an IOREQ end point.
>> No libxl here at all.
>>
>> As of Xen 4.15, there are enough stable interfaces to implement simple
>> IOERQ servers.
>>
>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fxapi-project%2Fvarstored%2Fcommit%2Ffde707c59f7a189e1d4e97c1a4ee1a2d0c378ad1&amp;data=04%7C01%7CAndrew.Cooper3%40citrix.com%7C08a3fe14704a4d6888cf08d9778ee5b2%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C637672277905441489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=%2B1pKhIuqzCGkgYD4snd6jnxjoEJzrCgUdol%2FfA2kwOk%3D&amp;reserved=0
>> was the commit where I removed the final unstable interface from
>> varstored (terrible name) which is a dom0 backend for UEFI secure
>> variable handling.  As such, it also serves as a (not totally simple)
>> reference of an IOERQ server.
>>
>>
>> There are a few bits and pieces of rust going on within Xen, and a whole
>> load of plans.  Also, there is a lot of interest from other downstreams
>> in being able to write Rust backends.
>>
>> We've got a placeholder xen and xen-sys crates, and placeholder work for
>> supporting cross-compile as x86 PV and PVH stubdomains.
> Are these in the rust-vmm project is elsewhere?

https://crates.io/crates/xen-sys

When I say placeholder, I really do mean placeholder.

To start this work meaningfully, we'd want to make a repo (or several)
in the xen-project organisation on github or gitlab (we have both, for
reasons), and set these as the upstream of the xen and xen-sys crates.

>> The want to have a simple IOREQ server compiled either as a dom0
>> backend, or as a PV or PVH stubdomains influences some of the design
>> decisions early on, but they're all no-brainers for the longevity of the
>> work.
> Just to clarify nomenclature is a PVH stubdomain what I'm referring to
> as a bare metal backend, i.e: a unikernel or RTOS image that implements
> the backend without having to transition between some sort of userspace
> and it's supporting kernel?

I think so, yes, although calling it "bare metal" seems misleading for
something which is a VM targetted at a specific hypervisor...


>> I started work on trying to reimplement varstored entirely in Rust as a
>> hackathon project, although ran out of time trying to make hypercall
>> buffers work (there is a bug with Box and non-global allocators causing
>> rustc to hit an assert().  In the short term, we'll have to implement
>> hypercall buffers in a slightly more irritating way).
>>
>> Furthermore, stick to the stable hypercalls only.  Xen's C libraries are
>> disaster for cross-version compatibility, and you absolutely do not want
>> to recompile your rust program just to run it against a different
>> version of the hypervisor.  The plan is to start with simple IOREQ
>> servers, which are on fully stable interfaces, then stabilise further
>> hypercalls as necessary to expand functionality.
> Are the hypercalls mediated by a kernel layer or are you making direct
> HVC calls (on ARM) to the hypervisor from userspace?

For a dom0 backends irrespective of architecture, you need to issue
ioctl()'s on the appropriate kernel device.

For a PV/PVH stubdom, you should make a call into the hypercall_page
https://xenbits.xen.org/docs/latest/guest-guide/x86/hypercall-abi.html
because Intel and AMD used different instructions for their equivalent
of HVC.

ARM doesn't have the hypercall page ABI, so I'd expect the hypercall
implementation to expand to HVC directly.


In terms of rust APIs, we'd want a crate which has target-specific
implementations so the caller need not worry about the implementation
details in the common case.

>
> Where would I look in the Xen code to find the hypercalls that are
> considered stable and won't change between versions?

I'm afraid that's mostly in developers heads right now.

For a first pass, you can look for __XEN_TOOLS__  (This is mis-named,
and ought to be called __XEN_UNSTABLE_INTERFACE__, because...) but be
aware that some things currently tagged __XEN_TOOLS__ are incorrect and
are in fact stable.

As a first pass, assume everything is unstable.  The things contained
within libxendevicemodel and libxenforeignmem are stable and were
specifically made so to try and get simple IOREQ server functionality
done and stable.

Almost everything else, particularly concerning the toolstack
operations, is unstable.  There is 15 years of organic growth and
dubious decisions here, and they need unpicking carefully.  We've got
some hypercalls which look like they're unstable, but are actually
stable (as they were exposed to guests), and therefore have ridiculous
interfaces.

The "ABI v2" work is massive and complicated, and picking at some of the
corners based on "what is needed to make new $FOO work" is a good way to
make some inroads.

>> It's high time the Xen Rust working group (which has been talked about
>> for several years now) actually forms...
> Indeed part of the purpose of this email was to smoke out those who are
> interested in the intersection of Xen, Rust and VirtIO ;-)

The conversation has come up quite a few times in the past, but mostly
by people who are also busy with other things.

~Andrew