RE: qemu and Xen ABI-unstable libs

[Paul Durrant <xadimgnik@xxxxxxxxx>]

> -----Original Message-----
> From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> Sent: 21 September 2020 11:16
> To: paul@xxxxxxx
> Cc: 'Ian Jackson' <iwj@xxxxxxxxxxxxxx>; 'Debian folks: Michael Tokarev' 
> <mjt@xxxxxxxxxx>; 'Hans van
> Kranenburg' <hans@xxxxxxxxxxx>; 'Xen upstream folks with an interest: Andrew 
> Cooper'
> <Andrew.Cooper3@xxxxxxxxxx>; pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx; 
> xen-devel@xxxxxxxxxxxxxxxxxxxx;
> 'My Xen upstream tools co-maintainer: Wei Liu' <wl@xxxxxxx>
> Subject: Re: qemu and Xen ABI-unstable libs
> 
> On Mon, Sep 21, 2020 at 08:36:55AM +0100, Paul Durrant wrote:
> > > -----Original Message-----
> > > From: Xen-devel <xen-devel-bounces@xxxxxxxxxxxxxxxxxxxx> On Behalf Of Ian 
> > > Jackson
> > > Sent: 18 September 2020 17:39
> > > To: Debian folks: Michael Tokarev <mjt@xxxxxxxxxx>; Hans van Kranenburg 
> > > <hans@xxxxxxxxxxx>; Xen
> > > upstream folks with an interest: Andrew Cooper 
> > > <Andrew.Cooper3@xxxxxxxxxx>; Roger Pau Monné
> > > <roger.pau@xxxxxxxxxx>
> > > Cc: pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx; 
> > > xen-devel@xxxxxxxxxxxxxxxxxxxx; My Xen upstream tools
> co-
> > > maintainer: Wei Liu <wl@xxxxxxx>
> > > Subject: RFC: qemu and Xen ABI-unstable libs
> > >
> > > Hi all.  Michael Tokarev has been looking into the problem that qemu
> > > is using Xen libraries with usntable ABIs.  We did an experiment to
> > > see which abi-unstable symbols qemu links to, by suppressing libxc
> > > from the link line.  The results are below.[1]
> > >
> > > Things are not looking too bad.  After some discussion on #xendevel I
> > > have tried to summarise the situation for each of the troublesome
> > > symbols.
> > >
> > > Also, we discovered that upstream qemu does not link against any
> > > abi-unstable Xen libraries if PCI passthrough is disabled.
> > >
> > > Please would my Xen colleages correct me if I have made any mistakes.
> > > Michael, I hope this is helpful and clear.
> > >
> > >
> > > In order from easy to hard:
> > >
> > >
> > > xc_domain_shutdown
> > >
> > > This call in qemu needs to be replaced with a call to the existing
> > > function xendevicemodel_shutdown in libxendevicemodel.  I think it is
> > > likely that this call is fixed in qemu upstream.
> > >
> >
> > I just pulled QEMU master and it appears that destroy_hvm_domain() is still 
> > calling
> xc_domain_shutdown().
> >
> > >
> > > xc_get_hvm_param
> > >
> > > There are three references in qemu's
> > > xen_get_default_ioreq_server_info, relating to ioreq servers.  These
> > > uses (and perhaps surrounding code at this function's call site)
> > > should be replaced by use of xendevicemodel_create_ioreq_server
> > > etc. from libxendevicemodel.  I think it is likely that this call is
> > > fixed in qemu upstream.
> > >
> >
> > These references are in compat code for Xen < 4.6. Use of (non-default) 
> > ioreq server has been
> present in the code for a long time.
> > We can remove them by retiring the compat code.
> >
> > >
> > > xc_physdev_map_pirq
> > > xc_physdev_map_pirq_msi
> > > xc_physdev_unmap_pirq
> > >
> > > These are all small wrappers for the PHYSDEVOP_map_pirq hypercall.
> > > PHYSDEVOP is already reasonably abi-stable at the hypervisor level (in
> > > theory it's versioned, but changing it would break all dom0's).
> >
> > The hypercalls are non-tools and directly called from the Linux kernel code 
> > so they are ABI.
> >
> > > These calls could just be provided as-is by a new stable abi
> > > entrypoint.  We think this should probably go in libxendevicemodel.
> > >
> >
> > Rather than simply moving this calls into libxendevicemodel, we should 
> > think about their
> interactions with calls such as
> > xc_domain_bind_pt_pci_irq() below and maybe have a stable library that 
> > actually provides a better
> API/ABI for interrupt
> > mapping/triggering although...
> 
> I've thought the same when speaking with Ian about this, as (for HVM
> passthrough) we use the physdev op to obtain a pirq from a physical
> device interrupt source (a MSI entry in the QEMU case, because the
> legacy interrupt is bound by the toolstack IIRC) and then use that
> pirq to bind it to a guest lapic vector.
> 
> I think in a sense such physical interrupt abstraction (the pirq) is
> helpful in order to simplify the binding, as you don't end up with a
> hypercall with a massive number of parameters to identify both the
> source and destination interrupt data. It's also helpful when the
> guest changes the interrupt binding, as you then only update the guest
> side and keep using the same pirq.
> 
> We might want however to have an interface more specific to
> passthrough, such that the pirqs (or maybe we could just call them
> handles) returned by such interface can only be used with guest
> specific bind hypercalls?

Yes, that sounds sensible: we have functions to allocare/free a pirq handle and 
then functions to bind/unbind such a handle to guest vector.

> 
> > I've long felt PCI pass-through should not be done by QEMU anyway (not 
> > least because we currently
> > have no mechanism for PCI pass-through to PVH domains).
> 
> Having xenpt in tree would be fine IMO. Now we have all the proper
> infrastructure in place to allow different pci devices to be handled
> by different emulators IIRC, which is all that's required for this to
> work correctly.

To be useful we need a way of selecting pass-through emulator in the toolstack 
and we also need to re-visit moving the PCI hotplug controller implementation 
into Xen (and augmenting the ioreq server API to add some sort of unplug 
request callback). I'll give it some thought.

  Paul

> 
> Thanks, Roger.