Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 25 Mar 2021, Roger Pau Monné wrote:
> On Thu, Mar 25, 2021 at 10:14:31AM +0100, George Dunlap wrote:
> > 
> > 
> > > On Mar 25, 2021, at 8:32 AM, Roger Pau Monne <roger.pau@xxxxxxxxxx> wrote:
> > > 
> > > On Wed, Mar 24, 2021 at 05:53:26AM -0700, Christopher Clark wrote:
> > >> On Wed, Mar 24, 2021 at 1:01 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> 
> > >> wrote:
> > >>> 
> > >>> On Tue, Mar 23, 2021 at 10:39:53AM -0700, Christopher Clark wrote:
> > >>>> On Thu, Mar 18, 2021 at 9:43 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> 
> > >>>> wrote:
> > >>> If you offload domain creation of guests with
> > >>> pci-passthrough devices to a control domain and/or hardware domain,
> > >>> I'm not sure I see the difference from normal domain creation, ie:
> > >>> it's no longer something specific to hyperlaunch, as I could achieve
> > >>> the same by using the existing xendomains init script.
> > >> 
> > >> So that's not what we've proposed, and hopefully not what we'll need to 
> > >> do.
> > >> 
> > >> Do you know if there is a need to perform work to support the
> > >> assignment of PCI devices at the point of domain creation (ie. in
> > >> domain_create), rather than handling it in a later step of domain
> > >> configuration, prior to the domain being started?
> > > 
> > > So while I think you could indeed create a domain from the hypervisor
> > > in a paused state and attach the pci devices later from a
> > > control/hardware domain, I don't see much benefit in doing it. If you
> > > need to end up waiting for a control/hardware domain to attach the
> > > devices and unpause you might as well do the whole domain creation
> > > from such control/hardware domain.
> > 
> > My understanding was that one of the primary advantages of domB was
> > that you could compile and run arbitrary code in whatever language
> > you wanted to, using already known tools.  If *all* we want to do is
> > to assign some pre-defined specific BDFs to specific domains, then
> > sure, we could add that capability to Xen.
> 
> Well, it's not so easy because we require QEMU or pciback ATM on x86
> in order to do pci passthrough to guests, so assigning BDFs to
> specific domains from the hypervisor would need to be done using vPCI
> (which is not yet ready for unprivileged guest usage) and only support
> HVM kind of guests strictly.
> 
> Doing passthrough to PV guests directly from the hypervisor is
> impossible AFAICT without putting pciback inside of the hypervisor.
> 
> > But suppose you wanted
> > to have the flexibility to handle changes in hardware at boot time?
> > “Scan through the PCI bus and assign anything that looks like a
> > network card to domNet, and anything that looks like a USB
> > controller to domUSB” is something you could easily do in domB, but
> > would be way too complicated to add to Xen.
> 
> Right, but then you might as well create the domain from domB instead
> of doing it in the hypervisor?
> 
> I'm not arguing about not using domB, I just don't see the benefit of
> creating a paused domain from the hypervisor that then requires the
> intervention of a further domain (domB) in order to finish creation.
> Won't it be simpler to just create the domain and attach the pci
> devices from domB?

I think that the ability of creating multiple VMs from Xen is actually a
very good one to have for a few reasons. It would align x86 with ARM, it
would be useful in cases where PCI passthrough is not involved, and it
is a powerful tool to have in our toolbox.

I see that handling PCI passthrough at domain creation time can be
difficult, so I think Christopher's solution is a good compromise.

FYI for dom0less/ARM we have been discussing doing device assignment at
creation time, but the idea was to provide the PCI topology in device
tree to Xen to help with discovery.