Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Tue, Mar 23, 2021 at 10:39:53AM -0700, Christopher Clark wrote:
> On Thu, Mar 18, 2021 at 9:43 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> >
> > Just took a quick look at it.
> >
> > On Mon, Mar 15, 2021 at 11:18:13PM -0400, Daniel P. Smith wrote:
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------------------+
> > > + | **Xen Dom0**  | **Linux** | **Late**   | **Jail**  | **Xen**     | 
> > > **Xen Hyperlaunch** |
> > > + | **(Classic)** | **KVM**   | **HW Dom** | **house** | 
> > > **dom0less**+---------+-----------+
> > > + |               |           |            |           |             | 
> > > Static  | Dynamic   |
> > > + 
> > > +===============+===========+============+===========+=============+=========+===========+
> > > + | Hypervisor able to launch multiple VMs during host boot               
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |     Y     |       Y     |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Hypervisor supports Static Partitioning                               
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |     Y     |       Y     |    
> > > Y    |           |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Able to launch VMs dynamically after host boot                        
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |       Y       |     Y     |      Y*    |     Y     |       Y*    |    
> > >      |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Supports strong isolation between all VMs started at host boot        
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |     Y     |       Y     |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Enables flexible sequencing of VM start during host boot              
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Prevent all-powerful static root domain being launched at boot        
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |       Y*    |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Operates without a Highly-privileged management VM (eg. Dom0)         
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |      Y*    |           |       Y*    |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Operates without a privileged toolstack VM (Control Domain)           
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |       Y*    |    
> > > Y    |           |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Extensible VM configuration applied before launch of VMs at host boot 
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Flexible granular assignment of permissions and functions to VMs      
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Supports extensible VM measurement architecture for DRTM and 
> > > attestation               |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | PCI passthrough configured at host boot                               
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> >
> > I'm curious about this, I assume this is done using vPCI so that
> > there's no hardware domain (and user-space device model) involved for
> > PCI passthrough?
> 
> That would be an incorrect assumption. See below for why.
> 
> > I'm also not sure how you are going to handle things like SR-IOV
> > devices. Right now SR-IOV capability is setup and initialized by the
> > hardware domain, and the new virtual devices are notified to Xen once
> > setup is done. Do you plan to move those bits into Xen, so that it can
> > setup and initialize the SR-IOV capability?
> 
> While you could do it with the vPCI, as you point out this will not work
> for SR-IOV. With hyperlaunch, these cases will require the use of a boot
> domain, which is for all intents and purposes, a lightweight/restricted
> toolstack domain.
> 
> The boot domain will have to do the necessary operations to ensure that
> when startup is finished, PCI passthrough will be successfully setup.
> Note, this may have to include the boot domain unpausing the hardware
> domain to help complete the setup before the boot domain can exit and
> allow the remaining domains to come online.

OK, I was expecting hyperlaunch to do all domain creation in the
hypervisor. If you offload domain creation of guests with
pci-passthrough devices to a control domain and/or hardware domain,
I'm not sure I see the difference from normal domain creation, ie:
it's no longer something specific to hyperlaunch, as I could achieve
the same by using the existing xendomains init script.

Also you need a way to pass the configuration from the hypervisor into
a control domain that would then wait for the hardware domain to come
up and afterwards launch a guest with a pci-passthorugh device. The
passing of this information from the hypervisor to the control domain
would need to be done in an OS agnostic way if possible.

Don't get me wrong, I don't think such approach is bad, I'm just
unsure whether such functionality is really part of hyperlaunch, or
instead something that you can achieve outside of hyperlaunch already.

Thanks, Roger.