[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document


  • To: Christopher Clark <christopher.w.clark@xxxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Wed, 24 Mar 2021 09:01:18 +0100
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zOChQOKi45cydSnmUvqH75LPIVSVtkYyJvlO+/i0S/k=; b=mFiiTqDg17FIw5qiqETnkrILuX02YTf2mVNtui6ZmqjWlo7n9NVcpikM+p79BxgA56T7jJNw034t0WoZNR3hi0fo4SWNZPpwO6xZGMf86w8RI61YWYhSto8YOdcpoedpBIPQLxVI3dAz5L1J2yzRByuzhWIpD0aLQRDKYHIfKjQqSofvG9C2Vt4cfOn+LX10H3ambupFiU14RSWRNE6uhhZpYQo7gvMYlq4MyE8cNnC3vLocHeht75Hc84tbH+fCPDqp8EQBrohifEBKV0R8O9E1SnXWvCmfT5T5NL5X0A2sS5VHLsdf/73Vs0urY57utLc7V/VMwGWZj46tb1XM7g==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Up0vJoI+99UHFGtqeIxi8r3TE2gT9ZRxDZJXHi95BaQKy2zSotxKcqdEgRbvwRGtwWSrzjTcN6qgZXVnVbgWmv6Lsze6OC0NhIY1XReO0GPuAi9/4DB1T6vjZmYQaErRMmjRwrKotoExTAL8uWAARTr86CWgq+INOLdBFZUPzlRG10+o0MoW6DBya1spexRe1oTpnRNGn4uqB8F+80kzQjQVya1XGQ4bJ/14CcXpzXhKy4vWxqEYXY70CnyteMb1PQzw5htQkG0nPZgJ2Kv3giEx81Vkt9wLIXi1vT7KPC/KsEdrbfhq4qni5IA5ulsG2rmXdgQnMWL3fgITBfeV7Q==
  • Authentication-results: esa2.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Cc: "Daniel P. Smith" <dpsmith@xxxxxxxxxxxxxxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>, Julien Grall <jgrall@xxxxxxxxxx>, Julien Grall <Julien.grall.oss@xxxxxxxxx>, <iwj@xxxxxxxxxxxxxx>, Wei Liu <wl@xxxxxxx>, George Dunlap <george.dunlap@xxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Rich Persaud <persaur@xxxxxxxxx>, Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, <luca.fancellu@xxxxxxx>, <paul@xxxxxxx>, Adam Schwalm <adam.schwalm@xxxxxxxxxx>
  • Delivery-date: Wed, 24 Mar 2021 08:01:58 +0000
  • Ironport-hdrordr: A9a23:f+I7nay08y14OvMaelZ0KrPxnO4kLtp033Aq2lEZdDV8Sebdv9 yynfgdyB//gCsQXnZlotybJKycWxrnmqJdybI6eZOvRhPvtmftFoFt6oP+3ybtcheTysd07o 0lSaR3DbTLYGRSpdrm4QW+DtYryMSG9qftvuvF03JxV2hRCp1IxS0RMHf9LmRdQg5aCZ0lUL ed/NNAvTq8eXIRB/7LfEUtde7FutHNidbaehYAHREq802jijmv5b78HXGjr2ojehlIxqov9n WArhzh6syYwoqG4zL/90uW1ZRZn9P91sBObfbjtuE5Iijh4zzYBrhJdKaFuFkO0YSSwXYs1O LBuhIxe/l0gkmhAl2dhTvI903e3C0163nkoGXo8UfLhcDiXjo1B45gqOtiA2LkwnEttt19z6 5Htljx3/E8bWKi7VbAzuPFWB1wmk2/rWBKq59qs1VlXZYDc7gUlIQD/SpuYeY9NRjn44MqGv QGNrC72N9qdzqhHhTkl1gq6tmtUnMvJwyBU0gPt+eEugIm4kxR/g82wtcSkWwH8494Y55Y5/ 7cOqAtr71WSNQKBJgNSNspcI+SMCjgUBjMOGWdLRDOE7wGAWvEr9rS7K8u7O+nVZQUxPIJ6d v8eWIdkVR3V1PlCMWI0pEO2AvKWn+BUTPkzdwbz4Rlu5XnLYCbcRGreRQLqY+Nsv8fCsrUV7 KYI5RNGcLuKmPoBMJgwxD+YZ9PMnMTOfdl++oTaharmIbmO4fqvuvUfLL4P7z2CwspXWv5Hz 8tRz72CMJc7l26e3PxjRTLMkmdOHDXzNZVKuz37uITwI8COslnqQ4Ok2m04cmNNHljv8UNDQ 5DCYKitpn+iXi9/G7O4WksEAFaFFxp7LLpVG4PgQcLNkjzYIsSotn3QxEX4FK3YjtEC+/GGg 9WoFp6vYitKYaL+CwkA9W7dkWXkmUUv3DPa5sHgKWM6YPEd/oDf9gbcZ00MT+OOw1+mA5spm sGQhQDXFXjGjTnjrjgqocVCuHZf9xVmxyqPsZQlHLauSyn1IISb0peewTrfd+cgA4oSTYRrE Z26bUjjL2JnivqFXEym90iMFpHaH2eBZVPCAjtXvQSppnbPCVLCUuajz2TjB8+Pk7n7V8biG DaISqIQv3TGVZGtndE0qHlzUNsegymDjFNQ0E/lbc4OXXNu3513+POXKa13meLQnYpw+0WMl j+EHAvCzIr4+ry+A+emT6EG3lj+44nOfbFCq8/N5vJ3Gm2FYGOnaYaPvNd8Zp/LuryuusTXe /3QX7SEBrIT8cSnyCFrHcsPyd57EQ+mfTzwRv/8SyW2mU8Dfe6GiURe5grZ/Wnq07qSPaD3M 8n0ZYbve6sPn7wbdDD46fNdDJHIg7Sp2nzb+xAk+EhgYsC8J9IW7/cWn/08VsC+jMUBsL9jl kfT6R2+6qpAP4mQ+UiPwZiumM0n9GOJnYxugP4AuUCbUgg5kWrSu+h0v7tk/4TGUWPqwv7BE mH/wBc9/nDWTGf1bRyMdNGHU1mLGw94m9l5uWMasn5DxirbfhK+DOBQzWAWY4YbKiOArMLqB lmp/mOgu+MbiL9nCTdpyFyLK4L02GpR6qJcU+xMN8N19yxIlKXhKS2pOa1kTfsUDO+L30iur ctTz1YUu1zzh84jIM21SCuSqv45mId+mEung1PpxrKwYip4GDSAEdcFxbW668mBAVuDg==
  • Ironport-sdr: KEVSbC3AjYXtqR6HsrrUyWeI+a9n7sHiYdNQMcCMcpzq5v2bK0b3xT+82tb83HtFQvummqcOkC gZm8UcLcA3ilXJef+aZce+PR6+7THaZaRrQIUdg2qfKDDxPbl2/NoCHp4oC/PLEwE6M9UJT85s f37oBAFZVlW5TMLOL2QzGeiODfPlSyD1GRLL/OawpXUiT5Txkb/V4m8C5oH1/OyQPMaO70L1YA p4Qc/yj8hPh3lAhJFwLgkxtoAhPN9LiaZ5B2xlp7UiZ5Bi1Srn0EXxCgwB/GlWRpDka+2uWVgN NAM=
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Tue, Mar 23, 2021 at 10:39:53AM -0700, Christopher Clark wrote:
> On Thu, Mar 18, 2021 at 9:43 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> >
> > Just took a quick look at it.
> >
> > On Mon, Mar 15, 2021 at 11:18:13PM -0400, Daniel P. Smith wrote:
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------------------+
> > > + | **Xen Dom0**  | **Linux** | **Late**   | **Jail**  | **Xen**     | 
> > > **Xen Hyperlaunch** |
> > > + | **(Classic)** | **KVM**   | **HW Dom** | **house** | 
> > > **dom0less**+---------+-----------+
> > > + |               |           |            |           |             | 
> > > Static  | Dynamic   |
> > > + 
> > > +===============+===========+============+===========+=============+=========+===========+
> > > + | Hypervisor able to launch multiple VMs during host boot               
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |     Y     |       Y     |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Hypervisor supports Static Partitioning                               
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |     Y     |       Y     |    
> > > Y    |           |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Able to launch VMs dynamically after host boot                        
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |       Y       |     Y     |      Y*    |     Y     |       Y*    |    
> > >      |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Supports strong isolation between all VMs started at host boot        
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |     Y     |       Y     |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Enables flexible sequencing of VM start during host boot              
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Prevent all-powerful static root domain being launched at boot        
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |       Y*    |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Operates without a Highly-privileged management VM (eg. Dom0)         
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |      Y*    |           |       Y*    |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Operates without a privileged toolstack VM (Control Domain)           
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |       Y*    |    
> > > Y    |           |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Extensible VM configuration applied before launch of VMs at host boot 
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Flexible granular assignment of permissions and functions to VMs      
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | Supports extensible VM measurement architecture for DRTM and 
> > > attestation               |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + | PCI passthrough configured at host boot                               
> > >                  |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > + |               |           |            |           |             |    
> > > Y    |     Y     |
> > > + 
> > > +---------------+-----------+------------+-----------+-------------+---------+-----------+
> >
> > I'm curious about this, I assume this is done using vPCI so that
> > there's no hardware domain (and user-space device model) involved for
> > PCI passthrough?
> 
> That would be an incorrect assumption. See below for why.
> 
> > I'm also not sure how you are going to handle things like SR-IOV
> > devices. Right now SR-IOV capability is setup and initialized by the
> > hardware domain, and the new virtual devices are notified to Xen once
> > setup is done. Do you plan to move those bits into Xen, so that it can
> > setup and initialize the SR-IOV capability?
> 
> While you could do it with the vPCI, as you point out this will not work
> for SR-IOV. With hyperlaunch, these cases will require the use of a boot
> domain, which is for all intents and purposes, a lightweight/restricted
> toolstack domain.
> 
> The boot domain will have to do the necessary operations to ensure that
> when startup is finished, PCI passthrough will be successfully setup.
> Note, this may have to include the boot domain unpausing the hardware
> domain to help complete the setup before the boot domain can exit and
> allow the remaining domains to come online.

OK, I was expecting hyperlaunch to do all domain creation in the
hypervisor. If you offload domain creation of guests with
pci-passthrough devices to a control domain and/or hardware domain,
I'm not sure I see the difference from normal domain creation, ie:
it's no longer something specific to hyperlaunch, as I could achieve
the same by using the existing xendomains init script.

Also you need a way to pass the configuration from the hypervisor into
a control domain that would then wait for the hardware domain to come
up and afterwards launch a guest with a pci-passthorugh device. The
passing of this information from the hypervisor to the control domain
would need to be done in an OS agnostic way if possible.

Don't get me wrong, I don't think such approach is bad, I'm just
unsure whether such functionality is really part of hyperlaunch, or
instead something that you can achieve outside of hyperlaunch already.

Thanks, Roger.



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.