[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Nested Virtualization of Hyper-V on Xen Not Working

  • To: Xentrigued <xentrigued@xxxxxxxxxxx>, <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
  • Date: Mon, 26 Jul 2021 14:55:14 +0100
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nPLanxlGLrFNWn3Yf8ofV/qFHuk2lJWJ9gIzpdPb7Uo=; b=OmyouAJ9C7ailBQhxh8ypddi+oDIWIgUskz6LiiYUJoJxhXMqul7MAPbdSIQfh4NgLq2jUAMXVl7Tc5AvIR6MW93oJt3ewSqsdcHU3KJRjVMJJ/NfJgVy9i6Vf+5JqZ7j9l+I5XP5H6vPfDHdouFbRoczj0mFFAho8kYIwEYaOW8bxWjfvrMO2VBFe4ytwjUvHv3oLfJH2ZLYXWDQ1pAy0jCnREVkHMh2t4GMkS54eOYrYekSbBUVS9IaqWwqNc63fpyLof2c+cNarb2+8QZCZRrFtjGdzWgN50KO12UkSPV2rY4XBuAnaHnqLO15L9+sOchIB2fg9Jvwp4NcMvzZQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gVaxjROAzUkCCZmgIzsItZeLKUFf7WqPOKsuDggSaywENgd0Z5ZhGJUcMltyZgtkRhAlw3H0Kqc3ezCtIIYlMfp+w8gsaz4Qro7X6wTUxNJoHPqkfj8lzvZ8rjKgVqNxwiGp6XyMQFEERKVSOD0o+hua0W8Z6l78nj8HYK4b7NaVxqhBoyzdkahLBrH9V7JxFhnBV63g2Pd7C7P0jZ9FaMMwAcizWPl0XOEM+pWMTv82bQpO+78MMygnvCkuiRl6mj9NU00uFzqHUcYXOYfWPqjAIJrHZ1IfjJZdUSUvX9KadA6rMbwfpJdq4li5a65ktfkTXAiWB+KJJbfUsrzfmw==
  • Authentication-results: esa5.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Delivery-date: Mon, 26 Jul 2021 13:55:43 +0000
  • Ironport-hdrordr: A9a23:EHE5DK82awY+GmAZuXluk+FKdb1zdoMgy1knxilNoENuHPBwxv rAoB1E73PJYVYqOE3I6urwR5VoJkmsiaKdgLNhQItKOTOGhILGFvAb0WKP+UyGJ8S6zJ8m6U 4CSdkPNDSTNykCsS+S2mDReLxMsbr3ktHbuQ6d9QYIcegDUdAQ0+4TMHf8LqQZfngiOXN0Lu vm2iIRzADQB0j/I/7LSkUtbqzmnZnmhZjmaRkJC1oO7xSPtyqh7PrfHwKD1hkTfjtTyfN6mF K13TDR1+GGibWW2xXc32jc49B/n8bg8MJKAIiphtIOIjvhpw60bMBKWqGEvhoyvOazgWxa3+ XkklMFBYBe+nnRdma6rV/E3BTh6i8n7zvYxVqRkRLY0IfEbQN/L/AEqZNScxPf5UZllsp7yr h302WQsIcSJQ/cnQzmjuK4FC1Cpw6Rmz4PgOQTh3tQXc81c7lKt7ES+0tTDdMpAD/60oY6C+ NjZfuspMq+SWnqKkwxg1MfhOBFBh8Ib1C7qwk5y42oOgFt7TJEJxBy/r1Yop9on6hNOKWt5I z/Q+xVff91P5YrhJlGdZM8qP2MeyXwqCL3QRevyGvcZdY60lL22tTKCeYOlayXkKJh9upFpH 2GaiIBiVIP
  • Ironport-sdr: m4W1CVQvQCUKk3u7h3Wf1l11wyBcJn/Ixg7hLXgsieQhpcQI2iRNjIGWTsbzvYY5hgR/DB5J4u Y+wpUk9FckOP0Hkw6dZZfUtNfI0JnjQLCQzm0hhsfbjG5X22I6cPtRDV3sqeyKEG3vn1QjPxUP jOAe1qI+fzWgiDEybSqztToBVYMCN5WSa2ES1PRPRxmLiB0RzDMhtS+LF/bxY8lwqL5y/DJJaZ 2KxgM+/YyNkB1+zExRkwJpWEGVh2FoSohme+KfFtrLs0PYNodGKv7l4vgExxKtvv7VcZQNIyVV St6osZ373G2dWCe5Fbp86o4x
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 21/07/2021 05:09, Xentrigued wrote:
> Clearly, much effort has already been expended to support the Viridian
> enlightenments that optimize running Windows on Xen.  It also looks like a
> significant amount of effort has been put forth to advance nested
> virtualization in general.
> Therefore, if it would be helpful, I am willing to perform testing and
> provide feedback and logs as appropriate in order to help get this working.
> While my day job is managing a heterogeneous collection of systems running
> on various hypervisors, I have learned the rudiments of integrating patches
> and rebuilding Xen from source so could no doubt be useful in assisting you
> with this worthwhile endeavor.


Thankyou for your interest and volunteering.

Nested virt under Xen is a disaster.  It has been bitrotting for 5
years, and was introduced in an ill-advised way to begin with.

With my Citrix Hypervisor hat on, getting Windows VBS working is a high
priority, but other security work keeps on taking priority.  The
non-security work I am managing to do is all about CPUID and MSR
handling at the toolstack level (rectifying some 15 years of accumulated
technical debt), which is a prerequisite to being able to support nested
virtualisation on Intel in a sustainable way.

There are two things which I know definitely don't work.
1) NMI Virtualisation isn't advertised (but is available if you ignore
the signs of its absence).  Most hypervisors refuse to function without it.
2) VMCS-based EFER loading/saving doesn't work on virtual vmentry/exit.

Fixing 1) is a online patch.

diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c
index e9f94daf6493..4c80912368d5 100644
--- a/xen/arch/x86/hvm/vmx/vvmx.c
+++ b/xen/arch/x86/hvm/vmx/vvmx.c
@@ -2237,6 +2237,7 @@ int nvmx_msr_read_intercept(unsigned int msr, u64
         /* 1-settings */
         data = PIN_BASED_EXT_INTR_MASK |
                PIN_BASED_NMI_EXITING |
+               PIN_BASED_VIRTUAL_NMIS |
         data = gen_vmx_msr(data, VMX_PINBASED_CTLS_DEFAULT1, host_data);

Fixing 2) is more tricky.  I "broke" it when I fixed a more serious bug
in Xen by making use of EFER-loading in the first place.  This patch
ought to revert to the old behaviour.

diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index f9f9bc18cdbc..e4c353202e2a 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -418,7 +418,7 @@ static int vmx_init_vmcs_config(bool bsp)
     min |= VM_EXIT_IA32E_MODE;
     _vmx_vmexit_control = adjust_vmx_controls(
         "VMExit Control", min, opt, MSR_IA32_VMX_EXIT_CTLS, &mismatch);
@@ -458,7 +458,7 @@ static int vmx_init_vmcs_config(bool bsp)
         _vmx_secondary_exec_control &=
     min = 0;
     _vmx_vmentry_control = adjust_vmx_controls(
         "VMEntry Control", min, opt, MSR_IA32_VMX_ENTRY_CTLS, &mismatch);

For how, the most important thing is to get one other "modern"
hypervisor working under Xen.  Nested-virt is "just an emulation" of
VT-x (Intel) / SVM (AMD), so it doesn't matter very much what hypervisor
you use in the VM if you're trying to debug why Xen's nested virt is broken.

Furthermore, you stand a far better chance of getting something working
by picking an old version of Windows/HyperV first, as it will use fewer
"new" features in hardware.  The only way we're going to fix things is




Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.