Re: Nested Virtualization of Hyper-V on Xen Not Working

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 21/07/2021 05:09, Xentrigued wrote:
> SUMMATION:
> Clearly, much effort has already been expended to support the Viridian
> enlightenments that optimize running Windows on Xen.  It also looks like a
> significant amount of effort has been put forth to advance nested
> virtualization in general.
>
> Therefore, if it would be helpful, I am willing to perform testing and
> provide feedback and logs as appropriate in order to help get this working.
>
> While my day job is managing a heterogeneous collection of systems running
> on various hypervisors, I have learned the rudiments of integrating patches
> and rebuilding Xen from source so could no doubt be useful in assisting you
> with this worthwhile endeavor.

Hello,

Thankyou for your interest and volunteering.

Nested virt under Xen is a disaster.  It has been bitrotting for 5
years, and was introduced in an ill-advised way to begin with.

With my Citrix Hypervisor hat on, getting Windows VBS working is a high
priority, but other security work keeps on taking priority.  The
non-security work I am managing to do is all about CPUID and MSR
handling at the toolstack level (rectifying some 15 years of accumulated
technical debt), which is a prerequisite to being able to support nested
virtualisation on Intel in a sustainable way.

There are two things which I know definitely don't work.
1) NMI Virtualisation isn't advertised (but is available if you ignore
the signs of its absence).  Most hypervisors refuse to function without it.
2) VMCS-based EFER loading/saving doesn't work on virtual vmentry/exit.

Fixing 1) is a online patch.

diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c
index e9f94daf6493..4c80912368d5 100644
--- a/xen/arch/x86/hvm/vmx/vvmx.c
+++ b/xen/arch/x86/hvm/vmx/vvmx.c
@@ -2237,6 +2237,7 @@ int nvmx_msr_read_intercept(unsigned int msr, u64
*msr_content)
         /* 1-settings */
         data = PIN_BASED_EXT_INTR_MASK |
                PIN_BASED_NMI_EXITING |
+               PIN_BASED_VIRTUAL_NMIS |
                PIN_BASED_PREEMPT_TIMER;
         data = gen_vmx_msr(data, VMX_PINBASED_CTLS_DEFAULT1, host_data);
         break;


Fixing 2) is more tricky.  I "broke" it when I fixed a more serious bug
in Xen by making use of EFER-loading in the first place.  This patch
ought to revert to the old behaviour.

diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index f9f9bc18cdbc..e4c353202e2a 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -418,7 +418,7 @@ static int vmx_init_vmcs_config(bool bsp)
 
     min = VM_EXIT_ACK_INTR_ON_EXIT;
     opt = (VM_EXIT_SAVE_GUEST_PAT | VM_EXIT_LOAD_HOST_PAT |
-           VM_EXIT_LOAD_HOST_EFER | VM_EXIT_CLEAR_BNDCFGS);
+           VM_EXIT_CLEAR_BNDCFGS);
     min |= VM_EXIT_IA32E_MODE;
     _vmx_vmexit_control = adjust_vmx_controls(
         "VMExit Control", min, opt, MSR_IA32_VMX_EXIT_CTLS, &mismatch);
@@ -458,7 +458,7 @@ static int vmx_init_vmcs_config(bool bsp)
         _vmx_secondary_exec_control &=
~SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS;
 
     min = 0;
-    opt = (VM_ENTRY_LOAD_GUEST_PAT | VM_ENTRY_LOAD_GUEST_EFER |
+    opt = (VM_ENTRY_LOAD_GUEST_PAT |
            VM_ENTRY_LOAD_BNDCFGS);
     _vmx_vmentry_control = adjust_vmx_controls(
         "VMEntry Control", min, opt, MSR_IA32_VMX_ENTRY_CTLS, &mismatch);


For how, the most important thing is to get one other "modern"
hypervisor working under Xen.  Nested-virt is "just an emulation" of
VT-x (Intel) / SVM (AMD), so it doesn't matter very much what hypervisor
you use in the VM if you're trying to debug why Xen's nested virt is broken.

Furthermore, you stand a far better chance of getting something working
by picking an old version of Windows/HyperV first, as it will use fewer
"new" features in hardware.  The only way we're going to fix things is
incrementally.

~Andrew