Re: [Xen-devel] Future support of 5-level paging in Xen:wq

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 09/12/16 19:31, Stefano Stabellini wrote:
> On Fri, 9 Dec 2016, Juergen Gross wrote:
>> On 09/12/16 00:50, Stefano Stabellini wrote:
>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>>>> On 08/12/2016 19:18, Stefano Stabellini wrote:
>>>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>>>>>> On 08/12/16 16:46, Juergen Gross wrote:
>>>>>>> The first round of (very preliminary) patches for supporting the new
>>>>>>> 5-level paging of future Intel x86 processors [1] has been posted to
>>>>>>> lkml:
>>>>>>>
>>>>>>> https://lkml.org/lkml/2016/12/8/378
>>>>>>>
>>>>>>> An explicit note has been added: "CONFIG_XEN is broken." and
>>>>>>> "I would appreciate help with the code."
>>>>>>>
>>>>>>> I think we should start a discussion what we want to do in future:
>>>>>>>
>>>>>>> - are we going to support 5-level paging for PV guests?
>>>>>>> - do we limit 5-level paging to PVH and HVM?
>>>>>> The 64bit PV ABI has 16TB of virtual address space just above the upper
>>>>>> 48-canonical boundary.
>>>>>>
>>>>>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
>>>>>> kernel with exactly the same amount of higher half space as it currently
>>>>>> has, or we'd have to recompile Xen as properly position-independent and
>>>>>> use a different virtual range in different paging mode.
>>>>>>
>>>>>> Another pain point is the quantity of virtual address space handed away
>>>>>> in the ABI.  We currently had 97% of the virtual address space away to
>>>>>> 64bit PV guests, and frankly this is too much.  This is the wrong way
>>>>>> around when Xen has more management to do than the guest.  If we were to
>>>>>> go along the 5-level PV guests route, I will insist that there is a
>>>>>> rather more even split of virtual address space baked into the ABI.
>>>>>>
>>>>>> However, a big question is whether any of this effort is worth doing, in
>>>>>> the light of PVH.
>>>>> With my Aporeto hat on, I'll say that given the overwhelming amount of
>>>>> hardware available out there without virtualization support, this work
>>>>> is worth doing. I am referring to all the public cloud virtual machines,
>>>>> which can support Xen PV guests but cannot support PVH guests.
>>>> Why is Xen supporting 5-level guests useful for running in a PV cloud
>>>> VM?  Xen doesn't run PV.
>>>>
>>>> I am not suggesting that we avoid adding 5-level support to Xen.  We
>>>> should absolutely do that.  The question is only whether we extend the
>>>> PV ABI to support 5-level PV guests.  Conceptually, its very easy to
>>>> have a 5-level Xen but only supporting 4-level PV guests.
>>>>
>>>> VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
>>>> surprised if you would find much hardware of this age in any cloud; you
>>>> can't by anything that old, and support contracts have probably run out
>>>> if you have owned that hardware for 10 years.
>>> I am thinking that in a couple of years, we might already find VMs so
>>> large that to use all the memory in a nested virt scenario, we need
>>> 5-level PV abi support.
>>>
>> No, I don't think so. I believe there will be no hardware capable of
>> 5-level paging but without VMX/SVM support. Support of PVH/HVM for such
>> large guests should be enough. We don't need to extend PV which we want
>> to get rid of in Linux anyway, no?
> I am talking about nested virtualization when the L1 virtual machine
> does not support nested VMX or SVM. No Amazon AWS virtual machines
> support nested VMX, but it is possible to run Xen PV virtual machines on
> top of any Amazon HVM instance.
>
> When 5-level pagetable hardware will become available on Amazon AWS, it
> might be possible to get virtual machines so large that in order to use
> all the memory, you need to use 5-level pagetables in L1 Xen. In this
> scenario, if we want to create a L2 virtual machine as large as possible
> we will need support for 5-level page tables in the PV ABI.
>
> Please correct me if I am wrong.

That is a valid scenario, although I don't think its very likely to happen.

Intel currently tops out at 46 bits physical (64TB), according to the
whitepaper, while a lot of AMD hardware has 48 bits physical (256TB). I
dread to think how much AWS would charge you for that much RAM, or how
much Amazon would be charged to buy such a server in the first place.

This is more RAM that Xen can currently handle, and isn't going to
change without breaking the current ABI.

Also, given the rise of virtualisation-based security solutions even by
Microsoft themselves in Windows 10, I think the chances are good that
you will be able to get VMs with nested virt before being able to get
VMs large enough to need 5-level paging.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel