Re: [Xen-devel] Bug: Limitation of <=2GB RAM in domU persists with 4.3.0

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 07/31/2013 06:53 PM, George Dunlap wrote:
> On Fri, Jul 26, 2013 at 2:11 PM, Gordan Bobic <gordan@xxxxxxxxxx> wrote:
> > > Now that is intereting - if this makes the memory holes the same between
> > > the guest and the host, does it also implicitly vBAR=pBAR?
> >
> > Another thing that occurred to me might be useful to check - it is
> > pretty easy to modify the BAR size on Nvidia cards. The defaults are
> > 64MB and 128MB for the two BARs. They can be made much, much larger,
> > and there is often advantage to enlarging them to at least be equal to
> > VRAM size. Soooooo... If I boost the BAR from 128MB to 2GB, being a
> > 64-bit BAR, it might make the BIOS do the sane thing and map it above
> > 4GB. With the other BAR also suitably enlarged and it being done on
> > the second GPU as well, there is no obvious option but to map them
> > above 4GB (unless the BIOS is broken, which it may well be, in
> > which case all bets are off).
> >
> > Which may just alleviate the memory issue if not completely fix
> > the problem.
> >
> > Will try this and see what happens.
> I believe XenServer has a patch that allows the toolstack (in this
> case xapi) to set the default size of the MMIO hole.  Andrew, did that
> ever make it upstream?
>
> Unfortunately, it is unlikely to work with upstream qemu until we fix
> the memory relocation issue...
Interesting you should mention something like this. I've been pondering 
whether it might be easier (even if it is a bodge) to simply always set 
the domU E820 map to have 0x80000000 - 0xFFFFFFFF (2GB->4GB) reserved. I 
have not yet seen a motherboard that maps 32-bit BARs below 2GB.
Note: Admittedly, I haven't tested what happens when you have multiple 
Nvidia cards each with a 1GB 32-bit BAR, though, I fully expect 
weirdness. And Nvidia cards have have the 32-bit BAR0 up to 2GB in size! 
But I cannot see a good reason to use such a configuration since it's 
the 64-bit BAR1 (up to 64GB in size) that provides the direct VRAM mapping.
Anyway, if the whole 2GB->4GB area was reserved, then presumably Xen 
would map the 32-bit bars below 2GB, which, provided there's enough 
memory for the OS kernel to load and the BARs, shouldn't be a problem (I 
cannot think of a sane case where this wouldn't hold). 64-bit BARs can 
get re-mapped somewhere sky-high in domU RAM (at the top of the 
addressable range sounds like a reasonable bet, BIOS (for non-broken 
BIOS implementations, of which there seem to be fewer than I'd like to 
believe) would probably set those just above the size of RAM in the 
machine, so to 2^48 minus BAR size would possibly be a safe place to map 
them.
Yes, I know it's a bodge. Yes, I know it wouldn't solve the GeForce 
passthrough problem. Yes, host E820 with vBAR = pBAR (possibly without 
IOMMU involvement) would be an awesome feature to have. But the bodge of 
just punching a 2GB hole at 2GB might just be a lot easier to implement 
as a quick fix.
Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel