Re: [Xen-devel] Bug: Limitation of <=2GB RAM in domU persists with 4.3.0

[Gordan Bobic <gordan@xxxxxxxxxx>]

On Fri, 26 Jul 2013 01:21:24 +0100, Ian Campbell 
<ian.campbell@xxxxxxxxxx> wrote:
> On Thu, 2013-07-25 at 23:23 +0100, Gordan Bobic wrote:
> > Now, if I am understanding the basic nature of the problem 
> > correctly,
> > this _could_ be worked around by ensuring that vBAR = pBAR since in 
> > that
> > case there is no room for the mis-mapped memory overwrites to occur. 
> > Is
> > that correct?
> AIUI (which is not very well...) it's not so much vBAR=pBAR but 
> making
> the guest e820 (memory map) have the same MMIO holes as the host so 
> that
> there can't be any clash between v- or p-BAR and RAM in the guest.
Sure, I understand that - but unless I am overlooking something,
vBAR=pBAR implicitly ensures that.

The question, then, is what happens in the null translation instance.
Specifically, if the PCIe bridge/router is broken (and NF200 is, it
seems), it would imply that when the driver talks to the device, the
operation will get sent to the vBAR (=pBAR, i.e. straight to the
hardware). This then gets translated to the pBAR. But - with a
broken bridge, and vBAR=pBAR, the MMIO request hits the pBAR
directly from the guest. Does it then still get intercepted by
the hypervisor, translated (null operation), and re-transmitted?
If so, this would lead to the card receiving everything twice,
resulting either in things outright breaking or going half as
fast at best.

Now, all this could be a good thing or a bad thing, depending on
how exactly you spin it. If the bridge is broken and doesn't
route all the way back to the root bridge, this could actually be
a performance optimizing feature. If we set vBAR=pBAR and disable
any translation thereafter, this avoids the overhead of passing
everything to/from the root PCIe bridge, and we can just directly
DMA everything.

I'm sure there are security implications here, but since NF200
doesn't do PCIe ACS either, any concept of security goes out
the window pre-emptively.

So, my question is:
1) If vBAR = pBAR, does the hypervisor still do any translation?
I presume it does because it expects the traffic to pass up
from the root bridge, to the hypervisor and then back, to
ensure security. If indeed it does do this, where could I
optionally disable it, and is there an easy to follow bit of
example code for how to plumb in a boot parameter option for
this?

2) Further, I'm finding myself motivated to write that
auto-set (as opposed to hard coded) vBAR=pBAR patch discussed
briefly a week or so ago (have an init script read the BAR
info from dom0 and put it in xenstore, plus a patch to
make pBAR=vBAR reservations built dynamically rather than
statically, based on this data. Now, I'm quite fluent in C,
but my familiarity with Xen soruce code is nearly non-existant
(limited to studying an old unsupported patch every now and then
in order to make it apply to a more recent code release).
Can anyone help me out with a high level view WRT where
this would be best plumbed in (which files and the flow of
control between the affected files)?

The added bonus of this (if it can be made to work) is that
it might just make unmodified GeForce cards work, too,
which probably makes it worthwhile on it's own.

> > I guess I could test this easily enough by applying the vBAR = pBAR 
> > hack.
> Does the e820_host=1 option help? That might be PV only though, I 
> can't
> remember...
Thanks for pointing this one out, I just found this post in the 
archives:
http://lists.xen.org/archives/html/xen-users/2012-08/msg00150.html

With a broken PCIe router, would I also need iommu=soft?

Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel