Re: [Xen-devel] Xen osstest on Calxeda midway progress (Was: Re: [xen-unstable test] 21486: tolerable FAIL - PUSHED)

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Mon, 2013-11-11 at 18:42 +0000, Stefano Stabellini wrote:
> On Thu, 7 Nov 2013, Ian Campbell wrote:
> > > > BTW, at one point we discussed relaxing the 1:1 mapping into just a
> > > > contiguous mapping and publishing the offset to the guest, whatever
> > > > became of that idea?
> > > > 
> > > > Julien solved his issue on Arndale by allocating the 1:1 from as close
> > > > to the start of physical RAM as he could, but having an offset would be
> > > > more elegant and less prone to spurious breakage I think.
> > > 
> > > I take that by "and publishing the offset to the guest" you mean
> > > adjusting the start of the memory region in the device tree?
> > 
> > No, that's what we do today. i.e. if RAM is at 0x80000000 and we
> > allocate a 1:1 region at 0x80800000 then we give the guest a RAM region
> > at 0x80800000. This caused breakage on Exynos, so Julien change things
> > (6c21cb36e263 "Allocate memory for dom0 from the bottom with the 1:1
> > Workaround") to increase the chances that the memory would end up
> > allocated from closer to 0x80000000, which mostly works because Xen
> > normally tends to allocate the highest address it can which meets the
> > callers constraints (bit width etc) and we allocate memory for dom0
> > fairly early. But it is potentially fragile.
> > 
> > What I'm suggesting is to allocate a contiguous block of memory from
> > wherever (e.g 0x80800000) but tell the guest it has RAM at 0x80000000
> > (and map it there in the p2m) and separately expose the 0x00800000
> > offset for use in the various phys_to_bus calculations (in the swiotlb
> > or wherever). IOW in the places which your series determines it is safe
> > to just return the pseudophysical address it would instead apply the
> > offset.
> > 
> > I'm thinking in terms of a property in the hypervisor node, e.g.
> > dma-offset = <0x00800000>;
> > 
> > Obviously we would omit this (or explicitly set it to zero) when SMMU is
> > present.  
> 
> Technically it would work, but it has the feeling of an ad-hoc hack.
> 
> Basically we are doing this to make it look like the RAM is starting in
> the same position as the native platform and at the same time have a
> bit more freedom in dom0 memory allocation in the hypervisor.

Correct.

> However why do we even have a device tree property to express where the
> memory start, when this property can only have a single value?
> 
> I think that Xen should be free to make the guest memory start at any
> address it can express in device tree, within reasonable limits.
> Otherwise we can go back to the world of add-hoc hardcoded constants and
> get rid of device tree entirely.
> If this doesn't work, why the kernel can't be fixed?
> Am I being unreasonable about this?

Unfortunately the reality is that the Linux kernel has various
constraints on where RAM might be and how it is aligned WRT to the
kernel load address. This is particularly bad for non-multiplatform
kernels (i.e. Julien had issues on Arndale) but there are aso some
constraints on non-multiplatform ones (I don't think we've tripped over
any of them).

We could fight this and try and fix it but it would basically be a case
of "Xen dom0 is special compared to the real hardware", and it could
involve some nasty start-of-day munging.

Overall a conceptually simple and well defined "ad-hoc hack" seems
preferable to me.

> > How are we intending to signal to dom0 that it needn't do all the
> > swiotlb magic even for foreign pages when an SMMU is present? We should
> > build that scheme in now. Presence/absence of dma-offset might be a
> > convenient hook, but maybe it needs to be per-device?
>  
> There there cases:
> 
> 1) no IOMMU
> 2) all the devices are behind an IOMMU
> 3) some devices are behind an IOMMU
> 
> 1) and 2) are easy and a global property under the Xen node would
> suffice.
> 3) is the difficult case. We would need to express that some devices
> need to go through swiotlb-xen while others don't.
> We could add a xen-dma-safe property under each dma capable
> device that is behind an IOMMU programmed by Xen.
> It might be best to wait for IOMMU support in the hypervisor before we
> get into this.

Sure, I didn't want to suggest otherwise, just that whatever schemes we
put in place today were designed to be forwards compatible. "build" was
probably a bad choice of words, "think about" might have been better.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel