Re: [Xen-devel] Upstream Dom0 DRM problems regarding swiotlb

[Juergen Gross <jgross@xxxxxxxx>]

On 14/02/2019 01:11, Andrew Cooper wrote:
> On 13/02/2019 21:08, Michael Labriola wrote:
>> On Wed, Feb 13, 2019 at 3:21 PM Andrew Cooper <andrew.cooper3@xxxxxxxxxx> 
>> wrote:
>>> On 13/02/2019 20:15, Michael Labriola wrote:
>>>> On Wed, Feb 13, 2019 at 2:16 PM Konrad Rzeszutek Wilk
>>>> <konrad.wilk@xxxxxxxxxx> wrote:
>>>>> On Wed, Feb 13, 2019 at 01:38:21PM -0500, Michael Labriola wrote:
>>>>>> On Wed, Feb 13, 2019 at 1:16 PM Michael Labriola
>>>>>> <michael.d.labriola@xxxxxxxxx> wrote:
>>>>>>> On Wed, Feb 13, 2019 at 11:57 AM Konrad Rzeszutek Wilk
>>>>>>> <konrad.wilk@xxxxxxxxxx> wrote:
>>>>>>>> On Wed, Feb 13, 2019 at 09:09:32AM -0700, Jan Beulich wrote:
>>>>>>>>>>>> On 13.02.19 at 17:00, <michael.d.labriola@xxxxxxxxx> wrote:
>>>>>>>>>> On Wed, Feb 13, 2019 at 9:28 AM Jan Beulich <JBeulich@xxxxxxxx> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> On 13.02.19 at 15:10, <michael.d.labriola@xxxxxxxxx> wrote:
>>>>>>>>>>>> Ah, so this isn't necessarily Xen-specific but rather any 
>>>>>>>>>>>> paravirtual
>>>>>>>>>>>> guest?  That hadn't crossed my mind.  Is there an easy way to find 
>>>>>>>>>>>> out
>>>>>>>>>>>> if we're a pv guest in the need_swiotlb conditionals?
>>>>>>>>>>> There's xen_pv_domain(), but I think xen_swiotlb would be more to
>>>>>>>>>>> the point if the check is already to be Xen-specific. There's no 
>>>>>>>>>>> generic
>>>>>>>>>>> "is PV" predicate that I'm aware of.
>>>>>>>>>> Well, that makes doing conditional code right more difficult.  I
>>>>>>>>>> assume since there isn't a generic predicate, and PV isn't new, that
>>>>>>>>>> it's absence is by design?  To reign in the temptation to sprinkle
>>>>>>>>>> conditional code all over the kernel?  ;-)
>>>>>>>>> Well, with lguest gone, Xen is the only PV environment the kernel
>>>>>>>>> can run in, afaik at least. I guess to decide between the suggested
>>>>>>>>> options or the need for some abstracting macro (or yet something
>>>>>>>>> else), you'll really need to ask the driver maintainers. Or simply
>>>>>>>>> send a patch their way implementing one of them, and see what
>>>>>>>>> their reaction is.
>>>>>>>> Could you try this out and see if it works and I will send it out:
>>>>>>>>
>>>>>> *snip*
>>>>>>> Yes, that works for me.  However, I feel like the conditional should
>>>>>>> be in drm_get_max_iomem() instead of directly after it everywhere it's
>>>>>>> used...  and is just checking xen_pv_domain() enough?  Jan made it
>>>>>>> sound like there were possibly other PV cases that would also still
>>>>>>> need swiotlb.
>>>>>> How about this?  It strcmp's pv_info to see if we're bare metal, does
>>>>>> the comparison in a single place, moves the bit shifting comparison
>>>>>> into the function (simplifying the drm driver code), and renames the
>>>>>> function to more aptly describe what's going on.
>>>>> <nods> That looks much better.
>>>> Great!  Now the only question left is:  KVM, VMware, Xen PVH, Xen HVM,
>>>> and Xen PV all populate pv_info.  Do any of those other than Xen PV
>>>> *really* need swiotlb.  That's slightly over my head.  As written, my
>>>> patch would require swiotlb for all of them because I was attempting
>>>> to not be Xen-specific.
>>> Its far more complicated that "Xen PV requires swiotlb".
>>>
>>> I presume the underlying problem here is DRM being special and not
>>> DMA-mapping its buffers, and trying to DMA to a buffer crossing a 4k
>>> boundary?
>> Well, I don't 100% understand how all these things work...  but here's
>> what I do know.  There are a series of commits in v4.17 that try to
>> optimize the radeon and amdgpu drivers by skipping calls to
>> ttm_dma_populate() and ttm_dma_unpopulate() unless they're "really
>> needed".  The original commit determines if swiotlb is needed by
>> checking to see if the max io mapping address of system memory is over
>> the video card's accessing range.  I can no longer log into Gnome on a
>> Xen dom0 after upgrading my kernel to v4.20 because the code that's no
>> longer happening was actually needed in a paravirtualized environment.
> 
> But from the log you provided, your bug was space exhaustion in the
> swiotlb, no?
> 
>> So, I'm trying to get all my details straight so I can submit a patch
>> to fix it w/out saying anything factually incorrect.
> 
> The thing which is different between Xen PV guests and most others (all
> others(?), now that Lguest and UML have been dropped) is that what Linux
> thinks of as PFN $N isn't necessarily adjacent to PFN $N+1 in system
> physical address space.
> 
> Therefore, code which has a buffer spanning a page boundary can't just
> convert a pointer to the buffer into a physical address, and hand that
> address to a device.  You generally end up with either memory corruption
> (DMA hitting the wrong page allocated to the guest), or an IOMMU fault
> (DMA hitting a pages which isn't allocated to the guest).
> 
> Xen PV is very good at finding DMA bugs in drivers.  The way to resolve
> this is to fix the driver to use the proper DMA APIs - not to add even
> more magic corner cases.
> 
> In general, a lot of devices can do 4k scatter/gather, or end up making
> requests to buffers which fit within a single page, but the SWIOTLB does
> act as a mechanism of last resort.  It has a massive performance penalty
> (due to double buffering), and does have a tendency to fragment (due to
> asymmetric size requests).
> 
> However, there is one DMA mode (in the process of getting properly
> upstream, but has been used for several years by various downstreams)
> where IOVA == Linux's idea of contiguous PFN space, so you can do odd
> sized DMAs which cross page boundaries.
> 
> The point is that the DMA ops (and *only* the DMA ops, from a
> correctness standpoint) know how to convert PFNs into IO-virtual
> addresses for devices, because it may not be a 1:1 mapping.  Nothing
> else in the kernel can legitimately be making decisions like this.

Correct. Adding Christoph who might want to add something.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel