[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-devel] PV drivers and zero copying
Hi, Julien!
On 07/31/2017 12:24 PM, Julien Grall wrote:
(+ Joao)
On 31/07/17 09:34, Oleksandr Andrushchenko wrote:
Hi, all!
Hi Oleksandr,
The aim of this mail is to highlight and discuss possible approaches to
implementing zero copying for PV drivers. Rationale behind this is that
there
are use-cases when drivers operate with big shared buffers, e.g.
display, when
memory copying from front’s buffer into back’s one may significantly hit
performance of the system (for example, for para-virtual display running
at full
HD resolution at 60Hz it is approximately 475MB/sec).
Assumptions (which actually fit ARM platforms, but can be extended to
other
platforms as well): Dom0 is a 1:1 mapped privileged domain, runs backend
driver/software DomU is an unprivileged domain without 1:1 memory
mapping, runs
frontend driver
I would rather avoid to stick with this assumption on ARM. This was
only meant to be a workaround for platform without IOMMU (see [1]) and
we will get into trouble when using IOMMU.
You are correct, thank you
For instance, there are no requirement to have the IOMMU supporting as
many as address bits than the processor. So 1:1 mapping here will not
be an option.
Buffer origin: while implementing zero copying the buffer allocation can
happen
either on DomU’s end or Dom0’s one depending on the use-case and HW
capabilities/availability: When DomU allocates: It cannot guarantee
physical
memory continuity of the buffers allocated Dom0’s HW *can* handle
non-contiguous
memory buffers allocated by DomU for memory operations (DMA, for
example), e.g.
either with IOMMU help or by any other means (HW block’s own MMU). When
Dom0
allocates as it is mapped 1:1 it can allocate physically contiguous
memory
Dom0’s HW *cannot* handle non-contiguous memory buffers allocated by
DomU for
memory operations by any means.
I am not sure to follow this. How zero copy is related to 1:1 mapping?
Is it because you have hardware that does not support scatter/gather
IO or IOMMU?
yes, you got it right
1 Sharing with granted references
==================================
1-1 Buffer allocated @DomU
--------------------------
@DomU
alloc_xenballooned_pages(nr_pages, pages);
cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
<pass grant_ref_t[] to Dom0>
@Dom0
alloc_xenballooned_pages(nr_pages, pages);
gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map |
GNTMAP_device_map,
grefs[i], otherend_id);
gnttab_map_refs(map_ops, NULL, pages, nr_pages);
1-2 Buffer allocated @Dom0
--------------------------
@Dom0
<the code below is equivalent to xen_alloc_ballooned_pages without
PV MMU support as seen in the balloon driver, the difference is
that
pages are explicitly allocated to be used for DMA>
dma_alloc_wc(dev, size, &dev_addr, GFP_KERNEL | __GFP_NOWARN);
HYPERVISOR_memory_op(XENMEM_populate_physmap, &reservation);
cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
<pass grant_ref_t[] to DomU>
@Dom0
alloc_xenballooned_pages(nr_pages, pages);
gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map, grefs[i],
otherend_id);
gnttab_map_refs(map_ops, NULL, pages, nr_pages);
2 Sharing with page transfers (GNTTABOP_transfer)
==================================================
FIXME: This use-case seems to be only needed while allocating physically
contiguous buffers at Dom0. For the reverse path 1-1 method can be used.
This approach relies on GNTTABOP_transfer API: “transfer <frame> to a
foreign
domain. The foreign domain has previously registered its interest in the
transfer via <domid, ref>”, for full documentation see [1]. The
process of
transferring pages is explained by Christopher Clark at [2] and is
available as
implementation at [3], [4]. The relevant logic is in:
xen/common/grant_table.c :
gnttab_transfer.
Basic workflow explained to me by Christopher:
- The mfn starts as owned by the sending domain, and that domain removes
any
mappings of it from its page tables. Xen will enforce that the
reference count
must be low enough for the transfer to succeed.
- The receiving domain indicates interest for receiving a page by
writing an
entry in its grant table.
- You'll need to communicate the grant ref from the receiver to the
sender (eg.
via xenstore or another existing channel)
- The sending domain invokes the hypercall, with the grant ref from the
receiving domain.
- The sending domain notifies the receiving domain somehow that the
transfer has
completed. (eg. send an event or via xenstore)
- Once the transfer has completed, the receiving domain will need to map
the
newly assigned page.
- Note: For the transfer, the receiving domain must have enough
headroom to
receive the new page, which means it must not have allocated all of
its memory
quota already prior to the transfer. Typically this can be ensured by
freeing
enough memory back to Xen before writing the grant ref.
3 Sharing with page exchange (XENMEM_exchange)
==============================================
This API was pointed to me by Stefano Stabellini as one of the possible
ways to
achieve zero copying and share physically contiguous buffers. It is used
by x86
SWIOTLB code (xen_create_contiguous_region, [5]), but as per my
understanding
this API cannot be used on ARM as of now [6]. Conclusion: not an option
for ARM
at the moment
Comparison for display use-case
===============================
1 Number of grant references used
1-1 grant references: nr_pages
1-2 GNTTABOP_transfer: nr_pages
1-3 XENMEM_exchange: not an option
2 Effect of DomU crash on Dom0 (its mapped pages)
2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
recovered
2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
for Dom0
2-3 XENMEM_exchange: not an option
3 Security issues from sharing Dom0 pages to DomU
1-1 grant references: none
1-2 GNTTABOP_transfer: none
1-3 XENMEM_exchange: not an option
At the moment approach 1 with granted references seems to be a winner
for
sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
Conclusion
==========
I would like to get some feedback from the community on which approach
is more
suitable for sharing large buffers and to have a clear vision on cons
and pros
of each one: please feel free to add other metrics I missed and correct
the ones
I commented on. I would appreciate help on comparing approaches 2 and 3
as I
have little knowledge of these APIs (2 seems to be addressed by
Christopher, and
3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
Thank you,
Oleksandr
[1]
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/grant_table.h;h=018036e825f8f2999812cdb089f7fa2195789231;hb=HEAD#l414
[2] https://xenbits.xen.org/docs/4.9-testing/misc/grant-tables.txt
[3]
https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netfront
[4]
https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netback
[5]
http://elixir.free-electrons.com/linux/latest/source/arch/x86/xen/mmu_pv.c#L2618
[6]
https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02110.html
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel
Thank you,
Oleksandr
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel
|
