[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Design session notes: GPU acceleration in Xen


  • To: Demi Marie Obenour <demi@xxxxxxxxxxxxxxxxxxxxxx>, Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>, Roger Pau Monné <roger.pau@xxxxxxxxxx>, "Pelloux-prayer, Pierre-eric" <Pierre-eric.Pelloux-prayer@xxxxxxx>
  • From: Christian König <christian.koenig@xxxxxxx>
  • Date: Wed, 19 Jun 2024 09:31:20 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FvNrR+GTYGF30Y9iOccAGxsRHNyVf6gXTJUZybTO+ww=; b=VOpeJZtqiUy4uWAqPjsDWWMMd5j6rOSNYul5BKZJhlau3oNdK0S67OwQ5qBIlzkFJnbVe5xDkQ83MGaxwmtSuK13IalOdwoT/BGmMwzUjZ8sS0AKf5hieXnkBiVPQdt5Ilgk6ZQlSSwFa0evLf48D6MBHvWnzAu5SkMnLiMH3gUvjARDWKrtdAIctCSlXqiYE8PvW81hX9vi6tFfLlLPCUWrL6CTpWCte6E/tuF43W2Oe+TGy1gaKrgfW3Ufv6dIzOCPs5Jn61PC7h7KrYYKLKfsLz1YvFbOzZhthPrAmVqVCur4KRNUypl5TotgpuiHxoMu7K8/y12ZR9khXXQqnw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ax0U8SUQOkU4K9AFH5oPHnVNTUAAqxbIbJBFMdsYKu0RepiCra9h9kTkaP71OUlZ6XTkE4SL5wO2F3qNaBjlzFRD7E94ENKcJ+37JgyKQj9N9TidDiZW8eUwtxpjQvmF67j1GJSkAFn88eyjJrT87jgawHjYOZvypC79Bien+pudrJ/EUUB4KkwbqMWFhElnHkXdE82evlZeGFSJ8XzNvBS+z0R1dJvUXDyYuytyMLny1U2Rzjs3vseAIUipnyggpSYr78ip0Sqbi289ly7vpPCbrKCFlAePhJ3sQfKDbAh2HQbPklr4ZwM59TFDgu+DTo0XjcyLNhwzwu6PPkwQ5g==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com;
  • Cc: Jan Beulich <jbeulich@xxxxxxxx>, Xenia Ragiadakou <burzalodowa@xxxxxxxxx>, Ray Huang <ray.huang@xxxxxxx>, Xen developer discussion <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Direct Rendering Infrastructure development <dri-devel@xxxxxxxxxxxxxxxxxxxxx>, Qubes OS Development Mailing List <qubes-devel@xxxxxxxxxxxxxxxx>
  • Delivery-date: Wed, 19 Jun 2024 07:31:40 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

Am 18.06.24 um 16:12 schrieb Demi Marie Obenour:
On Tue, Jun 18, 2024 at 08:33:38AM +0200, Christian König wrote:
> Am 18.06.24 um 02:57 schrieb Demi Marie Obenour:
>> On Mon, Jun 17, 2024 at 10:46:13PM +0200, Marek Marczykowski-Górecki
>> wrote:
>>> On Mon, Jun 17, 2024 at 09:46:29AM +0200, Roger Pau Monné wrote:
>>>> On Sun, Jun 16, 2024 at 08:38:19PM -0400, Demi Marie Obenour wrote:
>>>>> In both cases, the device physical
>>>>> addresses are identical to dom0’s physical addresses.
>>>>
>>>> Yes, but a PV dom0 physical address space can be very scattered.
>>>>
>>>> IIRC there's an hypercall to request physically contiguous memory for
>>>> PV, but you don't want to be using that every time you allocate a
>>>> buffer (not sure it would support the sizes needed by the GPU
>>>> anyway).
>>
>>> Indeed that isn't going to fly. In older Qubes versions we had PV
>>> sys-net with PCI passthrough for a network card. After some uptime it
>>> was basically impossible to restart and still have enough contagious
>>> memory for a network driver, and there it was about _much_ smaller
>>> buffers, like 2M or 4M. At least not without shutting down a lot more
>>> things to free some more memory.
>>
>> Ouch!  That makes me wonder if all GPU drivers actually need physically
>> contiguous buffers, or if it is (as I suspect) driver-specific. CCing
>> Christian König who has mentioned issues in this area.

> Well GPUs don't need physical contiguous memory to function, but if they
> only get 4k pages to work with it means a quite large (up to 30%)
> performance penalty.

The status quo is "no GPU acceleration at all", so 70% of bare metal
performance would be amazing right now.

Well AMD uses native context approach in XEN which which delivers over 90% of bare metal performance.

Pierre-Eric can tell you more, but we certainly have GPU solutions in productions with XEN which would suffer greatly if we see the underlying memory fragmented like this.

  However, the implementation
should not preclude eliminating this performance penalty in the future.

What size pages do GPUs need for good performance?  Is it the same as
CPU huge pages?

2MiB are usually sufficient.

Regards,
Christian.

  PV dom0 doesn't get huge pages at all, but PVH and HVM
guests do, and the goal is to move away from PV guests as they have lots
of unrelated problems.

> So scattering memory like you described is probably a very bad idea if you
> want any halve way decent performance.

For an initial prototype a 30% performance penalty is acceptable, but
it's good to know that memory fragmentation needs to be avoided.

> Regards,
> Christian

Thanks for the prompt response!




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.