[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: RMRRs and Phantom Functions


  • To: Roger Pau Monne <roger.pau@xxxxxxxxxx>
  • From: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>
  • Date: Wed, 27 Apr 2022 10:17:49 +0000
  • Accept-language: en-GB, en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=64WkSlMfxmMClRJnN3KRM3zv1EcSh3KQ22VB0ss8hyY=; b=cFqchBBi10wW3s+BmgJg3KtOW0siJXHiKQUHc45NSIpsatFXG9hQEqVG54FTmCL42y9bOAnKeYuEQIu0QObC8TNO/QiZ1YrDqEK4dEhFwxaaPC4Z1RfKRy/dodevTLinGBS57gGFNqxQqpMS2QzHWZJr2dYREwneGVDmb3VLl6ov5BsgKAfyhoOKR3nR8tuR280AbR6qGZp8G+qEDLpWQf8FHssQzQvC91MkG6JxPECe8EIu15AjTaIRQV8stBsap2CbtGA+OQaC3jaRWseEIADRn4RKxhz7HgDzA1b4TUJ5+LQ/mGMbfDYl8ivOkur8rWsTWr7hoplx2rY/dCHNxw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WcQURrpCnRL3BEgoif38/oEjcKzKjG4qlJpMIzp1ll51h6sWt4J4g7jdF9uaDk9GavZ/qZJyBPnGpM+FSAHTFOWJprsb3bQIpkwX3svtZBE2HFqO6gZwXC6t+ueUfzlFyamd1bwLdyznLetghxdnW/vSUPX8fVxX71PAy4DPVMSR7vN+dm+Y+D1q3CvacFnFpCnkObcCrastirIXWK33u8l3ULX52nh7g+KpeiIc04506SG11w2fseWT/6PI6C4oAdKujqlklGnkebh36RfZLQ7DjAR0QNes1sfo3iBDp4PVZa5HDEqjPhxQtcKAwxy1TBXeQ62l7b0QwwL5EZO2aw==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com;
  • Cc: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Kevin Tian <kevin.tian@xxxxxxxxx>, Edwin Torok <edvin.torok@xxxxxxxxxx>
  • Delivery-date: Wed, 27 Apr 2022 10:18:16 +0000
  • Ironport-data: A9a23:PQGFWqKbxipoRd2sFE+R1JQlxSXFcZb7ZxGr2PjKsXjdYENS0jcOy zNNX2DTOvjbYGOjc9lxbIyypkhUuZDRzt9qGgtlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s q3yv/GZdJhcokf0/0vrav67xZVF/fngqoDUUYYoAQgsA148IMsdoUg7wbRh3tQ22YPR7z6l4 rseneWOYDdJ5BYsWo4kw/rrRMRH5amaVJsw5zTSVNgT1LPsvyB94KE3fMldG0DQUIhMdtNWc s6YpF2PEsE1yD92Yj+tuu6TnkTn2dc+NyDW4pZdc/DKbhSvOkXee0v0XRYRQR4/ttmHozx+4 Ilqt8KLFwg0BPXjotUkfCBnLzshHqITrdcrIVDn2SCS52vvViOwhtBRVgQxN4Be/ftrC2ZT8 /BeMCoKch2Im+OxxvS8V/VogcMgasLsOevzuFk5lW2fUalgHsuFGvqSjTNb9G5YasRmNPDSf ccGLxFoawzNeUZnMVYLEpMu2uyvgxETdhUH8ArJ9fdqvQA/yiQv8/vcLsX3PeCsTMNPlUuUh HPqokDAV0Ry2Nu3jGDtHmiXrv/Cm2b3VZwfEJW89+V2mxuDy2oLEhoUWFCn5/6jhSaWWdhSN kgV8SoGtrUp+QqgSdyVdx61vn+C+AIdUtx4Eusm5QXLwa3Rizt1HUABRz9FLdk57sk/QGR60 kfTxoyxQztyrLeSVHSRsK+Oqi+/MjQUKmlEYjIYSQwC4J/op4RbYg/zc+uP2ZWd1rXdcQwcC RjQxMTir93/VfI26pg=
  • Ironport-hdrordr: A9a23:LuEZpKzLis7RLViPFhP5KrPxc+gkLtp133Aq2lEZdPULSKGlfp GV9sjziyWetN9IYgBapTiBUJPwIk81bfZOkMYs1MSZLXPbUQyTXc9fBOrZsnHd8kjFmtK1up 0QFJSWZOeQMbE+t7eD3ODaKadg/DDkytHPuQ629R4EIm9XguNbnn5E422gYy9LrXx9dP4E/e 2nl696TlSbGUg/X4CePD0oTuLDr9rEmNbNehgdHSMq7wGIkHeB9KP6OwLw5GZcbxp/hZMZtU TVmQ3w4auu99uhzAXH6mPV55NK3PP819p4AtCWgMR9EESvtu/oXvUlZ1SxhkFznAid0idtrD AKmWZ4Ay1H0QKUQohym2q05+Cv6kd015ao8y7kvZKqm72EeNt9MbsAuWsRSGqm16Jr1usMr5 5jziaXsYFaAgjHmzm479/UVwtynk7xunY6l/UP5kYvGrf2x4Uh37D30XklWavoJhiKoLzP0d Meef309bJTaxeXfnrZtm5gzJilWWkyBA6PRgwHttaO2zZbkXhlxw9ArfZv00so5dY4Ud1J9u 7EOqNnmPVHSdIXd7t0AKMETdGsAmLATBrQOCaZIEjhFqsAJ3XRwqSHqokd9aWvYtgF3ZEykJ POXBdRsnMzYVvnDYmU0JhC4nn2MROAtPTWu7ZjDrRCy83BreDQQFy+oXgV4ridiuRaBNHHUP CuP58TC+P/LALVaPJ04zE=
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Thread-index: AQHYWZZDIrEkmq9fL0+3qLNT2+7QyK0DZ3OAgAAld4A=
  • Thread-topic: RMRRs and Phantom Functions

On 27/04/2022 09:03, Roger Pau Monne wrote:
> On Tue, Apr 26, 2022 at 05:51:32PM +0000, Andrew Cooper wrote:
>> Hello,
>>
>> Edvin has found a machine with some very weird properties.  It is an HP
>> ProLiant BL460c Gen8 with:
>>
>>  \-[0000:00]-+-00.0  Intel Corporation Xeon E5/Core i7 DMI2
>>              +-01.0-[11]--
>>              +-01.1-[02]--
>>              +-02.0-[04]--+-00.0  Emulex Corporation OneConnect 10Gb NIC
>> (be3)
>>              |            +-00.1  Emulex Corporation OneConnect 10Gb NIC
>> (be3)
>>              |            +-00.2  Emulex Corporation OneConnect 10Gb
>> iSCSI Initiator (be3)
>>              |            \-00.3  Emulex Corporation OneConnect 10Gb
>> iSCSI Initiator (be3)
>>
>> yet all 4 other functions on the device periodically hit IOMMU faults
>> (~once every 5 mins, so definitely stats).
>>
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.4] fault addr
>> bdf80000
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.5] fault addr
>> bdf80000
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.6] fault addr
>> bdf80000
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.7] fault addr
>> bdf80000
>>
>> There are several RMRRs covering the these devices, with:
>>
>> (XEN) [VT-D]found ACPI_DMAR_RMRR:
>> (XEN) [VT-D] endpoint: 0000:03:00.0
>> (XEN) [VT-D] endpoint: 0000:01:00.0
>> (XEN) [VT-D] endpoint: 0000:01:00.2
>> (XEN) [VT-D] endpoint: 0000:04:00.0
>> (XEN) [VT-D] endpoint: 0000:04:00.1
>> (XEN) [VT-D] endpoint: 0000:04:00.2
>> (XEN) [VT-D] endpoint: 0000:04:00.3
>> (XEN) [VT-D]dmar.c:608:   RMRR region: base_addr bdf8f000 end_addr bdf92fff
>>
>> being the one relevant to these faults.  I've not manually decoded the
>> DMAR table because device paths are horrible to follow but there are at
>> least the correct number of endpoints.  The functions all have SR-IOV
>> (disabled) and ARI (enabled).  None have any Phantom functions described.
> According to the PCIe spec ARI capable devices must not have phantom
> functions:
>
> "With every Function in an ARI Device, the Phantom Functions Supported
> field must be set to 00b. The remainder of this field description
> applies only to non-ARI multi-Function devices."

Lovely...

>
>> Specifying pci-phantom=04:00,1 does appear to work around the faults,
>> but it's not right, because functions 1 thru 3 aren't actually phantom.
>>
>> Also, I don't see any logic which actually wires up phantom functions
>> like this to share RMRRs/IVMDs in IO contexts.  The faults only
>> disappear as a side effect of 04:00.0 and 04:00.4 being in dom0, as far
>> as I can tell.
> I think I'm slightly confused, so those faults only happen when the
> devices are assigned to domains different than dom0?
>
> It would seem to me that functions 4 to 7 not being recognized by Xen
> should also lead to their context entries not being setup in the dom0
> case, and thus the faults should also happen.

Functions 4 thru 7 do not exist in the system.  Their config space is
all ~0's.

As they appear to be non-existent, no IOMMU context is set up for them,
hence the DMA faults when their source id is actually used.

When specifying phantom, what we're saying is that "function $X uses $Y
as a source id too".  Or in other words, treat $Y as if it were $X.  In
a theoretical future with working IOMMU groups, this would force $X and
$Y into the same IOMMU group as they can't be separated.

~Andrew

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.