[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RMRRs and Phantom Functions
- To: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
- From: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>
- Date: Tue, 26 Apr 2022 17:51:32 +0000
- Accept-language: en-GB, en-US
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LZrlXuDr5SRMqD6btn73rSUnTkYmGZLru5ydx+LYhIo=; b=iWpnxLoJzy/tIYOsyjkC1JYD1SsGlg3EDw5NEuXPwqR7QQ2LTviIz7LPjpNNnmDo2eE7x/cNXJtp1Lci19gam4SadaB20ckwuFjUCJEmL9yFU8aEmAzDf7vzUJY7IYwwNWqiZerD5uDHPrGk0rLYdVZorvw+ixgPZCpMI4yrsbbfYCmNVPtAjAXoC//xH9wJ6CcqzdMki/yKY28DBAL54+wrenlxJAEpe7a+t6e25q6KzcfQ9j/87Mc1phBXhITe8Df6eT/mSwL6fPko23x5b6WwgeUQZA5dMI9Ibs9KEamzDZwMnGwIQyWiCicgOzjYoRl99Boic5z4qEiPn8WGcw==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AZ9wkzOOjmjv5d3wao2fkRSUw+WFEYo93PpOTfDLVBqh+JmOl9Nd+fkY1INEZ4UxReQuDLrH39qzO4P06s2HHXy1KG4FoE78xfuCzDCc3PM325QCB5BxzVpWZ28RFv/I8fHtmLSrXuLsyc+cHI8zktao07BD4GVja2qsyRHIiShOVf0ldQ52bVKc9qOCYw7EHO0iFgHJxbR/I45nwd8mnK0k2volOkrDVxOMvyqQl1jiLQEUYe2dep7Zb2oc4ZSWBcnr8xzaJDoE2fWSyhYrATk0O1cbCHxK+RvBAVLlsWk2BDxxUb8bZCQ2QdUdwwtCU/tWw6hAO2ymfMl7kBmG3Q==
- Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com;
- Cc: Roger Pau Monne <roger.pau@xxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Kevin Tian <kevin.tian@xxxxxxxxx>, Edwin Torok <edvin.torok@xxxxxxxxxx>
- Delivery-date: Tue, 26 Apr 2022 17:51:58 +0000
- Ironport-data: A9a23:w1ulCayVzbwwdNBAyHh6t+cyxyrEfRIJ4+MujC+fZmUNrF6WrkUFn TFNWjjQa62ON2P3ctsib9/i8E8B65CAx9U2SVNtrSAxQypGp/SeCIXCJC8cHc8zwu4v7q5Dx 59DAjUVBJlsFhcwnj/0bv656yMUOZigHtIQMsadUsxKbVIiGX9JZS5LwbZj2NY024bhWWthh PupyyHhEA79s9JLGjp8B5Kr8HuDa9yr5Vv0FnRnDRx6lAe2e0s9VfrzFonoR5fMeaFGH/bSe gr25OrRElU1XfsaIojNfr7TKiXmS1NJVOSEoiI+t6OK2nCuqsGuu0qS2TV1hUp/0l20c95NJ NpliruVQCoCN5H1yO0+UhwGFw1PO6NsweqSSZS/mZT7I0zuVVLJmq8rJmdmeIoS96BwHH1E8 uEeJHYVdBefiumqwbW9DO5xmsAkK8qtN4Qa0p1i5WiBUbB6HtaeHuOTuoAwMDQY36iiGd73Y cYDZCUpRxPHexBVYX8cCY4knffujX76G9FdgA3P+fBpvDmDpOB3+JrqOov0aNnQf9hYn2md/ jjmpGKlPyhPYbRzzhLAqBpAnNTnjS79HY4fCrC83vprm0GIgHweDgUMUlm2quX/jVSxM/pdI UEJ/islrYAp6VemCNL6WnWQo3OavxhaR9tZFcU77h2Azuzf5APxO4QfZjtIadhjvslmQzUvj wWNh4mwWmMpt6CJQ3WA8LvStSm1JSUeMW4FY2kDUBcB5N7g5oo0i3ojU+peLUJ8tfWtcRmY/ txAhHFWa2k75SLT65iGwA==
- Ironport-hdrordr: A9a23:FrujtKg28nzMwM1Bnko2KKuYn3BQX3l13DAbv31ZSRFFG/FwyP rCoB1L73XJYWgqM03IwerwQZVpQRvnhP1ICRF4B8buYOCUghrREGgE1/qv/9SAIVy1ygc578 tdmsdFebrN5DRB7PoSgzPIXOrIouP3lpxA7N22pxgCcegAUdAD0+4NMHf+LqQAfngiOXNWLv qhz/sCgwDlVWUcb8y9CHVAdfPEvcf3mJXvZgNDLwI76SGV5AnYqYLSIly95FMzQjlPybAt/S zuiAri/JiutPm911v1y3LT1ZJLg9Hso+EzRPBky/JlaQkEuDzYIbiJaIfy+AzdZ9vfr2rCpe O84SvI+f4DrU85MFvF+CcFkDOQqwrGo0WSs2Nwx0GT7/ARDQhKd/aoz+hiA2bkwltls9dm3K 1R2WWF85JREBPbhSz4o8PFThdwiyOP0DMfeEEo/gxiuKYlGchsRLYkjTVo+VY7bVHHwZFiFP MrANDX5f5Qf1/fZ3fFvnN3yNjpWngoBB+JTkULp8TQilFt7TtE5lpdwNZakmYL9Zo7RZUB7+ PYMr5wnLULSsMNd6pyCOoIXMPyAG3QRhDHNn6UPD3cZek6EmOIr4Sy7KQ+5emsdpBNxJwumI 7ZWFcdrmI2c1KGM7z74HSKyGG5fIyQZ0WS9igF3ekHhlTVfsubDQSTDFYzjsCnv/ITRsXGRv fbAuMlP8Pe
- List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
- Thread-index: AQHYWZZDIrEkmq9fL0+3qLNT2+7QyA==
- Thread-topic: RMRRs and Phantom Functions
Hello,
Edvin has found a machine with some very weird properties. It is an HP
ProLiant BL460c Gen8 with:
\-[0000:00]-+-00.0 Intel Corporation Xeon E5/Core i7 DMI2
+-01.0-[11]--
+-01.1-[02]--
+-02.0-[04]--+-00.0 Emulex Corporation OneConnect 10Gb NIC
(be3)
| +-00.1 Emulex Corporation OneConnect 10Gb NIC
(be3)
| +-00.2 Emulex Corporation OneConnect 10Gb
iSCSI Initiator (be3)
| \-00.3 Emulex Corporation OneConnect 10Gb
iSCSI Initiator (be3)
yet all 4 other functions on the device periodically hit IOMMU faults
(~once every 5 mins, so definitely stats).
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.4] fault addr
bdf80000
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.5] fault addr
bdf80000
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.6] fault addr
bdf80000
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.7] fault addr
bdf80000
There are several RMRRs covering the these devices, with:
(XEN) [VT-D]found ACPI_DMAR_RMRR:
(XEN) [VT-D] endpoint: 0000:03:00.0
(XEN) [VT-D] endpoint: 0000:01:00.0
(XEN) [VT-D] endpoint: 0000:01:00.2
(XEN) [VT-D] endpoint: 0000:04:00.0
(XEN) [VT-D] endpoint: 0000:04:00.1
(XEN) [VT-D] endpoint: 0000:04:00.2
(XEN) [VT-D] endpoint: 0000:04:00.3
(XEN) [VT-D]dmar.c:608: RMRR region: base_addr bdf8f000 end_addr bdf92fff
being the one relevant to these faults. I've not manually decoded the
DMAR table because device paths are horrible to follow but there are at
least the correct number of endpoints. The functions all have SR-IOV
(disabled) and ARI (enabled). None have any Phantom functions described.
Specifying pci-phantom=04:00,1 does appear to work around the faults,
but it's not right, because functions 1 thru 3 aren't actually phantom.
Also, I don't see any logic which actually wires up phantom functions
like this to share RMRRs/IVMDs in IO contexts. The faults only
disappear as a side effect of 04:00.0 and 04:00.4 being in dom0, as far
as I can tell.
Simply giving the RMRR via rmrr= doesn't work (presumably because of no
patching actual devices, but there's no warning), but it feels as if it
ought to.
~Andrew
|
