[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: RMRRs and Phantom Functions


  • To: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>
  • From: Jan Beulich <jbeulich@xxxxxxxx>
  • Date: Wed, 27 Apr 2022 12:20:15 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=suse.com; dmarc=pass action=none header.from=suse.com; dkim=pass header.d=suse.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jbQKfysjlsQbCx8NstU6XjoUwqPZYVhthJIJrWJvnH0=; b=YWdMMc0nZKAVNarB/Nm1miUMBNsBO+A+DmhvKiUYOKwJSUpn26qZtOkFeJx3JGpVNFW/tfKFVm9j8SuajOSmHBlS7oPG1tmnfI3Qub4IT3ze05v0btjX+0iv/eof+mM5LjNW88B7ASSQahCgtiwBCsarnq2ncmWAk8ckZf8sFQtBLOIxJ8Nvh3TOUK9z45hhV94AwTqUEX4ijrg1HgHh65xCGQba8ulDd89zp8L4cJVSBP1gUmq9HgebA4rAjEZhklMG1ONZ+mM5OzFflWVZ/bdIGffJdHmKKJ9gAeunwMWIdEua/zsmYE0whe7kAHeG1+RBIj1TkR2iOOKRypAo6g==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UBY8qcI1Z5Xk7YtnKYu+2TWqk/ioJRH/qOvTN/XrS/wpFwzsB43eVeHfiJTb7tcIKkOxqgc4ZTMm8Zbm8EkBcdnP1DhyyWaRh7I6flTk+QRlIowHE0FSYk1wpbQg6WoEz1VObsGPVEnVciQOWIFk6Rk40NnY6fdDka7F/DJxc4as1O/GdpL09m6JanNpc3Uo4b6IIL3ofLMeU0y5lN8tFI6fsOw7mIY6KX9Db6QWmAhmWFplg2gkMRgYJU6t/sXAfa6p7Cu63+KC7x3fO35e/kEiOvelCzKWIZeECzMNWPyekXgyN1uhrjkI+YwqMesJS8c4IpmLz4Mg9o80Z17lzA==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=suse.com;
  • Cc: Roger Pau Monne <roger.pau@xxxxxxxxxx>, Kevin Tian <kevin.tian@xxxxxxxxx>, Edwin Torok <edvin.torok@xxxxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Wed, 27 Apr 2022 10:20:24 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 27.04.2022 12:05, Andrew Cooper wrote:
> On 27/04/2022 07:59, Jan Beulich wrote:
>> On 26.04.2022 19:51, Andrew Cooper wrote:
>>> Hello,
>>>
>>> Edvin has found a machine with some very weird properties.  It is an HP
>>> ProLiant BL460c Gen8 with:
>>>
>>>  \-[0000:00]-+-00.0  Intel Corporation Xeon E5/Core i7 DMI2
>>>              +-01.0-[11]--
>>>              +-01.1-[02]--
>>>              +-02.0-[04]--+-00.0  Emulex Corporation OneConnect 10Gb NIC
>>> (be3)
>>>              |            +-00.1  Emulex Corporation OneConnect 10Gb NIC
>>> (be3)
>>>              |            +-00.2  Emulex Corporation OneConnect 10Gb
>>> iSCSI Initiator (be3)
>>>              |            \-00.3  Emulex Corporation OneConnect 10Gb
>>> iSCSI Initiator (be3)
>>>
>>> yet all 4 other functions on the device periodically hit IOMMU faults
>>> (~once every 5 mins, so definitely stats).
>>>
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.4] fault addr
>>> bdf80000
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.5] fault addr
>>> bdf80000
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.6] fault addr
>>> bdf80000
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.7] fault addr
>>> bdf80000
>>>
>>> There are several RMRRs covering the these devices, with:
>>>
>>> (XEN) [VT-D]found ACPI_DMAR_RMRR:
>>> (XEN) [VT-D] endpoint: 0000:03:00.0
>>> (XEN) [VT-D] endpoint: 0000:01:00.0
>>> (XEN) [VT-D] endpoint: 0000:01:00.2
>>> (XEN) [VT-D] endpoint: 0000:04:00.0
>>> (XEN) [VT-D] endpoint: 0000:04:00.1
>>> (XEN) [VT-D] endpoint: 0000:04:00.2
>>> (XEN) [VT-D] endpoint: 0000:04:00.3
>>> (XEN) [VT-D]dmar.c:608:   RMRR region: base_addr bdf8f000 end_addr bdf92fff
>>>
>>> being the one relevant to these faults.  I've not manually decoded the
>>> DMAR table because device paths are horrible to follow but there are at
>>> least the correct number of endpoints.  The functions all have SR-IOV
>>> (disabled) and ARI (enabled).  None have any Phantom functions described.
>>>
>>> Specifying pci-phantom=04:00,1 does appear to work around the faults,
>>> but it's not right, because functions 1 thru 3 aren't actually phantom.
>> Indeed, and I think you really mean "pci-phantom=04:00,4".
> 
> As a quick tangent, the cmdline docs for pci-phantom= are in desperate
> need of an example and a description of how stride works.  I've got some
> ideas and notes jotted down.
> 
> Do we really mean ,4 here?  What happens for function 1?

With stride 4 function 1's single phantom function is function 5. With
stride 1, as you had it before, functions 1...7 would all be considered
phantom functions of function 0.

Jan




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.