[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: RMRRs and Phantom Functions
- To: Kevin Tian <kevin.tian@xxxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
- From: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>
- Date: Wed, 27 Apr 2022 08:51:04 +0000
- Accept-language: en-GB, en-US
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bU/9Sn70oi4SD9AvXpyG3UJzes5PrW1s7e/YT2xyo2o=; b=I+nyq7bCc3/oms19XeCzDpGIxH0Hnl1C9OIWoAf8AniqWHhHZ7HGVUcuT114C58RYj/qjbV5J+SwmgTxYOQQ1xlT0B8mCA9Vv6fLSFkb9/1yRCY44rQq5YBzerZeYtXzvndT/GOZxGk6sJOi7PeJPWPVJAn3aOpLkTTTUcoOJcIpeXi33V6LrHUxI4N4laKVbtfJeuZtPBAkohTvtXUICZZHt7tm0IPVW44mEfNgly92PI1cvDASDnP5uT6RggRc3JX1liQHtPAQIJFGPnPt5fhE1P//JOjc/SCecl7uKh7O2NxJVJxLLKTwg3jK3nglODPqpYrHldRyo93R5K/uFA==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZlN5PrO6UYGb/C8dVTtRnRfEMaO0mN1UQjzRiLC2OdyT94UXmtZ6bNe9cmckdSYTCUCZBJUamqPRPiMAvFhi8BefGfMpGoiI/4UZU3yDcKwxv4D6cO7yy26AXbbBGR1y30l5HZN+O+rw1F1VuuGps2DmKjG/LE2Ojlurk4EEeFGJKvIec4O/25imdIFZqzJLQwfd1Z/kEJQiS31EdOb8KL30H3vF2SeLzoJm170ae+AkYvlTaEMfi3fbgfKwjm9qjDioTH+JgscErT9DampJsPdsNB85wwxBZLtgoTOogIWKnX5mBLHB118y5WCStPIADskPdB8G+lOWUfWINNB8rA==
- Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com;
- Cc: Roger Pau Monne <roger.pau@xxxxxxxxxx>, "Beulich, Jan" <JBeulich@xxxxxxxx>, Edwin Torok <edvin.torok@xxxxxxxxxx>
- Delivery-date: Wed, 27 Apr 2022 08:51:18 +0000
- Ironport-data: A9a23:uyq+G6trxDl9gwQ2ku/ingCegufnVFNfMUV32f8akzHdYApBsoF/q tZmKTiCafjZYGrzc4siOYzk900A7ZSDy98xTFY6+S1jEShB+JbJXdiXEBz9bniYRiHhoOOLz Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1/X5 4upyyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo Oihu6BcRi8GE7zcwLkRVSUAGnBmLZ9I3eDNPEGw5Jn7I03uKxMAwt1IJWRvZcg037gyBmtDs /sFNDoKcxaPwfqsx662QfVtgcJlK9T3OIQYuTdryjSx4fQOGMifBfmVo4IImm5o1qiiHt6HD yYdQRNpYA7NfFtkPVAPBYhltOypmmP+Y3tTr1f9Sa8fvDKPk1IuiOOF3Nz9edWGHfwPtGGjh nPK7Tr+PkBDd8fF1m/Qmp6rrqqV9c/hY6oRGaO58Lh2gVSV7m0VFBASE1C8pJGRmkO4Ht5SN UEQ0i4vtrQpslymSMHnWB+1q2LCuQQTM/JyOeAn7ACGyoLP/h2UQGMDS1Z8hMcOscY3QXkm0 wGPltawXTh36uTNGTSa66ueqi60NW4NN2geaCQYTAwDpd7+vIU0iRGJRdFmeEKosuDI9fjL6 2jihEADa38716bnC43TEYj7vg+R
- Ironport-hdrordr: A9a23:LOPDn608itE8oJ3ujNSzCgqjBEgkLtp133Aq2lEZdPU0SKGlfg 6V/MjztCWE7Ar5PUtLpTnuAsa9qB/nm6KdgrNhWItKPjOW21dARbsKheffKlXbcBEWndQtt5 uIHZIeNDXxZ2IK8PoT4mODYqodKA/sytHWuQ/cpU0dMz2Dc8tbnmBE4p7wKDwMeOFBb6BJcq a01458iBeLX28YVci/DmltZZm4mzWa/KiWGCLvHnQcmXGzsQ8=
- List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
- Thread-index: AQHYWZZDIrEkmq9fL0+3qLNT2+7QyK0DHAnAgABYo4A=
- Thread-topic: RMRRs and Phantom Functions
On 27/04/2022 04:39, Tian, Kevin wrote:
>> From: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>
>> Sent: Wednesday, April 27, 2022 1:52 AM
>>
>> Hello,
>>
>> Edvin has found a machine with some very weird properties. It is an HP
>> ProLiant BL460c Gen8 with:
>>
>> \-[0000:00]-+-00.0 Intel Corporation Xeon E5/Core i7 DMI2
>> +-01.0-[11]--
>> +-01.1-[02]--
>> +-02.0-[04]--+-00.0 Emulex Corporation OneConnect 10Gb NIC
>> (be3)
>> | +-00.1 Emulex Corporation OneConnect 10Gb NIC
>> (be3)
>> | +-00.2 Emulex Corporation OneConnect 10Gb
>> iSCSI Initiator (be3)
>> | \-00.3 Emulex Corporation OneConnect 10Gb
>> iSCSI Initiator (be3)
>>
>> yet all 4 other functions on the device periodically hit IOMMU faults
>> (~once every 5 mins, so definitely stats).
>>
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.4] fault addr
>> bdf80000
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.5] fault addr
>> bdf80000
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.6] fault addr
>> bdf80000
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.7] fault addr
>> bdf80000
>>
>> There are several RMRRs covering the these devices, with:
>>
>> (XEN) [VT-D]found ACPI_DMAR_RMRR:
>> (XEN) [VT-D] endpoint: 0000:03:00.0
>> (XEN) [VT-D] endpoint: 0000:01:00.0
>> (XEN) [VT-D] endpoint: 0000:01:00.2
>> (XEN) [VT-D] endpoint: 0000:04:00.0
>> (XEN) [VT-D] endpoint: 0000:04:00.1
>> (XEN) [VT-D] endpoint: 0000:04:00.2
>> (XEN) [VT-D] endpoint: 0000:04:00.3
>> (XEN) [VT-D]dmar.c:608: RMRR region: base_addr bdf8f000 end_addr
>> bdf92fff
>>
>> being the one relevant to these faults. I've not manually decoded the
>> DMAR table because device paths are horrible to follow but there are at
>> least the correct number of endpoints. The functions all have SR-IOV
>> (disabled) and ARI (enabled). None have any Phantom functions described.
>>
>> Specifying pci-phantom=04:00,1 does appear to work around the faults,
>> but it's not right, because functions 1 thru 3 aren't actually phantom.
>>
>> Also, I don't see any logic which actually wires up phantom functions
>> like this to share RMRRs/IVMDs in IO contexts. The faults only
>> disappear as a side effect of 04:00.0 and 04:00.4 being in dom0, as far
>> as I can tell.
>>
>> Simply giving the RMRR via rmrr= doesn't work (presumably because of no
>> patching actual devices, but there's no warning), but it feels as if it
>> ought to.
>>
> What is the Xen version? Does it include Jan's change for per-device
> quarantine?
It's an up-to-date XenServer, so Xen 4.13 based, but yes.
> btw it's weird why those NIC devices require RMRR in the first place...
It's stats to the BMC. This Emulex card is part of the default
configuration of the system from HP.
~Andrew
|
