[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 10/11] xen/arm: Do not map PCI ECAM space to Domain-0's p2m


  • To: Stefano Stabellini <sstabellini@xxxxxxxxxx>, Rahul Singh <rahul.singh@xxxxxxx>
  • From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>
  • Date: Wed, 15 Sep 2021 05:35:48 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=vPpE+tz+LC0WamEI+NnBHQ3+T2W7EesEGE/fp236j10=; b=FIZiGAvSfY7bK2d6HDLeRtaP57MmkxuMvzDbT2DSiBZNDAITpVio22zwD2gJ6TI9u0tuwrXK8pJB0fCW1DEqa1fqrdxUHVqbo3fA2doGkqHDqFq8We6jsJO635Iu9jJhZ9brmF12E3fVqJ7/6PEEdImPac3br/+8mpX10ypmYrmfbsCHUtohodvdU3QRwN4mR8jKjM54k1JuNYb/5xvq/K2MtUPiqjmgnH6OUGb/O1rrIpG9JCigDtmUHuL1++LM1s1ww0LmIP8pt2zK328v7GCwKBkxYkRu6dNVQ7vjY3PHCG2lOTRM9IqAC9KyDS7C1zbMucFmX+nIm/HJF0ZoBg==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gO+mc9O8Kx7jYbjiJiD3w87d5prvrEeTv6KFMVHRWKIKYQFMlN+Yv7TiLZbDUEvg3CUryfTGktzQVonEshCyeEQIL0MwDFguuaOclHyzTkfshgeGcitKQGAChR5J6ebodxbrAsL9aej72+MSMgppPr1B5V1WniyMtQEp7lEvbEuZcdUxFiDVbY8MdRmqjKzfxbqPjrK7xG9bgj7BRK4XOuwqaJlA2M7qF4jcMUWsAhIIJ9PJyK5c+/yPbvLnl/i5WBJbsqoIsewwyq8hVSSDwhTMOozNWs8hHoCA4FlKUIG6oe4Nye6vQVdP2+Ix58C9hSTqFkaTP0LsT3VEFQBmOw==
  • Authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=epam.com;
  • Cc: Julien Grall <julien.grall.oss@xxxxxxxxx>, Oleksandr Andrushchenko <andr2000@xxxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Oleksandr Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>, Volodymyr Babchuk <Volodymyr_Babchuk@xxxxxxxx>, Artem Mygaiev <Artem_Mygaiev@xxxxxxxx>, Roger Pau Monné <roger.pau@xxxxxxxxxx>, Bertrand Marquis <bertrand.marquis@xxxxxxx>
  • Delivery-date: Wed, 15 Sep 2021 05:36:02 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Thread-index: AQHXoJ50hf+zXQhc2ka7YfmmERKX+aucB06AgAE4wYCAAAtRgIAADDiAgAARZwCAAFsJAIAAFA4AgAO3eICAAc7agIAA860AgABTv4A=
  • Thread-topic: [PATCH 10/11] xen/arm: Do not map PCI ECAM space to Domain-0's p2m

Hi, Stefano, Rahul!

On 15.09.21 03:36, Stefano Stabellini wrote:
> On Tue, 14 Sep 2021, Oleksandr Andrushchenko wrote:
>> With the patch above I have the following log in Domain-0:
>>
>> generic-armv8-xt-dom0 login: root
>> root@generic-armv8-xt-dom0:~# (XEN) *** Serial input to Xen (type 'CTRL-a' 
>> three times to switch input)
>> (XEN) ==== PCI devices ====
>> (XEN) ==== segment 0000 ====
>> (XEN) 0000:03:00.0 - d0 - node -1
>> (XEN) 0000:02:02.0 - d0 - node -1
>> (XEN) 0000:02:01.0 - d0 - node -1
>> (XEN) 0000:02:00.0 - d0 - node -1
>> (XEN) 0000:01:00.0 - d0 - node -1
>> (XEN) 0000:00:00.0 - d0 - node -1
>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>
>> root@generic-armv8-xt-dom0:~# modprobe e1000e
>> [   46.104729] e1000e: Intel(R) PRO/1000 Network Driver
>> [   46.105479] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
>> [   46.107297] e1000e 0000:03:00.0: enabling device (0000 -> 0002)
>> (XEN) map [e0000, e001f] -> 0xe0000 for d0
>> (XEN) map [e0020, e003f] -> 0xe0020 for d0
>> [   46.178513] e1000e 0000:03:00.0: Interrupt Throttling Rate (ints/sec) set 
>> to dynamic conservative mode
>> [   46.189668] pci_msi_setup_msi_irqs
>> [   46.191016] nwl_compose_msi_msg msg at fe440000
>> (XEN) traps.c:2014:d0v0 HSR=0x00000093810047 pc=0xffff8000104b4b00 
>> gva=0xffff800010fa5000 gpa=0x000000e0040000
>> [   46.200455] Unhandled fault at 0xffff800010fa5000
>>
>> [snip]
>>
>> [   46.233079] Call trace:
>> [   46.233559]  __pci_write_msi_msg+0x70/0x180
>> [   46.234149]  pci_msi_domain_write_msg+0x28/0x30
>> [   46.234869]  msi_domain_activate+0x5c/0x88
>>
>>   From the above you can see that BARs are mapped for Domain-0 now
>>
>> only when an assigned PCI device gets enabled in Domain-0.
>>
>> Another thing to note is that we crash on MSI-X access as BARs are mapped
>>
>> to the domain while enabling memory decoding in the COMMAND register,
>>
>> but MSI-X are handled differently, e.g. we have MSI-X holes in the mappings.
>>
>> This is because before this change the whole PCI aperture was mapped into
>>
>> Domain-0 and it is not. Thus, MSI-X holes are left unmapped now and there
>>
>> was no need to do so, e.g. they were always mapped into Domain-0 and
>>
>> emulated for guests.
>>
>> Please note that one cannot use xl pci-attach in this case to attach the PCI 
>> device
>>
>> in question to Domain-0 as (please see the log) that device is already 
>> attached.
>>
>> Neither it can be detached and re-attached. So, without mapping MSI-X holes 
>> for
>>
>> Domain-0 the device becomes unusable in Domain-0. At the same time the device
>>
>> can be successfully passed to DomU.
>>
>>
>> Julien, Stefano! Please let me know how can we proceed with this.
> What was the plan for MSI-X in Dom0?
It just worked because we mapped everything
>
> Given that Dom0 interacts with a virtual-ITS and virtual-GIC rather than
> a physical-ITS and physical-GIC, I imagine that it wasn't correct for
> Dom0 to write to the real MSI-X table directly?
>
> Shouldn't Dom0 get emulated MSI-X tables like any DomU?
>
> Otherwise, if Dom0 is expected to have the real MSI-X tables mapped, then
> I would suggest to map them at the same time as the BARs. But I am
> thinking that Dom0 should get emulated MSI-X tables, not physical MSI-X
> tables.

Yes, it seems more than reasonable to enable emulation for Domain-0

as well. Other than that, Stefano, do you think we are good to go with

the changes I did in order to unmap everything for Domain-0?

Rahul, it seems we will need a change to vPCI/MSI-X so we can also

trap Domain-0 for MSI-X tables.

Thank you,

Oleksandr

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.