Re: [RFC PATCH 00/21] Add SMMUv3 Stage 1 Support for XEN guests

[Michal Orzel <michal.orzel@xxxxxxx>]

Hi Rahul,

On 01/12/2022 17:02, Rahul Singh wrote:
> 
> 
> The SMMUv3 supports two stages of translation. Each stage of translation can 
> be
> independently enabled. An incoming address is logically translated from VA to
> IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
> the output PA.
> 
> Stage 1 is intended to be used by a software entity to provide isolation or
> translation to buffers within the entity, for example DMA isolation within an
> OS. Stage 2 is intended to be available in systems supporting the
> Virtualization Extensions and is intended to virtualize device DMA to guest VM
> address spaces. When both stage 1 and stage 2 are enabled, the translation
> configuration is called nested.
> 
> Stage 1 translation support is required to provide isolation between different
> devices within OS. XEN already supports Stage 2 translation but there is no
> support for Stage 1 translation. The goal of this work is to support Stage 1
> translation for XEN guests. Stage 1 has to be configured within the guest to
> provide isolation.
> 
> We cannot trust the guest OS to control the SMMUv3 hardware directly as
> compromised guest OS can corrupt the SMMUv3 configuration and make the system
> vulnerable. The guest gets the ownership of the stage 1 page tables and also
> owns stage 1 configuration structures. The XEN handles the root configuration
> structure (for security reasons), including the stage 2 configuration.
> 
> XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the
> guest. Guest can use the native SMMUv3 driver to configure the stage 1
> translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap
> the access and configure hardware.
> 
> SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation ->
> XEN trap access -> XEN SMMUv3 driver configure the HW.
> 
> SMMUv3 driver has to be updated to support the Stage-1 translation support
> based on work done by the KVM team to support Nested Stage translation:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feauger%2Flinux%2Fcommits%2Fv5.11-stallv12-2stage-v14&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PdK4%2Bsps3%2FdXYJUDv3iCy%2Byaqbh1bOVb1AFzTtx1nts%3D&reserved=0
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F852299%2F&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5Kp7023HiA4Qbfi28wcPL20JyC2xLwwiyEUZcxTSCOA%3D&reserved=0
> 
> As the stage 1 translation is configured by XEN on behalf of the guest,
> translation faults encountered during the translation process need to be
> propagated up to the guest and re-injected into the guest. When the guest
> invalidates stage 1 related caches, invalidations must be forwarded to the
> SMMUv3 hardware.
> 
> This patch series is sent as RFC to get the initial feedback from the
> community. This patch series consists of 21 patches which is a big number for
> the reviewer to review the patches but to understand the feature end-to-end we
> thought of sending this as a big series. Once we will get initial feedback, we
> will divide the series into a small number of patches for review.

Due to the very limited availability of the board we have, that is equipped with
DMA platform devices and SMMUv3 (I know that you tested PCI use case 
thoroughly),
I managed for now to do the testing on dom0 only.

By commenting out the code in Linux responsible for setting up Xen SWIOTLB DMA 
ops, I was able
to successfully verify the nested SMMU working properly for DMA platform 
devices on the
example of using ZDMA. Both the upstream dmatest client app as well as the VFIO 
user space driver
that I wrote for ZDMA passed the test!

I added some logs to verify the sync up between Linux and Xen during a VFIO 
test:

LINUX: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210

Before transfer example:
 src value: 0xdb71faf
 dst value: 0
Waiting for transfer completion...
After transfer example:
 src value: 0xdb71faf
 dst value: 0xdb71faf
TEST RESULT: PASS

LINUX: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210

~Michal