Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Ayan Kumar Halder <ayankuma@xxxxxxx>]

On 27/10/2022 10:01, Ayan Kumar Halder wrote:
> On 26/10/2022 14:17, Rahul Singh wrote:
> > CAUTION: This message has originated from an External Source. Please 
> > use proper judgment and caution when opening attachments, clicking 
> > links, or responding to this email.
> >
> > Hi All,
> Hi Rahul,
>
> I have a very basic question.
>
> > At Arm, we started to implement the POC to support 2 levels of page 
> > tables/nested translation in SMMUv3.
> > To support nested translation for guest OS Xen needs to expose the 
> > virtual IOMMU. If we passthrough the
> > device to the guest that is behind an IOMMU and virtual IOMMU is 
> > enabled for the guest there is a need to
> > add IOMMU binding for the device in the passthrough node as per [1]. 
> > This email is to get an agreement on
> > how to add the IOMMU binding for guest OS.
> >
> > Before I will explain how to add the IOMMU binding let me give a 
> > brief overview of how we will add support for virtual
> > IOMMU on Arm. In order to implement virtual IOMMU Xen need SMMUv3 
> > Nested translation support. SMMUv3 hardware
> > supports two stages of translation. Each stage of translation can be 
> > independently enabled. An incoming address is logically
> > translated from VA to IPA in stage 1, then the IPA is input to stage 
> > 2 which translates the IPA to the output PA. Stage 1 is
> > intended to be used by a software entity( Guest OS) to provide 
> > isolation or translation to buffers within the entity, for example,
> > DMA isolation within an OS. Stage 2 is intended to be available in 
> > systems supporting the Virtualization Extensions and is
> > intended to virtualize device DMA to guest VM address spaces. When 
> > both stage 1 and stage 2 are enabled, the translation
> > configuration is called nesting.
> >
> > Stage 1 translation support is required to provide isolation between 
> > different devices within the guest OS. XEN already supports
> Doesn't this imply that there is support for Stage 1 translation for 
> guests ? Otherwise, how will the guest provide isolation between 
> different devices or dma-masters ?
Michal explained to me offline. This query is answered. Sorry for the noise.

- Ayan

> - Ayan
>
> > Stage 2 translation but there is no support for Stage 1 translation 
> > for guests.
> > We will add support for guests to configure
> > the Stage 1 transition via virtual IOMMU. XEN will emulate the SMMU 
> > hardware and exposes the virtual SMMU to the guest.
> > Guest can use the native SMMU driver to configure the stage 1 
> > translation. When the guest configures the SMMU for Stage 1,
> > XEN will trap the access and configure the hardware accordingly.
> >
> > Now back to the question of how we can add the IOMMU binding between 
> > the virtual IOMMU and the master devices so that
> > guests can configure the IOMMU correctly. The solution that I am 
> > suggesting is as below:
> > For dom0, while handling the DT node(handle_node()) Xen will replace 
> > the phandle in the "iommus" property with the virtual
> > IOMMU node phandle.
> >
> > For domU guests, when passthrough the device to the guest as per 
> > [2],  add the below property in the partial device tree
> > node that is required to describe the generic device tree binding for 
> > IOMMUs and their master(s)
> > "iommus = < &magic_phandle 0xvMasterID>
> >          • magic_phandle will be the phandle ( vIOMMU phandle in xl)  
> > that will be documented so that the user can set that in partial DT 
> > node (0xfdea).
> >          • vMasterID will be the virtual master ID that the user will 
> > provide.
> > The partial device tree will look like this:
> > /dts-v1/;
> >
> > / {
> >      /* #*cells are here to keep DTC happy */
> >      #address-cells = <2>;
> >      #size-cells = <2>;
> >
> >      aliases {
> >          net = &mac0;
> >      };
> >
> >      passthrough {
> >          compatible = "simple-bus";
> >          ranges;
> >          #address-cells = <2>;
> >          #size-cells = <2>;
> >          mac0: ethernet@10000000 {
> >              compatible = "calxeda,hb-xgmac";
> >              reg = <0 0x10000000 0 0x1000>;
> >              interrupts = <0 80 4  0 81 4  0 82 4>;
> >             iommus = <0xfdea 0x01>;
> >          };
> >      };
> > };
> >
> > In xl.cfg we need to define a new option to inform Xen about 
> > vMasterId to pMasterId mapping and to which IOMMU device this
> > the master device is connected so that Xen can configure the right 
> > IOMMU. This is required if the system has devices that have
> > the same master ID but behind a different IOMMU.
> >
> > iommu_devid_map = [ “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS” , 
> > “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS”]
> >          • PMASTER_ID is the physical master ID of the device from 
> > the physical DT.
> >          • VMASTER_ID is the virtual master Id that the user will 
> > configure in the partial device tree.
> >          • IOMMU_BASE_ADDRESS is the base address of the physical 
> > IOMMU device to which this device is connected.
> > Example: Let's say the user wants to assign the below physical device 
> > in DT to the guest.
> > iommu@4f000000 {
> >                  compatible = "arm,smmu-v3";
> >                  interrupts = <0x00 0xe4 0xf04>;
> >                  interrupt-parent = <0x01>;
> >                  #iommu-cells = <0x01>;
> >                  interrupt-names = "combined";
> >                  reg = <0x00 0x4f000000 0x00 0x40000>;
> >                  phandle = <0xfdeb>;
> >                  name = "iommu";
> > };
> >
> > test@10000000 {
> >          compatible = "viommu-test”;
> >          iommus = <0xfdeb 0x10>;
> >          interrupts = <0x00 0xff 0x04>;
> >          reg = <0x00 0x10000000 0x00 0x1000>;
> >          name = "viommu-test";
> > };
> >
> > The partial Device tree node will be like this:
> >
> > / {
> >      /* #*cells are here to keep DTC happy */
> >      #address-cells = <2>;
> >      #size-cells = <2>;
> >
> >      passthrough {
> >          compatible = "simple-bus";
> >          ranges;
> >          #address-cells = <2>;
> >          #size-cells = <2>;
> >
> >          test@10000000 {
> >                  compatible = "viommu-test";
> >                  reg = <0 0x10000000 0 0x1000>;
> >                  interrupts = <0 80 4  0 81 4  0 82 4>;
> >                  iommus = <0xfdea 0x01>;
> >          };
> >      };
> > };
> >
> >   iommu_devid_map = [ “0x10@0x01,0x4f000000”]
> >          • 0x10 is the real physical master id from the physical DT.
> >          • 0x01 is the virtual master Id that the user defines as a 
> > partial device tree.
> >          • 0x4f000000 is the base address of the IOMMU device.
> >
> > [1] 
> > https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
> > [2] https://xenbits.xen.org/docs/unstable/misc/arm/passthrough.txt
> >
> > Regards,
> > Rahul