Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Rahul Singh <Rahul.Singh@xxxxxxx>]

Hi All,

At Arm, we started to implement the POC to support 2 levels of page 
tables/nested translation in SMMUv3.
To support nested translation for guest OS Xen needs to expose the virtual 
IOMMU. If we passthrough the
device to the guest that is behind an IOMMU and virtual IOMMU is enabled for 
the guest there is a need to
add IOMMU binding for the device in the passthrough node as per [1]. This email 
is to get an agreement on
how to add the IOMMU binding for guest OS.

Before I will explain how to add the IOMMU binding let me give a brief overview 
of how we will add support for virtual
IOMMU on Arm. In order to implement virtual IOMMU Xen need SMMUv3 Nested 
translation support. SMMUv3 hardware
supports two stages of translation. Each stage of translation can be 
independently enabled. An incoming address is logically
translated from VA to IPA in stage 1, then the IPA is input to stage 2 which 
translates the IPA to the output PA. Stage 1 is
intended to be used by a software entity( Guest OS) to provide isolation or 
translation to buffers within the entity, for example,
DMA isolation within an OS. Stage 2 is intended to be available in systems 
supporting the Virtualization Extensions and is
intended to virtualize device DMA to guest VM address spaces. When both stage 1 
and stage 2 are enabled, the translation
configuration is called nesting.

Stage 1 translation support is required to provide isolation between different 
devices within the guest OS. XEN already supports
Stage 2 translation but there is no support for Stage 1 translation for guests. 
We will add support for guests to configure
the Stage 1 transition via virtual IOMMU. XEN will emulate the SMMU hardware 
and exposes the virtual SMMU to the guest.
Guest can use the native SMMU driver to configure the stage 1 translation. When 
the guest configures the SMMU for Stage 1,
XEN will trap the access and configure the hardware accordingly.

Now back to the question of how we can add the IOMMU binding between the 
virtual IOMMU and the master devices so that
guests can configure the IOMMU correctly. The solution that I am suggesting is 
as below:

For dom0, while handling the DT node(handle_node()) Xen will replace the 
phandle in the "iommus" property with the virtual
IOMMU node phandle.

For domU guests, when passthrough the device to the guest as per [2],  add the 
below property in the partial device tree
node that is required to describe the generic device tree binding for IOMMUs 
and their master(s)

"iommus = < &magic_phandle 0xvMasterID>  
        • magic_phandle will be the phandle ( vIOMMU phandle in xl)  that will 
be documented so that the user can set that in partial DT node (0xfdea).  
        • vMasterID will be the virtual master ID that the user will provide.

The partial device tree will look like this:
/dts-v1/;
 
/ {
    /* #*cells are here to keep DTC happy */
    #address-cells = <2>;
    #size-cells = <2>;
 
    aliases {
        net = &mac0;
    };
 
    passthrough {
        compatible = "simple-bus";
        ranges;
        #address-cells = <2>;
        #size-cells = <2>;
        mac0: ethernet@10000000 {
            compatible = "calxeda,hb-xgmac";
            reg = <0 0x10000000 0 0x1000>;
            interrupts = <0 80 4  0 81 4  0 82 4>;
           iommus = <0xfdea 0x01>;
        };
    };
};
 
In xl.cfg we need to define a new option to inform Xen about vMasterId to 
pMasterId mapping and to which IOMMU device this
the master device is connected so that Xen can configure the right IOMMU. This 
is required if the system has devices that have
the same master ID but behind a different IOMMU.
 
iommu_devid_map = [ “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS” , 
“PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS”]

        • PMASTER_ID is the physical master ID of the device from the physical 
DT.
        • VMASTER_ID is the virtual master Id that the user will configure in 
the partial device tree.
        • IOMMU_BASE_ADDRESS is the base address of the physical IOMMU device 
to which this device is connected. 
 
Example: Let's say the user wants to assign the below physical device in DT to 
the guest.
 
iommu@4f000000 {
                compatible = "arm,smmu-v3";
                interrupts = <0x00 0xe4 0xf04>;
                interrupt-parent = <0x01>;
                #iommu-cells = <0x01>;
                interrupt-names = "combined";
                reg = <0x00 0x4f000000 0x00 0x40000>;
                phandle = <0xfdeb>;
                name = "iommu";
};
 
test@10000000 {
        compatible = "viommu-test”;
        iommus = <0xfdeb 0x10>;
        interrupts = <0x00 0xff 0x04>;
        reg = <0x00 0x10000000 0x00 0x1000>;
        name = "viommu-test";
};
 
The partial Device tree node will be like this:
 
/ {
    /* #*cells are here to keep DTC happy */
    #address-cells = <2>;
    #size-cells = <2>;
 
    passthrough {
        compatible = "simple-bus";
        ranges;
        #address-cells = <2>;
        #size-cells = <2>;

        test@10000000 {
                compatible = "viommu-test";
                reg = <0 0x10000000 0 0x1000>;
                interrupts = <0 80 4  0 81 4  0 82 4>;
                iommus = <0xfdea 0x01>;
        };
    };
};
 
 iommu_devid_map = [ “0x10@0x01,0x4f000000”]
        • 0x10 is the real physical master id from the physical DT.
        • 0x01 is the virtual master Id that the user defines as a partial 
device tree.
        • 0x4f000000 is the base address of the IOMMU device.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
[2] https://xenbits.xen.org/docs/unstable/misc/arm/passthrough.txt

Regards,
Rahul