Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Julien,

> On 28 Oct 2022, at 14:06, Julien Grall <julien@xxxxxxx> wrote:
> 
> Hi Rahul,
> 
> On 28/10/2022 13:54, Rahul Singh wrote:
>>>>>>> For ACPI, I would have expected the information to be found in the 
>>>>>>> IOREQ.
>>>>>>> 
>>>>>>> So can you add more context why this is necessary for everyone?
>>>>>> We have information for IOMMU and Master-ID but we don’t have 
>>>>>> information for linking vMaster-ID to pMaster-ID.
>>>>> 
>>>>> I am confused. Below, you are making the virtual master ID optional. So 
>>>>> shouldn't this be mandatory if you really need the mapping with the 
>>>>> virtual ID?
>>>> vMasterID is optional if user knows pMasterID is unique on the system. But 
>>>> if pMasterId is not unique then user needs to provide the vMasterID.
>>> 
>>> So the expectation is the user will be able to know that the pMasterID is 
>>> uniq. This may be easy with a couple of SMMUs, but if you have 50+ (as 
>>> suggested above). This will become a pain on larger system.
>>> 
>>> IHMO, it would be much better if we can detect that in libxl (see below).
>> We can make the vMasterID compulsory to avoid complexity in libxl to solve 
>> this
> 
> In general, complexity in libxl is not too much of problem.

I am a bit unsure about this strategy.
Currently xl has one configuration file where you put all Xen parameters. The 
device tree is only needed by some guests to have a description of the system 
they run on.
If we change the model and say that Xen configuration parameters are both in 
the configuration and the device tree, we somehow enforce to have a device tree 
even though some guests do not need it at all (for example Zephyr).
I think we need to discuss that and make sure we stay coherent because right 
now the user will have to do things on the configuration and one thing in the 
device tree.

Cheers
Bertrand

> 
>>> 
>>>>> 
>>>>>> The device tree node will be used to assign the device to the guest and 
>>>>>> configure the Stage-2 translation. Guest will use the
>>>>>> vMaster-ID to configure the vIOMMU during boot. Xen needs information to 
>>>>>> link vMaster-ID to pMaster-ID to configure
>>>>>> the corresponding pIOMMU. As I mention we need vMaster-ID in case a 
>>>>>> system could have 2 identical Master-ID but
>>>>>> each one connected to a different SMMU and assigned to the guest.
>>>>> 
>>>>> I am afraid I still don't understand why this is a requirement. Libxl 
>>>>> could have enough knowledge (which will be necessarry for the PCI case) 
>>>>> to know the IOMMU and pMasterID associated with a device.
>>>>> 
>>>>> So libxl could allocate the vMasterID, tell Xen the corresponding mapping 
>>>>> and update the device-tree.
>>>>> 
>>>>> IOW, it doesn't seem to be necessary to involve the user in the process 
>>>>> here.
>>>> Yes, libxl could allocate the vMasterID but there is no way we can find 
>>>> the link b/w vMasterID created to pMasterID from dtdev.
>>>> What I understand from the code is that there is no link between the 
>>>> passthrough node and dtdev config option. The passthrough
>>>> node is directly copied to guest DT without any modification. Dtdev is 
>>>> used to add and assign the device to IOMMU.
>>>> Let's take an example if the user wants to assign two devices to the guest 
>>>> via passthrough node.
>>>> /dts-v1/;
>>>> / {
>>>>    /* #*cells are here to keep DTC happy */
>>>>    #address-cells = <2>;
>>>>    #size-cells = <2>;
>>>>    aliases {
>>>>        net = &mac0;
>>>>    };
>>>>    passthrough {
>>>>        compatible = "simple-bus";
>>>>        ranges;
>>>>        #address-cells = <2>;
>>>>        #size-cells = <2>;
>>>>        mac0: ethernet@10000000 {
>>>>            compatible = "calxeda,hb-xgmac";
>>>>            reg = <0 0x10000000 0 0x1000>;
>>>>            interrupts = <0 80 4  0 81 4  0 82 4>;
>>>>        };
>>>>      mac1: ethernet@20000000 {
>>>>            compatible = “r8169";
>>>>            reg = <0 0x10000000 0 0x1000>;
>>>>            interrupts = <0 80 4  0 81 4  0 82 4>;
>>>>        };
>>>>    };
>>>> };
>>>> dtdev = [ "/soc/ethernet@10000000”, “/soc/ethernet@f2000000” ]
>>>> There is no link which dtdev entry belongs to which node. Therefor there 
>>>> is no way to link the vMasterID created to pMasterID.
>>> 
>>> I agree there is no link today. But we could add a property in the partial 
>>> device-tree to mention which physical device is associated.
>>> 
>>> With that, I think all, the complexity is moved to libxl and it will be 
>>> easier for the user to use vIOMMU.
>>> 
>>> [...]
>> As of now libxl directly coping the partial DT to guest DT without any 
>> modification. If we have to go to this route libxl has to modify
>> the partial DT in libxl to include “iommus” or "iommu-map”. Is that okay to 
>> modify the partial DT in libxl 
> 
> I am not aware of any issue to modify the partial device-tree. In fact, I am 
> strongly in favor of libxl to modify it if it greatly improve the user 
> experience.
> 
> Cheers,
> 
> -- 
> Julien Grall
>