Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Julien Grall <julien@xxxxxxx>]

Hi Rahul,

On 28/10/2022 13:54, Rahul Singh wrote:
> > > > > > For ACPI, I would have expected the information to be found in the IOREQ.
> > > > > >
> > > > > > So can you add more context why this is necessary for everyone?
> > > > > We have information for IOMMU and Master-ID but we don’t have information for 
> > > > > linking vMaster-ID to pMaster-ID.
> > > > I am confused. Below, you are making the virtual master ID optional. So 
> > > > shouldn't this be mandatory if you really need the mapping with the virtual ID?
> > > vMasterID is optional if user knows pMasterID is unique on the system. But if 
> > > pMasterId is not unique then user needs to provide the vMasterID.
> > So the expectation is the user will be able to know that the pMasterID is uniq. 
> > This may be easy with a couple of SMMUs, but if you have 50+ (as suggested 
> > above). This will become a pain on larger system.
> >
> > IHMO, it would be much better if we can detect that in libxl (see below).
> We can make the vMasterID compulsory to avoid complexity in libxl to solve this
In general, complexity in libxl is not too much of problem.

> > > > > The device tree node will be used to assign the device to the guest and 
> > > > > configure the Stage-2 translation. Guest will use the
> > > > > vMaster-ID to configure the vIOMMU during boot. Xen needs information to link 
> > > > > vMaster-ID to pMaster-ID to configure
> > > > > the corresponding pIOMMU. As I mention we need vMaster-ID in case a system 
> > > > > could have 2 identical Master-ID but
> > > > > each one connected to a different SMMU and assigned to the guest.
> > > > I am afraid I still don't understand why this is a requirement. Libxl could 
> > > > have enough knowledge (which will be necessarry for the PCI case) to know the 
> > > > IOMMU and pMasterID associated with a device.
> > > >
> > > > So libxl could allocate the vMasterID, tell Xen the corresponding mapping and 
> > > > update the device-tree.
> > > >
> > > > IOW, it doesn't seem to be necessary to involve the user in the process here.
> > > Yes, libxl could allocate the vMasterID but there is no way we can find the 
> > > link b/w vMasterID created to pMasterID from dtdev.
> > > What I understand from the code is that there is no link between the 
> > > passthrough node and dtdev config option. The passthrough
> > > node is directly copied to guest DT without any modification. Dtdev is used to 
> > > add and assign the device to IOMMU.
> > > Let's take an example if the user wants to assign two devices to the guest via 
> > > passthrough node.
> > > /dts-v1/;
> > > / {
> > >     /* #*cells are here to keep DTC happy */
> > >     #address-cells = <2>;
> > >     #size-cells = <2>;
> > >     aliases {
> > >         net = &mac0;
> > >     };
> > >     passthrough {
> > >         compatible = "simple-bus";
> > >         ranges;
> > >         #address-cells = <2>;
> > >         #size-cells = <2>;
> > >         mac0: ethernet@10000000 {
> > >             compatible = "calxeda,hb-xgmac";
> > >             reg = <0 0x10000000 0 0x1000>;
> > >             interrupts = <0 80 4  0 81 4  0 82 4>;
> > >         };
> > >       mac1: ethernet@20000000 {
> > >             compatible = “r8169";
> > >             reg = <0 0x10000000 0 0x1000>;
> > >             interrupts = <0 80 4  0 81 4  0 82 4>;
> > >         };
> > >     };
> > > };
> > > dtdev = [ "/soc/ethernet@10000000”, “/soc/ethernet@f2000000” ]
> > > There is no link which dtdev entry belongs to which node. Therefor there is no 
> > > way to link the vMasterID created to pMasterID.
> > I agree there is no link today. But we could add a property in the partial 
> > device-tree to mention which physical device is associated.
> >
> > With that, I think all, the complexity is moved to libxl and it will be easier 
> > for the user to use vIOMMU.
> >
> > [...]
> As of now libxl directly coping the partial DT to guest DT without any 
> modification. If we have to go to this route libxl has to modify
> the partial DT in libxl to include “iommus” or "iommu-map”. Is that okay to modify the partial DT in libxl 
I am not aware of any issue to modify the partial device-tree. In fact, 
I am strongly in favor of libxl to modify it if it greatly improve the 
user experience.
Cheers,

--
Julien Grall