Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Julien Grall <julien@xxxxxxx>]

On 28/10/2022 15:37, Bertrand Marquis wrote:
> Hi Julien,
Hi Bertrand,

> > On 28 Oct 2022, at 14:27, Julien Grall <julien@xxxxxxx> wrote:
> >
> >
> >
> > On 28/10/2022 14:13, Bertrand Marquis wrote:
> > > Hi Julien,
> > Hi Bertrand,
> >
> > > > On 28 Oct 2022, at 14:06, Julien Grall <julien@xxxxxxx> wrote:
> > > >
> > > > Hi Rahul,
> > > >
> > > > On 28/10/2022 13:54, Rahul Singh wrote:
> > > > > > > > > > For ACPI, I would have expected the information to be found in the IOREQ.
> > > > > > > > > >
> > > > > > > > > > So can you add more context why this is necessary for everyone?
> > > > > > > > > We have information for IOMMU and Master-ID but we don’t have information for 
> > > > > > > > > linking vMaster-ID to pMaster-ID.
> > > > > > > > I am confused. Below, you are making the virtual master ID optional. So 
> > > > > > > > shouldn't this be mandatory if you really need the mapping with the virtual ID?
> > > > > > > vMasterID is optional if user knows pMasterID is unique on the system. But if 
> > > > > > > pMasterId is not unique then user needs to provide the vMasterID.
> > > > > > So the expectation is the user will be able to know that the pMasterID is uniq. 
> > > > > > This may be easy with a couple of SMMUs, but if you have 50+ (as suggested 
> > > > > > above). This will become a pain on larger system.
> > > > > >
> > > > > > IHMO, it would be much better if we can detect that in libxl (see below).
> > > > > We can make the vMasterID compulsory to avoid complexity in libxl to solve this
> > > > In general, complexity in libxl is not too much of problem.
> > > I am a bit unsure about this strategy.
> > > Currently xl has one configuration file where you put all Xen parameters. The 
> > > device tree is only needed by some guests to have a description of the system 
> > > they run on.
> > > If we change the model and say that Xen configuration parameters are both in 
> > > the configuration and the device tree, we somehow enforce to have a device tree 
> > > even though some guests do not need it at all (for example Zephyr).
> > I think my approach was misunderstood because there is no change in the 
> > existing model.
> >
> > What I am suggesting is to not introduce iommu_devid_map but instead let libxl 
> > allocate the virtual Master-ID and create the mapping with the physical 
> > Master-ID.
> >
> > Libxl would then update the property "iommus" in the device-tree with the 
> > allocated virtual Master-ID.
> Ok I understand now.
>
> > Each node in the partial device-tree would need to have a property
> > to refer to the physical device just so we know how to update the "iommus". The 
> > list of device passthrough will still be specified in the configuration file. IOW, the 
> > partial device-tree is not directly involved in the configuration of the guest.
> But we will generate it. How would something like Zephyr guest work ? Zephyr is 
> not using the device tree we pass, it has an embedded one.
In general, guest that don't use the device-tree/ACPI table to detect 
the layout are already in a bad situation because we don't guarantee 
that the layout (memory, interrupt...) will be stable across Xen 
version. Although, there are a implicit agreement that the layout will 
not change for minor release (i.e. 4.14.x).
But see below for some suggestions how this could be handled.

> > So far, I don't see a particular issue with this approach because the vMaster 
> > ID algorithm allocation should be generic. But please let me know if you think 
> > there are bits I am missing.
> I am a bit afraid of things that are “automatic”.
>
> For everything else we let the user in control (IPA for mapping, virtual 
> interrupt number) and in this case we switch to a model where we automatically 
> generated a vMaster ID.
We only let the user control where the device is mapped. But this is 
quite fragile... I think this should be generated at runtime.
> With this model, guest not using the device tree will have to guess the vMaster 
> ID or somehow know how the tools are generating it to use the right one.
To be honest, this is already the case today because the layout exposed 
to the guest is technically not fixed. Yes, so far, we haven't changed 
it too much. But sooner or later, this is going to bite because we made 
clear that the layout is not stable.
Now, if those projects are willing to rebuild for each version, then we 
could use the following approach:
  1) Write the xl.cfg
  2) Ask libxl to generate the device-tree
  3) Build Zephyr
  4) Create the domain

The expectation is for a given Xen version (and compatible), libxl will 
always generate the same Device-Tree.
Cheers,

--
Julien Grall