Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Michal Orzel <michal.orzel@xxxxxxx>]

On 28/10/2022 17:45, Bertrand Marquis wrote:
> 
> 
> Hi Julien,
> 
>> On 28 Oct 2022, at 16:01, Julien Grall <julien@xxxxxxx> wrote:
>>
>>
>>
>> On 28/10/2022 15:37, Bertrand Marquis wrote:
>>> Hi Julien,
>>
>> Hi Bertrand,
>>
>>>> On 28 Oct 2022, at 14:27, Julien Grall <julien@xxxxxxx> wrote:
>>>>
>>>>
>>>>
>>>> On 28/10/2022 14:13, Bertrand Marquis wrote:
>>>>> Hi Julien,
>>>>
>>>> Hi Bertrand,
>>>>
>>>>>> On 28 Oct 2022, at 14:06, Julien Grall <julien@xxxxxxx> wrote:
>>>>>>
>>>>>> Hi Rahul,
>>>>>>
>>>>>> On 28/10/2022 13:54, Rahul Singh wrote:
>>>>>>>>>>>> For ACPI, I would have expected the information to be found in the 
>>>>>>>>>>>> IOREQ.
>>>>>>>>>>>>
>>>>>>>>>>>> So can you add more context why this is necessary for everyone?
>>>>>>>>>>> We have information for IOMMU and Master-ID but we don’t have 
>>>>>>>>>>> information for linking vMaster-ID to pMaster-ID.
>>>>>>>>>>
>>>>>>>>>> I am confused. Below, you are making the virtual master ID optional. 
>>>>>>>>>> So shouldn't this be mandatory if you really need the mapping with 
>>>>>>>>>> the virtual ID?
>>>>>>>>> vMasterID is optional if user knows pMasterID is unique on the 
>>>>>>>>> system. But if pMasterId is not unique then user needs to provide the 
>>>>>>>>> vMasterID.
>>>>>>>>
>>>>>>>> So the expectation is the user will be able to know that the pMasterID 
>>>>>>>> is uniq. This may be easy with a couple of SMMUs, but if you have 50+ 
>>>>>>>> (as suggested above). This will become a pain on larger system.
>>>>>>>>
>>>>>>>> IHMO, it would be much better if we can detect that in libxl (see 
>>>>>>>> below).
>>>>>>> We can make the vMasterID compulsory to avoid complexity in libxl to 
>>>>>>> solve this
>>>>>>
>>>>>> In general, complexity in libxl is not too much of problem.
>>>>> I am a bit unsure about this strategy.
>>>>> Currently xl has one configuration file where you put all Xen parameters. 
>>>>> The device tree is only needed by some guests to have a description of 
>>>>> the system they run on.
>>>>> If we change the model and say that Xen configuration parameters are both 
>>>>> in the configuration and the device tree, we somehow enforce to have a 
>>>>> device tree even though some guests do not need it at all (for example 
>>>>> Zephyr).
>>>>
>>>> I think my approach was misunderstood because there is no change in the 
>>>> existing model.
>>>>
>>>> What I am suggesting is to not introduce iommu_devid_map but instead let 
>>>> libxl allocate the virtual Master-ID and create the mapping with the 
>>>> physical Master-ID.
>>>>
>>>> Libxl would then update the property "iommus" in the device-tree with the 
>>>> allocated virtual Master-ID.
>>> Ok I understand now.
>>>>
>>>> Each node in the partial device-tree would need to have a property
>>>> to refer to the physical device just so we know how to update the 
>>>> "iommus". The list of device passthrough will still be specified in the 
>>>> configuration file. IOW, the partial device-tree is not directly involved 
>>>> in the configuration of the guest.
>>> But we will generate it. How would something like Zephyr guest work ? 
>>> Zephyr is not using the device tree we pass, it has an embedded one.
>>
>> In general, guest that don't use the device-tree/ACPI table to detect the 
>> layout are already in a bad situation because we don't guarantee that the 
>> layout (memory, interrupt...) will be stable across Xen version. Although, 
>> there are a implicit agreement that the layout will not change for minor 
>> release (i.e. 4.14.x).
> 
> Well right now we have no ACPI support.
> But I still think that a non dtb guest is definitely a use case we need to 
> keep in mind for embedded and safety as most proprietary RTOS are not using a 
> device tree.
> 
>>
>> But see below for some suggestions how this could be handled.
>>
>>>>
>>>> So far, I don't see a particular issue with this approach because the 
>>>> vMaster ID algorithm allocation should be generic. But please let me know 
>>>> if you think there are bits I am missing.
>>> I am a bit afraid of things that are “automatic”.
>>> For everything else we let the user in control (IPA for mapping, virtual 
>>> interrupt number) and in this case we switch to a model where we 
>>> automatically generated a vMaster ID.
>>
>> We only let the user control where the device is mapped. But this is quite 
>> fragile... I think this should be generated at runtime.
>>
>>> With this model, guest not using the device tree will have to guess the 
>>> vMaster ID or somehow know how the tools are generating it to use the right 
>>> one.
>>
>> To be honest, this is already the case today because the layout exposed to 
>> the guest is technically not fixed. Yes, so far, we haven't changed it too 
>> much. But sooner or later, this is going to bite because we made clear that 
>> the layout is not stable.
>>
>> Now, if those projects are willing to rebuild for each version, then we 
>> could use the following approach:
>>  1) Write the xl.cfg
>>  2) Ask libxl to generate the device-tree
>>  3) Build Zephyr
>>  4) Create the domain
>>
>> The expectation is for a given Xen version (and compatible), libxl will 
>> always generate the same Device-Tree.
> 
> This is a good idea yes :-)

Zephyr still uses a device tree but in a static way - everything must be 
defined in a .dts before building it.
The steps mentioned by Julien are already followed by Zephyr when building it 
to run as a Xen VM.
You can take a look at the "Updating configuration" section at the bottom of 
the following site:
https://docs.zephyrproject.org/latest/boards/arm64/xenvm/doc/index.html

So, as we tend to use Zephyr as a de facto RTOS for Xen, it is already aware of 
possible changes to the layout.

> 
> Cheers
> Bertrand
> 
>>
>> Cheers,
>>
>> --
>> Julien Grall
> 

~Michal