Re: PVH Dom0 related UART failure

[Jan Beulich <jbeulich@xxxxxxxx>]

On 20.05.2023 00:44, Stefano Stabellini wrote:
> On Fri, 19 May 2023, Jan Beulich wrote:
>> On 18.05.2023 12:34, Roger Pau Monné wrote:
>>> On Wed, May 17, 2023 at 05:59:31PM -0700, Stefano Stabellini wrote:
>>>> I have run into another PVH Dom0 issue. I am trying to enable a PVH Dom0
>>>> test with the brand new gitlab-ci runner offered by Qubes. It is an AMD
>>>> Zen3 system and we already have a few successful tests with it, see
>>>> automation/gitlab-ci/test.yaml.
>>>>
>>>> We managed to narrow down the issue to a console problem. We are
>>>> currently using console=com1 com1=115200,8n1,pci,msi as Xen command line
>>>> options, it works with PV Dom0 and it is using a PCI UART card.
>>>>
>>>> In the case of Dom0 PVH:
>>>> - it works without console=com1
>>>> - it works with console=com1 and with the patch appended below
>>>> - it doesn't work otherwise and crashes with this error:
>>>> https://matrix-client.matrix.org/_matrix/media/r0/download/invisiblethingslab.com/uzcmldIqHptFZuxqsJtviLZK
>>>
>>> Jan also noticed this, and we have a ticket for it in gitlab:
>>>
>>> https://gitlab.com/xen-project/xen/-/issues/85
>>>
>>>> What is the right way to fix it?
>>>
>>> I think the right fix is to simply avoid hidden devices from being
>>> handled by vPCI, in any case such devices won't work propewrly with
>>> vPCI because they are in use by Xen, and so any cached information by
>>> vPCI is likely to become stable as Xen can modify the device without
>>> vPCI noticing.
>>>
>>> I think the chunk below should help.  It's not clear to me however how
>>> hidden devices should be handled, is the intention to completely hide
>>> such devices from dom0?
>>
>> No, Dom0 should still be able to see them in a (mostly) r/o fashion.
> 
> But why? If something is in-use by Xen (e.g. IOMMU, a serial PCI device,
> etc.) ideally Dom0 shouldn't even know of its existence because the
> device is not exposed to Dom0. Dom0 is not meant to use it. Why let Dom0
> know it exists if Dom0 should not use it?

Because we want to disturb the topology Dom0 sees as little as possible.
For example, imagine the device is func 0 of a multi-function device.
How would Dom0 know to even look at the other functions, when it can't
even read the relevant bit in func 0's config space?

Jan

> In Xen on ARM, initially we didn't expose devices used by Xen to Dom0
> at all.  However to hide them completely we had to make complex device
> tree manipulations. Now instead we leave the device nodes in device tree
> as is, but we change the "status" property to "disabled".
> 
> The idea is still that we completely hide Xen devices from Dom0, but
> because of implementation complexity, instead of completing taking away
> the corresponding nodes from device tree, we change them to disabled,
> which still leads to the same result: the guest OS will skip them.
> 
> I am saying this without being familiar with the x86 PVH implementation,
> so pardon my ignorance here, but it seems to me that as we are moving
> toward an better architecture with PVH, once that allows any OS to be
> Dom0, not just Linux, we would want to also completely hide devices
> owned by Xen from Dom0.
> 
> That way we don't need any workaround in the guest OS for it not to use
> them.