Re: PVH Dom0 related UART failure

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 19 May 2023, Jan Beulich wrote:
> On 18.05.2023 12:34, Roger Pau Monné wrote:
> > On Wed, May 17, 2023 at 05:59:31PM -0700, Stefano Stabellini wrote:
> >> I have run into another PVH Dom0 issue. I am trying to enable a PVH Dom0
> >> test with the brand new gitlab-ci runner offered by Qubes. It is an AMD
> >> Zen3 system and we already have a few successful tests with it, see
> >> automation/gitlab-ci/test.yaml.
> >>
> >> We managed to narrow down the issue to a console problem. We are
> >> currently using console=com1 com1=115200,8n1,pci,msi as Xen command line
> >> options, it works with PV Dom0 and it is using a PCI UART card.
> >>
> >> In the case of Dom0 PVH:
> >> - it works without console=com1
> >> - it works with console=com1 and with the patch appended below
> >> - it doesn't work otherwise and crashes with this error:
> >> https://matrix-client.matrix.org/_matrix/media/r0/download/invisiblethingslab.com/uzcmldIqHptFZuxqsJtviLZK
> > 
> > Jan also noticed this, and we have a ticket for it in gitlab:
> > 
> > https://gitlab.com/xen-project/xen/-/issues/85
> > 
> >> What is the right way to fix it?
> > 
> > I think the right fix is to simply avoid hidden devices from being
> > handled by vPCI, in any case such devices won't work propewrly with
> > vPCI because they are in use by Xen, and so any cached information by
> > vPCI is likely to become stable as Xen can modify the device without
> > vPCI noticing.
> > 
> > I think the chunk below should help.  It's not clear to me however how
> > hidden devices should be handled, is the intention to completely hide
> > such devices from dom0?
> 
> No, Dom0 should still be able to see them in a (mostly) r/o fashion.

But why? If something is in-use by Xen (e.g. IOMMU, a serial PCI device,
etc.) ideally Dom0 shouldn't even know of its existence because the
device is not exposed to Dom0. Dom0 is not meant to use it. Why let Dom0
know it exists if Dom0 should not use it?

In Xen on ARM, initially we didn't expose devices used by Xen to Dom0
at all.  However to hide them completely we had to make complex device
tree manipulations. Now instead we leave the device nodes in device tree
as is, but we change the "status" property to "disabled".

The idea is still that we completely hide Xen devices from Dom0, but
because of implementation complexity, instead of completing taking away
the corresponding nodes from device tree, we change them to disabled,
which still leads to the same result: the guest OS will skip them.

I am saying this without being familiar with the x86 PVH implementation,
so pardon my ignorance here, but it seems to me that as we are moving
toward an better architecture with PVH, once that allows any OS to be
Dom0, not just Linux, we would want to also completely hide devices
owned by Xen from Dom0.

That way we don't need any workaround in the guest OS for it not to use
them.