Re: PVH Dom0 related UART failure

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 18 May 2023, Roger Pau Monné wrote:
> On Wed, May 17, 2023 at 05:59:31PM -0700, Stefano Stabellini wrote:
> > Hi all,
> > 
> > I have run into another PVH Dom0 issue. I am trying to enable a PVH Dom0
> > test with the brand new gitlab-ci runner offered by Qubes. It is an AMD
> > Zen3 system and we already have a few successful tests with it, see
> > automation/gitlab-ci/test.yaml.
> > 
> > We managed to narrow down the issue to a console problem. We are
> > currently using console=com1 com1=115200,8n1,pci,msi as Xen command line
> > options, it works with PV Dom0 and it is using a PCI UART card.
> > 
> > In the case of Dom0 PVH:
> > - it works without console=com1
> > - it works with console=com1 and with the patch appended below
> > - it doesn't work otherwise and crashes with this error:
> > https://matrix-client.matrix.org/_matrix/media/r0/download/invisiblethingslab.com/uzcmldIqHptFZuxqsJtviLZK
> 
> Jan also noticed this, and we have a ticket for it in gitlab:
> 
> https://gitlab.com/xen-project/xen/-/issues/85
> 
> > What is the right way to fix it?
> 
> I think the right fix is to simply avoid hidden devices from being
> handled by vPCI, in any case such devices won't work propewrly with
> vPCI because they are in use by Xen, and so any cached information by
> vPCI is likely to become stable as Xen can modify the device without
> vPCI noticing.
> 
> I think the chunk below should help.  It's not clear to me however how
> hidden devices should be handled, is the intention to completely hide
> such devices from dom0?

I like the idea but the patch below still failed:

(XEN) Xen call trace:
(XEN)    [<ffff82d0402682b0>] R drivers/vpci/header.c#modify_bars+0x2b3/0x44d
(XEN)    [<ffff82d040268714>] F drivers/vpci/header.c#init_bars+0x2ca/0x372
(XEN)    [<ffff82d0402673b3>] F vpci_add_handlers+0xd5/0x10a
(XEN)    [<ffff82d0404408b9>] F 
drivers/passthrough/pci.c#setup_one_hwdom_device+0x73/0x97
(XEN)    [<ffff82d0404409b0>] F 
drivers/passthrough/pci.c#_setup_hwdom_pci_devices+0x63/0x15b
(XEN)    [<ffff82d04027df08>] F 
drivers/passthrough/pci.c#pci_segments_iterate+0x43/0x69
(XEN)    [<ffff82d040440e29>] F setup_hwdom_pci_devices+0x25/0x2c
(XEN)    [<ffff82d04043cb1a>] F 
drivers/passthrough/amd/pci_amd_iommu.c#amd_iommu_hwdom_init+0xd4/0xdd
(XEN)    [<ffff82d0404403c9>] F iommu_hwdom_init+0x49/0x53
(XEN)    [<ffff82d04045175e>] F dom0_construct_pvh+0x160/0x138d
(XEN)    [<ffff82d040468914>] F construct_dom0+0x5c/0xb7
(XEN)    [<ffff82d0404619c1>] F __start_xen+0x2423/0x272d
(XEN)    [<ffff82d040203344>] F __high_start+0x94/0xa0

I haven't managed to figure out why yet.


> --- 
> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
> index 652807a4a454..0baef3a8d3a1 100644
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -72,7 +72,12 @@ int vpci_add_handlers(struct pci_dev *pdev)
>      unsigned int i;
>      int rc = 0;
>  
> -    if ( !has_vpci(pdev->domain) )
> +    if ( !has_vpci(pdev->domain) ||
> +         /*
> +          * Ignore RO and hidden devices, those are in use by Xen and vPCI
> +          * won't work on them.
> +          */
> +         pci_get_pdev(dom_xen, pdev->sbdf) )
>          return 0;
>  
>      /* We should not get here twice for the same device. */

i