Re: PVH Dom0 related UART failure

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, May 18, 2023 at 06:46:52PM -0700, Stefano Stabellini wrote:
> On Thu, 18 May 2023, Roger Pau Monné wrote:
> > On Wed, May 17, 2023 at 05:59:31PM -0700, Stefano Stabellini wrote:
> > > Hi all,
> > > 
> > > I have run into another PVH Dom0 issue. I am trying to enable a PVH Dom0
> > > test with the brand new gitlab-ci runner offered by Qubes. It is an AMD
> > > Zen3 system and we already have a few successful tests with it, see
> > > automation/gitlab-ci/test.yaml.
> > > 
> > > We managed to narrow down the issue to a console problem. We are
> > > currently using console=com1 com1=115200,8n1,pci,msi as Xen command line
> > > options, it works with PV Dom0 and it is using a PCI UART card.
> > > 
> > > In the case of Dom0 PVH:
> > > - it works without console=com1
> > > - it works with console=com1 and with the patch appended below
> > > - it doesn't work otherwise and crashes with this error:
> > > https://matrix-client.matrix.org/_matrix/media/r0/download/invisiblethingslab.com/uzcmldIqHptFZuxqsJtviLZK
> > 
> > Jan also noticed this, and we have a ticket for it in gitlab:
> > 
> > https://gitlab.com/xen-project/xen/-/issues/85
> > 
> > > What is the right way to fix it?
> > 
> > I think the right fix is to simply avoid hidden devices from being
> > handled by vPCI, in any case such devices won't work propewrly with
> > vPCI because they are in use by Xen, and so any cached information by
> > vPCI is likely to become stable as Xen can modify the device without
> > vPCI noticing.
> > 
> > I think the chunk below should help.  It's not clear to me however how
> > hidden devices should be handled, is the intention to completely hide
> > such devices from dom0?
> 
> I like the idea but the patch below still failed:
> 
> (XEN) Xen call trace:
> (XEN)    [<ffff82d0402682b0>] R drivers/vpci/header.c#modify_bars+0x2b3/0x44d
> (XEN)    [<ffff82d040268714>] F drivers/vpci/header.c#init_bars+0x2ca/0x372
> (XEN)    [<ffff82d0402673b3>] F vpci_add_handlers+0xd5/0x10a
> (XEN)    [<ffff82d0404408b9>] F 
> drivers/passthrough/pci.c#setup_one_hwdom_device+0x73/0x97
> (XEN)    [<ffff82d0404409b0>] F 
> drivers/passthrough/pci.c#_setup_hwdom_pci_devices+0x63/0x15b
> (XEN)    [<ffff82d04027df08>] F 
> drivers/passthrough/pci.c#pci_segments_iterate+0x43/0x69
> (XEN)    [<ffff82d040440e29>] F setup_hwdom_pci_devices+0x25/0x2c
> (XEN)    [<ffff82d04043cb1a>] F 
> drivers/passthrough/amd/pci_amd_iommu.c#amd_iommu_hwdom_init+0xd4/0xdd
> (XEN)    [<ffff82d0404403c9>] F iommu_hwdom_init+0x49/0x53
> (XEN)    [<ffff82d04045175e>] F dom0_construct_pvh+0x160/0x138d
> (XEN)    [<ffff82d040468914>] F construct_dom0+0x5c/0xb7
> (XEN)    [<ffff82d0404619c1>] F __start_xen+0x2423/0x272d
> (XEN)    [<ffff82d040203344>] F __high_start+0x94/0xa0
> 
> I haven't managed to figure out why yet.

Do you have some other patches applied?

I've tested this by manually hiding a device on my system and can
confirm that without the fix I hit the ASSERT, but with the patch
applied I no longer hit it.  I have no idea how can you get into
init_bars if the device is hidden and thus belongs to dom_xen.

FWIW, I've used the following chunk to make a device RO and enable
memory decoding:

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 07d1986d330a..e4de372af7c9 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -1111,6 +1111,16 @@ static int __hwdom_init cf_check 
_setup_hwdom_pci_devices(
 {
     struct setup_hwdom *ctxt = arg;
     int bus, devfn;
+    pci_sbdf_t hide = {
+        .seg = 0,
+        .bus = 0,
+        .dev = 8,
+        .fn = 0,
+    };
+    uint16_t cmd = pci_conf_read16(hide, PCI_COMMAND);
+
+    pci_conf_write16(hide, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY);
+    printk("hide dev: %d\n", pci_ro_device(0, 0, PCI_DEVFN(8, 0)));
 
     for ( bus = 0; bus < 256; bus++ )
     {