Re: PVH Dom0 related UART failure

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 23 May 2023, Jan Beulich wrote:
> On 23.05.2023 00:20, Stefano Stabellini wrote:
> > On Sat, 20 May 2023, Roger Pau Monné wrote:
> >> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
> >> index ec2e978a4e6b..0ff8e940fa8d 100644
> >> --- a/xen/drivers/vpci/header.c
> >> +++ b/xen/drivers/vpci/header.c
> >> @@ -289,6 +289,13 @@ static int modify_bars(const struct pci_dev *pdev, 
> >> uint16_t cmd, bool rom_only)
> >>       */
> >>      for_each_pdev ( pdev->domain, tmp )
> >>      {
> >> +        if ( !tmp->vpci )
> >> +        {
> >> +            printk(XENLOG_G_WARNING "%pp: not handled by vPCI for %pd\n",
> >> +                   &tmp->sbdf, pdev->domain);
> >> +            continue;
> >> +        }
> >> +
> >>          if ( tmp == pdev )
> >>          {
> >>              /*
> >> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
> >> index 652807a4a454..0baef3a8d3a1 100644
> >> --- a/xen/drivers/vpci/vpci.c
> >> +++ b/xen/drivers/vpci/vpci.c
> >> @@ -72,7 +72,12 @@ int vpci_add_handlers(struct pci_dev *pdev)
> >>      unsigned int i;
> >>      int rc = 0;
> >>  
> >> -    if ( !has_vpci(pdev->domain) )
> >> +    if ( !has_vpci(pdev->domain) ||
> >> +         /*
> >> +          * Ignore RO and hidden devices, those are in use by Xen and vPCI
> >> +          * won't work on them.
> >> +          */
> >> +         pci_get_pdev(dom_xen, pdev->sbdf) )
> >>          return 0;
> >>  
> >>      /* We should not get here twice for the same device. */
> > 
> > 
> > Now this patch works! Thank you!! :-)
> > 
> > You can check the full logs here
> > https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/4329259080
> > 
> > Is the patch ready to be upstreamed aside from the commit message?
> 
> I don't think so. vPCI ought to work on "r/o" devices. Out of curiosity,
> have you also tried my (hackish and hence RFC) patch [1]?
> 
> [1] https://lists.xen.org/archives/html/xen-devel/2021-08/msg01489.html

I don't know the code well enough to discuss what is the best solution.
I'll let you and Roger figure it out. I would only kindly request to
solve this in few days so that we can enable the real hardware PVH test
in gitlab-ci as soon as possible. I think it is critical as it will
allow us to catch many real issues going forward.

For sure I can test your patch. BTW it is also really easy for you to do
it your simply by pushing a branch to a repo on gitlab-ci and watch for
the results. If you are interested let me know I can give you a
tutorial, you just need to create a repo, and register the gitlab runner
and voila'.

This is the outcome:

https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/876808194


(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:00.2
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:02.0
(XEN) Assertion 'd == dom_xen && system_state < SYS_STATE_active' failed at 
drivers/vpci/header.c:313
(XEN) ----[ Xen-4.18-unstable  x86_64  debug=y  Tainted:   C    ]----
(XEN) CPU:    14
(XEN) RIP:    e008:[<ffff82d04026839e>] 
drivers/vpci/header.c#modify_bars+0x3ba/0x4a3
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor (d0v0)
(XEN) rax: ffff830390164000   rbx: ffff83038bd8a7f0   rcx: 0000000000000010
(XEN) rdx: ffff83038e3b7fff   rsi: ffff83038e3c83a0   rdi: ffff83038e3c8398
(XEN) rbp: ffff83038e3b7c08   rsp: ffff83038e3b7b98   r8:  0000000000000001
(XEN) r9:  ffff83038dcfa000   r10: 000000000000000e   r11: 0000000000000000
(XEN) r12: 0000000000000007   r13: 00000000000dcc03   r14: ffff83039016ad10
(XEN) r15: 00000000000dcc00   cr0: 000000008005003b   cr4: 0000000000f506e0
(XEN) cr3: 000000038ddfe000   cr2: ffff88814e3ff000
(XEN) fsb: 0000000000000000   gsb: ffff888149a00000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d04026839e> 
(drivers/vpci/header.c#modify_bars+0x3ba/0x4a3):
(XEN)  3d c4 d1 37 00 02 76 c0 <0f> 0b 4c 8b 75 b0 0f ae e8 48 83 7d 98 00 74 2b
(XEN) Xen stack trace from rsp=ffff83038e3b7b98:
(XEN)    0000000000000002 ffff830390150230 ffff830390164000 ffff830390164158
(XEN)    ffff830390150230 00ff830300000000 ffff83038dcf8b00 0000000000000003
(XEN)    0000000000000206 ffff83038bd8c010 0000000000000000 0000000000000002
(XEN)    0000000000000002 0000000000000004 ffff83038e3b7c18 ffff82d040268909
(XEN)    ffff83038e3b7ca0 ffff82d040267998 000000118e3b7ca0 0000000000117803
(XEN)    0000000400000000 ffff830390150230 0000000000000000 0000000000000000
(XEN)    0000000400000002 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000002 0000000000000000 0000000000000004 0000000000000000
(XEN)    ffff82d04041df40 ffff83038e3b7cd0 ffff82d0402cb649 0000001140332c9e
(XEN)    ffff83038e3b7d88 0000000000000000 ffff83038dd094a0 ffff83038e3b7d30
(XEN)    ffff82d0402caa72 ffff83038e3b7ce0 00000cfc00000002 0000000000000000
(XEN)    0000000000000002 0000000000000000 ffff83038dd094a0 ffff83038e3b7d88
(XEN)    0000000000000001 0000000000000000 0000000000000000 ffff83038e3b7d58
(XEN)    ffff82d0402cab08 0000000000000002 ffff83038dcfa000 ffff83038e3b7e28
(XEN)    ffff83038e3b7dd0 ffff82d0402ba4ee 0000000000000cfc 0000000000000000
(XEN)    ffff83038e38f000 0000000000000000 0000000000000cfc 0000000000000000
(XEN)    0000000200000001 0001000000000000 0000000000000002 0000000000000002
(XEN)    0000000000000000 ffff83038e3b7e44 ffff83038dcfa000 ffff83038e3b7e10
(XEN)    ffff82d0402ba871 0000000000000000 ffff83038e3b7e44 0000000000000002
(XEN)    0000000000000cfc ffff83038dcf6000 0000000000000000 ffff83038e3b7e30
(XEN) Xen call trace:
(XEN)    [<ffff82d04026839e>] R drivers/vpci/header.c#modify_bars+0x3ba/0x4a3
(XEN)    [<ffff82d040268909>] F drivers/vpci/header.c#cmd_write+0x2c/0x3b
(XEN)    [<ffff82d040267998>] F vpci_write+0x14c/0x268
(XEN)    [<ffff82d0402cb649>] F arch/x86/hvm/io.c#vpci_portio_write+0xa0/0xa7
(XEN)    [<ffff82d0402caa72>] F hvm_process_io_intercept+0x203/0x26f
(XEN)    [<ffff82d0402cab08>] F hvm_io_intercept+0x2a/0x4c
(XEN)    [<ffff82d0402ba4ee>] F arch/x86/hvm/emulate.c#hvmemul_do_io+0x29a/0x5ee
(XEN)    [<ffff82d0402ba871>] F 
arch/x86/hvm/emulate.c#hvmemul_do_io_buffer+0x2f/0x6a
(XEN)    [<ffff82d0402bbd10>] F hvmemul_do_pio_buffer+0x33/0x35
(XEN)    [<ffff82d0402cb41d>] F handle_pio+0x6d/0x1b8
(XEN)    [<ffff82d0402a2e4d>] F svm_vmexit_handler+0x10fe/0x18e2
(XEN)    [<ffff82d0402034dc>] F svm_stgi_label+0x5/0x15


You can also check how I applied the patch here:
https://gitlab.com/xen-project/people/sstabellini/xen/-/commit/851e76bf0a1cf6f040b6e90795d216ebfcc069cc