Re: [Xen-devel] dom0 kernel - irq nobody cared ... the continuing saga ..

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Tue, Feb 10, 2015 at 04:35:47PM +0100, Sander Eikelenboom wrote:
> 
> Tuesday, February 10, 2015, 2:26:43 PM, you wrote:
> 
> >>>> On 10.02.15 at 14:07, <linux@xxxxxxxxxxxxxx> wrote:
> >> Tuesday, February 10, 2015, 10:35:36 AM, you wrote:
> >>>>>> On 10.02.15 at 10:19, <linux@xxxxxxxxxxxxxx> wrote:
> >>>>> I would have thought that xen-pciback would install an interrupt
> >>>>> handler here too when a device using IRQ 18 gets handed to a
> >>>>> guest. May there be something broken in xen_pcibk_control_isr()?
> >>>> 
> >>>> It seems to only do that for PV guests, not for HVM.
> >> 
> >>> Interesting; no idea why that would be.
> >> 
> >> Coming back to this part .. with some additional debugging on HVM guest 
> >> start i 
> >> get:
> >> [   84.362097] pciback 0000:02:00.0: resetting (FLR, D3,  bus) the device
> >> [   84.422884] pciback 0000:02:00.0: xen_pcibk_reset_device: me here
> >> [   84.452639] pciback 0000:02:00.0: xen_pcibk_control_isr: return !enable 
> >> && 
> >> !dev_data->isr_on enable:0 dev_data->isr_on:0
> >> [   85.636725] pciback 0000:00:19.0: resetting (FLR, D3,  ) the device
> >> [   85.774845] pciback 0000:00:19.0: xen_pcibk_reset_device: me here
> >> [   85.810857] pciback 0000:00:19.0: xen_pcibk_control_isr: return !enable 
> >> && 
> >> !dev_data->isr_on enable:0 dev_data->isr_on:0
> >> [   85.865075] xen_pciback: vpci: 0000:02:00.0: assign to virtual slot 0
> >> [   85.886926] pciback 0000:02:00.0: registering for 1
> >> [   85.907936] xen_pciback: vpci: 0000:00:19.0: assign to virtual slot 1
> >> [   85.929490] pciback 0000:00:19.0: registering for 1
> >>  
> >> So it bails out on:
> >>         /* Asked to disable, but ISR isn't runnig */
> >>         if (!enable && !dev_data->isr_on){
> >>                 dev_warn(&dev->dev, "%s: return !enable && 
> >> !dev_data->isr_on 
> >> enable:%d dev_data->isr_on:%d\n", __func__, enable, dev_data->isr_on ? 1 : 
> >> 0);
> >>                 return;
> >>         }
> 
> > I don't really know how this code is supposed to work (we don't use
> > it in our kernels), but it seems the more interesting question is what
> > happens when xen_pcibk_do_op() calls this function. I also note there
> > is a sysfs interface to control some aspects of this, but I can't see how
> > that would work when it only sets ->isr_on but doesn't install the IRQ
> > handler if it isn't already installed.
> 
> > Jan
> 
> Suse is still forward porting and not considering upstream/pvops ?
> 
> Hmmm it seems xen_pcibk_do_op() is never called ..
> 
> Turned up the pciback debug, and annotated xen_pcibk_do_op() it self with 
> more warnings, it should print something when invoked.
> complete dmesg attached.
> 
> Here the debug output from pciback:
> 
> seizing:
> [    9.113134] xen_pciback: wants to seize 0000:02:00.0
> [    9.113137] xen_pciback: wants to seize 0000:00:19.0
> [    9.113146] pciback 0000:00:00.0: probing...
> [    9.113154] pciback 0000:00:02.0: probing...
> [    9.113160] pciback 0000:00:14.0: probing...
> [    9.113165] pciback 0000:00:16.0: probing...
> [    9.113171] pciback 0000:00:16.3: probing...
> [    9.113176] pciback 0000:00:19.0: probing...
> [    9.113178] pciback 0000:00:19.0: seizing device
> [    9.113182] pciback 0000:00:19.0: pcistub_device_alloc
> [    9.113184] pciback 0000:00:19.0: deferring initialization
> [    9.113189] pciback 0000:00:1a.0: probing...
> [    9.113194] pciback 0000:00:1b.0: probing...
> [    9.113200] pciback 0000:00:1c.0: probing...
> [    9.113205] pciback 0000:00:1c.2: probing...
> [    9.113210] pciback 0000:00:1d.0: probing...
> [    9.113216] pciback 0000:00:1f.0: probing...
> [    9.113221] pciback 0000:00:1f.2: probing...
> [    9.113226] pciback 0000:00:1f.3: probing...
> [    9.113232] pciback 0000:02:00.0: probing...
> [    9.113234] pciback 0000:02:00.0: seizing device
> [    9.113237] pciback 0000:02:00.0: pcistub_device_alloc
> [    9.113239] pciback 0000:02:00.0: deferring initialization
> 
> initialization while dom0 is still booting:
> [   14.371824] pciback 0000:02:00.0: initializing...
> [   14.371829] pciback 0000:02:00.0: initializing config
> [   14.371879] pciback 0000:02:00.0: enabling device
> [   14.371944] xen: registering gsi 18 triggering 0 polarity 1
> [   14.371949] Already setup the GSI :18
> [   14.372457] pciback 0000:02:00.0: save state of device
> [   14.372582] pciback 0000:02:00.0: resetting (FLR, D3, etc) the device
> [   14.403788] pciback 0000:02:00.0: reset device
> [   14.403794] pciback 0000:02:00.0: xen_pcibk_reset_device: me heres
> [   14.404638] pciback 0000:02:00.0: xen_pcibk_control_isr: return !enable && 
> !dev_data->isr_on enable:0 dev_data->isr_on:0
> [   14.406186] pciback 0000:00:19.0: initializing...
> [   14.406189] pciback 0000:00:19.0: initializing config
> [   14.406225] pciback 0000:00:19.0: enabling device
> [   14.406320] xen: registering gsi 20 triggering 0 polarity 1
> [   14.406340] xen: --> pirq=20 -> irq=20 (gsi=20)
> [   14.406372] pciback 0000:00:19.0: save state of device
> [   14.406435] pciback 0000:00:19.0: resetting (FLR, D3, etc) the device
> [   14.507741] pciback 0000:00:19.0: reset device
> [   14.507747] pciback 0000:00:19.0: xen_pcibk_reset_device: me heres
> [   14.538842] pciback 0000:00:19.0: xen_pcibk_control_isr: return !enable && 
> !dev_data->isr_on enable:0 dev_data->isr_on:0
> [   14.571345] xen_pciback: backend is vpci
> 
> guest start:
> [  117.061148] pciback 0000:02:00.0: resetting (FLR, D3,  bus) the device
> [  117.092324] pciback 0000:02:00.0: OK
> [  117.123935] pciback 0000:02:00.0: xen_pcibk_reset_device: me heres
> [  117.153042] pciback 0000:02:00.0: xen_pcibk_control_isr: return !enable && 
> !dev_data->isr_on enable:0 dev_data->isr_on:0
> [  118.336807] pciback 0000:00:19.0: resetting (FLR, D3,  ) the device
> [  118.371580] pci 0000:00:00.0: not owned by us!
> [  118.475884] pciback 0000:00:19.0: xen_pcibk_reset_device: me heres
> [  118.497516] pciback 0000:00:19.0: xen_pcibk_control_isr: return !enable && 
> !dev_data->isr_on enable:0 dev_data->isr_on:0
> [  118.581239] xen-pciback pci-1-0: allocated pdev @ 0xffff88000007e9c0
> [  118.582466] xen-pciback pci-1-0: getting be setup
> [  118.582683] xen-pciback pci-1-0: exporting dom 0 bus 2 slot 0 func 0
> [  118.582688] xen_pciback: vpci: 0000:02:00.0: assign to virtual slot 0
> [  118.613355] pciback 0000:02:00.0: registering for 1
> [  118.635176] xen-pciback pci-1-0: exporting dom 0 bus 0 slot 19 func 0
> [  118.635180] xen_pciback: vpci: 0000:00:19.0: assign to virtual slot 1
> [  118.657663] pciback 0000:00:19.0: registering for 1
> [  118.682808] xen-pciback pci-1-0: Publishing pci roots
> [  118.682907] xen-pciback pci-1-0: writing root 0 at 0000:00
> [  118.686896] xen-pciback pci-1-0: fe state changed 1
> 
> 
> 
> So this one is up to Konrad i guess ...
> 
> I haven't checked the call chain of xen_pcibk_do_op .. but that could be a
> side effect of libxl not imitating pci-front good enough (since HVM guests
> don't use the pci-front driver, but instead rely on libxl and Qemu to play
> those parts.

Yup. It (the call to setup the ISR) is driven by the PV pci protocol
which of course HVM guests don't use.

But setting this ISR up needn't be tied in with the PV PCI protocol.

Thank you for narrowing this down!

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel