Xen project Mailing List

Re: [Xen-devel] Xen-unstable-staging: Xen BUG at iommu_map.c:455

To: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>

From: Sander Eikelenboom <linux@xxxxxxxxxxxxxx>

Date: Fri, 10 Apr 2015 21:27:32 +0200

Cc: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Paul Durrant <paul.durrant@xxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>

Delivery-date: Fri, 10 Apr 2015 19:27:59 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

Friday, April 10, 2015, 8:55:27 PM, you wrote: > On 10/04/15 11:24, Sander Eikelenboom wrote: >> Hi Andrew, >> >> Finally got some time to figure this out .. and i have narrowed it down to: >> git://xenbits.xen.org/staging/qemu-upstream-unstable.git >> commit 7665d6ba98e20fb05c420de947c1750fd47e5c07 "Xen: Use the ioreq-server >> API when available" >> A straight revert of this commit prevents the issue from happening. >> >> The reason i had a hard time figuring this out was: >> - I wasn't aware of this earlier, since git pulling the main xen tree, >> doesn't >> auto update the qemu-* trees. > This has caught me out so many times. It is very non-obvious behaviour. >> - So i happen to get this when i cloned a fresh tree to try to figure out >> the >> other issue i was seeing. >> - After that checking out previous versions of the main xen tree didn't >> resolve >> this new issue, because the qemu tree doesn't get auto updated and is set >> "master". >> - Cloning a xen-stable-4.5.0 made it go away .. because that has a specific >> git://xenbits.xen.org/staging/qemu-upstream-unstable.git tag which is not >> master. >> >> *sigh* >> >> This is tested with xen main tree at last commit >> 3a28f760508fb35c430edac17a9efde5aff6d1d5 >> (normal xen-unstable, not the staging branch) >> >> Ok so i have added some extra debug info (see attached diff) and this is the >> output when it crashes due to something the commit above triggered, the >> level is out of bounds and the pfn looks fishy too. >> Complete serial log from both bad and good (specific commit reverted) are >> attached. > Just to confirm, you are positively identifying a qemu changeset as > causing this crash? > If so, the qemu change has discovered a pre-existing issue in the > toolstack pci-passthrough interface. Whatever qemu is or isn't doing, > it should not be able to cause a crash like this. yeah when i revert this changeset it doesn't get triggered anymore, it could still be a bug in both the iommu code and qemu, but at least the iommu code shouldn't make the host crash regardless of any additional bugs in qemu. > With this in mind, I need to brush up on my AMD-Vi details. > In the meantime, can you run with the following patch to identify what > is going on, domctl wise? I assume it is the assign_device which is > failing, but it will be nice to observe the differences between the > working and failing case, which might offer a hint. Hmm you mean on the deassigning from dom0 or the assigning to the guest part ? From my serial-logs we have the bad: (XEN) [2015-04-10 09:57:59.407] io.c:429: d1: bind: m_gsi=37 g_gsi=36 dev=00.00.5 intx=0 (XEN) [2015-04-10 09:57:59.433] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:57:59.449] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:57:59.464] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:57:59.480] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:57:59.495] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:57:59.511] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:57:59.527] d1: hd->arch.paging_mode:2 <VERY BIG SNIP> (XEN) [2015-04-10 09:58:15.236] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:58:15.251] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:58:15.267] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:58:15.282] AMD-Vi: ?!?!? update_paging_mode level after:8 (XEN) [2015-04-10 09:58:15.303] AMD-Vi: ?!?!? amd_iommu_map_page level after update paging mode:8 (XEN) [2015-04-10 09:58:15.329] AMD-Vi: ?!?!? iommu_pde_from_gfn: domid:1 table:1 level:8 pfn:0xffffffffffffffff (XEN) [2015-04-10 09:58:15.359] Xen BUG at iommu_map.c:459 compared to the good: (XEN) [2015-04-10 09:35:40.643] io.c:429: d1: bind: m_gsi=37 g_gsi=36 dev=00.00.5 intx=0 (XEN) [2015-04-10 09:35:40.669] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.685] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.700] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.716] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.731] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.747] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.763] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.778] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.794] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.809] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:40.825] d1: hd->arch.paging_mode:2 <VERY BIG SNIP> (XEN) [2015-04-10 09:35:56.362] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.378] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.393] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.409] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.425] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.440] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.456] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.471] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.487] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.503] d1: hd->arch.paging_mode:2 (XEN) [2015-04-10 09:35:56.518] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.534] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.549] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.565] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.581] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.596] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.612] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.627] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.643] d1: hd->arch.paging_mode:3 (XEN) [2015-04-10 09:35:56.660] AMD-Vi: Disable: device id = 0x800, domain = 0, paging mode = 3 (XEN) [2015-04-10 09:35:56.685] AMD-Vi: Setup I/O page table: device id = 0x800, type = 0x1, root table = 0x11d55f000, domain = 1, paging mode = 3 (XEN) [2015-04-10 09:35:56.724] AMD-Vi: Re-assign 0000:08:00.0 from dom0 to dom1 I would say there is a good chance the bad is crashing at the point where the "good" one goes from paging_mode 2 to 3 .. Which in turn was a debug printk added in (from which i still wonder why that functions is called that many times ...): diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 4b83583..f247c6f 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -1357,6 +1357,8 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn) { if ( !iommu_use_hap_pt(d) ) { + printk(XENLOG_WARNING "d%d: hd->arch.paging_mode:%d\n", d->domain_id, hd->arch.paging_mode); + rc = arch_iommu_populate_page_table(d); if ( rc ) { I will give your debug patch a spin and report back tomorrow ! Is there an easy way to find out *what* changes the paging_mode to a non-valid 8 ? (dumping a stacktrace would probably end up at dom_ctl again) .. would be nice to know who was the "remote" caller of that. -- Sander > diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c > index 9f3413c..57eb311 100644 > --- a/xen/drivers/passthrough/pci.c > +++ b/xen/drivers/passthrough/pci.c > @@ -1532,6 +1532,11 @@ int iommu_do_pci_domctl( > max_sdevs = domctl->u.get_device_group.max_sdevs; > sdevs = domctl->u.get_device_group.sdev_array; > > + printk("*** %pv->d%d: get_device_group({%04x:%02x:%02x.%u, %u})\n", > + current, d->domain_id, > + seg, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), > + max_sdevs); > + > ret = iommu_get_device_group(d, seg, bus, devfn, sdevs, max_sdevs); > if ( ret < 0 ) > { > @@ -1558,6 +1563,10 @@ int iommu_do_pci_domctl( > bus = (domctl->u.assign_device.machine_sbdf >> 8) & 0xff; > devfn = domctl->u.assign_device.machine_sbdf & 0xff; > > + printk("*** %pv->d%d: test_assign_device({%04x:%02x:%02x.%u})\n", > + current, d->domain_id, > + seg, bus, PCI_SLOT(devfn), PCI_FUNC(devfn)); > + > if ( device_assigned(seg, bus, devfn) ) > { > printk(XENLOG_G_INFO > @@ -1582,6 +1591,10 @@ int iommu_do_pci_domctl( > bus = (domctl->u.assign_device.machine_sbdf >> 8) & 0xff; > devfn = domctl->u.assign_device.machine_sbdf & 0xff; > > + printk("*** %pv->d%d: assign_device({%04x:%02x:%02x.%u})\n", > + current, d->domain_id, > + seg, bus, PCI_SLOT(devfn), PCI_FUNC(devfn)); > + > ret = device_assigned(seg, bus, devfn) ?: > assign_device(d, seg, bus, devfn); > if ( ret == -ERESTART ) > @@ -1604,6 +1617,10 @@ int iommu_do_pci_domctl( > bus = (domctl->u.assign_device.machine_sbdf >> 8) & 0xff; > devfn = domctl->u.assign_device.machine_sbdf & 0xff; > > + printk("*** %pv->d%d: deassign_device({%04x:%02x:%02x.%u})\n", > + current, d->domain_id, > + seg, bus, PCI_SLOT(devfn), PCI_FUNC(devfn)); > + > spin_lock(&pcidevs_lock); > ret = deassign_device(d, seg, bus, devfn); > spin_unlock(&pcidevs_lock); _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.