Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Fri, Feb 16, 2018 at 07:02:39PM +0000, Andrew Cooper wrote:
> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
> > On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
> >> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> >>> Hi,
> >>>
> >>> As in the subject, the guest crashes on boot, before kernel output
> >>> anything. I've isolated this to the conditions below:
> >>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this 
> >>> case),
> >>>    without PCI device it works
> >>>  - Xen (in KVM) is started through OVMF; with seabios it works
> >>>  - nested HVM is disabled in KVM
> >>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
> >>>    boot (looks like qemu bug, unrelated to this one)
> >>>
> >>> Version info:
> >>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, 
> >>> ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
> >>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
> >>>  - Xen domU: Linux 4.14.13, direct boot
> >>>
> >>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> >>> /noexitboot and then dom0 kernel crashed saying something about conflict
> >>> between e820 and kernel mapping. But now those options are disabled.
> >>>
> >>> The crash message:
> >>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> >>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 
> >>> entry.o#create_bounce_frame+0x137/0x146
> >>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> >>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> >>> (XEN) CPU:    1
> >>> (XEN) RIP:    e033:[<ffffffff826d9156>]
> >> This is #UD, which is most probably hitting a BUG().  addr2line this ^
> >> to find some code to look at.
> > addr2line failed me
> 
> By default, vmlinux is stripped and compressed.  Ideally you want to
> addr2line the vmlinux artefact in the root of your kernel build, which
> is the plain elf with debugging symbols.

Yes, I've used it on vmlinux. Still got "??:?".

> Alternatively, use scripts/extract-vmlinux on the binary you actually
> booted, which might get you somewhere.

Interestingly, that fails too ("Cannot find vmlinux.").
But I don't care right now.

> > , but System.map says its xen_memory_setup. And it
> > looks like the BUG() is the same as I had in dom0 before:
> > "Xen hypervisor allocated kernel memory conflicts with E820 map".
> 
> Juergen: Is there anything we can do to try and insert some dummy
> exception handlers right at PV start, so we could at least print out a
> oneliner to the host console which is a little more helpful than Xen
> saying "something unknown went wrong" ?

Just before the BUG(), there is a call to xen_raw_console_write(). But
apparently it was too early...

> > Disabling e820_host in guest config solved the problem. Thanks!
> >
> > Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
> > should be avoided?
> 
> I don't really know.  e820_host is a gross hack which shouldn't really
> be present.  The actually problem is that Linux can't cope with the
> memory layout it was given (and I can't recall if there is anything
> Linux could potentially to do cope).  OTOH, the toolstack, which knew
> about e820_host and chose to lay the guest out in an overlapping way is
> probably also at fault.

Yes, probably. But note that the same happened to dom0, when /mapbs is
used. Toolstack wasn't involved there. But /mapbs is also a hack.

> IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions
> at all.
> 
> ~Andrew

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Attachment: signature.asc
Description: PGP signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel