[Xen-users] Advice on pcie cards acting up

[Håkon Alstadheim <hakon@xxxxxxxxxxxxxxxxxx>]

I have two "seats" domUs with vga and usb -cards passed through. 
Secondary VGA works great. Usb has always been acting up on this 
machine, I have gone through several cards and thrown out countless hubs 
and usb-devices. I have run this rig with Xen since Xen-4.6, same 
symptoms. I have tried disabling any "legacy usb" stuff in the BIOS. I 
am booting with EFI, but both VGA and USB cards are set to be 
initialized in BIOS mode (or whatever it is called). It seems I need 
this "mixed mode" boot to avoid my secondary graphics cards being 
touched before they are handed off to the VMs.
At the moment I have a mostly stable set-up with a windows domU that 
gets the USB-3.0 from the main board passed through, and a linux domU 
that gets an add-in usb-card. The windows VM often needs its usb-devices 
unplugged and replugged before they work. Sometimes USB dies on either 
or both VMS, requiring reboot of the entire machine. At the moment I 
have verbosity in the logs turned quite far down. I would like advice on 
what kind of logging to turn on, and possible tweaks to my config to get 
this set-up stable. First section below is from xl dmesg, while starting 
the two vms (and two others that work fine). The two usb-"cards" are 
02:00.0 and 06:00.0 . Could it be that usb in the two VMs is trampling 
on each other? How would I fix that if that is the problem? Relevant 
lines are near "end xl dmesg":
---------xl dmesg:----------

(XEN) parameter "xen.cfg" unknown!
(XEN) parameter "xen-marker-122" unknown!
(XEN) [00000582d537ed00] parameter "tmem" unknown!
 Xen 4.12.0-rc
(XEN) [00000582d5c396a4] Xen version 4.12.0-rc 
(hakon@xxxxxxxxxxxxxxxxxx) (gcc (Gentoo 8.2.0-r6 p1.7) 8.2.0) debug=n  
Sat Mar 30 00:20:11 CET 2019
(XEN) [00000582d98b29d8] Latest ChangeSet:
(XEN) [00000582dab217f4] Bootloader: EFI
(XEN) [00000582dbc4ae28] Command line: xen.cfg xen-marker-122 
console_timestamps=date iommu=1,intpost,verbose,debug 
iommu_inclusive_mapping=1 com1=57600,8n1 com2=57600,8n1 console=vga,com2 
dom0_max_vcpus=8 dom0_mem=5G,max:5G cpufreq=xen:performance,verbose 
smt=0 maxcpus=12 core_parking=performance nmi=dom0 gnttab_max_frames=256 
gnttab_max_maptrack_frames=1024 vcpu_migration_delay=2000 
tickle_one_idle_cpu=1 spec-ctrl=no-xen sched=credit2 max_cstate=1 
clocksource=tsc tsc=stable:socket loglvl=none/warning 
guest_loglvl=none/warning  tmem=1
(XEN) [00000582e9a56c64] Xen image load base address: 0x6fa00000
(XEN) [00000582eb54ebd4] Video information:
(XEN) [00000582ec7b2838]  VGA is graphics mode 1024x768, 32 bpp
(XEN) [00000582ee24b390] Disc information:
(XEN) [00000582ef451128]  Found 0 MBR signatures
(XEN) [ 2453.323263] printk: 102 messages suppressed.
(XEN) [ 2453.333662] Detected 2471.975 MHz processor.
(XEN) [ 2453.352019] Initing memory sharing.
(XEN) [ 2453.361965] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [ 2453.378711] Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [ 2453.395454] Intel VT-d Snoop Control enabled.
(XEN) [ 2453.406046] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [ 2453.418837] Intel VT-d Queued Invalidation enabled.
(XEN) [ 2453.430642] Intel VT-d Interrupt Remapping enabled.
(XEN) [ 2453.442446] Intel VT-d Posted Interrupt not enabled.
(XEN) [ 2453.454387] Intel VT-d Shared EPT tables enabled.
(XEN) [2019-03-30 07:57:40] printk: 31 messages suppressed.
(XEN) [2019-03-30 07:57:40] CMCI: threshold 0x2 too large for CPU6 bank 
17, using 0x1
(XEN) [2019-03-30 07:57:40] CMCI: threshold 0x2 too large for CPU6 bank 
18, using 0x1
(XEN) [2019-03-30 07:57:40] CMCI: threshold 0x2 too large for CPU6 bank 
19, using 0x1
(XEN) [2019-03-30 08:38:34] mwait-idle: max C-state 1 reached
(XEN) [2019-03-30 08:38:34] mwait-idle: max C-state 1 reached
(XEN) [2019-03-30 08:38:34] mwait-idle: max C-state 1 reached
(XEN) [2019-03-30 08:38:34] mwait-idle: max C-state 1 reached
(XEN) [2019-03-30 08:38:34] mwait-idle: max C-state 1 reached
(XEN) [2019-03-30 08:38:34] mwait-idle: max C-state 1 reached
(XEN) [2019-03-30 08:38:34] Brought up 12 CPUs
(XEN) [2019-03-30 09:19:27] Switched to Platform timer 2471.975MHz TSC
(XEN) [2019-03-30 09:19:27] Dom0 has maximum 840 PIRQs
(XEN) [2019-03-30 09:19:28]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [2019-03-30 09:19:28]  Dom0 kernel: 64-bit, PAE, lsb, paddr 
0x1000000 -> 0x342c000
(XEN) [2019-03-30 09:19:28] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [2019-03-30 09:19:28]  Dom0 alloc.: 
0000001050000000->0000001054000000 (1293641 pages to be allocated)
(XEN) [2019-03-30 09:19:28]  Init. ramdisk: 
000000107fd39000->000000107ffef560
(XEN) [2019-03-30 09:19:28] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [2019-03-30 09:19:28]  Loaded kernel: 
ffffffff81000000->ffffffff8342c000
(XEN) [2019-03-30 09:19:28]  Init. ramdisk: 
0000000000000000->0000000000000000
(XEN) [2019-03-30 09:19:37] printk: 176 messages suppressed.
(XEN) [2019-03-30 09:19:37] Set CPU acpi_id(0) cpuid(0) Px State info:
(XEN) [2019-03-30 09:19:37]     _PCT: descriptor=130, length=12, 
space_id=127, bit_width=0, bit_offset=0, reserved=0, address=0
(XEN) [2019-03-30 08:40:48] printk: 238 messages suppressed.
(XEN) [2019-03-30 08:40:48] [VT-D]d0:PCIe: unmap 0000:09:00.0
(XEN) [2019-03-30 08:40:48] [VT-D]d1:PCIe: map 0000:09:00.0
(XEN) [2019-03-30 08:42:59] [VT-D] It's risky to assign 0000:02:00.0 
with shared RMRR at 7db85000 for Dom2.
(XEN) [2019-03-30 08:42:59] [VT-D]d0:PCIe: unmap 0000:02:00.0
(XEN) [2019-03-30 08:42:59] [VT-D]d2:PCIe: map 0000:02:00.0
(XEN) [2019-03-30 08:43:00] [VT-D]d0:PCIe: unmap 0000:81:00.0
(XEN) [2019-03-30 08:43:00] [VT-D]d2:PCIe: map 0000:81:00.0
(XEN) [2019-03-30 08:43:01] [VT-D]d0:PCIe: unmap 0000:81:00.1
(XEN) [2019-03-30 08:43:01] [VT-D]d2:PCIe: map 0000:81:00.1
(XEN) [2019-03-30 08:43:10] [VT-D]d0:PCIe: unmap 0000:05:00.0
(XEN) [2019-03-30 08:43:10] [VT-D]d3:PCIe: map 0000:05:00.0
(XEN) [2019-03-30 08:43:18] [VT-D] It's risky to assign 0000:06:00.0 
with shared RMRR at 7db85000 for Dom4.
(XEN) [2019-03-30 08:43:18] [VT-D]d0:PCIe: unmap 0000:06:00.0
(XEN) [2019-03-30 08:43:18] [VT-D]d4:PCIe: map 0000:06:00.0
(XEN) [2019-03-30 08:43:19] [VT-D]d0:PCIe: unmap 0000:82:00.0
(XEN) [2019-03-30 08:43:20] printk: 1 messages suppressed.
(XEN) [2019-03-30 08:43:20] [VT-D]d0:PCIe: unmap 0000:82:00.1

---------end xl dmesg-----

Aside: Yes I know tmem is no longer, but I haven't updated my 
config-generated-scripts just yet. Other errors is because I like to put 
the file-name of the config in the config to see what file and what 
version of the config the machine was booted with.
Start commands:

--------------

xl create -c /etc/xen/windows.hvm 
pci=["02:00.0,rdm_policy=relaxed","81:00.0","81:00.1"]
xl create -c /etc/xen/gt.hvm 
pci=["06:00.0,rdm_policy=relaxed,permissive=1,msitranslate=1","82:00.0","82:00.1"]
-------------

-----windows.hvm:---

tsc_mode="native"
name = "windows.hvm"
type = "hvm"
nestedhvm = 1
xen_platform_pci = '1'
vcpus = 6
cpu_weight=5120
cpus_soft="node:1"
memory = 10240
mmio_hole=3072
pci_msitranslate=1
no_migrate = 1
localtime = 1
timer_mode = "no_delay_for_missed_ticks"
pci_permissive = 1
device_model_version="qemu-xen"

boot = 'cd'
disk = [ 'format=raw, vdev=xvda, access=rw, no-discard, direct-io-safe, 
target=/dev/md2p29'
        ,'format=raw, vdev=xvdc, access=rw, no-discard, direct-io-safe, 
target=/dev/md2p12'
,'file:/usr/local/share/xen/cd-images/win_8_1.img,hdb:cdrom,r'
    ]
vif = [ 'mac=02:16:3e:00:00:05,type=ioemu,model=e1000,bridge=br0']
on_poweroff = 'destroy'
on_reboot = 'destroy'
on_crash = 'destroy'
acpi = '1'
viridian=[ "defaults", "reference_tsc", "apic_assist", "crash_ctl"]
vnc = 1
spice = 0
sdl = 0
spicehost='0.0.0.0'
spiceport=6000
spicedisable_ticketing=1
spicevdagent=1
spiceagent_mouse=0
spice_clipboard_sharing=0
serial='pty'
---

---------gt.hvm:---------

tsc_mode="native"
name = "gt.hvm"
type = "hvm"
nestedhvm = 1
boot = 'n'
device_model_version="qemu-xen"
pvh = 1
xen_platform_pci = '1'
memory = 17408
mmio_hole=3072
vcpus = 6
cpus_soft="node:1"
cpu_weight=4096
no_migrate = 1
vif = ['mac=02:16:3e:00:00:04,bridge=br0']
disk = [ 'vdev=xvde, format=raw, no-discard, direct-io-safe, 
target=/dev/disk/by-label/SAS-GT'
        ,'vdev=xvdf, format=raw, no-discard, direct-io-safe, 
target=/dev/disk/by-label/SAS-GT-SWAP'
        ,'vdev=xvdg, format=raw, no-discard, direct-io-safe, 
target=/dev/disk/by-label/GT-TMP']
serial = 'pty'
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'destroy'
keymap="en-us"
sdl = 0
vnc = 0
spice = 0
-----xl info (hyper-threading is disabled in bios):----

xl info
host                   : gentoo
release                : 5.0.5-gentoo
version                : #1 SMP Wed Mar 27 17:10:04 CET 2019
machine                : x86_64
nr_cpus                : 12
max_cpu_id             : 11
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 2471.975
hw_caps                : 
bfebfbff:77fef3ff:2c100800:00000021:00000001:000037ab:00000000:00000100
virt_caps              : pv hvm hvm_directio pv_directio
total_memory           : 65379
free_memory            : 10904
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 12
xen_extra              : .0-rc
xen_version            : 4.12.0-rc
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit2
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : xen.cfg xen-marker-122 console_timestamps=date 
iommu=1,intpost,verbose,debug iommu_inclusive_mapping=1 com1=57600,8n1 
com2=57600,8n1 console=vga,com2 dom0_max_vcpus=8 dom0_mem=5G,max:5G 
cpufreq=xen:performance,verbose smt=0 maxcpus=12 
core_parking=performance nmi=dom0 gnttab_max_frames=256 
gnttab_max_maptrack_frames=1024 vcpu_migration_delay=2000 
tickle_one_idle_cpu=1 spec-ctrl=no-xen sched=credit2 max_cstate=1 
clocksource=tsc tsc=stable:socket loglvl=none/warning 
guest_loglvl=none/warning  tmem=1
cc_compiler            : gcc (Gentoo 8.2.0-r6 p1.7) 8.2.0
cc_compile_by          : hakon
cc_compile_domain      : alstadheim.priv.no
cc_compile_date        : Sat Mar 30 00:20:11 CET 2019
build_id               : 34189a9d7db499d10a3ef59a53680573
xend_config_format     : 4

----


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users