|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Problems with pci passthrough after upgrade (Ubuntu 18.04 --> Ubuntu 20.04 with Xen 4.11)
Hello,I updated my Ubuntu server from 18.04 with Xen 4.9 to 20.04 with Xen 4.11. Dom0 and all domU instances worked as expected. Next I updated first domU from 18.04 to 20.04. The update went well, but the domU crashed when booting with pci passthrough of a SR-IOV NIC based on 4-port Intel i350. The kernel in the domU is 5.4 (Linux smarthome 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux) after the update. Change back to kernel 4.15 or 5.0 the domU booted and the VF NIC is present (lspci) and the igbvf module is loaded as before the update. Also booting without pci passthrough the 5.4 kernel boots. The grub command line of the dom0:GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=2048M,max:2048M dom0_max_vcpus=4 dom0_vcpus_pin=true \ iommu=on iommu=verbose \ console=vga ucode=scan smt=true xpti=dom0=false,domu=true \ conring_size=2097152 loglvl=all guest_loglvl=all" The configuration of the domU looks like this: # # Hostname # name = 'smarthome' # # Kernel + memory size # kernel = '/usr/lib/grub-xen/grub-x86_64-xen.bin' vcpus = '2' cpus = '6-11' memory = '4096' localtime = 0 # # Disk device(s). # disk = [ '/dev/pulsar02-vg/Xsmarthome-disk,,xvdb', '/dev/pulsar02-vg/Xsmarthome-swap,,xvda', ] # # Networking # pci = [ "0000:02:12.1,permissive=1" # SRV (VF) ] # # Behaviour # on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'destroy' The grub command line of the domU looks like this: GRUB_CMDLINE_LINUX_DEFAULT="iommu=soft console=hvc0 earlyprintk=xen"The kernel message when starting with xl create -c /etc/xen/Xsmarthome.cfg is: (early) [ 1.652005] PM: Registered nosave memory: [mem 0xfed92000-0xfedfffff] (early) [ 1.652008] PM: Registered nosave memory: [mem 0xfee00000-0xfeefffff] (early) [ 1.652012] PM: Registered nosave memory: [mem 0xfef00000-0xfeffffff] (early) [ 1.652015] PM: Registered nosave memory: [mem 0xff000000-0xffffffff] (early) [ 1.652019] [mem 0x90000000-0xdfffffff] available for PCI devices (early) [ 1.652023] Booting paravirtualized kernel on Xen (early) [ 1.652026] Xen version: 4.11.4-pre (preserve-AD)(early) [ 1.652032] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns (early) [ 1.652040] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 (early) [ 1.652133] percpu: Embedded 54 pages/cpu s184320 r8192 d28672 u1048576 (early) [ 1.652174] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) (early) [ 1.652180] Built 1 zonelists, mobility grouping on. Total pages: 1032073 (early) [ 1.652184] Policy zone: Normal(early) [ 1.652188] Kernel command line: root=UUID=024e1144-d306-4e5f-b67e-01db80827787 ro mitigations=off iommu=soft console=hvc0 earlyprintk=xen (early) [ 1.652374] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear) (early) [ 1.652439] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) (early) [ 1.652660] mem auto-init: stack:off, heap alloc:on, heap free:off (early) [ 1.698831] software IO TLB: mapped [mem 0x178200000-0x17c200000] (64MB) (early) [ 1.706256] Memory: 3960888K/4193916K available (14339K kernel code, 2398K rwdata, 4956K rodata, 2716K init, 4988K bss, 233028K reserved, 0K cma-reserved) (early) [ 1.706269] random: get_random_u64 called from kmem_cache_open+0x2d/0x410 with crng_init=0 (early) [ 1.706471] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 (early) [ 1.706906] ftrace: allocating 44527 entries in 174 pages (early) [ 1.718159] rcu: Hierarchical RCU implementation.(early) [ 1.718164] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=2. (early) [ 1.718168] Tasks RCU enabled.(early) [ 1.718171] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. (early) [ 1.718174] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 (early) [ 1.721252] Using NULL legacy PIC(early) [ 1.721257] NR_IRQS: 524544, nr_irqs: 48, preallocated irqs: 0 (early) [ 1.721304] xen:events: Using FIFO-based ABI (early) [ 1.721478] random: crng done (trusting CPU's manufacturer) (early) [ 1.721508] Console: colour dummy device 80x25 (early) [ 1.721597] printk: console [tty0] enabled [ 1.721604] printk: console [hvc0] enabled (early) [ 1.721604] printk: console [hvc0] enabled [ 1.721610] printk: bootconsole [xenboot0] disabled (early) [ 1.721610] printk: bootconsole [xenboot0] disabled[ 1.721636] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 1.721650] installing Xen timer for CPU 0[ 1.721677] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2fc04a0142b, max_idle_ns: 440795346615 ns [ 1.721687] Calibrating delay loop (skipped), value calculated using timer frequency.. 6625.46 BogoMIPS (lpj=13250936) [ 1.721694] pid_max: default: 32768 minimum: 301 [ 1.721752] LSM: Security Framework initializing [ 1.721772] Yama: becoming mindful. [ 1.721818] AppArmor: AppArmor initialized[ 1.721900] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 1.721910] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 1.721943] *** VALIDATE tmpfs *** [ 1.722115] *** VALIDATE proc *** [ 1.722214] *** VALIDATE cgroup1 *** [ 1.722218] *** VALIDATE cgroup2 ***(early) Poking(early) KASLR using(early) RDRAND(early) RDTSC(early) ... [ 1.722386] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8 [ 1.722391] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4 [ 1.722398] Speculative Store Bypass: Vulnerable [ 1.722402] SRBDS: Unknown: Dependent on hypervisor status [ 1.762961] cpu 0 spinlock event irq 1 [ 1.762967] VPMU disabled by hypervisor.[ 1.763121] Performance Events: unsupported p6 CPU model 158 no PMU driver, software events only. [ 1.763173] rcu: Hierarchical SRCU implementation. [ 1.763631] NMI watchdog: Perf NMI watchdog permanently disabled [ 1.763674] smp: Bringing up secondary CPUs ... [ 1.763790] installing Xen timer for CPU 1 [ 1.763811] SMP alternatives: switching to SMP code [ 1.803767] cpu 1 spinlock event irq 13 [ 1.803767] smp: Brought up 1 node, 2 CPUs [ 1.803767] smpboot: Max logical packages: 1 [ 1.803767] devtmpfs: initialized [ 1.803767] x86/mm: Memory block size: 128MB[ 1.803767] PM: Registering ACPI NVS region [mem 0x836a9000-0x836a9fff] (4096 bytes) [ 1.803767] PM: Registering ACPI NVS region [mem 0x8ccc5000-0x8cda6fff] (925696 bytes) [ 1.803767] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 1.803767] futex hash table entries: 512 (order: 3, 32768 bytes, linear) [ 1.803767] pinctrl core: initialized pinctrl subsystem [ 1.822672] PM: RTC time: 165:165:165, date: 2065-165-165 [ 1.822823] NET: Registered protocol family 16 [ 1.822837] xen:grant_table: Grant tables using version 1 layout [ 1.849698] Grant table initialized [ 1.849738] audit: initializing netlink subsys (disabled)[ 1.849751] audit: type=2000 audit(1603042631.327:1): state=initialized audit_enabled=0 res=1 [ 1.849751] EISA bus registered [ 1.850622] PCI: setting up Xen PCI frontend stub [ 1.853869] fbcon: Taking over console [ 1.853869] ACPI: Interpreter disabled. [ 1.853869] xen:balloon: Initialising balloon driver [ 1.853869] iommu: Default domain type: Translated [ 1.853869] SCSI subsystem initialized [ 1.853869] vgaarb: loaded [ 1.853869] usbcore: registered new interface driver usbfs [ 1.853869] usbcore: registered new interface driver hub [ 1.853869] usbcore: registered new device driver usb [ 1.853869] pps_core: LinuxPPS API ver. 1 registered[ 1.853869] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@xxxxxxxx> [ 1.853869] PTP clock support registered [ 1.853869] EDAC MC: Ver: 3.0.0 [ 1.853869] PCI: System does not support PCI [ 1.853869] NetLabel: Initializing [ 1.853869] NetLabel: domain hash size = 128 [ 1.853869] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.853869] NetLabel: unlabeled traffic allowed by default [ 1.858197] clocksource: Switched to clocksource xen [ 1.864969] *** VALIDATE bpf *** [ 1.865005] VFS: Disk quotas dquot_6.6.0[ 1.865017] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.865033] *** VALIDATE ramfs ***[ 1.865036] hugetlbfs: disabling because there are no supported hugepage sizes [ 1.865085] AppArmor: AppArmor Filesystem Enabled [ 1.865099] pnp: PnP ACPI: disabled [ 1.866763] thermal_sys: Registered thermal governor 'fair_share' [ 1.866763] thermal_sys: Registered thermal governor 'bang_bang' [ 1.866785] thermal_sys: Registered thermal governor 'step_wise' [ 1.866790] thermal_sys: Registered thermal governor 'user_space'[ 1.866794] thermal_sys: Registered thermal governor 'power_allocator' [ 1.866838] NET: Registered protocol family 2[ 1.866947] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear) [ 1.866968] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear) [ 1.867010] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear) [ 1.867047] TCP: Hash tables configured (established 32768 bind 32768) [ 1.867066] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear) [ 1.867077] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear) [ 1.867105] NET: Registered protocol family 1 [ 1.867112] NET: Registered protocol family 44 [ 1.867117] PCI: CLS 0 bytes, default 64 [ 1.867138] Trying to unpack rootfs image as initramfs... [ 2.218397] Freeing initrd memory: 43228K[ 2.218495] check: Scanning for low memory corruption every 60 seconds [ 2.218936] Initialise system trusted keyrings [ 2.218950] Key type blacklist registered[ 2.219018] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.219708] zbud: loaded [ 2.219896] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 2.220102] fuse: init (API version 7.31) [ 2.220121] *** VALIDATE fuse *** [ 2.220125] *** VALIDATE fuse *** [ 2.220207] Platform Keyring initialized [ 2.222191] Key type asymmetric registered [ 2.222195] Asymmetric key parser 'x509' registered[ 2.222204] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 244) [ 2.222244] io scheduler mq-deadline registered[ 2.222316] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.222711] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 2.223862] Linux agpgart interface v0.103 [ 2.224871] loop: module loaded [ 2.224876] Invalid max_queues (4), will use default max: 2. [ 2.225814] libphy: Fixed MDIO Bus: probed [ 2.225821] tun: Universal TUN/TAP device driver, 1.6 [ 2.225857] PPP generic driver version 2.4.2 [ 2.225890] xen_netfront: Initialising Xen virtual ethernet driver [ 2.225922] VFIO - User Level meta-driver version: 0.3[ 2.225977] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.225983] ehci-pci: EHCI PCI platform driver [ 2.225991] ehci-platform: EHCI generic platform driver [ 2.226002] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.226011] ohci-pci: OHCI PCI platform driver [ 2.226018] ohci-platform: OHCI generic platform driver [ 2.226023] uhci_hcd: USB Universal Host Controller Interface driver [ 2.226060] i8042: PNP: No PS/2 controller found. [ 2.226064] i8042: Probing ports directly. [ 3.238830] i8042: No controller found[ 3.238859] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2fc04a0142b, max_idle_ns: 440795346615 ns [ 3.238984] mousedev: PS/2 mouse device common for all mice [ 3.239072] i2c /dev entries driver [ 3.239103] device-mapper: uevent: version 1.0.3[ 3.239168] device-mapper: ioctl: 4.41.0-ioctl (2019-09-16) initialised: dm-devel@xxxxxxxxxx [ 3.239189] platform eisa.0: Probing EISA bus 0 [ 3.239213] platform eisa.0: EISA: Detected 0 cards [ 3.239222] intel_pstate: CPU model not supported [ 3.239263] ledtrig-cpu: registered to indicate activity on CPUs[ 3.239301] BUG: unable to handle page fault for address: ffffc900401d3818 [ 3.239308] #PF: supervisor read access in kernel mode [ 3.239312] #PF: error_code(0x0000) - not-present page[ 3.239315] PGD 7ec72067 P4D 7ec72067 PUD 177d50067 PMD 177d51067 PTE 0 [ 3.239321] Oops: 0000 [#1] SMP NOPTI[ 3.239325] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-51-generic #56-Ubuntu [ 3.239333] RIP: e030:pmc_core_probe+0x134/0x17f[ 3.239337] Code: 82 48 c7 c7 68 70 d7 82 e8 a9 e5 80 ff 48 8b 05 82 2c 48 01 48 c7 83 88 00 00 00 40 70 d7 82 48 63 40 44 48 03 05 64 2c 48 01 <8b> 00 48 8b 15 63 2c 48 01 48 c7 c7 60 db 14 82 8b 4a 48 ba 01 00 [ 3.239349] RSP: e02b:ffffc9004000bbc8 EFLAGS: 00010286[ 3.239353] RAX: ffffc900401d3818 RBX: ffffffff827dbe80 RCX: 80000000fe001073 [ 3.239358] RDX: ffffffff82d77020 RSI: ffffffff8242e365 RDI: ffffffff82d77068 [ 3.239363] RBP: ffffc9004000bbe0 R08: 0000000000000000 R09: ffffc9004000ba80 [ 3.239368] R10: 0000000000007ff0 R11: ffff888177f96900 R12: ffffffff827dbe90 [ 3.239373] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3.239384] FS: 0000000000000000(0000) GS:ffff88817c600000(0000) knlGS:0000000000000000 [ 3.239389] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033[ 3.239394] CR2: ffffc900401d3818 CR3: 000000000260a000 CR4: 0000000000040660 [ 3.239402] Call Trace: [ 3.239408] platform_drv_probe+0x3b/0x80 [ 3.239412] really_probe+0x2b3/0x3e0 [ 3.239415] driver_probe_device+0xbc/0x100 [ 3.239419] __device_attach_driver+0x71/0xd0 [ 3.239423] ? driver_allows_async_probing+0x50/0x50 [ 3.239427] bus_for_each_drv+0x84/0xd0 [ 3.239431] __device_attach+0xed/0x170 [ 3.239434] device_initial_probe+0x13/0x20 [ 3.239437] bus_probe_device+0x8f/0xa0 [ 3.239441] device_add+0x3c7/0x6b0 [ 3.239444] platform_device_add+0xf9/0x240 [ 3.239448] platform_device_register+0x6b/0x70 [ 3.239455] ? pmc_core_driver_init+0x19/0x19 [ 3.239459] pmc_core_platform_init+0x43/0x45 [ 3.239464] do_one_initcall+0x4a/0x1fa [ 3.239469] kernel_init_freeable+0x1b2/0x255 [ 3.239474] ? rest_init+0xb0/0xb0 [ 3.239478] kernel_init+0xe/0x100 [ 3.239481] ret_from_fork+0x1f/0x40 [ 3.239485] Modules linked in: [ 3.239488] CR2: ffffc900401d3818 [ 3.239494] ---[ end trace c5dde7a582f9e4e9 ]--- [ 3.239499] RIP: e030:pmc_core_probe+0x134/0x17f[ 3.239502] Code: 82 48 c7 c7 68 70 d7 82 e8 a9 e5 80 ff 48 8b 05 82 2c 48 01 48 c7 83 88 00 00 00 40 70 d7 82 48 63 40 44 48 03 05 64 2c 48 01 <8b> 00 48 8b 15 63 2c 48 01 48 c7 c7 60 db 14 82 8b 4a 48 ba 01 00 [ 3.239513] RSP: e02b:ffffc9004000bbc8 EFLAGS: 00010286[ 3.239517] RAX: ffffc900401d3818 RBX: ffffffff827dbe80 RCX: 80000000fe001073 [ 3.239522] RDX: ffffffff82d77020 RSI: ffffffff8242e365 RDI: ffffffff82d77068 [ 3.239527] RBP: ffffc9004000bbe0 R08: 0000000000000000 R09: ffffc9004000ba80 [ 3.239532] R10: 0000000000007ff0 R11: ffff888177f96900 R12: ffffffff827dbe90 [ 3.239538] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3.239547] FS: 0000000000000000(0000) GS:ffff88817c600000(0000) knlGS:0000000000000000 [ 3.239552] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033[ 3.239556] CR2: ffffc900401d3818 CR3: 000000000260a000 CR4: 0000000000040660 [ 3.239565] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 [ 3.239574] Kernel Offset: disabledFurther testing with several command line options for domU like "pic=nobar" didn't solve the problem. I also tried without passing the VF trough by guest config file, but with attaching the pci device after booting the kernel 5.4. Doitn that wit xl pci-attach I got the following message from dmesg in domU: [ 367.486057] pcifront pci-0: Installing PCI frontend [ 367.486256] pcifront pci-0: Creating PCI Frontend Bus 0000:02 [ 367.486283] pcifront pci-0: PCI host bridge to bus 0000:02 [ 367.486284] pci_bus 0000:02: root bus resource [io 0x0000-0xffff][ 367.486285] pci_bus 0000:02: root bus resource [mem 0x00000000-0x7fffffffff] [ 367.486286] pci_bus 0000:02: root bus resource [bus 00-ff] [ 367.494571] pci 0000:02:12.1: [8086:1520] type 00 class 0x020000[ 367.495865] pci 0000:02:12.1: reg 0x10: [mem 0x949b0000-0x949b3fff 64bit pref] [ 367.496963] pci 0000:02:12.1: reg 0x1c: [mem 0x94990000-0x94993fff 64bit pref] [ 367.500708] pcifront pci-0: New device on 0000:02:12.1 found. [ 367.502079] pcifront pci-0: claiming resource 0000:02:12.1/0[ 367.502080] pci 0000:02:12.1: can't claim BAR 0 [mem 0x949b0000-0x949b3fff 64bit pref]: address conflict with System RAM [mem 0x00100000-0xffffffff] [ 367.502081] pcifront pci-0: Could not claim resource 0000:02:12.1/0! Device offline. Try using e820_host=1 in the guest config. [ 367.502093] pcifront pci-0: claiming resource 0000:02:12.1/3[ 367.502094] pci 0000:02:12.1: can't claim BAR 3 [mem 0x94990000-0x94993fff 64bit pref]: address conflict with System RAM [mem 0x00100000-0xffffffff] [ 367.502094] pcifront pci-0: Could not claim resource 0000:02:12.1/3! Device offline. Try using e820_host=1 in the guest config. [ 367.596326] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k [ 367.596327] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.[ 367.596449] igbvf 0000:02:12.1: can't enable device: BAR 0 [mem size 0x00004000 64bit pref] not assigned [ 367.596483] igbvf: probe of 0000:02:12.1 failed with error -22Using e820_host=1 in the guest config makes the kernel crash immidiately while booting. Searching the internet I found very old bug reports abour BAR and memory < 3GB, but no hints to actual Xen or kernel versions. Using less than 4 GB memory for the domU crashes the kernel even without pci passthrough. Can anybody help me? Anybod have similar problems? Any other command line option in dom0 or domU I can try? Best regards Torsten || -- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |