[Xen-users] Create domU with pciback fails, then my system re-boots! xen-hypervisor-4.3-amd64

[Jo Mills <jo@xxxxxxxxxxxxxxxx>]

Hi,

I am having difficulty in getting my domU with pciback to start and 
would be very grateful for any help or advice on where I am going 
wrong.  My system is a GenuineIntel Intel(R) Core(TM)2 Quad CPU, 
motherboard in Intel S32000SHV, 8GB ram.  BIOS is the latest released 
version. 

dom0 is Debian Jessie:

    Linux version 3.13-1-amd64 (debian-kernel@xxxxxxxxxxxxxxxx) (gcc
      version 4.8.2 (Debian 4.8.2-16) ) #1 SMP Debian 3.13.10-1
      (2014-04-15)

    xen-hypervisor-4.3-amd64            4.3.0-3+b1
    xen-system-amd64                    4.3.0-3+b1
    xen-tools                           4.4-1
    xen-utils-4.3                       4.3.0-3+b1
    xen-utils-common                    4.3.0-3
    xenstore-utils                      4.3.0-3+b1


There are four ethernet devices fitted, and after the various udev 
renaming of Ethernet interfaces these come out as being:

    eth0 via-rhine 0000:04:00.0 assigned for zone LOC xenbr0
    
    eth1 via-rhine 0000:04:01.0 assigned for zone DMZ (pci-passthrough)
    
    eth2 e1000 0000:04:02.0 used for DBRB
    
    eth3 e1000e 0000:01:00.0 planned for Windows client domU
    

  Aside: In "xl dmesg" I get:

       (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
       (XEN) Intel VT-d Snoop Control not enabled.
       (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
       (XEN) Intel VT-d Queued Invalidation not enabled.
       (XEN) Intel VT-d Interrupt Remapping not enabled.
       (XEN) Intel VT-d Shared EPT tables not enabled.
       (XEN) I/O virtualisation enabled
       
  so I am hopeful of at some stage, getting a windows client to run.



On power up, dom0 starts OK, but I have never managed to get my file

    /etc/modprobe.d/xen-pciback.conf 
    
to work.  It contains the following:  

  install via-rhine /sbin/modprobe pciback ; /sbin/modprobe --first-time 
--ignore-install via_rhine
  options xen-pciback hide=(0000:04:01.0) 
  install e1000 /sbin/modprobe pciback ; /sbin/modprobe --first-time 
--ignore-install e1000
  options xen-pciback hide=(0000:01:00.0) 


However, if I issue the commands:
    
  ~#  xl pci-assignable-add 04:01.0
  ~#  xl pci-assignable-add 01:00.0
  ~#  xl pci-assignable-list

I get the following:

    0000:01:00.0
    0000:04:01.0

In syslog I can also see:

   pciback 0000:04:01.0: seizing device
   xen: registering gsi 17 triggering 0 polarity 1
   Already setup the GSI :17
   e1000e 0000:01:00.0 eth3: removed PHC
   pciback 0000:01:00.0: seizing device
   xen: registering gsi 16 triggering 0 polarity 1
   Already setup the GSI :16

so this would seem to be OK.  


 ~# ifconfig
 
then shows:

eth0      Link encap:Ethernet  HWaddr 00:24:01:a3:b9:ef  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2813 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1323 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:256913 (250.8 KiB)  TX bytes:331402 (323.6 KiB)

eth2      Link encap:Ethernet  HWaddr 00:15:17:ca:37:d5  
          inet addr:192.168.10.202  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:feca:37d5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:103247 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24142 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:128195865 (122.2 MiB)  TX bytes:2626900 (2.5 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:88 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6468 (6.3 KiB)  TX bytes:6468 (6.3 KiB)

xenbr0    Link encap:Ethernet  HWaddr 00:24:01:a3:b9:ef  
          inet addr:192.168.1.202  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::224:1ff:fea3:b9ef/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2813 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1323 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:217381 (212.2 KiB)  TX bytes:331402 (323.6 KiB)


DRBD and xenbr0 are working so it all seems OK so far.  My config file 
for the domU is:

~# cat /etc/xen/vm-server-21.cfg
  #
  # Configuration file for the Xen instance vm-server-21, created
  # by xen-tools 3.9 on Thu Apr  8 12:37:42 2010.
  #
  # Modified according to xl.cfg on 20-APR-2104
  #
  
  #
  #  Hostname
  #
  name        = 'vm-server-21'
  
  #
  # Builder
  #
  ##builder    = 'generic'
  
  #
  # cpus
  #
  cpus        = '1-3'
  
  #
  # Memory in MB
  #
  memory      = '1024'
  
  #
  # Bootloader  (Not the previous path via /etc/xen/default/...) 
  #
  ### bootloader  = 'pygrub'
  
  #
  # Kernel 
(https://www.mail-archive.com/debian-bugs-dist@xxxxxxxxxxxxxxxx/msg1210293.html)
  #
  kernel = '/boot/vmlinuz-2.6.26-2-xen-amd64'
  ramdisk = '/boot/initrd.img-2.6.26-2-xen-amd64'
  
  #
  #  Disk device(s).
  #
  root        = '/dev/xvdb ro'
  
  #
  # Pygrub needs the system partition to appear first
  #
  disk        = [
                    'drbd:vm-21-disk,xvdb,w',
                    'phy:/dev/green/vm-server-21-swap,xvda,w',
                ]
  
  
  
  #
  #  Networking
  #
  ## vif         = [ 'ip=192.168.2.221,mac=00:16:3e:dd:02:00' ]
  #
  # Add support for looped through pci NIC
  # (Same device number on both blue and green
  #
  pci = [ '04:01.0' ]
  
  #
  #  Behaviour
  #
  on_poweroff = 'destroy'
  on_reboot   = 'restart'
  on_crash    = 'restart'
  
  extra = 'xencons=tty swiotlb=force'



When I issue the command

~# xl -vvv create -c /etc/xen/vm-server-21.cfg

all hell breaks lose and after a bit the server re-boots.

The output from the xl command is:

Parsing config from /etc/xen/vm-server-21.cfg
libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x9d71f0: create: 
how=(nil) callback=(nil) poller=0x9d6ba0
libxl: verbose: libxl_create.c:130:libxl__domain_build_info_setdefault: 
qemu-xen is unavailable, use qemu-xen-traditional instead: No such file or 
directory
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvdb 
spec.backend=unknown
libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvdb, uses 
script=... assuming phy backend
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk 
vdev=xvdb, using backend phy
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda 
spec.backend=unknown
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk 
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader 
configured, using user supplied kernel
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x9d7608: 
deregister unregistered
libxl: debug: libxl_x86.c:82:e820_sanitize: Memory: 1048576kB End of RAM: 
0x40000 (PFN) Delta: 0kB, PCI start: 3665832kB (0xdfbea PFN), Balloon 0kB

libxl: debug: libxl_x86.c:201:e820_sanitize: :  [0 -> 40000] RAM
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [40000 -> dfbea] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfbea -> dfc96] ACPI NVS
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfc96 -> dfcfa] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfcfa -> dfd5f] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfd5f -> dfd69] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfd69 -> dfddf] ACPI NVS
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfddf -> dfde5] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfde5 -> dfdff] ACPI
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfdff -> dfe00] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfe00 -> dff00] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [f0000 -> f4000] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [fee00 -> fee01] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [fff80 -> fff8c] Reserved
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvdb ro xencons=tty 
swiotlb=force", features="(null)"
libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 0 path 
/boot/vmlinuz-2.6.26-2-xen-amd64

domainbuilder: detail: xc_dom_kernel_file: 
filename="/boot/vmlinuz-2.6.26-2-xen-amd64"
domainbuilder: detail: xc_dom_malloc_filemap    : 1666 kB
domainbuilder: detail: xc_dom_malloc            : 7801 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x1a0b72 -> 0x79e530
domainbuilder: detail: xc_dom_ramdisk_file: 
filename="/boot/initrd.img-2.6.26-2-xen-amd64"
domainbuilder: detail: xc_dom_malloc_filemap    : 7926 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps xen-3.0-x86_64 
xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... 
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ... 
domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ... 
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x200000 memsz=0x2fe000
xc: detail: elf_parse_binary: phdr: paddr=0x4fe000 memsz=0x529a8
xc: detail: elf_parse_binary: phdr: paddr=0x551000 memsz=0x888
xc: detail: elf_parse_binary: phdr: paddr=0x552000 memsz=0xdf918
xc: detail: elf_parse_binary: memory: 0x200000 -> 0x631918
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff80200000
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80208000
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: FEATURES = 
"writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff80200000
xc: detail:     virt_kend        = 0xffffffff80631918
xc: detail:     virt_entry       = 0xffffffff80200000
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 
0xffffffff80200000 -> 0xffffffff80631918
domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k 
each
domainbuilder: detail: xc_dom_mem_init: 0x40000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 2048 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 
0xffffffff80200000 -> 0xffffffff80632000  (pfn 0x200 + 0x432 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x200+0x432 at 0x7fea4385c000
xc: detail: elf_load_binary: phdr 0 at 0x7fea4385c000 -> 0x7fea43b5a000
xc: detail: elf_load_binary: phdr 1 at 0x7fea43b5a000 -> 0x7fea43bac9a8
xc: detail: elf_load_binary: phdr 2 at 0x7fea43bad000 -> 0x7fea43bad888
xc: detail: elf_load_binary: phdr 3 at 0x7fea43bae000 -> 0x7fea43bec6b0
domainbuilder: detail: xc_dom_alloc_segment:   ramdisk      : 
0xffffffff80632000 -> 0xffffffff81d14000  (pfn 0x632 + 0x16e2 pages)
domainbuilder: detail: xc_dom_malloc            : 137 kB
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x632+0x16e2 at 0x7fea4217a000
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x7bd8b0 -> 0x16e1610
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    : 
0xffffffff81d14000 -> 0xffffffff81f14000  (pfn 0x1d14 + 0x200 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x1d14+0x200 at 0x7fea41f7a000
domainbuilder: detail: xc_dom_alloc_page   :   start info   : 
0xffffffff81f14000 (pfn 0x1f14)
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     : 
0xffffffff81f15000 (pfn 0x1f15)
domainbuilder: detail: xc_dom_alloc_page   :   console      : 
0xffffffff81f16000 (pfn 0x1f16)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 
0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 
0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 
0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 
0xffffffff80000000 -> 0xffffffff81ffffff, 16 table(s)
domainbuilder: detail: xc_dom_alloc_segment:   page tables  : 
0xffffffff81f17000 -> 0xffffffff81f2a000  (pfn 0x1f17 + 0x13 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x1f17+0x13 at 0x7fea46b00000
domainbuilder: detail: xc_dom_alloc_page   :   boot stack   : 
0xffffffff81f2a000 (pfn 0x1f2a)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0xffffffff81f2b000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0xffffffff82000000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x40000
domainbuilder: detail: clear_page: pfn 0x1f16, mfn 0x1ddb0e
domainbuilder: detail: clear_page: pfn 0x1f15, mfn 0x1ddb0f
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1f14+0x1 
at 0x7fea46cdf000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff80208000 pfn=0x208
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 10027 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 9593 kB
domainbuilder: detail:       domU mmap          : 29856 kB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xdfc99
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x1f17 mfn 0x1ddb0d
domainbuilder: detail: launch_vm: called, ctxt=0x7fffdf91bd50
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvdb 
spec.backend=phy
libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvdb, uses 
script=... assuming phy backend
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x9d8548 
wpath=/local/domain/0/backend/vbd/2/51728/state token=3/0: register slotnum=3
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda 
spec.backend=phy
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x9dfb98 
wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: register slotnum=2
libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x9d71f0: inprogress: 
poller=0x9d6ba0, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x9d8548 
wpath=/local/domain/0/backend/vbd/2/51728/state token=3/0: event 
epath=/local/domain/0/backend/vbd/2/51728/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/2/51728/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x9d8548 
wpath=/local/domain/0/backend/vbd/2/51728/state token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x9d8548: 
deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: 
/etc/xen/scripts/block-drbd add
libxl: debug: libxl_event.c:472:watchfd_callback: watch 
epath=/local/domain/0/backend/vbd/2/51728/state token=3/0: empty slot
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x9dfb98 
wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: event 
epath=/local/domain/0/backend/vbd/2/51712/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/2/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x9dfb98 
wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: deregister slotnum=2
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x9dfb98: 
deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: 
/etc/xen/scripts/block add
libxl: debug: libxl_event.c:472:watchfd_callback: watch 
epath=/local/domain/0/backend/vbd/2/51712/state token=2/1: empty slot

    (It just hangs at this point)
    
    
On the server's console I get:

    e1000 0000:04:02.0 eth2: Detected Tx Unit Hang
      Tx Queue             <0>
      TDH                  <0>
      TDT                  <2>
      next_to_use          <2>
      next_to_clean        <0>
    buffer_info[next_to_clean]
      time_stamp           <100002ef6>
      next_to_watch        <0>
      jiffies              <1000034d3>
      next_to_watch.status <0>

looping over and over again, and then as mentioned above, the server 
hangs.  I have modified /etc/xen/scripts/block-drbd as per a xen bug 
report (I cannot recall at the moment which one), but the change was:


 case $t in 
      drbd)
      
      
to become

 case $t in 
      drbd|phy)



Any help will be greatly appreciated.

Regards,

Jo.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users