[Xen-users] VTd not showing PCI device in VM

["Rafael Ayala" <rafael_ayala@xxxxxxxxxxxxxx>]

I'm having trouble actually seeing a PCI device in my VMs.  I've resolved several of my issues using previous posts and using the VTdHowTo wiki page.  I have both VT and VTd BIOS options enabled.  I have pciback hiding the devices and xm can list and assign the devices to VMs.  I don't see the PCI devices in my VMs though.  I have tried a windows and a Linux vm, without success. I also made sure I removed the drivers for the devices to make sure dom0 was not using them. 

Grub command:
title Xen 3.3 (2.6.18.8)
    root (hd0,1)
    kernel /boot/xen-3.3.0.gz vtd=1 iommu=1
    module /boot/vmlinuz-2.6.18.8-xen root=/dev/disk/by-id/scsi-SATA_ST3250310NS_9SF0LZAP-part2 vga=0x317    resume=/dev/sda1 splash=silent showopts pciback.hide=(01:00.0)(01:00.1)(06:00.0)

    module /boot/initrd-2.6.18.8-xen

The PCI devices I'm trying to assign:
# lspci
01:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
01:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
06:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)

Xen says I can assign them
# xm pci-list-assignable-devices
0000:06:00.0
0000:01:00.0 0000:01:00.1

Create my VM:
# xm create w2k3.vxm pci=01:00.0 pci=01:00.1
Using config file "./w2k3.vxm".
Started domain w2k3

I can confirm Xen thinks I have assigned the devices
# xm pci-list w2k3
domain   bus   slot   func
0x0  0x01   0x00    0x0   
0x0  0x01   0x00    0x1   

# xm create rhel5.vxm.vxm pci=01:00.0 pci=01:00.1
Using config file "./rhel5.vxm".
Started domain rhel5

# xm pci-list rhel5
domain   bus   slot   func
0x0  0x06   0x00    0x0   

Again, neither windows (device manager) nor Linux (lspci) VMs can see the assigned devices.  Below is "xm dmesg" showing VTd is enabled.  Another curious fact is that I can assign the same devices to 2 VMs.  I would have expected an error, but perhaps it is allowed to share devices.  Anyone have any thoughts on what I'm missing?

(XEN) Xen version 3.3.0 (root@xxxxxxxxxxx) (gcc version 4.1.2 20070115 (SUSE Linux)) Wed Dec 10 10:59:53 EST 2008
(XEN) Latest ChangeSet: unavailable
(XEN) Command line: vtd=1 iommu=1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009cc00 (usable)
(XEN)  000000000009cc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cfef0000 (usable)
(XEN)  00000000cfef0000 - 00000000cff03000 (ACPI data)
(XEN)  00000000cff03000 - 00000000cff04000 (ACPI NVS)
(XEN)  00000000cff04000 - 00000000d0000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec10000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000230000000 (usable)
(XEN) System RAM: 8190MB (8387120kB)
(XEN) ACPI: RSDP 000F6530, 0024 (r2 PTLTD )
(XEN) ACPI: XSDT CFEFA758, 00CC (r1 SMCSLP SUPERMIC  6040000  LTP        0)
(XEN) ACPI: FACP CFF022FC, 00F4 (r3 INTEL  STOAKLEY  6040000 PTL         3)
(XEN) ACPI: DSDT CFEFD9BC, 48CC (r1  Intel SEABURG   6040000 MSFT  3000001)
(XEN) ACPI: FACS CFF03FC0, 0040
(XEN) ACPI: DMAR CFF023F0, 0108 (r1 Intel  OEMDMAR   6040000 LOHR        1)
(XEN) ACPI: TCPA CFF024F8, 0032 (r1 Intel  STOAKLEY  6040000 LOHR       5A)
(XEN) ACPI: APIC CFF0252A, 00D4 (r1 PTLTD        APIC    6040000  LTP        0)
(XEN) ACPI: MCFG CFF025FE, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
(XEN) ACPI: BOOT CFF0263A, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
(XEN) ACPI: SPCR CFF02662, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL         1)
(XEN) ACPI: ERST CFF026B2, 0590 (r1 SMCI   ERSTTBL   6040000 SMCI        1)
(XEN) ACPI: HEST CFF02C42, 00A8 (r1 SMCI   HESTTBL   6040000 SMCI        1)
(XEN) ACPI: BERT CFF02CEA, 0030 (r1 SMCI   BERTTBL   6040000 SMCI        1)
(XEN) ACPI: EINJ CFF02D1A, 0170 (r1 SMCI   EINJTBL   6040000 SMCI        1)
(XEN) ACPI: SLIC CFF02E8A, 0176 (r1 SMCSLP SUPERMIC  6040000  LTP        0)
(XEN) ACPI: SSDT CFEFC0B3, 025F (r1  PmRef  Cpu0Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFC00D, 00A6 (r1  PmRef  Cpu7Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFBF67, 00A6 (r1  PmRef  Cpu6Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFBEC1, 00A6 (r1  PmRef  Cpu5Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFBE1B, 00A6 (r1  PmRef  Cpu4Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFBD75, 00A6 (r1  PmRef  Cpu3Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFBCCF, 00A6 (r1  PmRef  Cpu2Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFBC29, 00A6 (r1  PmRef  Cpu1Tst     3000 INTL 20050228)
(XEN) ACPI: SSDT CFEFA824, 1405 (r1  PmRef    CpuPm     3000 INTL 20050228)
(XEN) Xen heap: 14MB (14500kB)
(XEN) Domain heap initialised
(XEN) Processor #0 7:7 APIC version 20
(XEN) Processor #4 7:7 APIC version 20
(XEN) Processor #1 7:7 APIC version 20
(XEN) Processor #5 7:7 APIC version 20
(XEN) Processor #2 7:7 APIC version 20
(XEN) Processor #6 7:7 APIC version 20
(XEN) Processor #3 7:7 APIC version 20
(XEN) Processor #7 7:7 APIC version 20
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec86000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec89000, GSI 48-71
(XEN) Enabling APIC mode:  Flat.  Using 3 I/O APICs
(XEN) Intel VT-d has been enabled
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2666.833 MHz processor.
(XEN) HVM: VMX enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 1/4 eip 8c000
(XEN) CPU1: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 2/1 eip 8c000
(XEN) CPU2: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 3/5 eip 8c000
(XEN) CPU3: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 4/2 eip 8c000
(XEN) CPU4: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 5/6 eip 8c000
(XEN) CPU5: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 6/3 eip 8c000
(XEN) CPU6: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 7/7 eip 8c000
(XEN) CPU7: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Total of 8 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) checking TSC synchronization across 8 CPUs: passed.
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) Brought up 8 CPUs
(XEN) I/O virtualisation enabled
(XEN) I/O virtualisation for PV guests disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff805c148c
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000224000000->0000000226000000 (2017483 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff805c148c
(XEN)  Init. ramdisk: ffffffff805c2000->ffffffff80d07e00
(XEN)  Phys-Mach map: ffffffff80d08000->ffffffff81c7c658
(XEN)  Start info:    ffffffff81c7d000->ffffffff81c7d4a4
(XEN)  Page tables:   ffffffff81c7e000->ffffffff81c91000
(XEN)  Boot stack:    ffffffff81c91000->ffffffff81c92000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82000000
(XEN)  ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 108kB init memory.
(XEN) mtrr: type mismatch for d0000000,2000000 old: write-back new: write-combining

-----------------------------------------
Rafael E. Ayala
Senior Software Engineer
Xyratex International, Inc.

______________________________________________________________________
This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it.

Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses.

Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.

The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan.
______________________________________________________________________

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users