Re: [Xen-users] Quadrified GTX 480 VT-d passthrough. CUDA 5.5 in Linux partial success

[Gordan Bobic <gordan@xxxxxxxxxx>]

Does the nvidia binary driver provide a reset handle for the device via 
sysfs?
If you echo 1 into it, does it help or does it crash things?


On Tue, 19 Nov 2013 10:32:31 +0100, Tamas Lengyel 
<tamas.lengyel@xxxxxxxxxxxx> wrote:
> Hi everyone,
> after following in the footsteps of the following discussion
>  
> (http://lists.xenproject.org/archives/html/xen-users/2013-09/msg00106.html
> [1])
>  I had been able to turn my GTX 480 into a Quadro 6000. When I VT-d
> passthrough it to a Debian jessie VM it shows up fine and CUDA 5.5
>  seems to function properly up to a point:
>
> lspci -v:
>
> 00:04.0 VGA compatible controller: NVIDIA Corporation GF100GL [Quadro
>  6000] (rev a3) (prog-if 00 [VGA controller])
> Subsystem: ASUSTeK Computer Inc. Device 075f
>  Physical Slot: 4
> Flags: bus master, fast devsel, latency 0, IRQ 32
>  Memory at ee000000 (32-bit, non-prefetchable) [size=32M]
> Memory at e0000000 (64-bit, prefetchable) [size=128M]
>  Memory at e8000000 (64-bit, prefetchable) [size=64M]
> I/O ports at c100 [size=128]
>  Expansion ROM at f1000000 [disabled] [size=512K]
> Capabilities: [60] Power Management version 3
>  Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [78] Express Endpoint, MSI 00
>  Capabilities: [b4] Vendor Specific Information: Len=14
> Kernel driver in use: nvidia
>
> 00:05.0 Audio device: NVIDIA Corporation GF100 High Definition Audio
>  Controller (rev a1)
> Subsystem: ASUSTeK Computer Inc. Device 075f
>  Physical Slot: 5
> Flags: bus master, fast devsel, latency 0, IRQ 37
>  Memory at f1080000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: [60] Power Management version 3
>  Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [78] Express Endpoint, MSI 00
>  Kernel driver in use: snd_hda_intel
>
> NVIDIA_CUDA-5.5_Samples/1_Utilities/deviceQuery# ./deviceQuery
>  ./deviceQuery Starting...
>
> ÂCUDA Device Query (Runtime API) version (CUDART static linking)
>
> Detected 1 CUDA Capable device(s)
>
>  Device 0: "Quadro 6000"
>  CUDA Driver Version / Runtime Version     Â6.0 / 5.5
>  Â CUDA Capability Major/Minor version number: Â Â2.0
> Â Total amount of global memory: Â Â Â Â Â Â Â Â 1536 MBytes
> (1610285056 bytes)
>  Â (15) Multiprocessors, ( 32) CUDA Cores/MP: Â Â 480 CUDA Cores
> Â GPU Clock rate: Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â1401
> MHz (1.40 GHz)
>  Â Memory Clock rate: Â Â Â Â Â Â Â Â Â Â Â Â Â Â 1848
> Mhz
> Â Memory Bus Width: Â Â Â Â Â Â Â Â Â Â Â Â Â Â
> Â384-bit
>  Â L2 Cache Size: Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
> 786432 bytes
> Â Maximum Texture Dimension Size (x,y,z) Â Â Â Â 1D=(65536),
>  2D=(65536, 65535), 3D=(2048, 2048, 2048)
> Â Maximum Layered 1D Texture Size, (num) layers Â1D=(16384), 2048
> layers
>  Â Maximum Layered 2D Texture Size, (num) layers Â2D=(16384, 16384),
> 2048 layers
> Â Total amount of constant memory: Â Â Â Â Â Â Â 65536 bytes
>  Â Total amount of shared memory per block: Â Â Â 49152 bytes
> Â Total number of registers available per block: 32768
>  Â Warp size: Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
> 32
> Â Maximum number of threads per multiprocessor: Â1536
>  Â Maximum number of threads per block: Â Â Â Â Â 1024
> Â Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
>   Max dimension size of a grid size  Â(x,y,z): (65535, 65535,
> 65535)
> Â Maximum memory pitch: Â Â Â Â Â Â Â Â Â Â Â Â
> Â2147483647 bytes
>  Â Texture alignment: Â Â Â Â Â Â Â Â Â Â Â Â Â Â 512
> bytes
> Â Concurrent copy and kernel execution: Â Â Â Â ÂYes with 2 copy
> engine(s)
>  Â Run time limit on kernels: Â Â Â Â Â Â Â Â Â Â No
> Â Integrated GPU sharing Host Memory: Â Â Â Â Â ÂNo
>  Â Support host page-locked memory mapping: Â Â Â Yes
> Â Alignment requirement for Surfaces: Â Â Â Â Â ÂYes
>  Â Device has ECC support: Â Â Â Â Â Â Â Â Â Â Â
> ÂDisabled
> Â Device supports Unified Addressing (UVA): Â Â ÂYes
>  Â Device PCI Bus ID / PCI location ID: Â Â Â Â Â 0 / 4
> Â Compute Mode:
>  Â Â Â< Default (multiple host threads can use ::cudaSetDevice()
> with
> device simultaneously) >
>
> deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 6.0, CUDA
>  Runtime Version = 5.5, NumDevs = 1, Device0 = Quadro 6000
> Result = PASS
>
> Unfortunately if I try to run any CUDA app or even nvidia-smi
>  afterwards, I get the following errors:
>
> NVIDIA_CUDA-5.5_Samples/1_Utilities/deviceQuery# ./deviceQuery
>  ./deviceQuery Starting...
>
> ÂCUDA Device Query (Runtime API) version (CUDART static linking)
>
> cudaGetDeviceCount returned 10
> -> invalid device ordinal
>  Result = FAIL
>
> # nvidia-smi
>  Unable to determine the device handle for GPU 0000:00:04.0: The
> NVIDIA
> kernel module detected an issue with GPU interrupts.Consult the
>  "Common Problems" Chapter of the NVIDIA Driver README for
> details and steps that can be taken to resolve this issue.
>
> If I restart the VM I can run a single CUDA app again, once. It's
>  still pretty impressive to be able to do that without having to 
> patch
> Xen or reboot the entire machine =)ÂIt doesn't seem to matter what
> CUDA app I'm running, here is matrixMul
>  for example:
>
> matrixMul# ./matrixMul
>  [Matrix Multiply Using CUDA] - Starting...
> GPU Device 0: "Quadro 6000" with compute capability 2.0
>
> MatrixA(320,320), MatrixB(640,320)
> Computing result using CUDA Kernel...
>  done
> Performance= 227.22 GFlop/s, Time= 0.577 msec, Size= 131072000 Ops,
>  WorkgroupSize= 1024 threads/block
> Checking computed result for correctness: Result = PASS
>
> Note: For peak performance, please refer to the matrixMulCUBLAS
> example.
>
> Anyhoo, does anyone have any idea what might I be able to tweak so I
> can
>  avoid this issue? The setup clearly seems to work for the most
> part.
>
> My domU config:
>
>  arch = 'x86_64'
> name = "debian-miner"
>  builder = "hvm"
> maxmem = 512
>  memory = 512
> vcpus = 1
>  maxcpus = 1
> boot = "cd"
>  pae=1
> acpi = 1
>  apic = 1
> hap=1
>  hpet=1
> shadow_memory = 32
>  on_poweroff = "destroy"
> on_reboot = "restart"
>  on_crash = "restart"
> vnc=1
>  vncunused=1
> vnclisten="0.0.0.0"
>  vif = [ 'type=netfront,bridge=xenbr0,mac=00:16:3e:12:c3:fa']
>  device_model_version="qemu-xen-traditional"
>  gfx_passthru=0
> xen_platform_pci=1
>  pci Â= [ '01:00.0', '01:00.1' ]
> pci_msitranslate = 1
>  pci_power_mgmt = 1
> pci_permissive = 1
>  xen_extended_power_mgmt = 1
> acpi_s3 = 1
>  acpi_s4 = 1
> disk = [ Â Â Â Â'phy:/dev/t0vg/debian-testing,xvda,w'];
>
> And I'm running on Xen 4.3.1 with NVIDIA driver 331.20 x86_64 in the
> domU.
>
> Thanks and cheers!
>
>
> Links:
> ------
> [1]
>
> http://lists.xenproject.org/archives/html/xen-users/2013-09/msg00106.html

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users