Re: [Xen-devel] Vt-d not working with 3.4.1

[Andrew Lyon <andrew.lyon@xxxxxxxxx>]

On Thu, Aug 20, 2009 at 5:41 PM, Mr. Teo En Ming (Zhang
Enming)<enming.teo@xxxxxxxxxxxxxxx> wrote:
> Dear All,
>
> I followed the XEN - NVIDIA - STEP BY STEP instructions at nV news forums
> but I still couldn't get X to work under Xen pv-ops domain 0 kernel. It
> still shows me a freezed blank screen. I couldn't switch to any tty (1 to 7
> inclusive).
>
> Here are the scenarios I have tried:
>
> 1)     IGNORE_XEN_PRESENCE=y ./NVIDIA-Linux-x86_64-185.18.31-pkg2.run
>
> 2)     IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT
> -DNV_SIGNAL_STRUCT_RLIM" ./NVIDIA-Linux-x86_64-185.18.31-pkg2.run
>
> 3)
>
> ./NVIDIA-Linux-x86_64-185.18.31-pkg2.run --extract-only
>
> cd usr/src/nv
>
> IGNORE_XEN_PRESENCE=y CC="gcc -DNV_VMAP_4_PRESENT -DNV_SIGNAL_STRUCT_RLIM"
> make SYSSRC=/usr/src/kernels/2.6.30-rc3-enming.teo-tip module
>
> cp nvidia.ko /lib/modules/2.6.30-rc3-enming.teo-tip/kernel/drivers/video
>
> cd /lib/modules/2.6.30-rc3-enming.teo-tip/kernel/drivers/video
>
> depmod -a
>
> rmmod nvidia
>
> lsmod | grep nvidia
>
> modprobe nvidia
>
> lsmod | grep nvidia
>
> None of the above three scenarios I have attempted worked for me.
>
> When I tried to load the newly compiled nvidia driver (against the Xen
> Domain 0 paravirt operations kernel), it gives me the following message:
>
> NVRM: loading NVIDIA UNIX x86_64 Kernel Module 185.18.31 Tue Jul 28 17:52:27
> PDT 2009
>
> NVRM: PAT configuration unsupported, falling back to MTRRs.
>
> Any more ideas?
>
> My home PC configuration:
>
> Intel Pentium processor E6300 LGA775
> Intel Desktop Board DQ45CB
> BIOS Version 0093
> 8 GB DDR2-800
> NVIDIA Geforce 8400 GS 512 MB DDR2 PCI Express x16
>
> Fedora 11 64-bit Host operating system
> Xen hypervisor 3.4.1
> Jeremy Fitzhardinge's pv-ops dom 0 Xen kernel 2.6.30-rc3 and 2.6.31-rc6
> Default shipping X server
> mesa-lib upgraded
>
> --
> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics Engineering)
> BEng(Hons)(Mechanical Engineering)
> Technical Support Engineer
> Information Technology Department
> Asiasoft Online Pte Ltd
> Tampines Central 1 #04-01 Tampines Plaza
> Singapore 529541
> Republic of Singapore
> Mobile: +65-9648-9798
> MSN: teoenming@xxxxxxxxxxx
> Alma Maters: Singapore Polytechnic, National University of Singapore
>
>
>
> On 08/21/2009 12:23 AM, Teo En Ming (Zhang Enming) wrote:
>>
>> This certainly looks promising. No need to apply any patch to NVIDIA
>> graphics drivers for it to be Xen-aware.
>>
>> Link: http://www.nvnews.net/vbulletin/showthread.php?t=122900
>>
>> Regards,
>>
>> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics Engineering)
>> BEng(Hons)(Mechanical Engineering)
>> Technical Support Engineer
>> Information Technology Department
>> Asiasoft Online Pte Ltd
>> Tampines Central 1 #04-01 Tampines Plaza
>> Singapore 529541
>> Republic of Singapore
>> Mobile: +65-9648-9798
>> MSN: teoenming@xxxxxxxxxxx
>>
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Teo En Ming
>> (Zhang Enming)
>> Sent: Thursday, August 20, 2009 9:21 PM
>> To: enming.teo@xxxxxxxxxxxxxxx; mad@xxxxxx
>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; weidong.han@xxxxxxxxx; 'Bonenkamp,Ralf'
>> Subject: RE: [Xen-devel] Vt-d not working with 3.4.1
>>
>> <QUOTE drivesoslow>
>>
>> SOLVED!
>>
>> It was a kernel issue. The current Fedora 8 XEN kernel was the issue.
>> Since
>> there was not a newer one available I enabled the development repo and
>> installed the Fedora 9 XEN kernel that was in it. A couple of small hacks
>> with the drive source and I have a working NVIDIA 169.07 driver. I assume
>> when the next kernel is release for Fedora 8 it will resolve this issue.
>>
>> Was:
>> Code:
>>
>> kernel-xen-2.6.21-2952.fc8
>> kernel-xen-devel-2.6.21-2952.fc8
>>
>> Now:
>> Code:
>>
>> kernel-xen-2.6.21.7-2892.fc9
>> kernel-xen-devel-2.6.21.7-2892.fc9
>>
>> </QUOTE>
>>
>>
>> Regards,
>>
>> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics Engineering)
>> BEng(Hons)(Mechanical Engineering)
>> Technical Support Engineer
>> Information Technology Department
>> Asiasoft Online Pte Ltd
>> Tampines Central 1 #04-01 Tampines Plaza
>> Singapore 529541
>> Republic of Singapore
>> Mobile: +65-9648-9798
>> MSN: teoenming@xxxxxxxxxxx
>>
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Teo En Ming
>> (Zhang Enming)
>> Sent: Thursday, August 20, 2009 9:10 PM
>> To: enming.teo@xxxxxxxxxxxxxxx; mad@xxxxxx
>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; weidong.han@xxxxxxxxx; 'Bonenkamp,Ralf'
>> Subject: RE: [Xen-devel] Vt-d not working with 3.4.1
>>
>> <QUOTE drivesoslow>
>>
>> I was able to get it to compile and install no problem finally, but it
>> locks
>> up when I startx. I can compile it for non-XEN 64bit no problem. The
>> non-XEN
>> kernel is significantly newer (2.6.23 vs 2.6.21)
>>
>> </QUOTE>
>>
>> Link: http://www.nvnews.net/vbulletin/showthread.php?t=106395
>>
>> Same problem as what I am getting.
>>
>> I am using the NVidia Geforce 8 series drivers for 64-bit.
>>
>> Regards,
>>
>> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics Engineering)
>> BEng(Hons)(Mechanical Engineering)
>> Technical Support Engineer
>> Information Technology Department
>> Asiasoft Online Pte Ltd
>> Tampines Central 1 #04-01 Tampines Plaza
>> Singapore 529541
>> Republic of Singapore
>> Mobile: +65-9648-9798
>> MSN: teoenming@xxxxxxxxxxx
>>
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Teo En Ming
>> (Zhang Enming)
>> Sent: Thursday, August 20, 2009 8:52 PM
>> To: enming.teo@xxxxxxxxxxxxxxx; mad@xxxxxx
>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; weidong.han@xxxxxxxxx; 'Bonenkamp,Ralf'
>> Subject: RE: [Xen-devel] Vt-d not working with 3.4.1
>>
>> <QUOTE monz at nV News Forums>
>>
>> The problem with video on Xen is that the driver needs to be Xen-aware.
>>
>> </QUOTE>
>>
>> Link: http://www.nvnews.net/vbulletin/showthread.php?t=60125
>>
>> Regards,
>>
>> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics Engineering)
>> BEng(Hons)(Mechanical Engineering)
>> Technical Support Engineer
>> Information Technology Department
>> Asiasoft Online Pte Ltd
>> Tampines Central 1 #04-01 Tampines Plaza
>> Singapore 529541
>> Republic of Singapore
>> Mobile: +65-9648-9798
>> MSN: teoenming@xxxxxxxxxxx
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Teo En Ming
>> (Zhang Enming)
>> Sent: Thursday, August 20, 2009 8:49 PM
>> To: enming.teo@xxxxxxxxxxxxxxx; mad@xxxxxx
>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; weidong.han@xxxxxxxxx; 'Bonenkamp,Ralf'
>> Subject: RE: [Xen-devel] Vt-d not working with 3.4.1
>>
>> Will the Xen patch for NVIDIA graphics drivers mentioned in the below
>> opensuse.org link work with Xen 3.4.1 rc and Jeremy's pv-ops dom 0 kernel
>> 2.6.30-rc3 or 2.6.31-rc6 under Fedora 11 64-bit? Please see the patch code
>> below.
>>
>> <CODE>
>>
>> diff -urN nv-1.0-9625/nv.c nv-1.0-9625-xenrt/nv.c
>> --- nv-1.0-9625/nv.c    2006-09-26 21:33:35.000000000 +0200
>> +++ nv-1.0-9625-xenrt/nv.c      2006-10-03 01:15:42.000000000 +0200
>> @@ -42,8 +42,26 @@
>>
>>  int nv_pat_enabled = 0;
>>
>> +/*
>> + * disable PAT support if XEN or PREEMPT_RT is configured in kernel
>> + */
>> +
>> +#if defined(CONFIG_XEN) || defined(CONFIG_PREEMPT_RT)
>> +static int nv_disable_pat = 1;
>> +#else
>>  static int nv_disable_pat = 0;
>> +#endif
>> +
>> +/*
>> + * you can re-enable PAT support for PREEMPT_RT when applying
>> + * "nv_disable_pat=0" as kernel parameter for the sake of slightly
>> + * better 3D performance but at the expense of higher latencies.
>> + * if XEN is configured, then PAT support can't be enabled!
>> + */
>> +
>> +#if !defined(CONFIG_XEN)
>>  NV_MODULE_PARAMETER(nv_disable_pat);
>> +#endif
>>
>>  #if defined(NVCPU_X86) || defined(NVCPU_X86_64)
>>  NvU64 __nv_supported_pte_mask = ~_PAGE_NX;
>> diff -urN nv-1.0-9625/nv-linux.h nv-1.0-9625-xenrt/nv-linux.h
>> --- nv-1.0-9625/nv-linux.h      2006-09-26 21:33:37.000000000 +0200
>> +++ nv-1.0-9625-xenrt/nv-linux.h        2006-10-03 01:15:42.000000000
>> +0200
>> @@ -226,7 +226,7 @@
>>   * tiny, and the kernel panics when it is exhausted. try to warn the user
>> that
>>   * they need to boost the size of their pool.
>>   */
>> -#if defined(CONFIG_SWIOTLB)&&  !defined(GFP_DMA32)
>> +#if defined(CONFIG_SWIOTLB)&&  !defined(GFP_DMA32)&&
>>  !defined(CONFIG_XEN)
>>  #define NV_SWIOTLB 1
>>  #endif
>>
>> @@ -734,7 +734,10 @@
>>  #define NV_VM_INSERT_PAGE(vma, addr, page) \
>>      vm_insert_page(vma, addr, page)
>>  #endif
>> -#if defined(NV_REMAP_PFN_RANGE_PRESENT)
>> +#if defined(CONFIG_XEN)
>> +#define NV_REMAP_PAGE_RANGE(from, offset, x...) \
>> +    io_remap_pfn_range(vma, from, ((offset)>>  PAGE_SHIFT), x)
>> +#elif defined(NV_REMAP_PFN_RANGE_PRESENT)
>>  #define NV_REMAP_PAGE_RANGE(from, offset, x...) \
>>      remap_pfn_range(vma, from, ((offset)>>  PAGE_SHIFT), x)
>>  #elif defined(NV_REMAP_PAGE_RANGE_5_PRESENT)
>> @@ -746,6 +749,9 @@
>>  #define NV_REMAP_PAGE_RANGE(x...) remap_page_range(x)
>>  #endif
>>
>> +#if !defined(CONFIG_XEN)
>> +#define phys_to_machine(x) x
>> +#endif
>>
>>  #define NV_PGD_OFFSET(address, kernel, mm)              \
>>     ({                                                   \
>> diff -urN nv-1.0-9625/nv-vm.c nv-1.0-9625-xenrt/nv-vm.c
>> --- nv-1.0-9625/nv-vm.c 2006-09-26 21:33:37.000000000 +0200
>> +++ nv-1.0-9625-xenrt/nv-vm.c   2006-10-03 01:24:31.000000000 +0200
>> @@ -352,6 +352,9 @@
>>
>>  static void nv_flush_caches(void)
>>  {
>> +#if defined(CONFIG_PREEMPT_RT)
>> +    if(!nv_pat_enabled) return;
>> +#endif
>>  #if defined(KERNEL_2_4)
>>      // for 2.4 kernels, just automatically flush the caches and
>> invalidate
>> tlbs
>>  #ifdef CONFIG_SMP
>> @@ -508,7 +511,7 @@
>>          page_ptr->phys_addr = phys_addr;
>>          page_ptr->page_count = NV_GET_PAGE_COUNT(page_ptr);
>>          page_ptr->virt_addr = virt_addr;
>> -        page_ptr->dma_addr = page_ptr->phys_addr;
>> +        page_ptr->dma_addr = phys_to_machine(page_ptr->phys_addr);
>>
>>          /* lock the page for dma purposes */
>>          nv_lock_page(page_ptr);
>> diff -urN nv-1.0-9625/os-agp.c nv-1.0-9625-xenrt/os-agp.c
>> --- nv-1.0-9625/os-agp.c        2006-09-26 21:33:37.000000000 +0200
>> +++ nv-1.0-9625-xenrt/os-agp.c  2006-10-03 01:15:42.000000000 +0200
>> @@ -286,7 +286,7 @@
>>
>>           page_ptr->phys_addr = (ptr->memory[i]&  PAGE_MASK);
>>           page_ptr->virt_addr = (unsigned long) __va(page_ptr->phys_addr);
>> -         page_ptr->dma_addr  = page_ptr->phys_addr;
>> +         page_ptr->dma_addr  = phys_to_machine(page_ptr->phys_addr);
>>      }
>>
>>      return RM_OK;
>> diff -urN nv-1.0-9625/os-interface.c nv-1.0-9625-xenrt/os-interface.c
>> --- nv-1.0-9625/os-interface.c  2006-09-26 21:33:37.000000000 +0200
>> +++ nv-1.0-9625-xenrt/os-interface.c    2006-10-03 01:15:42.000000000
>> +0200
>> @@ -527,6 +527,7 @@
>>      MicroSeconds = MilliSeconds * 1000;
>>      tm_end.tv_usec = MicroSeconds;
>>      tm_end.tv_sec = 0;
>> +#if !defined(CONFIG_XEN)
>>      NV_TIMERADD(&tm_aux,&tm_end,&tm_end);
>>
>>      /* do we have a full jiffie to wait? */
>> @@ -564,6 +565,7 @@
>>                  MicroSeconds = 0;
>>          } while ((jiffies = NV_USECS_TO_JIFFIES(MicroSeconds)) != 0);
>>      }
>> +#endif
>>
>>      if (MicroSeconds>  1000)
>>      {
>>
>> </CODE>
>>
>> Regards,
>>
>> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics Engineering)
>> BEng(Hons)(Mechanical Engineering)
>> Technical Support Engineer
>> Information Technology Department
>> Asiasoft Online Pte Ltd
>> Tampines Central 1 #04-01 Tampines Plaza
>> Singapore 529541
>> Republic of Singapore
>> Mobile: +65-9648-9798
>> MSN: teoenming@xxxxxxxxxxx
>>
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Teo En Ming
>> (Zhang Enming)
>> Sent: Thursday, August 20, 2009 8:37 PM
>> To: enming.teo@xxxxxxxxxxxxxxx; mad@xxxxxx
>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; weidong.han@xxxxxxxxx; 'Bonenkamp,Ralf'
>> Subject: RE: [Xen-devel] Vt-d not working with 3.4.1
>>
>> No wonder compiling official NVidia drivers against the Xen domain 0
>> paravirt operations kernel doesn't work. It causes my X server to show a
>> blank screen and freeze. NVidia drivers need to be patched for it to work
>> with Xen kernels.
>>
>> http://en.opensuse.org/Use_Nvidia_driver_with_Xen
>>
>> May I know where I can get the patch?
>>
>> Regards,
>>
>> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics Engineering)
>> BEng(Hons)(Mechanical Engineering)
>> Technical Support Engineer
>> Information Technology Department
>> Asiasoft Online Pte Ltd
>> Tampines Central 1 #04-01 Tampines Plaza
>> Singapore 529541
>> Republic of Singapore
>> Mobile: +65-9648-9798
>> MSN: teoenming@xxxxxxxxxxx
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Mr. Teo En
>> Ming
>> (Zhang Enming)
>> Sent: Thursday, August 20, 2009 1:06 AM
>> To: mad@xxxxxx
>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; weidong.han@xxxxxxxxx; 'Bonenkamp,Ralf'
>> Subject: Re: [Xen-devel] Vt-d not working with 3.4.1
>>
>> Dear All,
>>
>> I can confirm that recompiling the Xen dom 0 paravirt-ops kernel using
>> gzip compression works. I can use the generated bzImage to boot up Domain
>> 0.
>>
>> If you attempt to compile the Xen Dom 0 pv-ops kernel using bzip2
>> compression, you won't be able to boot up Domain 0 using the generated
>> bzImage. You will get an ELF error and kernel panic on CPU 0.
>>
>> As for X server, I haven't got much luck yet.
>>
>> I was able to download and compile NVIDIA's kernel module directly from
>> official NVIDIA website for Fedora 11 Default shipping kernel. I was
>> able to start X server successfully and launch a desktop environment
>> using nvidia.ko.
>>
>> However, if I want to compile the nvidia kernel module against Xen dom 0
>> pv_ops kernel sources, the kernel module compiles successfully, but when
>> I start X, the screen becomes blank and the whole system freezes there.
>>
>>
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>

I am fairly sure that IGNORE_XEN_PRESENCE=y only makes the driver work
with a traditional (not pvops) patched xen kernel, I don't think it
will help to make the module compile or work with a pv_ops kernel.

Andy

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel