Re: [Xen-users] DomU sees only 512MB RAM with PCI-passthrough

[David Vrabel <david.vrabel@xxxxxxxxxx>]

On 13/12/12 11:57, Ian Campbell wrote:
> On Thu, 2012-12-13 at 11:43 +0000, Christian Holpert wrote:
>> Hello Ian,
>>
>> At 10:38 13.12.2012, you wrote:
>>> On Wed, 2012-12-12 at 17:04 +0000, Christian Holpert wrote:
>>>> Please tell me, if you need more logs.
>>>
>>> dmesg of the guest booting both with and without the pci device might be
>>> interesting. So would the output of "xl -vvv create" in both cases.
>>
>> attached.
> 
> In the diff of the dmesg I see:
>         --- dmesg_without_pci.log       2012-12-13 11:50:00.000000000 +0000
>         +++ dmesg_with_pci.log  2012-12-13 11:50:00.000000000 +0000
>         @@ -1,72 +1,92 @@
>          Reserving virtual address space above 0xf5800000
>          Linux version 3.6.6-gentoo (root@xen) (gcc version 4.5.4 (Gentoo 
> 4.5.4 p1.0, pie-0.4.7) ) #1 SMP Tue Nov 13 17:47:58 CET 2012
>          ACPI in unprivileged domain disabled
>         +Freeing 20000-80000 pfn range: 393216 pages freed
>         +1-1 mapping on 20000->100000
>         +Released 393216 pages of unused memory
>         +Set 917504 page(s) to 1-1 mapping
>          e820: BIOS-provided physical RAM map:
>          Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
>          Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
>         -Xen: [mem 0x0000000000100000-0x000000007fffffff] usable
>         +Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
>         +Xen: [mem 0x0000000020000000-0x00000000201fffff] reserved
>         +Xen: [mem 0x0000000020200000-0x0000000040003fff] unusable
>         +Xen: [mem 0x0000000040004000-0x0000000040004fff] reserved
>         +Xen: [mem 0x0000000040005000-0x00000000d9cbefff] unusable
>         +Xen: [mem 0x00000000d9cbf000-0x00000000da285fff] reserved
>         +Xen: [mem 0x00000000da286000-0x00000000da505fff] ACPI NVS
>         +Xen: [mem 0x00000000da506000-0x00000000da50afff] ACPI data
>         +Xen: [mem 0x00000000da50b000-0x00000000da54dfff] ACPI NVS
>         +Xen: [mem 0x00000000da54e000-0x00000000dad55fff] unusable
>         +Xen: [mem 0x00000000dad56000-0x00000000daff1fff] reserved
>         +Xen: [mem 0x00000000daff2000-0x00000000daffffff] unusable
>         +Xen: [mem 0x00000000db800000-0x00000000df9fffff] reserved
>         +Xen: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
>         +Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
>         +Xen: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
>         +Xen: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
>         +Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
>         +Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
>          NX (Execute Disable) protection: active
>          MPS support code is not built-in.
>          Using acpi=off or acpi=noirq or pci=noacpi may have problem
>          DMI not present or invalid.
>          e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
>          e820: remove [mem 0x000a0000-0x000fffff] usable
>         -e820: last_pfn = 0x80000 max_arch_pfn = 0x1000000
>         +e820: last_pfn = 0x20000 max_arch_pfn = 0x1000000

I think the kernel has done the correct thing with the supplied e820 map.

> On the libxl side with pci I also see:
>         +libxl: debug: libxl_pci.c:85:libxl__create_pci_backend: Creating pci 
> backend
>         +libxl: debug: libxl_x86.c:82:e820_sanitize: Memory: 2097152kB End of 
> RAM: 0x20000 (PFN) Delta: 1572864kB, PCI start: 524288kB (0x20000 PFN), 
> Balloon 0kB
>         +
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [0 -> 20000] RAM

This is the 512 MiB limit here, so I think the bug is in libxl.

Why has it placed the PCI window at 512 MiB anyway?  3 GiB would be more
conventional.

>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [20000 -> 20200] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [20200 -> 40004] 
> Unusable
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [40004 -> 40005] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [40005 -> d9cbf] 
> Unusable
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [d9cbf -> da286] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da286 -> da506] ACPI 
> NVS
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da506 -> da50b] ACPI
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da50b -> da54e] ACPI 
> NVS
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da54e -> dad56] 
> Unusable
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [dad56 -> daff2] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [daff2 -> db000] 
> Unusable
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [db800 -> dfa00] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [f8000 -> fc000] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fec00 -> fec01] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed00 -> fed04] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed1c -> fed20] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fee00 -> fee01] 
> Reserved
>         +libxl: debug: libxl_x86.c:201:e820_sanitize: : [ff000 -> 100000] 
> Reserved
> 
> So I suspect this is some interaction between xl's e820_host option
> (which gets automatically enabled if you give a PCI device) and the
> kernels early memory layout stuff. I'm CCing David and Konrad in the
> hopes they have an idea what is happening.

David

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users