[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] DomU sees only 512MB RAM with PCI-passthrough



On Thu, 2012-12-13 at 11:43 +0000, Christian Holpert wrote:
> Hello Ian,
> 
> At 10:38 13.12.2012, you wrote:
> >On Wed, 2012-12-12 at 17:04 +0000, Christian Holpert wrote:
> > > Please tell me, if you need more logs.
> >
> >dmesg of the guest booting both with and without the pci device might be
> >interesting. So would the output of "xl -vvv create" in both cases.
> 
> attached.

In the diff of the dmesg I see:
        --- dmesg_without_pci.log       2012-12-13 11:50:00.000000000 +0000
        +++ dmesg_with_pci.log  2012-12-13 11:50:00.000000000 +0000
        @@ -1,72 +1,92 @@
         Reserving virtual address space above 0xf5800000
         Linux version 3.6.6-gentoo (root@xen) (gcc version 4.5.4 (Gentoo 4.5.4 
p1.0, pie-0.4.7) ) #1 SMP Tue Nov 13 17:47:58 CET 2012
         ACPI in unprivileged domain disabled
        +Freeing 20000-80000 pfn range: 393216 pages freed
        +1-1 mapping on 20000->100000
        +Released 393216 pages of unused memory
        +Set 917504 page(s) to 1-1 mapping
         e820: BIOS-provided physical RAM map:
         Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
         Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
        -Xen: [mem 0x0000000000100000-0x000000007fffffff] usable
        +Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
        +Xen: [mem 0x0000000020000000-0x00000000201fffff] reserved
        +Xen: [mem 0x0000000020200000-0x0000000040003fff] unusable
        +Xen: [mem 0x0000000040004000-0x0000000040004fff] reserved
        +Xen: [mem 0x0000000040005000-0x00000000d9cbefff] unusable
        +Xen: [mem 0x00000000d9cbf000-0x00000000da285fff] reserved
        +Xen: [mem 0x00000000da286000-0x00000000da505fff] ACPI NVS
        +Xen: [mem 0x00000000da506000-0x00000000da50afff] ACPI data
        +Xen: [mem 0x00000000da50b000-0x00000000da54dfff] ACPI NVS
        +Xen: [mem 0x00000000da54e000-0x00000000dad55fff] unusable
        +Xen: [mem 0x00000000dad56000-0x00000000daff1fff] reserved
        +Xen: [mem 0x00000000daff2000-0x00000000daffffff] unusable
        +Xen: [mem 0x00000000db800000-0x00000000df9fffff] reserved
        +Xen: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
        +Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
        +Xen: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
        +Xen: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
        +Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
        +Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
         NX (Execute Disable) protection: active
         MPS support code is not built-in.
         Using acpi=off or acpi=noirq or pci=noacpi may have problem
         DMI not present or invalid.
         e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
         e820: remove [mem 0x000a0000-0x000fffff] usable
        -e820: last_pfn = 0x80000 max_arch_pfn = 0x1000000
        +e820: last_pfn = 0x20000 max_arch_pfn = 0x1000000
        
On the libxl side with pci I also see:
        +libxl: debug: libxl_pci.c:85:libxl__create_pci_backend: Creating pci 
backend
        +libxl: debug: libxl_x86.c:82:e820_sanitize: Memory: 2097152kB End of 
RAM: 0x20000 (PFN) Delta: 1572864kB, PCI start: 524288kB (0x20000 PFN), Balloon 
0kB
        +
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [0 -> 20000] RAM
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [20000 -> 20200] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [20200 -> 40004] 
Unusable
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [40004 -> 40005] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [40005 -> d9cbf] 
Unusable
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [d9cbf -> da286] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da286 -> da506] ACPI 
NVS
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da506 -> da50b] ACPI
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da50b -> da54e] ACPI 
NVS
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [da54e -> dad56] 
Unusable
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [dad56 -> daff2] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [daff2 -> db000] 
Unusable
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [db800 -> dfa00] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [f8000 -> fc000] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fec00 -> fec01] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed00 -> fed04] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed1c -> fed20] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [fee00 -> fee01] 
Reserved
        +libxl: debug: libxl_x86.c:201:e820_sanitize: : [ff000 -> 100000] 
Reserved

So I suspect this is some interaction between xl's e820_host option
(which gets automatically enabled if you give a PCI device) and the
kernels early memory layout stuff. I'm CCing David and Konrad in the
hopes they have an idea what is happening.

Ian.

> 
> >Is the 512MB number coming only from within the guest or is it also
> >visible in e.g. xl list?
> 
> with pci:
> xen xen # xl list
> Name                                        ID   Mem VCPUs      State   
> Time(s)
> Domain-0                                     0  2557     4     r-----    
> 2928.2
> nodi                                        27   512     3     r-----      
> 34.1
> 
> --------------
> 
> without pci:
> xen xen # xl list
> Name                                        ID   Mem VCPUs      State   
> Time(s)
> Domain-0                                     0  2557     4     r-----    
> 2953.2
> nodi                                        28  2048     3     -b----       
> 0.6
> 
> >You could try ballooning up inside the guest,
> >by writing to the /sys/devices/system/xen_memory/xen_memory0/target_kb
> >sysfs node.
> 
> (with pci:)
> No effect. sysfs already shows 2GB, but is ignored for values above 512MB.
> 
> nodi ~ # cat /sys/devices/system/xen_memory/xen_memory0/target_kb
> 2097152
> nodi ~ # cat /proc/meminfo
> MemTotal:         511268 kB
> MemFree:          266328 kB
> Buffers:           16572 kB
> Cached:           152604 kB
> SwapCached:            0 kB
> Active:            65400 kB
> Inactive:         157904 kB
> Active(anon):      54180 kB
> Inactive(anon):      332 kB
> Active(file):      11220 kB
> Inactive(file):   157572 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> HighTotal:             0 kB
> HighFree:              0 kB
> LowTotal:         511268 kB
> LowFree:          266328 kB
> SwapTotal:        983036 kB
> SwapFree:         983036 kB
> Dirty:                 0 kB
> Writeback:             0 kB
> AnonPages:         54208 kB
> Mapped:            33852 kB
> Shmem:               388 kB
> Slab:               8552 kB
> SReclaimable:       3284 kB
> SUnreclaim:         5268 kB
> KernelStack:         760 kB
> PageTables:         1076 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:     1238668 kB
> Committed_AS:     237812 kB
> VmallocTotal:     339960 kB
> VmallocUsed:        7816 kB
> VmallocChunk:     331464 kB
> DirectMap4k:      524288 kB
> DirectMap2M:           0 kB
> nodi ~ # echo 1048576 > /sys/devices/system/xen_memory/xen_memory0/target_kb
> nodi ~ # cat /sys/devices/system/xen_memory/xen_memory0/target_kb
> 1048576
> nodi ~ # cat /proc/meminfo
> MemTotal:         511268 kB
> MemFree:          242920 kB
> Buffers:           16580 kB
> [...]
> 
> I was able to change to values below 512MB though. So the interface 
> is working at least.
> 
> >Lastly you should try a more recent domU kernel.
> 
> today all tests above done with 3.6.6-gentoo.
> Tried 2.6.38 and 3.4.11 yesterday.
> 
> Thanks for taking time,
> Christian
> 
> 



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.