[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] PCI passthrough for domU allocated with more than 4G memory



Hi,

 

I have encountered some strange problems when I was trying to PCI passthrough Broadcom 5709/5716 NICs to domUs allocated with more than 4G memory. Please see below for details.

 

My environment is:

Hardware Platform: DELL R210 with 2 Broadcom 5709 NICs and 2 Broadcom 5716 NICs

Xen: xen 4.2 unstable (64bits for hypervisor and 32bit for tools)

Kernel for both dom0 and domUs: xenified kernel 2.6.32.57 (32bit)

OS: CentOS 6.2 (32bit)

 

The general info regarding to xen can be get via below command

# xl info

host                   : 7.8

release                : 2.6.32.57

version                : #1 SMP Fri Jul 6 18:44:16 CST 2012

machine                : i686

nr_cpus                : 8

max_cpu_id             : 31

nr_nodes               : 1

cores_per_socket       : 4

threads_per_core       : 2

cpu_mhz                : 2660

hw_caps                : bfebfbff:28100800:00000000:00003b40:0098e3fd:00000000:00000001:00000000

virt_caps              : hvm hvm_directio

total_memory           : 8182

free_memory            : 7046

sharing_freed_memory   : 0

sharing_used_memory    : 0

free_cpus              : 0

xen_major              : 4

xen_minor              : 2

xen_extra              : -unstable

xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64

xen_scheduler          : credit

xen_pagesize           : 4096

platform_params        : virt_start=0xff400000

xen_changeset          : unavailable

xen_commandline        : dom0_mem=1024M dom0_max_vcpus=2 dom0_vcpus_pin

cc_compiler            : gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)

cc_compile_by          : root

cc_compile_domain      :

cc_compile_date        : Thu Jul 12 11:20:56 CST 2012

xend_config_format     : 4

 

case 1> I specified PV-domU config as below

memory=3072

maxmem=6144

name="bs"

vif=['ip=169.254.254.1,script=vif-nat',]

disk=['file:/root/bs.img,xvda1,w']

kernel='/root/vmlinuz'

extra="iommu=soft console=hvc0"

ramdisk='/root/initrd.img'

root="/dev/xvda1 ro"

pci=['01:00.0','01:00.1']

 

and then start domU. After that I executed below command

# xl list

Name                                        ID   Mem VCPUs      State   Time(s)

Domain-0                                     0  1024     2     r-----      88.7

bs                                           2  3072     1     -b----       1.1

 

It seemed normal to me. But when I logon bs domU, and executed below command

# cat /proc/meminfo | head

MemTotal:        6158940 kB

MemFree:         2944776 kB

Buffers:            5108 kB

Cached:            32292 kB

SwapCached:            0 kB

Active:            21456 kB

Inactive:          22936 kB

Active(anon):       7000 kB

Inactive(anon):      108 kB

Active(file):      14456 kB

 

It indicated the total memory was 6G, why?

When I back to dom0, I executed below command

# xl mem-set bs 6144

# xl list

Name                                        ID   Mem VCPUs      State   Time(s)

Domain-0                                     0  1024     2     r-----      93.5

bs                                           2  6144     1     -b----      10.5

It seemed normal to me. But when I logon bs domU again and executed below command

# cat /proc/meminfo | head

MemTotal:        9304668 kB

MemFree:         6087540 kB

Buffers:            5168 kB

Cached:            32464 kB

SwapCached:            0 kB

Active:            22300 kB

Inactive:          22408 kB

Active(anon):       7080 kB

Inactive(anon):      108 kB

Active(file):      15220 kB

 

It indicated total memory was 9G (6G + 3G). It was wired. Any idea about this?

Case 2> I specified PV-domU config as below

memory=6144

maxmem=6144

name="bs"

vif=['ip=169.254.254.1,script=vif-nat',]

disk=['file:/root/bs.img,xvda1,w']

kernel='/root/vmlinuz'

extra="iommu=soft console=hvc0"

ramdisk='/root/initrd.img'

root="/dev/xvda1 ro"

pci=['01:00.0','01:00.1']

 

and then start domU. After that I executed below command

# xl list

Name                                        ID   Mem VCPUs      State   Time(s)

Domain-0                                     0   648     2     r-----     120.5

bs                                           3  3360     1     -b----       7.0

 

the output was very confusing. Why dom0 memory had been shrank to 648M and only 3360M assigned to bs domU?

 

My own analysis:

I extracted the bios e820 memory map on bs domU as below

[    0.000000] BIOS-provided physical RAM map:

[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)

[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)

[    0.000000]  Xen: 0000000000100000 - 00000000bf699000 (usable)

[    0.000000]  Xen: 00000000bf699000 - 00000000bf6af000 (reserved)

[    0.000000]  Xen: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)

[    0.000000]  Xen: 00000000bf6ce000 - 00000000c0000000 (reserved)

[    0.000000]  Xen: 00000000e0000000 - 00000000f0000000 (reserved)

[    0.000000]  Xen: 00000000fe000000 - 0000000100000000 (reserved)

[    0.000000]  Xen: 0000000180000000 - 00000001c33ec000 (usable)

 

I think the root cause might be related to the holes between c0000000 and e0000000 and between f0000000 and fe000000 and between 100000000 and 180000000. And I think the e820_host option set according to my tracking.

 

Thanks in advance

HUAXIANG FAN
Software Engineer II

WEBSENSE NETWORK SECURITY TECHNOLOGY R&D (BEIJING) CO. LTD.
ph: +8610.5884.4327
fax: +8610.5884.4727
www.websense.cn

Websense TRITON™
For Essential Information Protection™
Web Security | Data Security | Email Security



Protected by Websense Hosted Email Security — www.websense.com

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.