Hi,
I have encountered some strange problems when I was trying to PCI passthrough Broadcom 5709/5716 NICs to domUs allocated with more than 4G memory. Please see below for details.
My environment is:
Hardware Platform: DELL R210 with 2 Broadcom 5709 NICs and 2 Broadcom 5716 NICs
Xen: xen 4.2 unstable (64bits for hypervisor and 32bit for tools)
Kernel for both dom0 and domUs: xenified kernel 2.6.32.57 (32bit)
OS: CentOS 6.2 (32bit)
The general info regarding to xen can be get via below command
# xl info
host : 7.8
release : 2.6.32.57
version : #1 SMP Fri Jul 6 18:44:16 CST 2012
machine : i686
nr_cpus : 8
max_cpu_id : 31
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2660
hw_caps : bfebfbff:28100800:00000000:00003b40:0098e3fd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 8182
free_memory : 7046
sharing_freed_memory : 0
sharing_used_memory : 0
free_cpus : 0
xen_major : 4
xen_minor : 2
xen_extra : -unstable
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xff400000
xen_changeset : unavailable
xen_commandline : dom0_mem=1024M dom0_max_vcpus=2 dom0_vcpus_pin
cc_compiler : gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Thu Jul 12 11:20:56 CST 2012
xend_config_format : 4
case 1> I specified PV-domU config as below
memory=3072
maxmem=6144
name="bs"
vif=['ip=169.254.254.1,script=vif-nat',]
disk=['file:/root/bs.img,xvda1,w']
kernel='/root/vmlinuz'
extra="iommu=soft console=hvc0"
ramdisk='/root/initrd.img'
root="/dev/xvda1 ro"
pci=['01:00.0','01:00.1']
and then start domU. After that I executed below command
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 2 r----- 88.7
bs 2 3072 1 -b---- 1.1
It seemed normal to me. But when I logon bs domU, and executed below command
# cat /proc/meminfo | head
MemTotal: 6158940 kB
MemFree: 2944776 kB
Buffers: 5108 kB
Cached: 32292 kB
SwapCached: 0 kB
Active: 21456 kB
Inactive: 22936 kB
Active(anon): 7000 kB
Inactive(anon): 108 kB
Active(file): 14456 kB
It indicated the total memory was 6G, why?
When I back to dom0, I executed below command
# xl mem-set bs 6144
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 2 r----- 93.5
bs 2 6144 1 -b---- 10.5
It seemed normal to me. But when I logon bs domU again and executed below command
# cat /proc/meminfo | head
MemTotal: 9304668 kB
MemFree: 6087540 kB
Buffers: 5168 kB
Cached: 32464 kB
SwapCached: 0 kB
Active: 22300 kB
Inactive: 22408 kB
Active(anon): 7080 kB
Inactive(anon): 108 kB
Active(file): 15220 kB
It indicated total memory was 9G (6G + 3G). It was wired. Any idea about this?
Case 2> I specified PV-domU config as below
memory=6144
maxmem=6144
name="bs"
vif=['ip=169.254.254.1,script=vif-nat',]
disk=['file:/root/bs.img,xvda1,w']
kernel='/root/vmlinuz'
extra="iommu=soft console=hvc0"
ramdisk='/root/initrd.img'
root="/dev/xvda1 ro"
pci=['01:00.0','01:00.1']
and then start domU. After that I executed below command
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 648 2 r----- 120.5
bs 3 3360 1 -b---- 7.0
the output was very confusing. Why dom0 memory had been shrank to 648M and only 3360M assigned to bs domU?
My own analysis:
I extracted the bios e820 memory map on bs domU as below
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
[ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 00000000bf699000 (usable)
[ 0.000000] Xen: 00000000bf699000 - 00000000bf6af000 (reserved)
[ 0.000000] Xen: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
[ 0.000000] Xen: 00000000bf6ce000 - 00000000c0000000 (reserved)
[ 0.000000] Xen: 00000000e0000000 - 00000000f0000000 (reserved)
[ 0.000000] Xen: 00000000fe000000 - 0000000100000000 (reserved)
[ 0.000000] Xen: 0000000180000000 - 00000001c33ec000 (usable)
I think the root cause might be related to the holes between c0000000 and e0000000 and between f0000000 and fe000000 and between 100000000 and 180000000. And I think the e820_host option set according to my tracking.
Thanks in advance
HUAXIANG FAN
Software Engineer II
WEBSENSE NETWORK SECURITY TECHNOLOGY R&D (BEIJING) CO. LTD.
ph: +8610.5884.4327
fax: +8610.5884.4727
www.websense.cn
Websense TRITON™
For Essential Information Protection™
Web Security |
Data Security |
Email Security
