Re: [Xen-devel] Xen & I/O in clusters - problems! New Information

[Rune Johan Andresen <runejoha@xxxxxxxxxxx>]

Hi, we have some additional information about our problem benchmarking 
Xen in clusters:
Between two Xen dom0 domains (between two physical computers in the 
cluster) we got these strange results:
(We use ttcp socketbuffsize, 10^4 -> 10^6)

Kernel 2.4:
Xen Dom0 -> Xen Dom0: ca. 65 000 KB/s

Kernel 2.6:
Xen Dom0 -> Xen Dom0: ca. 80 000 KB/s (?)

Native Linux:
Native Linux -> Native Linux: ca. 114 000 KB/s

What is new and strage is that Xen Dom0 use about 60% of the CPU when 
transfering or receiving, while
Native Linux ony use 6-7%(!) It seems like we have a problem with the 
DMA here(?). We use Xen 2.0,
Gigabit ethernet.

I tried 'mv /lib/tls /lib/tls.disabled' on each node without success

Under is the dmesg for the Xen node:

Linux version 2.6.8.1-xen0 (root@xxxxxxxxxxxxxxxxxxxx) (gcc version 
3.2.3 20030502 (Red Hat Linux 3.2.3-34)) #1 Tue Oct 12 14:10:47 GMT 
2004
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000008000000 (usable)
128MB LOWMEM available.
On node 0 totalpages: 32768
  DMA zone: 4096 pages, LIFO batch:1
  Normal zone: 28672 pages, LIFO batch:7
  HighMem zone: 0 pages, LIFO batch:1
DMI not present.
Built 1 zonelists
Kernel command line: root=/dev/sda1 ro console=tty0
Initializing CPU#0
PID hash table entries: 1024 (order 10: 8192 bytes)
Xen reported: 3400.171 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 125260k/131072k available (2636k kernel code, 5624k reserved, 
834k data, 396k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor 
mode... Ok.
Calibrating delay loop... 6789.52 BogoMIPS
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000
CPU: After vendor identify, caps:  bfebfbff 00000000 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: After all inits, caps:        beebcbe1 00000000 00000000 00000080
CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz stepping 09
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
NET: Registered protocol family 16
PCI: Using configuration type Xen
SCSI subsystem initialized
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Probing PCI hardware (bus 01)
PCI: Probing PCI hardware (bus 02)
PCI: Probing PCI hardware (bus 03)
PCI: Probing PCI hardware
Initializing Cryptographic API
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
Using anticipatory io scheduler
nbd: registered device at major 43
Intel(R) PRO/1000 Network Driver - version 5.2.52-k4
Copyright (c) 1999-2004 Intel Corporation.
PCI: Obtained IRQ 18 for device 0000:01:01.0
PCI: Setting latency timer of device 0000:01:01.0 to 64
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
PCI: Obtained IRQ 21 for device 0000:03:02.0
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
pcnet32.c:v1.30i 06.28.2004 tsbogend@xxxxxxxxxxxxxxxx
e100: Intel(R) PRO/100 Network Driver, 3.0.18
e100: Copyright(c) 1999-2004 Intel Corporation
Xen virtual console successfully installed as ttyS
Event-channel device installed.
Initialising Xen netif backend
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with 
idebus=xx
hda: SAMSUNG CD-ROM SN-124, ATAPI CD/DVD-ROM drive
ide1: I/O resource 0x170-0x177 not free.
ide1: ports already in use, skipping probe
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: ATAPI 24X CD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.20
PCI: Obtained IRQ 24 for device 0000:02:01.0
PCI: Obtained IRQ 25 for device 0000:02:01.1
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
        <Adaptec 3960D Ultra160 SCSI adapter>
        aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs

(scsi0:A:0): 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
  Vendor: SEAGATE   Model: ST336607LW        Rev: DS09
  Type:   Direct-Access                      ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
        <Adaptec 3960D Ultra160 SCSI adapter>
        aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs

Red Hat/Adaptec aacraid driver (1.1.2-lk2 Oct 12 2004)
3ware Storage Controller device driver for Linux v1.26.00.039.
3w-xxxx: No cards found.
libata version 1.02 loaded.
ata_piix version 1.02
ata_piix: combined mode detected
PCI: Obtained IRQ 17 for device 0000:00:1f.2
ata: 0x1f0 IDE port busy
PCI: Setting latency timer of device 0000:00:1f.2 to 64
ata1: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xFEA8 irq 15
ata1: SATA port has no device.
scsi2 : ata_piix
SCSI device sda: 71132959 512-byte hdwr sectors (36420 MB)
SCSI device sda: drive cache: write through
 sda: sda1 sda2 sda3 sda4 < sda5 >
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
mice: PS/2 mouse device common for all mice
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
md: raid0 personality registered as nr 2
md: raid1 personality registered as nr 3
md: raid5 personality registered as nr 4
raid5: automatically using best checksumming function: pIII_sse
   pIII_sse  :   440.400 MB/sec
raid5: using function: pIII_sse (440.400 MB/sec)
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
device-mapper: 4.1.0-ioctl (2003-12-10) initialised: dm@xxxxxxxxxxxxxx
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 8Kbytes
TCP: Hash tables configured (established 8192 bind 16384)
NET: Registered protocol family 1
NET: Registered protocol family 17
Bridge firewalling registered
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: sda1: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 4718
EXT3-fs: sda1: 1 orphan inode deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 396k freed

  ***************************************************************
  ***************************************************************
  ** WARNING: Currently emulating unsupported memory accesses  **
  **          in /lib/tls libraries. The emulation is very     **
  **          slow, and may not work correctly with all        **
  **          programs (e.g., some may 'Segmentation fault').  **
  **          TO ENSURE FULL PERFORMANCE AND CORRECT FUNCTION, **
  **          YOU MUST EXECUTE THE FOLLOWING AS ROOT:          **
  **          mv /lib/tls /lib/tls.disabled                    **
  ***************************************************************
  ***************************************************************

Pausing... 5Pausing... 4Pausing... 
3Pausing... 2Pausing... 
1Continuing...
EXT3 FS on sda1, internal journal
Adding 1020116k swap on /dev/sda3.  Priority:-1 extents:1
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
process `syslogd' is using obsolete setsockopt SO_BSDCOMPAT
process `snmpd' is using obsolete setsockopt SO_BSDCOMPAT

Cheers,
Rune




On Oct 15, 2004, at 1:24 AM, Mark A. Williamson wrote:

> > When we benchmark from a virtual domain  (running on Xen on a physical
> > node) to an another virtual domain (on another physical node) we get
> > 56.480 MByte/s (1:16)
> Ouch.  How are you benchmarking this?  (what tool, what parameters, 
> etc.).
> It'll help me reproduce this on our test systems.  Then we'll know if 
> it's
> your config or if there's something to track down.
>
> We did see some weird performance for small packets at one stage and 
> I'm not
> sure if that was ever resolved.  If it's the same problem, I can do a 
> binary
> chop search of changesets in order to locate it.
>
> Cheers,
> Mark
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on 
> ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give 
> us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out 
> more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxxxx
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel