[Xen-users] Spectacularly disappointing disk throughput

[Chris Marget <chris@xxxxxxxxxx>]

Greetings!

I've got a FreeBSD-based (FreeNAS) appliance running as an HVM DomU.

Dom0 is Debian Squeeze on an AMD990 chipset system with IOMMU enabled.

The DomU sees six physical drives: one of them is a USB stick that I've passed through in its entirety as a block device. The other five are SATA drives attached to a controller that I've handed to the DomU with PCI passthrough.

The relevant parts of the DomU configuration are:

name = 'freenas-hvm'

kernel = "/usr/lib/xen-4.0/boot/hvmloader"

builder = 'hvm'

memory = 1024

shadow_memory = 8

vcpus = 1

device_model = '/usr/lib/xen-4.0/bin/qemu-dm'

disk = [ 'phy:/dev/sdc,hda,w' ]             # /dev/sdc is the USB stick

pci = [ '00:11.0' ]                                # This is the SATA controller with 5 drives

vif = ['bridge=vlan14' ]

boot='dc'

sdl=0

vnc=1

vnclisten='0.0.0.0'

vncconsole=1

stdvga=0

The SATA controller according to 'lspci':

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)

Everything "works", but is it painfully slow.

Reading from a single SATA drive within the DomU gives me about 0.5MB/s:

[root@freenas /dev]# dd if=/dev/ada1 of=/dev/null skip=100000 bs=4096 count=1000

1000+0 records in

1000+0 records out

4096000 bytes transferred in 8.058105 secs (508308 bytes/sec)

Concurrent reads from all five SATA drives show that they're able to achieve this speed all at the same time:

[root@freenas /dev]# for disk in ada1 ada2 ada3 ada4 ada5

> do dd if=/dev/$disk of=/dev/null bs=4096 count=1000 &

> done

4096000 bytes transferred in 8.049052 secs (508880 bytes/sec)

4096000 bytes transferred in 8.070050 secs (507556 bytes/sec)

4096000 bytes transferred in 8.071446 secs (507468 bytes/sec)

4096000 bytes transferred in 8.447751 secs (484863 bytes/sec)

4096000 bytes transferred in 8.501915 secs (481774 bytes/sec)

The USB stick, OTOH, passed through as a block device? It reads 18x faster at around 9MB/sec

[root@freenas /dev]# dd if=/dev/ada0 of=/dev/null bs=4096 count=1000

1000+0 records in

1000+0 records out

4096000 bytes transferred in 0.458198 secs (8939370 bytes/sec)

From the Dom0 I can read from the USB stick at around 15Mb/s (slow media), and I can read from all SATA drives at around 80-100MB/s concurrently (after un-hiding the PCI device).

If I pass the drives through individually (as I have done with the USB stick) the DomU reveals a 10MB/s ceiling. I can read from one disk at 10MB/s, or I can read from all at 2MB/s each.

Thoughts? Does this rotten behavior even make sense?

FreeBSD doesn't support PV mode on amd64, so that's out, but there are some PV drivers within HVM mode that I could be playing with. I don't really grok the details of it, but I don't think I have them working right now. I wonder if this is the ticket?

I'd appreciate any advice that would help me to improve the situation.

Thank you!

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users