[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Spectacularly disappointing disk throughput


I've got a FreeBSD-based (FreeNAS) appliance running as an HVM DomU.

Dom0 is Debian Squeeze on an AMD990 chipset system with IOMMU enabled.

The DomU sees six physical drives: one of them is a USB stick that I've passed through in its entirety as a block device. The other five are SATA drives attached to a controller that I've handed to the DomU with PCI passthrough.

The relevant parts of the DomU configuration are:

name = 'freenas-hvm'
kernel = "/usr/lib/xen-4.0/boot/hvmloader"
builder = 'hvm'
memory = 1024
shadow_memory = 8
vcpus = 1
device_model = '/usr/lib/xen-4.0/bin/qemu-dm'
disk = [ 'phy:/dev/sdc,hda,w' ]             # /dev/sdc is the USB stick
pci = [ '00:11.0' ]                                # This is the SATA controller with 5 drives
vif = ['bridge=vlan14' ]

The SATA controller according to 'lspci':
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)

Everything "works", but is it painfully slow.

Reading from a single SATA drive within the DomU gives me about 0.5MB/s:
[root@freenas /dev]# dd if=/dev/ada1 of=/dev/null skip=100000 bs=4096 count=1000
1000+0 records in
1000+0 records out
4096000 bytes transferred in 8.058105 secs (508308 bytes/sec)

Concurrent reads from all five SATA drives show that they're able to achieve this speed all at the same time:
[root@freenas /dev]# for disk in ada1 ada2 ada3 ada4 ada5
> do dd if=/dev/$disk of=/dev/null bs=4096 count=1000 &
> done
4096000 bytes transferred in 8.049052 secs (508880 bytes/sec)
4096000 bytes transferred in 8.070050 secs (507556 bytes/sec)
4096000 bytes transferred in 8.071446 secs (507468 bytes/sec)
4096000 bytes transferred in 8.447751 secs (484863 bytes/sec)
4096000 bytes transferred in 8.501915 secs (481774 bytes/sec)

The USB stick, OTOH, passed through as a block device? It reads 18x faster at around 9MB/sec
[root@freenas /dev]# dd if=/dev/ada0 of=/dev/null bs=4096 count=1000
1000+0 records in
1000+0 records out
4096000 bytes transferred in 0.458198 secs (8939370 bytes/sec)

From the Dom0 I can read from the USB stick at around 15Mb/s (slow media), and I can read from all SATA drives at around 80-100MB/s concurrently (after un-hiding the PCI device).

If I pass the drives through individually (as I have done with the USB stick) the DomU reveals a 10MB/s ceiling. I can read from one disk at 10MB/s, or I can read from all at 2MB/s each.

Thoughts? Does this rotten behavior even make sense?

FreeBSD doesn't support PV mode on amd64, so that's out, but there are some PV drivers within HVM mode that I could be playing with. I don't really grok the details of it, but I don't think I have them working right now. I wonder if this is the ticket?

I'd appreciate any advice that would help me to improve the situation.

Thank you!
Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.