[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Spectacularly disappointing disk throughput

  • To: <xen-users@xxxxxxxxxxxxxxxxxxx>
  • From: <admin@xxxxxxxxxxx>
  • Date: Fri, 3 Feb 2012 12:14:00 -0600
  • Delivery-date: Fri, 03 Feb 2012 18:16:00 +0000
  • Importance: Normal
  • List-id: Xen user discussion <xen-users.lists.xensource.com>
  • Thread-index: AczinUFJZkA3MQnESOid65Svolb22AAAca4g

I would not read too much into these performance numbers.  I have found that FreeNAS is pretty slow even directly on physical hardware.  Using exactly the same hardware, OpenSolaris or Nexenta is way faster than FreeNAS.



Why do you want to run FreeNAS in a VM.  Is it for production purposes or testing purposes?  I would not recommend running a file server as a VM if it is a performance sensitive production situation.


-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Chris Marget
Friday, February 03, 2012 11:54 AM
To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] Spectacularly disappointing disk throughput




I've got a FreeBSD-based (FreeNAS) appliance running as an HVM DomU.


Dom0 is Debian Squeeze on an AMD990 chipset system with IOMMU enabled.


The DomU sees six physical drives: one of them is a USB stick that I've passed through in its entirety as a block device. The other five are SATA drives attached to a controller that I've handed to the DomU with PCI passthrough.


The relevant parts of the DomU configuration are:


name = 'freenas-hvm'

kernel = "/usr/lib/xen-4.0/boot/hvmloader"

builder = 'hvm'

memory = 1024

shadow_memory = 8

vcpus = 1

device_model = '/usr/lib/xen-4.0/bin/qemu-dm'

disk = [ 'phy:/dev/sdc,hda,w' ]             # /dev/sdc is the USB stick

pci = [ '00:11.0' ]                                # This is the SATA controller with 5 drives

vif = ['bridge=vlan14' ]








The SATA controller according to 'lspci':

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)


Everything "works", but is it painfully slow.


Reading from a single SATA drive within the DomU gives me about 0.5MB/s:

[root@freenas /dev]# dd if=/dev/ada1 of=/dev/null skip=100000 bs=4096 count=1000

1000+0 records in

1000+0 records out

4096000 bytes transferred in 8.058105 secs (508308 bytes/sec)


Concurrent reads from all five SATA drives show that they're able to achieve this speed all at the same time:

[root@freenas /dev]# for disk in ada1 ada2 ada3 ada4 ada5

> do dd if=/dev/$disk of=/dev/null bs=4096 count=1000 &

> done

4096000 bytes transferred in 8.049052 secs (508880 bytes/sec)

4096000 bytes transferred in 8.070050 secs (507556 bytes/sec)

4096000 bytes transferred in 8.071446 secs (507468 bytes/sec)

4096000 bytes transferred in 8.447751 secs (484863 bytes/sec)

4096000 bytes transferred in 8.501915 secs (481774 bytes/sec)


The USB stick, OTOH, passed through as a block device? It reads 18x faster at around 9MB/sec

[root@freenas /dev]# dd if=/dev/ada0 of=/dev/null bs=4096 count=1000

1000+0 records in

1000+0 records out

4096000 bytes transferred in 0.458198 secs (8939370 bytes/sec)


From the Dom0 I can read from the USB stick at around 15Mb/s (slow media), and I can read from all SATA drives at around 80-100MB/s concurrently (after un-hiding the PCI device).


If I pass the drives through individually (as I have done with the USB stick) the DomU reveals a 10MB/s ceiling. I can read from one disk at 10MB/s, or I can read from all at 2MB/s each.


Thoughts? Does this rotten behavior even make sense?


FreeBSD doesn't support PV mode on amd64, so that's out, but there are some PV drivers within HVM mode that I could be playing with. I don't really grok the details of it, but I don't think I have them working right now. I wonder if this is the ticket?


I'd appreciate any advice that would help me to improve the situation.


Thank you!

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.