Re: [Xen-users] Spectacularly disappointing disk throughput

[<admin@xxxxxxxxxxx>]

I would not read too much into these performance numbers.  I have found that FreeNAS is pretty slow even directly on physical hardware.  Using exactly the same hardware, OpenSolaris or Nexenta is way faster than FreeNAS.

http://www.zfsbuild.com/2010/09/10/freenas-vs-opensolaris-zfs-benchmarks/

 

Why do you want to run FreeNAS in a VM.  Is it for production purposes or testing purposes?  I would not recommend running a file server as a VM if it is a performance sensitive production situation.

 

-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Chris Marget
Sent: Friday, February 03, 2012 11:54 AM
To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] Spectacularly disappointing disk throughput

 

Greetings!

 

I've got a FreeBSD-based (FreeNAS) appliance running as an HVM DomU.

 

Dom0 is Debian Squeeze on an AMD990 chipset system with IOMMU enabled.

 

The DomU sees six physical drives: one of them is a USB stick that I've passed through in its entirety as a block device. The other five are SATA drives attached to a controller that I've handed to the DomU with PCI passthrough.

 

The relevant parts of the DomU configuration are:

 

name = 'freenas-hvm'

kernel = "/usr/lib/xen-4.0/boot/hvmloader"

builder = 'hvm'

memory = 1024

shadow_memory = 8

vcpus = 1

device_model = '/usr/lib/xen-4.0/bin/qemu-dm'

disk = [ 'phy:/dev/sdc,hda,w' ]             # /dev/sdc is the USB stick

pci = [ '00:11.0' ]                                # This is the SATA controller with 5 drives

vif = ['bridge=vlan14' ]

boot='dc'

sdl=0

vnc=1

vnclisten='0.0.0.0'

vncconsole=1

stdvga=0

 

The SATA controller according to 'lspci':

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)

 

Everything "works", but is it painfully slow.

 

Reading from a single SATA drive within the DomU gives me about 0.5MB/s:

[root@freenas /dev]# dd if=/dev/ada1 of=/dev/null skip=100000 bs=4096 count=1000

1000+0 records in

1000+0 records out

4096000 bytes transferred in 8.058105 secs (508308 bytes/sec)

 

Concurrent reads from all five SATA drives show that they're able to achieve this speed all at the same time:

[root@freenas /dev]# for disk in ada1 ada2 ada3 ada4 ada5

> do dd if=/dev/$disk of=/dev/null bs=4096 count=1000 &

> done

4096000 bytes transferred in 8.049052 secs (508880 bytes/sec)

4096000 bytes transferred in 8.070050 secs (507556 bytes/sec)

4096000 bytes transferred in 8.071446 secs (507468 bytes/sec)

4096000 bytes transferred in 8.447751 secs (484863 bytes/sec)

4096000 bytes transferred in 8.501915 secs (481774 bytes/sec)

 

The USB stick, OTOH, passed through as a block device? It reads 18x faster at around 9MB/sec

[root@freenas /dev]# dd if=/dev/ada0 of=/dev/null bs=4096 count=1000

1000+0 records in

1000+0 records out

4096000 bytes transferred in 0.458198 secs (8939370 bytes/sec)

 

From the Dom0 I can read from the USB stick at around 15Mb/s (slow media), and I can read from all SATA drives at around 80-100MB/s concurrently (after un-hiding the PCI device).

 

If I pass the drives through individually (as I have done with the USB stick) the DomU reveals a 10MB/s ceiling. I can read from one disk at 10MB/s, or I can read from all at 2MB/s each.

 

Thoughts? Does this rotten behavior even make sense?

 

FreeBSD doesn't support PV mode on amd64, so that's out, but there are some PV drivers within HVM mode that I could be playing with. I don't really grok the details of it, but I don't think I have them working right now. I wonder if this is the ticket?

 

I'd appreciate any advice that would help me to improve the situation.

 

Thank you!

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users