Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail <benoit.depail@xxxxxxxxxxxxxx>]

On 07/20/17 10:52, Roger Pau Monné wrote:
> For the sake of the people that I've added, would you mind describing
> the issues you are seeing and the tests that you performed?
> 
> Thanks, Roger.
> 

Sure.

The main issue we are seeing is degraded write performance on storage
devices of Xen PV DomUs, about half (or even a third on our production
setup where NFS is involved) of what we used to have.

Our test setup is as follow (we left the NFS out as it has nothing to do
with our problem):

On a physical server running debian stretch as a Xen 4.8 hypervisor, we
create a PV domU, also running debian stretch. The storage of this domU
resides on the server's local disk as raw image files, which will be
setup as loop-devices before being exposed to the domU as /dev/xvd{a,b}.

the domU configuration (relevant part):

disk        = [
   'file:/mnt/domu_root.img,xvda,w',
   'file:/mnt/d-anb-nab2.img,xvdb,w'
]


When the domU is running, a loop-device is created:

/dev/loop1: [65027]:12 (/mnt/d-anb-nab2.img)

We perform a simple dd to test the sequential writing speed:

##### Dom0 with a kernel < 4.4.2 (here, 4.1.42, we had similar results
with 3.16.0)

# dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 47.5052 s, 113 MB/s

# dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.8227 s, 115 MB/s

On the domU:
# dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.7245 s, 115 MB/s


##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
d2081cfe624b5decaaf68088ca256ed1b140672c

On the dom0:
# dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s

# dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s

On the domU:
# dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s


For completeness sake, I'll put my findings below:

> I used git bisect on the linux-stable source tree to build (a lot of)
> tests kernels, and was able to find this commit as the first one
> introducing the regression :
>
> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
> commit d2081cfe624b5decaaf68088ca256ed1b140672c
> Author: Keith Busch <keith.busch@xxxxxxxxx>
> Date:   Tue Jan 12 15:08:39 2016 -0700
>
>     block: split bios to max possible length
>
> In term of kernel version, the first one showing bad performances in my
> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
>
> Interestingly, this commit is an improvement of
> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
> but do not show any performance issue in our case.
>
> I can also confirm that this issue is still present in the latest
> unstable kernel (we tested 4.13-rc1).


Thanks,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users