Re: [Xen-devel] Read Performance issue when Xen Hypervisor is activated

[Michael Schinzel <schinzel@xxxxxxxxxxxxxx>]

Good Morning,

we have done some more tests in the last days. Tested different Mainboard 
Settings, ... 


As test we also look for some more points. We test the following commands also 
to get the information of the I/O Performance.


dd if=/dev/zero of=/root/testfile bs=1G count=2 oflag=direct
dd if=/dev/zero of=/root/testfile bs=512 count=4000 oflag=direct
ioping /dev/sda
hdparm -tT --direct /dev/sda

these is a better test case. 


When we boot the Hostsystem without Xen Support, we get the following results 
as reference:

root@v7:~# dd if=/dev/zero of=/root/testfile bs=1G count=2 oflag=direct

2+0 Datensätze ein
2+0 Datensätze aus
2147483648 Bytes (2,1 GB) kopiert, 5,16157 s, 416 MB/s

root@v7:~# dd if=/dev/zero of=/root/testfile bs=512 count=4000 oflag=direct
4000+0 Datensätze ein
4000+0 Datensätze aus
2048000 Bytes (2,0 MB) kopiert, 0,510736 s, 4,0 MB/s

root@v7:~# ioping /dev/sda
--- /dev/sda (block device 8.19 TiB) ioping statistics ---
21 requests completed in 20.7 s, 59 iops, 236.8 KiB/s
min/avg/max/mdev = 4.17 ms / 16.9 ms / 87.5 ms / 16.3 ms


root@v7:~# hdparm -tT --direct /dev/sda

/dev/sda:
 Timing O_DIRECT cached reads:   8394 MB in  2.00 seconds = 4198.15 MB/sec
 Timing O_DIRECT disk reads: 1414 MB in  3.00 seconds = 470.99 MB/sec





After this, i have also tested another Debian 8 Standard System with other 
Hardware specifications to check, if there is maybe an issue with Hardware 
side. 

Intel Xeon E5-2620v4
64 GB DDR4 ECC Reg
LSI 9361-8i RAID
BBU
2x 4 TB WD RE
4x 2 TB WD RE
2x 512 GB SSD

This Hardware is such the same like the Xen Node has. The Server is installed 
with Proxmox and Debian 8.  The HDDs are not connected by Backplane, they are 
added direct attached to the RAID Controller.

root@node-2-newcolo:~# dd if=/dev/zero of=/root/testfile bs=1G count=2 
oflag=direct
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 5.76812 s, 372 MB/s

root@node-2-newcolo:~# dd if=/dev/zero of=/root/testfile bs=512 count=4000 
oflag=direct
4000+0 records in
4000+0 records out
2048000 bytes (2.0 MB) copied, 0.491542 s, 4.2 MB/s

root@node-2-newcolo:~# ioping /dev/sdb
--- /dev/sdb (block device 3.64 TiB) ioping statistics ---
13 requests completed in 12.5 s, 25 iops, 103.0 KiB/s
min/avg/max/mdev = 149 us / 38.8 ms / 170.2 ms / 48.5 ms

root@node-2-newcolo:~# hdparm -tT --direct /dev/sdb

/dev/sdb:
 Timing O_DIRECT cached reads:   1294 MB in  2.00 seconds = 646.29 MB/sec
 Timing O_DIRECT disk reads: 1068 MB in  3.00 seconds = 355.87 MB/sec

This is ok, because there are some - 2/3 - VMs running.



After this, i have bootet the vServer Xen Node with the Xen Hypervisor Kernel. 

root@v7:~# dd if=/dev/zero of=/root/testfile bs=1G count=2 oflag=direct
2+0 Datensätze ein
2+0 Datensätze aus
2147483648 Bytes (2,1 GB) kopiert, 5,63201 s, 381 MB/s

root@v7:~# dd if=/dev/zero of=/root/testfile bs=512 count=4000 oflag=direct
4000+0 Datensätze ein
4000+0 Datensätze aus
2048000 Bytes (2,0 MB) kopiert, 0,522344 s, 3,9 MB/s

root@v7:~# ioping /dev/sda
--- /dev/sda (block device 8.19 TiB) ioping statistics ---
10 requests completed in 10.2 s, 41 iops, 165.3 KiB/s
min/avg/max/mdev = 9.26 ms / 24.2 ms / 115.1 ms / 30.4 ms

root@v7:~# hdparm -tT --direct /dev/sda

/dev/sda:
 Timing O_DIRECT cached reads:   4814 MB in  1.99 seconds = 2414.92 MB/sec
 Timing O_DIRECT disk reads: 1386 MB in  3.00 seconds = 461.41 MB/sec


You can see, in default Xen configuration, the most important thing at read 
performance test -> 2414.92 MB/sec <- the used cache is half of the cache like 
the same host is bootet without hypervisor. We now searched and searched and 
searched and find the Case: xen_acpi_processor

Xen is manageing the CPU Performance default with 1.200 Mhz. It is like you are 
driving a Ferrari all the time with 30 miles/h :) So we changed the Performance 
parameter to

 xenpm set-scaling-governor all performance

After this, the benchmark result speed a little bit up:

root@v7:~# dd if=/dev/zero of=/root/testfile bs=1G count=2 oflag=direct

2+0 Datensätze ein
2+0 Datensätze aus
2147483648 Bytes (2,1 GB) kopiert, 5,25885 s, 408 MB/s

root@v7:~# dd if=/dev/zero of=/root/testfile bs=512 count=4000 oflag=direct
4000+0 Datensätze ein
4000+0 Datensätze aus
2048000 Bytes (2,0 MB) kopiert, 0,23312 s, 8,8 MB/s

root@v7:~# ioping /dev/sda
--- /dev/sda (block device 8.19 TiB) ioping statistics ---
9 requests completed in 8.69 s, 73 iops, 293.9 KiB/s
min/avg/max/mdev = 10.3 ms / 13.6 ms / 22.3 ms / 3.65 ms

root@v7:~# hdparm -tT --direct /dev/sda

/dev/sda:
 Timing O_DIRECT cached reads:   6398 MB in  2.00 seconds = 3206.43 MB/sec
 Timing O_DIRECT disk reads: 1396 MB in  3.01 seconds = 464.53 MB/sec


There you see, the write performance speed up very nice. 8,8 MB/s and 293.9 
Kib/s for IOPS are such good performance information. The read Performance is a 
bit better, but not as good as with standard Kernel. You must see, there is 
nothing running at this Server.

After a little bit searching around, i also find a parameter for the scheduler.

root@v7:~# cat /sys/block/sda/queue/scheduler
noop deadline [cfq]

I changed the scheduler to deadline.  After this Change


root@v7:~#  dd if=/dev/zero of=/root/testfile bs=1G count=2 oflag=direct
2+0 Datensätze ein
2+0 Datensätze aus
2147483648 Bytes (2,1 GB) kopiert, 5,29884 s, 405 MB/s

root@v7:~# dd if=/dev/zero of=/root/testfile bs=512 count=4000 oflag=direct
4000+0 Datensätze ein
4000+0 Datensätze aus
2048000 Bytes (2,0 MB) kopiert, 0,209492 s, 9,8 MB/s

root@v7:~# ioping /dev/sda
--- /dev/sda (block device 8.19 TiB) ioping statistics ---
16 requests completed in 15.5 s, 52 iops, 210.8 KiB/s
min/avg/max/mdev = 4.44 ms / 19.0 ms / 109.2 ms / 23.8 ms

root@v7:~# hdparm -tT --direct /dev/sda

/dev/sda:
 Timing O_DIRECT cached reads:   6418 MB in  2.00 seconds = 3215.70 MB/sec
 Timing O_DIRECT disk reads: 1406 MB in  3.00 seconds = 468.61 MB/sec


It seems the write performance get a bit worser, but with small files, it 
speeds a little bit up. But ioping is now 210 instead of 293. At read 
Performance, there were no change.


With standard reading a file of 10 GB, i get now

root@v7:~# time dd if=datei of=/dev/null
20971520+0 Datensätze ein
20971520+0 Datensätze aus
10737418240 Bytes (11 GB) kopiert, 51,2435 s, 210 MB/s

real    0m51.246s
user    0m10.464s
sys     0m33.880s


all the time 210 MB/s. With default Kernel something arround over 400 MB/s. 
This is at reading a big file. I dont know what hdparm makes other instead of 
dd to get 468 MB/s. It seems there is already somethink in the hypervisor which 
makes for example irq management or sth else. 



The Kernel start Parameter is: GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=16192M 
dom0_max_vcpus=4 dom0_vcpus_pin cpufreq=dom0"

We have already tried to remove the CPU reservation, memory limit and so on but 
this don't change anythink. Also upgrading the Hypervisor dont change anythink 
at this performance issue. 
 


Mit freundlichen Grüßen

Michael Schinzel
- Geschäftsführer -


IP-Projects GmbH & Co. KG
Am Vogelherd 14
D - 97295 Waldbrunn 
Telefon: 09306 - 76499-0
FAX: 09306 - 76499-15
E-Mail: info@xxxxxxxxxxxxxx
Geschäftsführer: Michael Schinzel
Registergericht Würzburg: HRA 6798
Komplementär: IP-Projects Verwaltungs GmbH



-----Ursprüngliche Nachricht-----
Von: Dario Faggioli [mailto:dario.faggioli@xxxxxxxxxx] 
Gesendet: Freitag, 30. Dezember 2016 17:35
An: Michael Schinzel <schinzel@xxxxxxxxxxxxxx>; Xen-devel 
<xen-devel@xxxxxxxxxxxxxxxxxxxx>
Cc: Roger Pau Monne <roger.paumonne@xxxxxxxxxx>; Bob Liu <bob.liu@xxxxxxxxxx>
Betreff: Re: [Xen-devel] RE: Read Performance issue when Xen Hypervisor is 
activated

[Cc-ing someone which have done disk benchmark in somewhat recent time]

On Tue, 2016-12-27 at 14:26 +0000, Michael Schinzel wrote:
> We have searched in the last days more and more for the cause of this 
> performance issue.
>  
> In cooperation with the datacenter, we change some hardware to check, 
> if the problem already proceeds. We put the RAID Controller included 
> all RAID Arrays to another Supermicro Mainboard: X10SLM-F with only 
> one CPU. The result was, we got 400 MB/s read Speed. So it seems there 
> is an issue with the Servers Mainboard / CPU and the Xen Hypervisor 
> but, we also change the Mainboard to an Supermicro X9DR3-F with the 
> actual BIOS Version 3.2a – these also do not solved the problem with 
> the performance.
>  
> What we also have done:
> -          Upgraded Hypervisor from default Debian 8 – 4.4.1 to 4.8.
> -          Tested some kernel boot configurations\
>
I think it would be useful to know more about your configuration, e.g., are 
these tests being done in Dom0? How many vCPUs and memory does Dom0 have?

> With an non hypervisor Kernel, the system also uses the read Cache of 
> the controller and after some read operations at the same file, it 
> gets 1.2 G/s back from the Cache. At Xen Hypervisor Kernel, it seems 
> the system do not use any caching operations. I also tested a bit with 
> hdparm:
>  
> root@v7:~# hdparm -Tt /dev/sdb
>  
> /dev/sdb:
> Timing cached reads:   14060 MB in  1.99 seconds = 7076.16 MB/sec 
> Timing buffered disk reads: 304 MB in  3.01 seconds = 100.85 MB/sec
>  
> This Performance is horrable. It is a RAID 10 with read/write cache 
> and SSD Caching functions.
>  
> Does somebody know how Xen proceeds with such Caching Systems?
>  
>  
> Yours sincerely
>  
> Michael Schinzel
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, 
Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel