[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Slow performance inside one VM


  • To: Adam Goryachev <mailinglists@xxxxxxxxxxxxxxxxxxxxxx>
  • From: Wiebe Cazemier <wiebe@xxxxxxxxxxxx>
  • Date: Mon, 9 Apr 2018 22:44:53 +0200 (CEST)
  • Cc: Xen-users <Xen-users@xxxxxxxxxxxxx>
  • Delivery-date: Mon, 09 Apr 2018 20:46:38 +0000
  • List-id: Xen user discussion <xen-users.lists.xenproject.org>
  • Thread-index: N56b/uW8G7FLP8h8MS5l9chCa0qVLg==
  • Thread-topic: Slow performance inside one VM

----- Original Message -----
> From: "Adam Goryachev" <mailinglists@xxxxxxxxxxxxxxxxxxxxxx>
> To: "Xen-users" <Xen-users@xxxxxxxxxxxxx>
> Sent: Friday, 6 April, 2018 03:33:03
> Subject: [Xen-users] Slow performance inside one VM

> I have three windows VM's (domU's) running on the one physical box (dom0).
> 
> 2 of them are Windows Server 2008R2, one of these is hardly used, but
> the other is heavily used.
> 
> 1 of them is Windows Server 2016, this one is reasonably heavily used
> (number of users) but performs well.
> 
> The VM with the performance problem has been happening for around 3
> months now, but I can't seem to find the issue. I've added additional
> memory, but then saw that this wasn't used, so reduced it back a bit.
> I've seen all CPU's (within the VM - taskmanager) pegged at 100% for
> periods of time, so I've added additional vCPU's and this has stopped
> that, but the "performance" issue is still present.
> 
> Currently, the config file looks like this:
> 
> builder        = 'hvm'
> memory        = 20480
> name        = "server1"
> hostname    = "server1"
> localtime    = 1
> vcpus        = 22        # How many Virtual CPU's to present
> viridian    = 1
> disk        = [ 'phy:/dev/ssd/server1,xvda,w' ]
> boot        = 'dc'
> vif        = ['bridge=xenbr0, mac=00:16:3e:43:ab:94']
> vnc        = 1
> vncunused    = 0
> vncdisplay    = 5
> vnclisten    = '0.0.0.0'
> stdvga        = 1
> #videoram    = 16
> usb        = 1
> usbdevice    = 'tablet'
> acpi        = 1
> apic        = 1
> on_reboot    = 'restart'
> on_poweroff    = 'destroy'
> on_crash    = 'restart'
> 
> The disk is a whole LV, using a RAID5 (Linux MD) from 3 x Intel SSD
> 
> I suspect the issue is a disk issue based on Windows Resource Monitor
> showing disk "Response Times" of over 30+ ms, up to 120ms I've seen.
> 
> From the dom0 I can see one of the qemu-system-i386 processes is
> sometimes over 50% cpu, but of the 4 CPU's allocated to dom0, none of
> them are over 20% utilisation.
> 
>  2133 ?        RLsl 1117:45 /usr/bin/qemu-system-i386 -xen-domid 1
> -chardev socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1,server,nowait
> -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev
> socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1,server,nowait
> -mon chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config
> -name server3 -vnc 0.0.0.0:7 -display none -device
> cirrus-vga,vgamem_mb=8 -boot order=cd -usb -usbdevice tablet -smp
> 12,maxcpus=12 -device rtl8139,id=nic0,netdev=net0,mac=00:16:3e:3c:d7:16
> -netdev type=tap,id=net0,ifname=vif1.0-emu,script=no,downscript=no
> -machine xenfv -m 24568 -drive
> file=/dev/ssd/server3,if=ide,index=0,media=disk,format=raw,cache=writeback
> 
> However, that process is from server3 (the win2016 domU)
> 
> All software is installed from debian packages, versions are:
> 
> dpkg -l | grep xen
> ii  grub-xen-bin 2.02~beta3-5                            amd64
> GRand Unified Bootloader, version 2 (Xen binaries)
> ii  grub-xen-host 2.02~beta3-5                            amd64
> GRand Unified Bootloader, version 2 (Xen host version)
> ii  libxen-4.4:amd64 4.4.1-9+deb8u10
> amd64        Public libs for Xen
> ii  libxen-4.8:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Public libs for Xen
> ii  libxenstore3.0:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Xenstore communications library for Xen
> ii  xen-hypervisor-4.4-amd64 4.4.1-9+deb8u10
> amd64        Xen Hypervisor on AMD64
> ii  xen-hypervisor-4.8-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Xen Hypervisor on AMD64
> ii  xen-linux-system-3.16.0-4-amd64
> 3.16.51-3                               amd64        Xen system with
> Linux 3.16 on 64-bit PCs (meta-package)
> ii  xen-linux-system-3.16.0-5-amd64
> 3.16.51-3+deb8u1                        amd64        Xen system with
> Linux 3.16 on 64-bit PCs (meta-package)
> ii  xen-linux-system-amd64 4.9+80+deb9u4
> amd64        Xen system with Linux for 64-bit PCs (dummy package)
> ii  xen-system-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Xen System on AMD64 (meta-package)
> ii  xen-utils-4.4 4.4.1-9+deb8u10                         amd64
> XEN administrative tools
> ii  xen-utils-4.8 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64
> XEN administrative tools
> ii  xen-utils-common 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> all          Xen administrative tools - common files
> ii  xenstore-utils 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64
> Xenstore command line utilities for Xen
> 
> xl info:
> 
> host                   : xen3
> release                : 4.9.0-6-amd64
> version                : #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02)
> machine                : x86_64
> nr_cpus                : 40
> max_cpu_id             : 39
> nr_nodes               : 2
> cores_per_socket       : 10
> threads_per_core       : 2
> cpu_mhz                : 2200
> hw_caps                :
> b7ebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
> virt_caps              : hvm hvm_directio
> total_memory           : 65403
> free_memory            : 8841
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 8
> xen_extra              : .3
> xen_version            : 4.8.3
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          :
> xen_commandline        : placeholder dom0_mem=6144M,max:6144M
> dom0_max_vcpus=4 dom0_vcpus_pin dom0_mem=6144M,max:6144M
> dom0_max_vcpus=4 dom0_vcpus_pin gnttab_max_frames=256
> cc_compiler            : gcc (Debian 6.3.0-18) 6.3.0 20170516
> cc_compile_by          : ijackson
> cc_compile_domain      : chiark.greenend.org.uk
> cc_compile_date        : Fri Mar  2 16:10:09 UTC 2018
> build_id               : dff6bad5189f35adc717d7989e1e2c87b87860cc
> xend_config_format     : 4
> 
> Within the VM, I've installed the drivers
> "Disk drive" displays as "XENSRC PVDISK SCSI Disk Device"
> "Storage Controller" displays as "Xen PV Storage Host Adapter" where the
> Driver shows Xen Project date 28/2/2017 version 8.2.0.9
> 
> Using iostat: /usr/bin/iostat -dmx /dev/sda /dev/sdb /dev/sdc /dev/md0
> /dev/md1 /dev/ssd/server1 /dev/ssd/server3 /dev/ssd/server2 1
> 
> Shows "% Util" is well under 20% (usually under 5%)
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdc               1.00    18.00    2.00   47.00     0.01 0.21
> 9.41     0.01    0.16    0.00    0.17   0.16   0.80
> sda               0.00    18.00    2.00   37.00     0.01 0.17
> 9.56     0.01    0.31    0.00    0.32   0.31   1.20
> sdb               0.00     2.00    2.00   39.00     0.01 0.12
> 6.37     0.02    0.39    0.00    0.41   0.39   1.60
> md1               0.00     0.00    0.00   41.00     0.00 0.19
> 9.24     0.00    0.00    0.00    0.00   0.00   0.00
> md0               0.00     0.00    0.00    0.00     0.00 0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> dm-6              0.00     0.00    0.00   14.00     0.00 0.04
> 6.50     0.02    1.14    0.00    1.14   0.57   0.80
> dm-10             0.00     0.00    0.00    0.00     0.00 0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> dm-18             0.00     0.00    0.00   20.00     0.00 0.14
> 14.00     0.01    0.60    0.00    0.60   0.60   1.20
> 
> I'm not sure what other information I should provide, and to be honest,
> I'm not sure what the bottleneck is. Users are seeing wait times of
> greater than 10 seconds to open an application, or new email at times,
> while other times it is near instant (less than 2s). All users are
> connecting via RDP to this server.
> 
> Any hints on how to prove that this is a disk performance issue, or CPU
> or memory, or my suspicion, is some xen overhead/inefficiency issue
> probably caused by misconfiguration on my part. The system did work
> really well prior to January, but we did also add a number of additional
> users to the system around that time.
> 
> Thank you for any assistance or ideas.
> 
> Regards,
> Adam
> 

You seem to have covered a lot of bases and I'm not sure what I can offer, 
especially without being at the server, but perhaps this issue I had will help 
you:

https://serverfault.com/questions/660575/which-scheduler-to-change-on-lvm-to-benefit-virtual-machines

Look at my own answer to the question.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.