|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] Slow performance inside one VM
I have three windows VM's (domU's) running on the one physical box (dom0).2 of them are Windows Server 2008R2, one of these is hardly used, but the other is heavily used. 1 of them is Windows Server 2016, this one is reasonably heavily used (number of users) but performs well. The VM with the performance problem has been happening for around 3 months now, but I can't seem to find the issue. I've added additional memory, but then saw that this wasn't used, so reduced it back a bit. I've seen all CPU's (within the VM - taskmanager) pegged at 100% for periods of time, so I've added additional vCPU's and this has stopped that, but the "performance" issue is still present. Currently, the config file looks like this: builder = 'hvm' memory = 20480 name = "server1" hostname = "server1" localtime = 1 vcpus = 22 # How many Virtual CPU's to present viridian = 1 disk = [ 'phy:/dev/ssd/server1,xvda,w' ] boot = 'dc' vif = ['bridge=xenbr0, mac=00:16:3e:43:ab:94'] vnc = 1 vncunused = 0 vncdisplay = 5 vnclisten = '0.0.0.0' stdvga = 1 #videoram = 16 usb = 1 usbdevice = 'tablet' acpi = 1 apic = 1 on_reboot = 'restart' on_poweroff = 'destroy' on_crash = 'restart' The disk is a whole LV, using a RAID5 (Linux MD) from 3 x Intel SSDI suspect the issue is a disk issue based on Windows Resource Monitor showing disk "Response Times" of over 30+ ms, up to 120ms I've seen. From the dom0 I can see one of the qemu-system-i386 processes is sometimes over 50% cpu, but of the 4 CPU's allocated to dom0, none of them are over 20% utilisation. 2133 ? RLsl 1117:45 /usr/bin/qemu-system-i386 -xen-domid 1 -chardev socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1,server,nowait -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1,server,nowait -mon chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config -name server3 -vnc 0.0.0.0:7 -display none -device cirrus-vga,vgamem_mb=8 -boot order=cd -usb -usbdevice tablet -smp 12,maxcpus=12 -device rtl8139,id=nic0,netdev=net0,mac=00:16:3e:3c:d7:16 -netdev type=tap,id=net0,ifname=vif1.0-emu,script=no,downscript=no -machine xenfv -m 24568 -drive file=/dev/ssd/server3,if=ide,index=0,media=disk,format=raw,cache=writeback However, that process is from server3 (the win2016 domU) All software is installed from debian packages, versions are: dpkg -l | grep xenii grub-xen-bin 2.02~beta3-5 amd64 GRand Unified Bootloader, version 2 (Xen binaries) ii grub-xen-host 2.02~beta3-5 amd64 GRand Unified Bootloader, version 2 (Xen host version) ii libxen-4.4:amd64 4.4.1-9+deb8u10 amd64 Public libs for Xen ii libxen-4.8:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 Public libs for Xen ii libxenstore3.0:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 Xenstore communications library for Xen ii xen-hypervisor-4.4-amd64 4.4.1-9+deb8u10 amd64 Xen Hypervisor on AMD64 ii xen-hypervisor-4.8-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 Xen Hypervisor on AMD64 ii xen-linux-system-3.16.0-4-amd64 3.16.51-3 amd64 Xen system with Linux 3.16 on 64-bit PCs (meta-package) ii xen-linux-system-3.16.0-5-amd64 3.16.51-3+deb8u1 amd64 Xen system with Linux 3.16 on 64-bit PCs (meta-package) ii xen-linux-system-amd64 4.9+80+deb9u4 amd64 Xen system with Linux for 64-bit PCs (dummy package) ii xen-system-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 Xen System on AMD64 (meta-package) ii xen-utils-4.4 4.4.1-9+deb8u10 amd64 XEN administrative tools ii xen-utils-4.8 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 XEN administrative tools ii xen-utils-common 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 all Xen administrative tools - common files ii xenstore-utils 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 Xenstore command line utilities for Xen xl info: host : xen3 release : 4.9.0-6-amd64 version : #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) machine : x86_64 nr_cpus : 40 max_cpu_id : 39 nr_nodes : 2 cores_per_socket : 10 threads_per_core : 2 cpu_mhz : 2200hw_caps : b7ebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100 virt_caps : hvm hvm_directio total_memory : 65403 free_memory : 8841 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 8 xen_extra : .3 xen_version : 4.8.3xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset :xen_commandline : placeholder dom0_mem=6144M,max:6144M dom0_max_vcpus=4 dom0_vcpus_pin dom0_mem=6144M,max:6144M dom0_max_vcpus=4 dom0_vcpus_pin gnttab_max_frames=256 cc_compiler : gcc (Debian 6.3.0-18) 6.3.0 20170516 cc_compile_by : ijackson cc_compile_domain : chiark.greenend.org.uk cc_compile_date : Fri Mar 2 16:10:09 UTC 2018 build_id : dff6bad5189f35adc717d7989e1e2c87b87860cc xend_config_format : 4 Within the VM, I've installed the drivers "Disk drive" displays as "XENSRC PVDISK SCSI Disk Device""Storage Controller" displays as "Xen PV Storage Host Adapter" where the Driver shows Xen Project date 28/2/2017 version 8.2.0.9 Using iostat: /usr/bin/iostat -dmx /dev/sda /dev/sdb /dev/sdc /dev/md0 /dev/md1 /dev/ssd/server1 /dev/ssd/server3 /dev/ssd/server2 1 Shows "% Util" is well under 20% (usually under 5%)Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdc 1.00 18.00 2.00 47.00 0.01 0.21 9.41 0.01 0.16 0.00 0.17 0.16 0.80 sda 0.00 18.00 2.00 37.00 0.01 0.17 9.56 0.01 0.31 0.00 0.32 0.31 1.20 sdb 0.00 2.00 2.00 39.00 0.01 0.12 6.37 0.02 0.39 0.00 0.41 0.39 1.60 md1 0.00 0.00 0.00 41.00 0.00 0.19 9.24 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-6 0.00 0.00 0.00 14.00 0.00 0.04 6.50 0.02 1.14 0.00 1.14 0.57 0.80 dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-18 0.00 0.00 0.00 20.00 0.00 0.14 14.00 0.01 0.60 0.00 0.60 0.60 1.20 I'm not sure what other information I should provide, and to be honest, I'm not sure what the bottleneck is. Users are seeing wait times of greater than 10 seconds to open an application, or new email at times, while other times it is near instant (less than 2s). All users are connecting via RDP to this server. Any hints on how to prove that this is a disk performance issue, or CPU or memory, or my suspicion, is some xen overhead/inefficiency issue probably caused by misconfiguration on my part. The system did work really well prior to January, but we did also add a number of additional users to the system around that time. Thank you for any assistance or ideas. Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au -- The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this message in error, please notify us immediately. Please also destroy and delete the message from your computer. Viruses - Any loss/damage incurred by receiving this email is not the sender's responsibility. _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxxx https://lists.xenproject.org/mailman/listinfo/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |