Re: [win-pv-devel] Windows on Xen bad IO performance

[Paul Durrant <Paul.Durrant@xxxxxxxxxx>]

> -----Original Message-----
> From: win-pv-devel [mailto:win-pv-devel-bounces@xxxxxxxxxxxxxxxxxxxx] On
> Behalf Of Jakub Kulesza
> Sent: 31 July 2018 10:02
> To: win-pv-devel@xxxxxxxxxxxxxxxxxxxx
> Subject: Re: [win-pv-devel] Windows on Xen bad IO performance
> 
> 2018-07-31 9:51 GMT+02:00 Paul Durrant <Paul.Durrant@xxxxxxxxxx>:
> >
> > De-htmling... Responses below...
> >
> > -----
> > From: win-pv-devel [mailto:win-pv-devel-bounces@xxxxxxxxxxxxxxxxxxxx]
> On Behalf Of Jakub Kulesza
> > Sent: 30 July 2018 16:08
> > To: win-pv-devel@xxxxxxxxxxxxxxxxxxxx
> > Subject: [win-pv-devel] Windows on Xen bad IO performance
> >
> > I have a number of different hosts with different xen and windows
> versions, but they all share the same thing. Each time I install xen windows 
> pv
> drivers 8.2.0 from here: https://www.xenproject.org/developer...v-
> drivers.html I'm getting worse IO performance than before, on standard
> Windows drivers.
> >
> [cut]
> >
> > I found out that I need to modify the gnttab_max_frames parameter to
> the xen hypervisor at boottime. A lot of links and reading starts here:
> https://wiki.gentoo.org/wiki/Xen#Xen..._kernel_4.3.2B
> >
> > I did some testing and I am very confused right now. The
> gnttab_max_frames is by default 32 (increased to 64 in some xen version),
> and to solve the issues i would need to set it higher to 256. The results I 
> get
> seem to show something totally different.
> >
> > New test rig:
> > • ubuntu 18.04 LTS with everything from normal repositories, updated, xen
> 4.9
> > • i5-8500, 16GB ram, Samsung 850 evo SSD,
> > • windows 2016 installed on a LVM volume,
> > • xen pv drivers 8.2.0 installed on Windows,
> > • logged to the VM using VNC from a laptop in the same local network.
> >
> > I've tested this at a number of values of gnttab_max_frames from 4 to
> 4096.
> >
> > Passmark provides consistent results at around 510 MB/s READ, 305 MB/s
> WRITE, 330 MB/s Random ReadWrite, regardless of the setting of
> gnttab_max_frames. I guess that it does not saturate the grant tables
> mechanism of XEN that much. But with ATTO, the situation is sooo different.
> > • gnttab_max_frames = 4
> > o Windows is very snappy, responsive, even under heavy load from ATTO.
> > o Atto shows good results, with some signs of saturation with packets
> bigger than 512KB.
> > • gnttab_max_frames = 10
> > o Windows is very snappy but stops being responsive, even under heavy
> load from ATTO.
> > o Atto shows mediocre results, saturation is very high with packets bigger
> than 512KB.
> > • gnttab_max_frames = 64
> > o You can feel that the windows windows open a little bit slower, system
> feels dead with high load from ATTO.
> > o Atto shows bad results, saturation kills the system with packets bigger
> than 512KB. System is getting back OK after ATTO finishes.
> > • gnttab_max_frames = 256
> > o Even worse than 64, the results show similarity to 64, but the system just
> did not react. I fed up with waiting.
> > • gnttab_max_frames = 4096
> > o Windows did not boot. I just got fed up with waiting.
> [cut]
> 
> >
> > As discussed on IRC, it would be useful if you tried the 8.2.2 drivers and 
> > also
> highly useful if you could capture logging from QEMU.
> >
> > One other thing that occurs to me is that XENVBD implements indirect
> granting but this is relatively under tested because the only backend that
> implements it is blkback, and we don't use that in XenServer. Whilst is may
> be slower overall, you might get more stability using QEMU qdisk. (We have a
> couple of performance fixes for this in the pipeline in Citrix as we are now
> starting to use it as our default backend, but it should be reasonable as-is).
> >
> >   Paul
> 
> I did test 8.2.2 PV drivers. Did not managed to get QEMU logging thou.
> Will read more and retry.
> 
> Results on the i5-8500 rig - everything set the same as in the tests
> mentioned above:
> 
> https://imgur.com/gallery/PTm5f4G
> 
> gnttab_max_frames = 4:
> no signs or very little signs of saturation, everything is flying,
> scores are better than with 8.2.0
> 
> gnttab_max_frames = default for ubuntu 18.04 (so 32 or 64)
> saturation, system goes unresponsive, as bad as before
> 
> gnttab_max_frames = 256
> saturation, system goes unresponsive, as bad as before
> 
> Passmark shows better results on all gnttab_max_frames settings:
> Read: 514-515 (same as 8.2.0)
> Write: 477 (better!)
> Random ReadWrite: 300-360 (same as 8.2.0)
> 
> Is this behaviour (lowering max frames to get better results) working
> as expected?
> 
> How low should I NOT go with max_frames?

In general you should not be lowering it from the default. The only thing that 
will achieve is starving the guest frontend of grants. If it has having a 
positive impact then that indicates a problem with the frontend.

> 
> Does XenServer recommend any windows guest drivers if used with qemu
> backend?
> 

XenServer is basically using 8.2.1 plus some branding and workaround patches. 
We're likely to move to an 8.2.2 XENVBD though.

  Paul

> 
> --
> Pozdrawiam
> Jakub Kulesza
> 
> _______________________________________________
> win-pv-devel mailing list
> win-pv-devel@xxxxxxxxxxxxxxxxxxxx
> https://lists.xenproject.org/mailman/listinfo/win-pv-devel
_______________________________________________
win-pv-devel mailing list
win-pv-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/win-pv-devel