Slow HVM start with direct kernel boot

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I observe much slower HVM start time than PVH, on Xen 4.14. It is using
qemu in (linux) stubdomain and direct kernel boot. The overall
difference is ~50s vs ~20s on some not particularly new laptop. On a
newer one I get 24s vs 6s, but I choose to debug it on the older one, as
the difference is more apparent. My measurements are taken with `time
qvm-start <vmname>`, which includes some (constant time) preparation,
starting the domain via libvirt and then waiting until system is fully
started.

I've identified several factors:
1. qemu is running in stubdomain there - it needs to start, and also
   get all domU's disk attached - slowness of block script is doubled
   here (4 disks, configured as block devices so the simplest case - it
   takes nearly 2s to run all the scripts)
2. Linux in HVM takes 64MB for SWIOTLB, which is significant given the
   domU has 400MB in total - this makes later system startup slower
   (less page cache, some swapping)
3. Finally, there is a huge delay between starting qemu (actually,
   last line of SeaBIOS output) and first output from Linux - about 10s or
   more.

For part of the debugging I've switched to qemu in dom0 - which reduced
point 1 above, but other points still applies.

What I've got so far for the above:

1. Some solutions discussed in
https://lore.kernel.org/xen-devel/YGyCECo8O0rwS8Z5@mail-itl/ and
https://xen.markmail.org/thread/ytikjjbs4kpihzi5. I've made the simplest
test of replacing the whole block script with a one-liner xenstore-write
(no locking, no sharing check etc), but it didn't helped much.

2. I don't have PCI passthrough there. Is the SWIOTLB needed there? Can
I disable it (I know how to reduce its size with swiotlb=, but is there
an option for disabling?)

3. This one, I discussed on IRC today and Anthony gave me this link:
https://lore.kernel.org/xen-devel/20180216204031.000052e9@xxxxxxxxx/
I've applied the change to qemu (fw_cfg_init_io -> fw_cfg_init_io_dma),
without any mentioned further work, and it does wonders already. The
kernel loading time is reduced from 10-15s, down to 1-2s.

That said, I/O handling in qemu seems to be slow. For example, a large
amount of time is spent on locking/unlocking map-cache. I believe this
may add to subjectively HVM feeling slower than PVH. Here is 'perf'
report collected while domU was loading kernel (via fw_cfg apparently):
https://gist.github.com/marmarek/350158269b77ebb8492d7c9392db686f


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature