[Xen-devel] [Xen Hackathon] benchmarks

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

Present:
- Konrad Wilk,
- Boris Ostrovsky,
- David Vrabel,
- Stefano Stabellini,
- Anthony Perard,
- George Dunlap,
- Dario Faggioli,
- Marc Zyngier,
- Srinivas Kalaga.

Oracle is investingating running benchmarks automatically against
upstream. Maybe with XenRT.

Need benchmarks testing integrated with OSSTest. We can identify
slowdowns: we don't want a commit that causes a slowdown by more than 5%
to be pushed. Dario is looking into this.

Which benchmarks?
1) CPU and memory
specint, specjbb, kernbench, hackbench
Need to test virtualization overhead with no contention and with
contention, test with many VMs to check contention.
Need to test native and KVM as comparison. We don't need to run
benchmarks on KVM and native for every commit.

2) Disk
Need to use SSDs or ramdisks or null block driver.
FIO, randread and randwrite separate would help identify regressions
but randrw would be more efficient.
Need to test with contention and without contention.

3) Network
Need 40gbe NIC. 
Need to test with contention and without contention.
VM to offhost transmit, VM from offhost receive, VM to VM on the same host.
Then the same again with contention (aggregate values for all of these).

4) Xen feature specific benchmarks
live-migration, save/restore, life cycle operations

Other interesting tests:
- bootstorm
lots of VMs starting at the same time
- PCI passthrough
Is there anything to test there? Maybe not.

Maybe we want a shorter set of tests for the push gate and a longer set
that are run only weekly.
Which ones we should run in OSSTests? Maybe kernbench and specint?
How do we compare to the previous run? Each machine has different
numbers. We need to calculate regressions per machine.

Test with a small dom0 (small amount of rams and vcpus) or a big dom0
(all the ram and as many vcpus as pcpus in the system)?

Need a wait to display the results, especially over time to see the
trends.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel