Re: [MirageOS-devel] performance regression tests

[Jon Ludlam <jjl25@xxxxxxxxx>]

On Sun, Feb 15, 2015 at 02:16:15PM +0000, Richard Mortier wrote:
> oh cool-- i'll give that a go sometime then (recently nuked all my
> vbox vms so have to rebuild anyway).
> 
> blobs might be useful, or perhaps update the vagrant-vms repo with
> appropriate build scripts?
>
 
I've updated imduffy15's xenserver packer scripts to the latest 6.5 release of 
XS - the result is here:

https://github.com/jonludlam/packer-xenserver

I can probably integrate it with the vagrant-vms repo once I've ironed out the 
kinks. I don't think I have access to blobs now, though I've just had to 
reinstall this computer so it's conceivable I've lost an ssh key somewhere. In 
any case, the box doesn't take long to build. It boots OK and the toolstack is 
all running and I can run unikernels, but I'm having difficulty with forwarded 
ports right now. I'll give it another go tomorrow.

Jon


> On 15 February 2015 at 13:54, Jon Ludlam <jjl25@xxxxxxxxx> wrote:
> > Virtualbox should work actually. I did have a vagrant box with xenserver 
> > 6.2 sp1 on it, but it was hosted on jon.recoil.org which has now been 
> > replaced with a tiny unikernel. I can try to dig it up this evening to see 
> > if it still works. I could stick it on blobs.openmirage.org maybe?
> >
> > Jon
> >
> > Sent from my iPad
> >
> >> On 15 Feb 2015, at 12:50, Richard Mortier <richard.mortier@xxxxxxxxxxxx> 
> >> wrote:
> >>
> >> yes, it's still the 2014/15 academic year :p
> >>
> >> i suspect i know the answer to this but just in case: i presume by
> >> "bare metal" you really do mean bare metal? ie., inside a virtualbox
> >> is not going to work?
> >>
> >> masoud-- if that's the case, i'll pick this up with you offline.
> >> there's a server in nottingham that you should be able to get access
> >> to and do what you want with. i originally put xen on it by hand and
> >> used the xm/xl tools, but it sounds like it might be time to wipe it
> >> and start again with xenserver.
> >>
> >>
> >>> On 15 February 2015 at 12:18, David Scott <scott.dj@xxxxxxxxx> wrote:
> >>>
> >>>
> >>> On Sun, Feb 15, 2015 at 11:52 AM, Richard Mortier
> >>> <richard.mortier@xxxxxxxxxxxx> wrote:
> >>>>
> >>>>> On 15 February 2015 at 11:28, Anil Madhavapeddy <anil@xxxxxxxxxx> wrote:
> >>>>> On 15 Feb 2015, at 11:19, Richard Mortier <richard.mortier@xxxxxxxxxxxx>
> >>>>> wrote:
> >>>>>>
> >>>>>> On 15 February 2015 at 11:15, Anil Madhavapeddy <anil@xxxxxxxxxx>
> >>>>>> wrote:
> >>>>>>> This is great to see -- thanks for working on this Masoud!
> >>>>>>>
> >>>>>>> In particular, having even a simple iperf test would let us test
> >>>>>>> several interesting combinations straight away:
> >>>>>>>
> >>>>>>> - vary OCaml version (4.01 vs 4.02 is now supported), and there
> >>>>>>> is an experimental inlining branch in 4.03dev that could directly
> >>>>>>> be tested using this infrastructure.
> >>>>>>
> >>>>>> ok-- so i guess that needs a switch/env var to specify the `opam
> >>>>>> switch` to use?
> >>>>>
> >>>>> Yeah.  Although from the performance scripts' perspective, it's better
> >>>>> if they just assume that there is a working OPAM environment.  It would
> >>>>> be easier to control these parameters from outside, and keep the perf
> >>>>> scripts as easy to run as possible.
> >>>>
> >>>> ah-- original thinking was to make `mir-perf.sh` a harness that might
> >>>> be fired by `git-bisect`.
> >>>>
> >>>> if it's just to be a script to run a single experiment (parameter set)
> >>>> in a pre-configured environment, then that's a lot simpler.
> >>>>
> >>>>> A couple of other things that might help:
> >>>>>
> >>>>> - Luke Dunstan has a rather comprehensive acceptance test suite for
> >>>>>  MDNS:
> >>>>> https://github.com/infidel/ocaml-mdns/tree/master/lib_test/acceptance
> >>>>>
> >>>>> - OCamlPro has a benchmarking system for core OCaml here that may have
> >>>>>  some useful libraries: https://github.com/OCamlPro/operf-macro
> >>>>>
> >>>>> - Performance tests could be wrapped using Core_bench, which
> >>>>>  does linear regression across runs.
> >>>>>
> >>>>> https://realworldocaml.org/v1/en/html/understanding-the-garbage-collector.html#the-mutable-write-barrier
> >>>>>
> >>>>
> >>>> cool, ta.
> >>>>
> >>>>>>> - evaluate the impact of some features incoming such as the open
> >>>>>>> RFC for checksum offload.
> >>>>>>
> >>>>>> how are they specified -- as a PR?
> >>>>>> in which case, masoud-- i guess that
> >>>>>> https://help.github.com/articles/checking-out-pull-requests-locally/
> >>>>>> is a starting point for how to specify a particular PR rather than
> >>>>>> simply a commit rev.
> >>>>>
> >>>>> Yes, although again this would be better done outside the performance
> >>>>> harness as an OPAM pin for the local environment.  Just having the
> >>>>> ability to quickly run a performance test would be invaluable at this
> >>>>> stage.
> >>>>
> >>>> so the caller of the script will
> >>>>
> >>>> + select their opam switch
> >>>> + pin any libraries, whether to PRs or particular commit-revs
> >>>> + record the environment configuration
> >>>>
> >>>> + then run the script, specifying the test to run, which will
> >>>>  + start the unikernel-under-test
> >>>>  + start any testing-unikernels (eg., iperf-client, iperf-server)
> >>>>  + collect and record test results
> >>>>
> >>>> ...right?
> >>>>
> >>>> ultimately i was thinking to have the recorded test results put in a
> >>>> repo somewhere too, so that they could be visualised, compared to a
> >>>> benchmark, etc.
> >>>>
> >>>>>> just to be clear -- you mean does iperf via unikernels, ie., all the
> >>>>>> tests that are executed should also be unikernels rather than standard
> >>>>>> tools so that we don't take dependencies on an underlying platform
> >>>>>> like the dom0?
> >>>>>
> >>>>> Yes -- Xen unikernels would be the primary target.
> >>>>
> >>>> cool. less shell hacking, more unikernel hacking :)
> >>>>
> >>>>> Starting and stopping VMs in open source Xen can be bit of a pain, so
> >>>>> it would be ok if the test harness used XenServer (which the ARM SDcard
> >>>>> images now include).  Jon or Dave could comment on the state of the
> >>>>> XMLRPC
> >>>>> OCaml bindings to XenServer...
> >>>>
> >>>> that would be useful. is it easy to get xenserver installed on x86 as
> >>>> well? (that's the platform masoud is mostly working on.)
> >>>
> >>>
> >>> x86 as well? What is this, 2014? :-) Your options are:
> >>>
> >>> 1. install the latest 6.5 release of xenserver on bare metal
> >>>
> >>> Pros: this is well tested and will work very well as a virtualisation
> >>> platform
> >>> Cons: it takes over a whole machine. To do unikernel development you'll 
> >>> need
> >>> to install a dev VM i.e. you can't do it in dom0. It's good practice to
> >>> leave dom0 alone anyway. Think of it like a black box which can run VMs.
> >>>
> >>> 2. build the latest version from source and install on a CentOS 6/7 or 
> >>> maybe
> >>> Ubuntu box
> >>>
> >>> Pros: you can do whatever you want with dom0
> >>> Cons: some aspect of running VMs/ managing networks / managing storage is
> >>> bound to not work and you'll need to debug it. There is little automated
> >>> testing of this configuration, I (also Euan, Jon) try to fix bugs when 
> >>> I/we
> >>> encounter them on a best-effort basis.
> >>>
> >>> If you want to focus on unikernel dev and test, I recommend option 1. If 
> >>> you
> >>> want a new hobby of debugging VM hosting software, then try option 2.
> >>>
> >>> In the long run ... we'll converge option 1 and option 2... but I wouldn't
> >>> wait.
> >>>
> >>> Cheers,
> >>> Dave
> >>>
> >>>>
> >>>>
> >>>>>> cool... (one day i must learn about the cambridge infrastructure
> >>>>>> machines :)
> >>>>>
> >>>>> Be careful what you wish for :)
> >>>>
> >>>> now i'm genuinely curious... :)
> >>>>
> >>>> --
> >>>> Richard Mortier
> >>>> richard.mortier@xxxxxxxxxxxx
> >>>>
> >>>> _______________________________________________
> >>>> MirageOS-devel mailing list
> >>>> MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> >>>> http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Dave Scott
> >>
> >>
> >>
> >> --
> >> Richard Mortier
> >> richard.mortier@xxxxxxxxxxxx
> >>
> >> _______________________________________________
> >> MirageOS-devel mailing list
> >> MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> >> http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
> 
> 
> 
> -- 
> Richard Mortier
> richard.mortier@xxxxxxxxxxxx

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel