Re: [Xen-devel] Call to write tests for osstest

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Friday, August 9, 2013, 2:00:08 PM, you wrote:

> On Fri, Aug 09, 2013 at 01:46:42PM +0200, Sander Eikelenboom wrote:
>> 
>> Friday, August 9, 2013, 1:19:54 PM, you wrote:
>> 
>> > On gio, 2013-08-08 at 19:32 +0200, Sander Eikelenboom wrote:
>> >> * Some performance testing (network, block, cpu/mem/fork/real apps 
>> >> benchmarks, some other metrics)
>> >>    - perhaps makes separate graphs of these, so one can see performance 
>> >> increase or decrease over a larger time frame, and see at around what 
>> >> commits that occurred.
>> >>    - only after all basic tests succeeded and a push was done.
>> >> 
>> > We are after this too. I am looking at how to make it possible to do
>> > something like that on top of OSSTest.
>> 
>> > Perf benchmarking is a little bit different from regression
>> > smoke-testing, but still I think (hope? :-)) that most of the
>> > infrastructure can be reused.
>> 
>> Yes it does require to keep the rest of the circumstances the same for a 
>> longer period of time.
>> But it could be quite valuable, slow introduced and minor performance 
>> regressions are hard to discover.
>> 
>> > Also, what hardware to use and how to properly schedule these kind of
>> > "tests" is something that needs a bit more of thinking/discussion, I
>> > think.
>> 
>> Yes since you have to keep the "environment" the same, the machine should 
>> only run the perf tests at that time.
>> 

> This worries me. It is really hard to keep the "environment" the same.
> AIUI network / block etc. performance can be affected by other kernel
> subsystems. Also the kernel config could also have impact on
> performance.

True, though you probably should be able to make out a general trend when 
putting it in graphs.
And add a milestone when you significantly alter the config of a machine.

> Wei.

>> > Anyway, although I'm not committing to have all it 100% ready for 4.4,
>> > George, feel free to add this to your tracking list and put m name on
>> > it.
>> 
>> > Regards,
>> > Dario
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxx
>> http://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel