From wg-test-framework-bounces@lists.xenproject.org Mon Nov 11 12:56:56 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 11 Nov 2013 12:56:56 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vfr32-0008Bi-Cf for archives@lists.xenproject.org; Mon, 11 Nov 2013 12:56:56 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vfr30-0008BH-RA; Mon, 11 Nov 2013 12:56:55 +0000 Received: from [85.158.139.211:63618] by server-12.bemta-5.messagelabs.com id 6E/2C-15833-614D0825; Mon, 11 Nov 2013 12:56:54 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-4.tower-206.messagelabs.com!1384174610!1253430!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 8093 invoked from network); 11 Nov 2013 12:56:52 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-4.tower-206.messagelabs.com with RC4-SHA encrypted SMTP; 11 Nov 2013 12:56:52 -0000 X-IronPort-AV: E=Sophos;i="4.93,677,1378857600"; d="scan'208,217";a="70554333" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 11 Nov 2013 12:56:50 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL01.citrite.net (10.13.107.78) with Microsoft SMTP Server (TLS) id 14.2.342.4; Mon, 11 Nov 2013 07:56:49 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Mon, 11 Nov 2013 13:55:05 +0100 From: Lars Kurth To: "wg-test-framework@lists.xenproject.org" , "advisory-board@lists.xenproject.org" Thread-Topic: [Minutes] First Test WG framework meeting in October Thread-Index: Ac7cliyYXPUFa+qhSeiqvi3390N/2A== Date: Mon, 11 Nov 2013 12:55:04 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.168] MIME-Version: 1.0 X-DLP: MIA2 Cc: "Slutz, Donald Christopher \(dslutz@verizon.com\)" , "Hart, Harry D \(hhart@verizon.com\)" , "Coulis, Demetrios \(Demetrios.Coulis@ca.com\)" Subject: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0800111578980940768==" Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org --===============0800111578980940768== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_" --_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all, sorry it has taken me so long to put this together. I set up the mailing li= st, and signed you all up. Please read the minutes AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences. Lars =3D=3D Attendees =3D=3D * Aravind Gopalakrishnan (AMD) * Anthony Liguori (AWS), Matt Wilson (AWS) * Chris Sheperd (Citrix) * Demetrios Coulis standing in for Allan Roberto (CA) * Greg Lutostanski (Calxeda) * Konrad Wilk (Oracle) * Lars Kurth (Chair, Xen Project) * Will Auld (Intel) I believe, but am not 100% sure that *Harry Hart and Don Slutz (Verizon) also stayed at the meeting {{Action|Don}} Please confirm whether you were at the meeting and whether V= erizon would in fact also participate in the WG and who the respective comp= any rep would be =3D=3D Agenda =3D=3D * Introductions * Purpose and Scope of Working Group :Lars: My view is that the WG provides oversight and guidance for creating = a test infrastructure for the Xen Project on behalf of the Xen Community. T= his is merely providing a stake in the ground to start a discussion. As such, the group needs to **Make proposals to the Advisory Board for funding, for example *** What do we need to set up such a system in terms of hosting space, mach= ines, ... *** Do we need a full-time resource employed by the LF to maintain and evol= ve the framework, ... ** Make proposals to the community and work with the community on a solutio= n that works for developers in the community *** Without community buy-in and creating something that the community actu= ally wants, we won't be able to improve upstream quality ** Make decisions to help bootstrap demo systems, proof of concepts, etc. ** The group will also need to approve Statement of Works or provide input = on hiring contractors/resource ** Influence or set ground rules to ensure that the money that the Advisory= Board invests delivers value to the community and the Advisory Board ** Highlight focus areas for investment: e.g. the group would decide on que= stions such as *** Are there focus areas for test coverage the WG/AB cares about? *** Can WG/AB members find resources to cover these areas? *** If not, do we need to use project funds to seed such areas? :Comment by Matt Wilson (AWS): I'd like to suggest that the AB and working = groups try to avoid making technology decisions. It makes sense for the WG = to decide where to make investments on behalf of Xen Project members (e.g. = where to invest through contractors / grants, capital investments, etc.). W= hile this might influence technology decisions, it should avoid trouble som= e engineering-by-committee problems. :My proposal would be to go through the list above and add/remove items and= examples. We do not need to make decisions on items listed above at this m= eeting (I see these as example to clarify the scope/purpose/boundaries of = the WG). I would then take the output and draft a charter which we can vote= on in the WG subsequently and to put it forward to the Advisory Board and = the wider community. * Status Update and scenarios going forward (Chris Shepherd) :We probably need to give a quick overview of what we have (there are quite= a few new people on the list) :Status update: ** The OSU Open Source Lab in principle agreed to set up a Test-As-A-Servic= e rig for Xen for a limited time ** The proposal is based on Citrix's internal XenRT system. This would be a= proof of concept aimed at informing further investment decisions. ** OSSTest is already running (however on Citrix servers, with limited acce= ss by the community) ... see *** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osst= est/ *** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-b= y-step/ * Meeting Mechanics : ** Meeting cadence/time/etc. ** Mailing lists =3D=3D=3D Introductions =3D=3D=3D Please feel free to expand and ask questions, etc. * Aravind Gopalakrishnan, works in the Server OS team at AMD and specialise= s in RAS features * Chris Sheperd, leads the Test Department for XenServer at Citrix * Will Auld is Performance Architect and Principal Engineer at Intel, but = also works closely with the Cloud Virtualization Test Group * Anthony Liguori is the QEMU project lead at Amazon * Demetrios Coulis, is product manager for AppLogic at CA * Greg Lutostanski is working in the Validation team (sorry, may have gotte= n this wrong) at Calxeda and is new to Xen * Konrad Wilk is Software Development Manager at Oracle, but also Xen maint= ainer in Linux * Lars Kurth is the community manager for the Xen Project and is volunteeri= ng to chair this working group until it becomes self-sustaining =3D=3D=3D Purpose and Scope of Working Group =3D=3D=3D We didn't really work through the items in the list in the agenda. We ended= up having a discussion. I didn't capture all of it - the discussion got q= uite engaged - and I was at taking notes. Please feel free to augment on th= e list and correct me. Lars: We started with the list of items in the agenda, but realized that ma= ybe we need to take a step back and check our assumptions. Anthony: * Raised the point that the introduction of KVM autotest (http://www.linux-= kvm.org/page/KVM-Autotest) has been problematic. Developers generally tend = to write test code if it fits into their development workflow (i.e. they ca= n run tests easily locally on their branch very easily). * System Test frameworks (such as Xen OSSTest and XenRT) which are run *aft= er* submission are more problematic and many devs tend to ignore them * From Anthony's experience, we also approached the OSSTest / XenRT discuss= ions wrongly. We should *not* just ask developers to write tests, but under= stand better what tests framework they would find attractive and provide so= mething which helps them. Lars: * We can fix the last point on engaging the community We then covered an angle on what is wrong with OSS test right now Konrad: stated that OSSTest is not well enough maintained and owned right = now (IanJ does this in his spare time). OSSTest or other systems the AB su= pports will need to be properly owned and funded, otherwise the system woul= d lose trust. Matt commented that he doesn't like the pushgate mechanism in OSSTest (we d= idn't cover this in detail) On the other hand ... running Coverity on the code base has been a huge suc= cess. Konrad: * In 2 months we had > 215 bug fixes * the quality of the xen core components according to coverity is now highe= r than that in the Linux kernel This implies that there is a willingness in the community to use tools to i= mprove quality. We then slipped into a discussion about goals: Anthony raised a few questio= ns * How do we determine how we (or what we propose) provides value to members= and the community * We would need to define measurable success criteria The conclusion we came to (note that we didn't vote) was * Understand what member companies want to get out of the framework * Understand what the community wants to get out of it (as value for compan= ies depends on community buy in). {{Action|Lars}} Carefully draft a mail to the devel lists (after sending to= this list for approval) along the lines of: * As AB we have resources that we can use to help the community * We are only willing to spend money, if we are confident enough that this = helps the community and is likely to be adopted * Here are a few options and what we think would help you [intended to seed= the discussion] * We are looking for volunteers in the community to work with the WG {{Action|All}} Each member company should provide a statement of what they = are trying to achieve, whether there are any specific test related items, c= onstraints, platforms, goals, etc. that they are care about '''Additional Notes: ''' I am also including some notes and discussions I had outside the WG meeting= at the bottom of this section. I had a conversation with a number of developers after the test talks * Anthony stated that maybe to have a test tool based on QEMU that allows p= eople to run some relevant tests locally would be best (problem: performanc= e). He also raised the point that both OSSTest and XenRT are just like auto= test. Why do we need yet another new test framework rather than using somet= hing which is out there already? * Ian Jackson stated that the main issue right now is that OSSTest is not p= roperly resourced and does have not good coverage. He spends 50% of his tim= e keeping OSSTest running and sometimes tracking down hardware issues. So w= hatever we do, resourcing needs to be resolved. * When I talked to Citrix Platform team members, it turned out that only a = few have actually used OSSTest (lack of documentation is the key issue). It= 's too hard to get started with OSSTest right now. * Another developer stated (source not disclosed): why don't the member com= panies fund the "creation of tests (regardless of test framework) if they c= are about quality" * Ian Jackson: there are some interesting properties about OSSTest (e.g. th= e capability to submit test code with a spec on the fly). [Aside by Lars: t= his may be close enough to providing the capability to try something locall= y, if we there also was a capability to test a devs personal git branch wit= h some test code] * Ian J also stated that he would look at XenRT, if the code was made avail= able in a Git repo rather than a tarball. Any issues and discussions could = then happen using the normal ways of discussing larger code contributions o= n xen-devel This clearly shows that we may have an issue winning the community over. =3D=3D=3D Status Update and scenarios going forward =3D=3D=3D (mostly by Chris Sheperd) http://www.slideshare.net/xen_com_mgr/xen-summit-taas-and-xenrt-0 shows a s= tatus update Documentation at http://wiki.xenproject.org/wiki/Category:XenRT Harry Hart: mentioned that Verizon tried XenRT but that they got stuck (Asi= de Lars: the fact that right now XenRT is sort of outside the Xen Project c= ommunity does not lend itself to discuss and raise issues about it) The main benefit of XenRT is that we inherit a large number of existing tes= ts (including for example performance tests and others) At the time of the meeting, the intention was to show a demo running at htt= p://osuosl.org/, but OSL had not set this up '''Update: 8/11/13: Sorry for dropping the ball on this. We have the system= s basically ready to go outside of getting public internet plumbed. Are you= still in need of them? Please let me know!''' The question now is whether a) Citrix is willing to spend some time on setting a test environment up (a= nd add support for xl) b) Whether others on this list think that a XenRT demo instance is a good i= dea My question to Chris: {{Action|Chris}} Determine whether Citrix is willing to set up XenRT on htt= p://osuosl.org/ (and add support for "xl") {{Action|Chris}} Let Lance Albertson from http://osuosl.org/ know Now as we do have a WG, I suggest to CC it if the answer to the The other question I have to the group (in light of the previous discussion= ) is: {{Action|All} Let me know by "replying +1" to this item whether you feel th= ere is value in setting up a XenRT demo instance on http://osuosl.org/; "re= plying 0" means you don't care; "replying -1" means you believe this is a b= ad idea (to satisfy Xen Project conventions you would have to justify why y= ou think this) =3D=3D=3D Meeting Mechaniscs =3D=3D=3D {{Action|All} Let me know what day and time of week you would be available= for a monthly call. All people on the list are based in the EU or US (East= to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Ple= ase state 2-3 preferences. --_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

 

Hi all,

sorry it has taken me so long to put this together. = I set up the mailing list, and signed you all up. Please read the minutes
AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences.

 

Lars

 

=3D=3D Attendees =3D=3D

* Aravind Gopalakrishnan (AMD)

* Anthony Liguori (AWS), Matt Wilson (AWS)

* Chris Sheperd (Citrix)

* Demetrios Coulis standing in for Allan Roberto (CA= )

* Greg Lutostanski (Calxeda)

* Konrad Wilk (Oracle)

* Lars Kurth (Chair, Xen Project)

* Will Auld (Intel)

I believe, but am not 100% sure that

*Harry Hart and Don Slutz (Verizon) also stayed at t= he meeting

 

{{Action|Don}} Please confirm whether you were at th= e meeting and whether Verizon would in fact also participate in the WG and = who the respective company rep would be

 

=3D=3D Agenda =3D=3D

 

* Introductions

* Purpose and Scope of Working Group

 

:Lars: My view is that the WG provides oversight and= guidance for creating a test infrastructure for the Xen Project on behalf = of the Xen Community. This is merely providing a stake in the ground to sta= rt a discussion.

 

As such, the group needs to

**Make proposals to the Advisory Board for funding, = for example

*** What do we need to set up such a system in terms= of hosting space, machines, …

*** Do we need a full-time resource employed by the = LF to maintain and evolve the framework, …

** Make proposals to the community and work with the= community on a solution that works for developers in the community

*** Without community buy-in and creating something = that the community actually wants, we won’t be able to improve upstre= am quality

** Make decisions to help bootstrap demo systems, pr= oof of concepts, etc.

** The group will also need to approve Statement of = Works or provide input on hiring contractors/resource

** Influence or set ground rules to ensure that the = money that the Advisory Board invests delivers value to the community and t= he Advisory Board

** Highlight focus areas for investment: e.g. the gr= oup would decide on questions such as

*** Are there focus areas for test coverage the WG/A= B cares about?

*** Can WG/AB members find resources to cover these = areas?

*** If not, do we need to use project funds to seed = such areas?

 

:Comment by Matt Wilson (AWS): I'd like to sugges= t that the AB and working groups try to avoid making technology decisions. = It makes sense for the WG to decide where to make investments on behalf of = Xen Project members (e.g. where to invest through contractors / grants, capital investments, etc.). While thi= s might influence technology decisions, it should avoid trouble some engine= ering-by-committee problems.

 

:My proposal would be to go through the list abov= e and add/remove items and examples. We do not need to make decisions on it= ems listed above at this meeting (I see these as example to clarify  t= he scope/purpose/boundaries of the WG). I would then take the output and draft a charter which we can vote on in t= he WG subsequently and to put it forward to the Advisory Board and the wide= r community.

 

* Status Update and scenarios going forward (Chris S= hepherd)

:We probably need to give a quick overview of what w= e have (there are quite a few new people on the list)

 

:Status update:

** The OSU Open Source Lab in principle agreed to se= t up a Test-As-A-Service rig for Xen for a limited time

** The proposal is based on Citrix’s internal = XenRT system. This would be a proof of concept aimed at informing further i= nvestment decisions.

** OSSTest is already running (however on Citrix ser= vers, with limited access by the community) … see

*** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/=

*** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/

 

* Meeting Mechanics :

** Meeting cadence/time/etc.

** Mailing lists

 

=3D=3D=3D Introductions =3D=3D=3D

Please feel free to expand and ask questions, etc.

* Aravind Gopalakrishnan, works in the Server OS tea= m at AMD and specialises in RAS features

* Chris Sheperd, leads the Test Department for XenSe= rver at Citrix

* Will Auld is Performance Architect and  Princ= ipal Engineer at Intel, but also works closely with the Cloud Virtualizatio= n Test Group

* Anthony Liguori is the QEMU project lead at Amazon=

* Demetrios Coulis, is product manager for AppLogic = at CA

* Greg Lutostanski is working in the Validation team= (sorry, may have gotten this wrong) at Calxeda and is new to Xen

* Konrad Wilk is Software Development Manager at Ora= cle, but also Xen maintainer in Linux

* Lars Kurth is the community manager for the Xen Pr= oject and is volunteering to chair this working group until it becomes self= -sustaining

 

=3D=3D=3D Purpose and Scope of Working Group  = =3D=3D=3D

 

We didn’t really work through the items in the= list in the agenda. We ended up having a discussion.  I didn’t = capture all of it - the discussion got quite engaged – and I was at t= aking notes. Please feel free to augment on the list and correct me.

 

Lars: We started with the list of items in the agend= a, but realized that maybe we need to take a step back and check our assump= tions.

 

Anthony:

* Raised the point that the introduction of KVM auto= test (http://www.lin= ux-kvm.org/page/KVM-Autotest) has been problematic. Developers generall= y tend to write test code if it fits into their development workflow (i.e. they can run tests easily locally on= their branch very easily).

* System Test frameworks (such as Xen OSSTest and Xe= nRT) which are run *after* submission are more problematic and many = devs tend to ignore them

* From Anthony’s experience, we also approache= d the OSSTest / XenRT discussions wrongly. We should *not* just ask = developers to write tests, but understand better what tests framework they = would find attractive and provide something which helps them.

 

Lars:

* We can fix the last point on engaging the communit= y

 

We then covered an angle on what is wrong with OSS t= est right now

 

Konrad:  stated that OSSTest is not well enough= maintained and owned right now (IanJ does this in his spare time).  O= SSTest or other systems the AB supports will need to be properly owned and = funded, otherwise the system would lose trust.

 

Matt commented that he doesn’t like the pushga= te mechanism in OSSTest (we didn’t cover this in detail)

 

On the other hand … running Coverity on the co= de base has been a huge success.

 

Konrad:

* In 2 months we had > 215 bug fixes

* the quality of the xen core components according t= o coverity is now higher than that in the Linux kernel

 

This implies that there is a willingness in the comm= unity to use tools to improve quality.

 

We then slipped into a discussion about goals: Antho= ny raised a few questions

* How do we determine how we (or what we propose) pr= ovides value to members and the community

* We would need to define measurable success criteri= a

 

The conclusion we came to (note that we didn’t= vote) was

* Understand what member companies want to get out o= f the framework

* Understand what the community wants to get out of = it (as value for companies depends on community buy in).

 

{{Action|Lars}} Carefully draft a mail to the devel = lists (after sending to this list for approval) along the lines of:

* As AB we have resources that we can use to help th= e community

* We are only willing to spend money, if we are conf= ident enough that this helps the community and is likely to be adopted=

* Here are a few options and what we think would hel= p you [intended to seed the discussion]

* We are looking for volunteers in the community to = work with the WG

 

{{Action|All}} Each member company should provide a = statement of what they are trying to achieve, whether there are any specifi= c test related items, constraints, platforms, goals, etc. that they are car= e about

 

’’’Additional Notes: ’’= ;’

I am also including some notes and discussions I had= outside the WG meeting at the bottom of this section.

 

I had a conversation with a number of developers aft= er the test talks

* Anthony stated that maybe to have a test tool base= d on QEMU that allows people to run some relevant tests locally would be be= st (problem: performance). He also raised the point that both OSSTest and X= enRT are just like autotest. Why do we need yet another new test framework rather than using something which i= s out there already?

* Ian Jackson stated that the main issue right now i= s that OSSTest is not properly resourced and does have not good coverage. H= e spends 50% of his time keeping OSSTest running and sometimes tracking dow= n hardware issues. So whatever we do,  resourcing needs to be resolved.

* When I talked to Citrix Platform team members, it = turned out that only a few have actually used OSSTest (lack of documentatio= n is the key issue). It’s too hard to get started with OSSTest right = now.

* Another developer stated (source not disclosed): w= hy don’t the member companies fund the “creation of tests (rega= rdless of test framework) if they care about quality”

* Ian Jackson: there are some interesting properties= about OSSTest (e.g. the capability to submit test code with a spec on the = fly). [Aside by Lars: this may be close enough to providing the capability = to try something locally, if we there also was a capability to test a devs personal git branch with some test co= de]

* Ian J also stated that he would look at XenRT, if = the code was made available in a Git repo rather than a tarball. Any issues= and discussions could then happen using the normal ways of discussing larg= er code contributions on xen-devel

 

This clearly shows that we may have an issue winning= the community over.

 

=3D=3D=3D Status Update and scenarios going forward&= nbsp; =3D=3D=3D

(mostly by Chris Sheperd)

 

http://www.slideshare.net/xen_com_mgr/xen-summit-= taas-and-xenrt-0 shows a status update

Documentation at http://wiki.xenproject.org/wiki/Category:XenRT

 

Harry Hart: mentioned that Verizon tried XenRT but t= hat they got stuck (Aside Lars: the fact that right now XenRT is sort of ou= tside the Xen Project community does not lend itself to discuss and raise i= ssues about it)

The main benefit of XenRT is that we inherit a large= number of existing tests (including for example performance tests and othe= rs)

At the time of the meeting, the intention was to sho= w a demo running at http://osuosl.org/, but OSL had not set = this up

 

‘’’Update: 8/11/13: Sorry for drop= ping the ball on this. We have the systems basically ready to go outside of= getting public internet plumbed. Are you still in need of them? Please let= me know!’’’

 

The question now is whether

a) Citrix is willing to spend some time on setting a= test environment up (and add support for xl)

b) Whether others on this list think that a XenRT de= mo instance is a good idea

 

My question to Chris:

{{Action|Chris}} Determine whether Citrix is willing= to set up XenRT on http://osuosl.org/ (and add support for = “xl”)

{{Action|Chris}} Let Lance Albertson from http://osuosl.org/  know

 

Now as we do have a WG, I suggest to CC it if the an= swer to the

 

The other question I have to the group (in light of = the previous discussion) is:

{{Action|All} Let me know by “replying +1&= #8221; to this item whether you feel there is value in setting up a XenRT d= emo instance on http://osuosl.org/; “replying 0= 221; means you don’t care; “replying -1“ means you believ= e this is a bad idea (to satisfy Xen Project conventions you would have to = justify why you think this)

 

=3D=3D=3D Meeting Mechaniscs  =3D=3D=3D

 

{{Action|All}  Let me know what day and time of= week you would be available for a monthly call. All people on the list are= based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot w= ould probably work best. Please state 2-3 preferences.

 

--_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_-- --===============0800111578980940768== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework --===============0800111578980940768==-- From wg-test-framework-bounces@lists.xenproject.org Mon Nov 11 12:56:56 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 11 Nov 2013 12:56:56 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vfr32-0008Bi-Cf for archives@lists.xenproject.org; Mon, 11 Nov 2013 12:56:56 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vfr30-0008BH-RA; Mon, 11 Nov 2013 12:56:55 +0000 Received: from [85.158.139.211:63618] by server-12.bemta-5.messagelabs.com id 6E/2C-15833-614D0825; Mon, 11 Nov 2013 12:56:54 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-4.tower-206.messagelabs.com!1384174610!1253430!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 8093 invoked from network); 11 Nov 2013 12:56:52 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-4.tower-206.messagelabs.com with RC4-SHA encrypted SMTP; 11 Nov 2013 12:56:52 -0000 X-IronPort-AV: E=Sophos;i="4.93,677,1378857600"; d="scan'208,217";a="70554333" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 11 Nov 2013 12:56:50 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL01.citrite.net (10.13.107.78) with Microsoft SMTP Server (TLS) id 14.2.342.4; Mon, 11 Nov 2013 07:56:49 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Mon, 11 Nov 2013 13:55:05 +0100 From: Lars Kurth To: "wg-test-framework@lists.xenproject.org" , "advisory-board@lists.xenproject.org" Thread-Topic: [Minutes] First Test WG framework meeting in October Thread-Index: Ac7cliyYXPUFa+qhSeiqvi3390N/2A== Date: Mon, 11 Nov 2013 12:55:04 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.168] MIME-Version: 1.0 X-DLP: MIA2 Cc: "Slutz, Donald Christopher \(dslutz@verizon.com\)" , "Hart, Harry D \(hhart@verizon.com\)" , "Coulis, Demetrios \(Demetrios.Coulis@ca.com\)" Subject: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0800111578980940768==" Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org --===============0800111578980940768== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_" --_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all, sorry it has taken me so long to put this together. I set up the mailing li= st, and signed you all up. Please read the minutes AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences. Lars =3D=3D Attendees =3D=3D * Aravind Gopalakrishnan (AMD) * Anthony Liguori (AWS), Matt Wilson (AWS) * Chris Sheperd (Citrix) * Demetrios Coulis standing in for Allan Roberto (CA) * Greg Lutostanski (Calxeda) * Konrad Wilk (Oracle) * Lars Kurth (Chair, Xen Project) * Will Auld (Intel) I believe, but am not 100% sure that *Harry Hart and Don Slutz (Verizon) also stayed at the meeting {{Action|Don}} Please confirm whether you were at the meeting and whether V= erizon would in fact also participate in the WG and who the respective comp= any rep would be =3D=3D Agenda =3D=3D * Introductions * Purpose and Scope of Working Group :Lars: My view is that the WG provides oversight and guidance for creating = a test infrastructure for the Xen Project on behalf of the Xen Community. T= his is merely providing a stake in the ground to start a discussion. As such, the group needs to **Make proposals to the Advisory Board for funding, for example *** What do we need to set up such a system in terms of hosting space, mach= ines, ... *** Do we need a full-time resource employed by the LF to maintain and evol= ve the framework, ... ** Make proposals to the community and work with the community on a solutio= n that works for developers in the community *** Without community buy-in and creating something that the community actu= ally wants, we won't be able to improve upstream quality ** Make decisions to help bootstrap demo systems, proof of concepts, etc. ** The group will also need to approve Statement of Works or provide input = on hiring contractors/resource ** Influence or set ground rules to ensure that the money that the Advisory= Board invests delivers value to the community and the Advisory Board ** Highlight focus areas for investment: e.g. the group would decide on que= stions such as *** Are there focus areas for test coverage the WG/AB cares about? *** Can WG/AB members find resources to cover these areas? *** If not, do we need to use project funds to seed such areas? :Comment by Matt Wilson (AWS): I'd like to suggest that the AB and working = groups try to avoid making technology decisions. It makes sense for the WG = to decide where to make investments on behalf of Xen Project members (e.g. = where to invest through contractors / grants, capital investments, etc.). W= hile this might influence technology decisions, it should avoid trouble som= e engineering-by-committee problems. :My proposal would be to go through the list above and add/remove items and= examples. We do not need to make decisions on items listed above at this m= eeting (I see these as example to clarify the scope/purpose/boundaries of = the WG). I would then take the output and draft a charter which we can vote= on in the WG subsequently and to put it forward to the Advisory Board and = the wider community. * Status Update and scenarios going forward (Chris Shepherd) :We probably need to give a quick overview of what we have (there are quite= a few new people on the list) :Status update: ** The OSU Open Source Lab in principle agreed to set up a Test-As-A-Servic= e rig for Xen for a limited time ** The proposal is based on Citrix's internal XenRT system. This would be a= proof of concept aimed at informing further investment decisions. ** OSSTest is already running (however on Citrix servers, with limited acce= ss by the community) ... see *** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osst= est/ *** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-b= y-step/ * Meeting Mechanics : ** Meeting cadence/time/etc. ** Mailing lists =3D=3D=3D Introductions =3D=3D=3D Please feel free to expand and ask questions, etc. * Aravind Gopalakrishnan, works in the Server OS team at AMD and specialise= s in RAS features * Chris Sheperd, leads the Test Department for XenServer at Citrix * Will Auld is Performance Architect and Principal Engineer at Intel, but = also works closely with the Cloud Virtualization Test Group * Anthony Liguori is the QEMU project lead at Amazon * Demetrios Coulis, is product manager for AppLogic at CA * Greg Lutostanski is working in the Validation team (sorry, may have gotte= n this wrong) at Calxeda and is new to Xen * Konrad Wilk is Software Development Manager at Oracle, but also Xen maint= ainer in Linux * Lars Kurth is the community manager for the Xen Project and is volunteeri= ng to chair this working group until it becomes self-sustaining =3D=3D=3D Purpose and Scope of Working Group =3D=3D=3D We didn't really work through the items in the list in the agenda. We ended= up having a discussion. I didn't capture all of it - the discussion got q= uite engaged - and I was at taking notes. Please feel free to augment on th= e list and correct me. Lars: We started with the list of items in the agenda, but realized that ma= ybe we need to take a step back and check our assumptions. Anthony: * Raised the point that the introduction of KVM autotest (http://www.linux-= kvm.org/page/KVM-Autotest) has been problematic. Developers generally tend = to write test code if it fits into their development workflow (i.e. they ca= n run tests easily locally on their branch very easily). * System Test frameworks (such as Xen OSSTest and XenRT) which are run *aft= er* submission are more problematic and many devs tend to ignore them * From Anthony's experience, we also approached the OSSTest / XenRT discuss= ions wrongly. We should *not* just ask developers to write tests, but under= stand better what tests framework they would find attractive and provide so= mething which helps them. Lars: * We can fix the last point on engaging the community We then covered an angle on what is wrong with OSS test right now Konrad: stated that OSSTest is not well enough maintained and owned right = now (IanJ does this in his spare time). OSSTest or other systems the AB su= pports will need to be properly owned and funded, otherwise the system woul= d lose trust. Matt commented that he doesn't like the pushgate mechanism in OSSTest (we d= idn't cover this in detail) On the other hand ... running Coverity on the code base has been a huge suc= cess. Konrad: * In 2 months we had > 215 bug fixes * the quality of the xen core components according to coverity is now highe= r than that in the Linux kernel This implies that there is a willingness in the community to use tools to i= mprove quality. We then slipped into a discussion about goals: Anthony raised a few questio= ns * How do we determine how we (or what we propose) provides value to members= and the community * We would need to define measurable success criteria The conclusion we came to (note that we didn't vote) was * Understand what member companies want to get out of the framework * Understand what the community wants to get out of it (as value for compan= ies depends on community buy in). {{Action|Lars}} Carefully draft a mail to the devel lists (after sending to= this list for approval) along the lines of: * As AB we have resources that we can use to help the community * We are only willing to spend money, if we are confident enough that this = helps the community and is likely to be adopted * Here are a few options and what we think would help you [intended to seed= the discussion] * We are looking for volunteers in the community to work with the WG {{Action|All}} Each member company should provide a statement of what they = are trying to achieve, whether there are any specific test related items, c= onstraints, platforms, goals, etc. that they are care about '''Additional Notes: ''' I am also including some notes and discussions I had outside the WG meeting= at the bottom of this section. I had a conversation with a number of developers after the test talks * Anthony stated that maybe to have a test tool based on QEMU that allows p= eople to run some relevant tests locally would be best (problem: performanc= e). He also raised the point that both OSSTest and XenRT are just like auto= test. Why do we need yet another new test framework rather than using somet= hing which is out there already? * Ian Jackson stated that the main issue right now is that OSSTest is not p= roperly resourced and does have not good coverage. He spends 50% of his tim= e keeping OSSTest running and sometimes tracking down hardware issues. So w= hatever we do, resourcing needs to be resolved. * When I talked to Citrix Platform team members, it turned out that only a = few have actually used OSSTest (lack of documentation is the key issue). It= 's too hard to get started with OSSTest right now. * Another developer stated (source not disclosed): why don't the member com= panies fund the "creation of tests (regardless of test framework) if they c= are about quality" * Ian Jackson: there are some interesting properties about OSSTest (e.g. th= e capability to submit test code with a spec on the fly). [Aside by Lars: t= his may be close enough to providing the capability to try something locall= y, if we there also was a capability to test a devs personal git branch wit= h some test code] * Ian J also stated that he would look at XenRT, if the code was made avail= able in a Git repo rather than a tarball. Any issues and discussions could = then happen using the normal ways of discussing larger code contributions o= n xen-devel This clearly shows that we may have an issue winning the community over. =3D=3D=3D Status Update and scenarios going forward =3D=3D=3D (mostly by Chris Sheperd) http://www.slideshare.net/xen_com_mgr/xen-summit-taas-and-xenrt-0 shows a s= tatus update Documentation at http://wiki.xenproject.org/wiki/Category:XenRT Harry Hart: mentioned that Verizon tried XenRT but that they got stuck (Asi= de Lars: the fact that right now XenRT is sort of outside the Xen Project c= ommunity does not lend itself to discuss and raise issues about it) The main benefit of XenRT is that we inherit a large number of existing tes= ts (including for example performance tests and others) At the time of the meeting, the intention was to show a demo running at htt= p://osuosl.org/, but OSL had not set this up '''Update: 8/11/13: Sorry for dropping the ball on this. We have the system= s basically ready to go outside of getting public internet plumbed. Are you= still in need of them? Please let me know!''' The question now is whether a) Citrix is willing to spend some time on setting a test environment up (a= nd add support for xl) b) Whether others on this list think that a XenRT demo instance is a good i= dea My question to Chris: {{Action|Chris}} Determine whether Citrix is willing to set up XenRT on htt= p://osuosl.org/ (and add support for "xl") {{Action|Chris}} Let Lance Albertson from http://osuosl.org/ know Now as we do have a WG, I suggest to CC it if the answer to the The other question I have to the group (in light of the previous discussion= ) is: {{Action|All} Let me know by "replying +1" to this item whether you feel th= ere is value in setting up a XenRT demo instance on http://osuosl.org/; "re= plying 0" means you don't care; "replying -1" means you believe this is a b= ad idea (to satisfy Xen Project conventions you would have to justify why y= ou think this) =3D=3D=3D Meeting Mechaniscs =3D=3D=3D {{Action|All} Let me know what day and time of week you would be available= for a monthly call. All people on the list are based in the EU or US (East= to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Ple= ase state 2-3 preferences. --_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

 

Hi all,

sorry it has taken me so long to put this together. = I set up the mailing list, and signed you all up. Please read the minutes
AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences.

 

Lars

 

=3D=3D Attendees =3D=3D

* Aravind Gopalakrishnan (AMD)

* Anthony Liguori (AWS), Matt Wilson (AWS)

* Chris Sheperd (Citrix)

* Demetrios Coulis standing in for Allan Roberto (CA= )

* Greg Lutostanski (Calxeda)

* Konrad Wilk (Oracle)

* Lars Kurth (Chair, Xen Project)

* Will Auld (Intel)

I believe, but am not 100% sure that

*Harry Hart and Don Slutz (Verizon) also stayed at t= he meeting

 

{{Action|Don}} Please confirm whether you were at th= e meeting and whether Verizon would in fact also participate in the WG and = who the respective company rep would be

 

=3D=3D Agenda =3D=3D

 

* Introductions

* Purpose and Scope of Working Group

 

:Lars: My view is that the WG provides oversight and= guidance for creating a test infrastructure for the Xen Project on behalf = of the Xen Community. This is merely providing a stake in the ground to sta= rt a discussion.

 

As such, the group needs to

**Make proposals to the Advisory Board for funding, = for example

*** What do we need to set up such a system in terms= of hosting space, machines, …

*** Do we need a full-time resource employed by the = LF to maintain and evolve the framework, …

** Make proposals to the community and work with the= community on a solution that works for developers in the community

*** Without community buy-in and creating something = that the community actually wants, we won’t be able to improve upstre= am quality

** Make decisions to help bootstrap demo systems, pr= oof of concepts, etc.

** The group will also need to approve Statement of = Works or provide input on hiring contractors/resource

** Influence or set ground rules to ensure that the = money that the Advisory Board invests delivers value to the community and t= he Advisory Board

** Highlight focus areas for investment: e.g. the gr= oup would decide on questions such as

*** Are there focus areas for test coverage the WG/A= B cares about?

*** Can WG/AB members find resources to cover these = areas?

*** If not, do we need to use project funds to seed = such areas?

 

:Comment by Matt Wilson (AWS): I'd like to sugges= t that the AB and working groups try to avoid making technology decisions. = It makes sense for the WG to decide where to make investments on behalf of = Xen Project members (e.g. where to invest through contractors / grants, capital investments, etc.). While thi= s might influence technology decisions, it should avoid trouble some engine= ering-by-committee problems.

 

:My proposal would be to go through the list abov= e and add/remove items and examples. We do not need to make decisions on it= ems listed above at this meeting (I see these as example to clarify  t= he scope/purpose/boundaries of the WG). I would then take the output and draft a charter which we can vote on in t= he WG subsequently and to put it forward to the Advisory Board and the wide= r community.

 

* Status Update and scenarios going forward (Chris S= hepherd)

:We probably need to give a quick overview of what w= e have (there are quite a few new people on the list)

 

:Status update:

** The OSU Open Source Lab in principle agreed to se= t up a Test-As-A-Service rig for Xen for a limited time

** The proposal is based on Citrix’s internal = XenRT system. This would be a proof of concept aimed at informing further i= nvestment decisions.

** OSSTest is already running (however on Citrix ser= vers, with limited access by the community) … see

*** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/=

*** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/

 

* Meeting Mechanics :

** Meeting cadence/time/etc.

** Mailing lists

 

=3D=3D=3D Introductions =3D=3D=3D

Please feel free to expand and ask questions, etc.

* Aravind Gopalakrishnan, works in the Server OS tea= m at AMD and specialises in RAS features

* Chris Sheperd, leads the Test Department for XenSe= rver at Citrix

* Will Auld is Performance Architect and  Princ= ipal Engineer at Intel, but also works closely with the Cloud Virtualizatio= n Test Group

* Anthony Liguori is the QEMU project lead at Amazon=

* Demetrios Coulis, is product manager for AppLogic = at CA

* Greg Lutostanski is working in the Validation team= (sorry, may have gotten this wrong) at Calxeda and is new to Xen

* Konrad Wilk is Software Development Manager at Ora= cle, but also Xen maintainer in Linux

* Lars Kurth is the community manager for the Xen Pr= oject and is volunteering to chair this working group until it becomes self= -sustaining

 

=3D=3D=3D Purpose and Scope of Working Group  = =3D=3D=3D

 

We didn’t really work through the items in the= list in the agenda. We ended up having a discussion.  I didn’t = capture all of it - the discussion got quite engaged – and I was at t= aking notes. Please feel free to augment on the list and correct me.

 

Lars: We started with the list of items in the agend= a, but realized that maybe we need to take a step back and check our assump= tions.

 

Anthony:

* Raised the point that the introduction of KVM auto= test (http://www.lin= ux-kvm.org/page/KVM-Autotest) has been problematic. Developers generall= y tend to write test code if it fits into their development workflow (i.e. they can run tests easily locally on= their branch very easily).

* System Test frameworks (such as Xen OSSTest and Xe= nRT) which are run *after* submission are more problematic and many = devs tend to ignore them

* From Anthony’s experience, we also approache= d the OSSTest / XenRT discussions wrongly. We should *not* just ask = developers to write tests, but understand better what tests framework they = would find attractive and provide something which helps them.

 

Lars:

* We can fix the last point on engaging the communit= y

 

We then covered an angle on what is wrong with OSS t= est right now

 

Konrad:  stated that OSSTest is not well enough= maintained and owned right now (IanJ does this in his spare time).  O= SSTest or other systems the AB supports will need to be properly owned and = funded, otherwise the system would lose trust.

 

Matt commented that he doesn’t like the pushga= te mechanism in OSSTest (we didn’t cover this in detail)

 

On the other hand … running Coverity on the co= de base has been a huge success.

 

Konrad:

* In 2 months we had > 215 bug fixes

* the quality of the xen core components according t= o coverity is now higher than that in the Linux kernel

 

This implies that there is a willingness in the comm= unity to use tools to improve quality.

 

We then slipped into a discussion about goals: Antho= ny raised a few questions

* How do we determine how we (or what we propose) pr= ovides value to members and the community

* We would need to define measurable success criteri= a

 

The conclusion we came to (note that we didn’t= vote) was

* Understand what member companies want to get out o= f the framework

* Understand what the community wants to get out of = it (as value for companies depends on community buy in).

 

{{Action|Lars}} Carefully draft a mail to the devel = lists (after sending to this list for approval) along the lines of:

* As AB we have resources that we can use to help th= e community

* We are only willing to spend money, if we are conf= ident enough that this helps the community and is likely to be adopted=

* Here are a few options and what we think would hel= p you [intended to seed the discussion]

* We are looking for volunteers in the community to = work with the WG

 

{{Action|All}} Each member company should provide a = statement of what they are trying to achieve, whether there are any specifi= c test related items, constraints, platforms, goals, etc. that they are car= e about

 

’’’Additional Notes: ’’= ;’

I am also including some notes and discussions I had= outside the WG meeting at the bottom of this section.

 

I had a conversation with a number of developers aft= er the test talks

* Anthony stated that maybe to have a test tool base= d on QEMU that allows people to run some relevant tests locally would be be= st (problem: performance). He also raised the point that both OSSTest and X= enRT are just like autotest. Why do we need yet another new test framework rather than using something which i= s out there already?

* Ian Jackson stated that the main issue right now i= s that OSSTest is not properly resourced and does have not good coverage. H= e spends 50% of his time keeping OSSTest running and sometimes tracking dow= n hardware issues. So whatever we do,  resourcing needs to be resolved.

* When I talked to Citrix Platform team members, it = turned out that only a few have actually used OSSTest (lack of documentatio= n is the key issue). It’s too hard to get started with OSSTest right = now.

* Another developer stated (source not disclosed): w= hy don’t the member companies fund the “creation of tests (rega= rdless of test framework) if they care about quality”

* Ian Jackson: there are some interesting properties= about OSSTest (e.g. the capability to submit test code with a spec on the = fly). [Aside by Lars: this may be close enough to providing the capability = to try something locally, if we there also was a capability to test a devs personal git branch with some test co= de]

* Ian J also stated that he would look at XenRT, if = the code was made available in a Git repo rather than a tarball. Any issues= and discussions could then happen using the normal ways of discussing larg= er code contributions on xen-devel

 

This clearly shows that we may have an issue winning= the community over.

 

=3D=3D=3D Status Update and scenarios going forward&= nbsp; =3D=3D=3D

(mostly by Chris Sheperd)

 

http://www.slideshare.net/xen_com_mgr/xen-summit-= taas-and-xenrt-0 shows a status update

Documentation at http://wiki.xenproject.org/wiki/Category:XenRT

 

Harry Hart: mentioned that Verizon tried XenRT but t= hat they got stuck (Aside Lars: the fact that right now XenRT is sort of ou= tside the Xen Project community does not lend itself to discuss and raise i= ssues about it)

The main benefit of XenRT is that we inherit a large= number of existing tests (including for example performance tests and othe= rs)

At the time of the meeting, the intention was to sho= w a demo running at http://osuosl.org/, but OSL had not set = this up

 

‘’’Update: 8/11/13: Sorry for drop= ping the ball on this. We have the systems basically ready to go outside of= getting public internet plumbed. Are you still in need of them? Please let= me know!’’’

 

The question now is whether

a) Citrix is willing to spend some time on setting a= test environment up (and add support for xl)

b) Whether others on this list think that a XenRT de= mo instance is a good idea

 

My question to Chris:

{{Action|Chris}} Determine whether Citrix is willing= to set up XenRT on http://osuosl.org/ (and add support for = “xl”)

{{Action|Chris}} Let Lance Albertson from http://osuosl.org/  know

 

Now as we do have a WG, I suggest to CC it if the an= swer to the

 

The other question I have to the group (in light of = the previous discussion) is:

{{Action|All} Let me know by “replying +1&= #8221; to this item whether you feel there is value in setting up a XenRT d= emo instance on http://osuosl.org/; “replying 0= 221; means you don’t care; “replying -1“ means you believ= e this is a bad idea (to satisfy Xen Project conventions you would have to = justify why you think this)

 

=3D=3D=3D Meeting Mechaniscs  =3D=3D=3D

 

{{Action|All}  Let me know what day and time of= week you would be available for a monthly call. All people on the list are= based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot w= ould probably work best. Please state 2-3 preferences.

 

--_000_B5EC053C78C5D746BD8AC3386F2303C90DA2D6AMSPEX01CL01citri_-- --===============0800111578980940768== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework --===============0800111578980940768==-- From wg-test-framework-bounces@lists.xenproject.org Fri Nov 15 08:05:08 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 15 Nov 2013 08:05:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VhEOq-0006Fa-MX for archives@lists.xenproject.org; Fri, 15 Nov 2013 08:05:08 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VhEOo-0006Ew-FG for wg-test-framework@lists.xenproject.org; Fri, 15 Nov 2013 08:05:07 +0000 Received: from [85.158.143.35:33525] by server-1.bemta-4.messagelabs.com id B2/6E-02132-1B5D5825; Fri, 15 Nov 2013 08:05:05 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-14.tower-21.messagelabs.com!1384502694!3384608!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 23619 invoked from network); 15 Nov 2013 08:04:55 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-14.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 15 Nov 2013 08:04:55 -0000 X-IronPort-AV: E=Sophos;i="4.93,705,1378857600"; d="scan'208,217";a="72259526" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 15 Nov 2013 08:04:53 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL03.citrite.net (10.13.107.80) with Microsoft SMTP Server (TLS) id 14.2.342.4; Fri, 15 Nov 2013 03:04:53 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Fri, 15 Nov 2013 09:04:51 +0100 From: Lars Kurth To: "Gopalakrishnan, Aravind" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: AQHO4ZUJbVlsdHNVqUeeF7Fxi5fPz5ol7B+D Date: Fri, 15 Nov 2013 08:04:50 +0000 Message-ID: References: , <4923C2DE085EEB4FAB1D375DD09D0BA61AD3D37C@SATLEXDAG01.amd.com> In-Reply-To: <4923C2DE085EEB4FAB1D375DD09D0BA61AD3D37C@SATLEXDAG01.amd.com> Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.69.46.1] MIME-Version: 1.0 X-DLP: MIA2 Cc: "wg-test-framework@lists.xenproject.org" Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============6750491874880416836==" Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org --===============6750491874880416836== Content-Language: en-GB Content-Type: multipart/alternative; boundary="_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_" --_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Aravaind, thanks for getting back. I have followed up a little about the system test = vs. unit test argument with the developers in Cambridge at least. I think w= hat this comes down to is really a question about workflow. Right now, OSSTest as well as Xen RT require (in my understanding) a) code to be submitted for review b) code to be reviewed and be submitted by the committer to the staging bra= nch c) only then tests are run d) if stuff fails, it has to be taken out and the whole process starts agai= n Whereas what would be really desirable and IMHO attractive to devs is the f= ollowing work-flow a) developer has a well working prototype on their personal git branch some= where b) developer can run a set of interesting (or new) tests on some machines o= n different architectures locally (which is somewhat impractical) c) alternatively developer points a to his branch, plus some interesting te= sts and specifies interesting machines and tests are quickly run on a centr= al test farm (the question is how quickly and smoothly this process would b= e) d) if all works well, code is submitted for review (and test results could = be attached) I think this is what Anthony's argument ultimately comes down to. The quest= ion is whether the second workflow is achievable with something like OSSTes= t and XenRT. Of course the system testing approach is also needed. Regards Lars ________________________________ From: Gopalakrishnan, Aravind [Aravind.Gopalakrishnan@amd.com] Sent: 14 November 2013 23:54 To: Lars Kurth Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting = in October Hi Lars, I am available on all days (except Thursday) between 10am to 12noon And, I am =93+1=94 for a demo instance setup at OSU. I do agree with Anthony=92s comment that XenRT is similar to autotest.. but= AFAICT, autotest=92s xen testing support is mostly non-existent.. Maybe understanding more about the feature-set of XenRT could go some way t= o help figure out if we can just leverage XenRT or need to beef up Xen supp= ort in autotest.. (Apologies for the late reply) Thanks, -Aravind. From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framew= ork-bounces@lists.xenproject.org] On Behalf Of Lars Kurth Sent: Monday, November 11, 2013 6:55 AM To: wg-test-framework@lists.xenproject.org; advisory-board@lists.xenproject= .org Cc: Slutz, Donald Christopher (dslutz@verizon.com); Hart, Harry D (hhart@ve= rizon.com); Coulis, Demetrios (Demetrios.Coulis@ca.com) Subject: [Wg-test-framework] [Minutes] First Test WG framework meeting in O= ctober Hi all, sorry it has taken me so long to put this together. I set up the mailing li= st, and signed you all up. Please read the minutes AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences. Lars =3D=3D Attendees =3D=3D * Aravind Gopalakrishnan (AMD) * Anthony Liguori (AWS), Matt Wilson (AWS) * Chris Sheperd (Citrix) * Demetrios Coulis standing in for Allan Roberto (CA) * Greg Lutostanski (Calxeda) * Konrad Wilk (Oracle) * Lars Kurth (Chair, Xen Project) * Will Auld (Intel) I believe, but am not 100% sure that *Harry Hart and Don Slutz (Verizon) also stayed at the meeting {{Action|Don}} Please confirm whether you were at the meeting and whether V= erizon would in fact also participate in the WG and who the respective comp= any rep would be =3D=3D Agenda =3D=3D * Introductions * Purpose and Scope of Working Group :Lars: My view is that the WG provides oversight and guidance for creating = a test infrastructure for the Xen Project on behalf of the Xen Community. T= his is merely providing a stake in the ground to start a discussion. As such, the group needs to **Make proposals to the Advisory Board for funding, for example *** What do we need to set up such a system in terms of hosting space, mach= ines, =85 *** Do we need a full-time resource employed by the LF to maintain and evol= ve the framework, =85 ** Make proposals to the community and work with the community on a solutio= n that works for developers in the community *** Without community buy-in and creating something that the community actu= ally wants, we won=92t be able to improve upstream quality ** Make decisions to help bootstrap demo systems, proof of concepts, etc. ** The group will also need to approve Statement of Works or provide input = on hiring contractors/resource ** Influence or set ground rules to ensure that the money that the Advisory= Board invests delivers value to the community and the Advisory Board ** Highlight focus areas for investment: e.g. the group would decide on que= stions such as *** Are there focus areas for test coverage the WG/AB cares about? *** Can WG/AB members find resources to cover these areas? *** If not, do we need to use project funds to seed such areas? :Comment by Matt Wilson (AWS): I'd like to suggest that the AB and working = groups try to avoid making technology decisions. It makes sense for the WG = to decide where to make investments on behalf of Xen Project members (e.g. = where to invest through contractors / grants, capital investments, etc.). W= hile this might influence technology decisions, it should avoid trouble som= e engineering-by-committee problems. :My proposal would be to go through the list above and add/remove items and= examples. We do not need to make decisions on items listed above at this m= eeting (I see these as example to clarify the scope/purpose/boundaries of = the WG). I would then take the output and draft a charter which we can vote= on in the WG subsequently and to put it forward to the Advisory Board and = the wider community. * Status Update and scenarios going forward (Chris Shepherd) :We probably need to give a quick overview of what we have (there are quite= a few new people on the list) :Status update: ** The OSU Open Source Lab in principle agreed to set up a Test-As-A-Servic= e rig for Xen for a limited time ** The proposal is based on Citrix=92s internal XenRT system. This would be= a proof of concept aimed at informing further investment decisions. ** OSSTest is already running (however on Citrix servers, with limited acce= ss by the community) =85 see *** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osst= est/ *** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-b= y-step/ * Meeting Mechanics : ** Meeting cadence/time/etc. ** Mailing lists =3D=3D=3D Introductions =3D=3D=3D Please feel free to expand and ask questions, etc. * Aravind Gopalakrishnan, works in the Server OS team at AMD and specialise= s in RAS features * Chris Sheperd, leads the Test Department for XenServer at Citrix * Will Auld is Performance Architect and Principal Engineer at Intel, but = also works closely with the Cloud Virtualization Test Group * Anthony Liguori is the QEMU project lead at Amazon * Demetrios Coulis, is product manager for AppLogic at CA * Greg Lutostanski is working in the Validation team (sorry, may have gotte= n this wrong) at Calxeda and is new to Xen * Konrad Wilk is Software Development Manager at Oracle, but also Xen maint= ainer in Linux * Lars Kurth is the community manager for the Xen Project and is volunteeri= ng to chair this working group until it becomes self-sustaining =3D=3D=3D Purpose and Scope of Working Group =3D=3D=3D We didn=92t really work through the items in the list in the agenda. We end= ed up having a discussion. I didn=92t capture all of it - the discussion g= ot quite engaged =96 and I was at taking notes. Please feel free to augment= on the list and correct me. Lars: We started with the list of items in the agenda, but realized that ma= ybe we need to take a step back and check our assumptions. Anthony: * Raised the point that the introduction of KVM autotest (http://www.linux-= kvm.org/page/KVM-Autotest) has been problematic. Developers generally tend = to write test code if it fits into their development workflow (i.e. they ca= n run tests easily locally on their branch very easily). * System Test frameworks (such as Xen OSSTest and XenRT) which are run *aft= er* submission are more problematic and many devs tend to ignore them * From Anthony=92s experience, we also approached the OSSTest / XenRT discu= ssions wrongly. We should *not* just ask developers to write tests, but und= erstand better what tests framework they would find attractive and provide = something which helps them. Lars: * We can fix the last point on engaging the community We then covered an angle on what is wrong with OSS test right now Konrad: stated that OSSTest is not well enough maintained and owned right = now (IanJ does this in his spare time). OSSTest or other systems the AB su= pports will need to be properly owned and funded, otherwise the system woul= d lose trust. Matt commented that he doesn=92t like the pushgate mechanism in OSSTest (we= didn=92t cover this in detail) On the other hand =85 running Coverity on the code base has been a huge suc= cess. Konrad: * In 2 months we had > 215 bug fixes * the quality of the xen core components according to coverity is now highe= r than that in the Linux kernel This implies that there is a willingness in the community to use tools to i= mprove quality. We then slipped into a discussion about goals: Anthony raised a few questio= ns * How do we determine how we (or what we propose) provides value to members= and the community * We would need to define measurable success criteria The conclusion we came to (note that we didn=92t vote) was * Understand what member companies want to get out of the framework * Understand what the community wants to get out of it (as value for compan= ies depends on community buy in). {{Action|Lars}} Carefully draft a mail to the devel lists (after sending to= this list for approval) along the lines of: * As AB we have resources that we can use to help the community * We are only willing to spend money, if we are confident enough that this = helps the community and is likely to be adopted * Here are a few options and what we think would help you [intended to seed= the discussion] * We are looking for volunteers in the community to work with the WG {{Action|All}} Each member company should provide a statement of what they = are trying to achieve, whether there are any specific test related items, c= onstraints, platforms, goals, etc. that they are care about =92=92=92Additional Notes: =92=92=92 I am also including some notes and discussions I had outside the WG meeting= at the bottom of this section. I had a conversation with a number of developers after the test talks * Anthony stated that maybe to have a test tool based on QEMU that allows p= eople to run some relevant tests locally would be best (problem: performanc= e). He also raised the point that both OSSTest and XenRT are just like auto= test. Why do we need yet another new test framework rather than using somet= hing which is out there already? * Ian Jackson stated that the main issue right now is that OSSTest is not p= roperly resourced and does have not good coverage. He spends 50% of his tim= e keeping OSSTest running and sometimes tracking down hardware issues. So w= hatever we do, resourcing needs to be resolved. * When I talked to Citrix Platform team members, it turned out that only a = few have actually used OSSTest (lack of documentation is the key issue). It= =92s too hard to get started with OSSTest right now. * Another developer stated (source not disclosed): why don=92t the member c= ompanies fund the =93creation of tests (regardless of test framework) if th= ey care about quality=94 * Ian Jackson: there are some interesting properties about OSSTest (e.g. th= e capability to submit test code with a spec on the fly). [Aside by Lars: t= his may be close enough to providing the capability to try something locall= y, if we there also was a capability to test a devs personal git branch wit= h some test code] * Ian J also stated that he would look at XenRT, if the code was made avail= able in a Git repo rather than a tarball. Any issues and discussions could = then happen using the normal ways of discussing larger code contributions o= n xen-devel This clearly shows that we may have an issue winning the community over. =3D=3D=3D Status Update and scenarios going forward =3D=3D=3D (mostly by Chris Sheperd) http://www.slideshare.net/xen_com_mgr/xen-summit-taas-and-xenrt-0 shows a s= tatus update Documentation at http://wiki.xenproject.org/wiki/Category:XenRT Harry Hart: mentioned that Verizon tried XenRT but that they got stuck (Asi= de Lars: the fact that right now XenRT is sort of outside the Xen Project c= ommunity does not lend itself to discuss and raise issues about it) The main benefit of XenRT is that we inherit a large number of existing tes= ts (including for example performance tests and others) At the time of the meeting, the intention was to show a demo running at htt= p://osuosl.org/, but OSL had not set this up =91=92=92Update: 8/11/13: Sorry for dropping the ball on this. We have the = systems basically ready to go outside of getting public internet plumbed. A= re you still in need of them? Please let me know!=92=92=92 The question now is whether a) Citrix is willing to spend some time on setting a test environment up (a= nd add support for xl) b) Whether others on this list think that a XenRT demo instance is a good i= dea My question to Chris: {{Action|Chris}} Determine whether Citrix is willing to set up XenRT on htt= p://osuosl.org/ (and add support for =93xl=94) {{Action|Chris}} Let Lance Albertson from http://osuosl.org/ know Now as we do have a WG, I suggest to CC it if the answer to the The other question I have to the group (in light of the previous discussion= ) is: {{Action|All} Let me know by =93replying +1=94 to this item whether you fee= l there is value in setting up a XenRT demo instance on http://osuosl.org/;= =93replying 0=94 means you don=92t care; =93replying -1=93 means you belie= ve this is a bad idea (to satisfy Xen Project conventions you would have to= justify why you think this) =3D=3D=3D Meeting Mechaniscs =3D=3D=3D {{Action|All} Let me know what day and time of week you would be available= for a monthly call. All people on the list are based in the EU or US (East= to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Ple= ase state 2-3 preferences. --_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable = object[type$=3D"x-shockwave-fla= sh"]:not([classid]),object[type$=3D"futuresplash"]:not([classid]),embed[typ= e$=3D"x-shockwave-flash"],embed[type$=3D"futuresplash"]{display:none !impor= tant}
Aravaind,

thanks for getting back. I have followed up a little about the system = test vs. unit test argument with the developers in Cambridge at least. I th= ink what this comes down to is really a question about workflow.

Right now, OSSTest as well as Xen RT require (in my understanding)
a) code to be submitted for review
b) code to be reviewed and be submitted by the committer to the stagin= g branch
c) only then tests are run
d) if stuff fails, it has to be taken out and the whole process starts= again

Whereas what would be really desirable and IMHO attractive to devs is = the following work-flow
a) developer has a well working prototype on their personal git branch= somewhere
b) developer can run a set of interesting (or new) tests on some machi= nes on different architectures locally (which is somewhat impractical)
c) alternatively developer points a to his branch, plus some interesti= ng tests and specifies interesting machines and tests are quickly run on a = central test farm (the question is how quickly and smoothly this process wo= uld be)
d) if all works well, code is submitted for review (and test results c= ould be attached)

I think this is what Anthony's argument ultimately comes down to. The = question is whether the second workflow is achievable with something like O= SSTest and XenRT. Of course the system testing approach is also needed.

Regards
Lars

From: Gopalakrishnan, Aravind [Aravind.Go= palakrishnan@amd.com]
Sent: 14 November 2013 23:54
To: Lars Kurth
Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework m= eeting in October

Hi Lars,

I am available on all = days (except Thursday) between 10am to 12noon

 

And, I am =93+1=94= for a demo instance setup at OSU.

 

I do agree with Anthon= y=92s comment that XenRT is similar to autotest.. but AFAICT, autotest=92s = xen testing support  is mostly non-existent..

Maybe understanding mo= re about the feature-set of XenRT could go some way to help figure out if w= e can just leverage XenRT or need to beef up Xen support in autotest..

 

(Apologies for the lat= e reply)

 

Thanks,

-Aravind.

 

From: wg-tes= t-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@= lists.xenproject.org] On Behalf Of Lars Kurth
Sent: Monday, November 11, 2013 6:55 AM
To: wg-test-framework@lists.xenproject.org; advisory-board@lists.xen= project.org
Cc: Slutz, Donald Christopher (dslutz@verizon.com); Hart, Harry D (h= hart@verizon.com); Coulis, Demetrios (Demetrios.Coulis@ca.com)
Subject: [Wg-test-framework] [Minutes] First Test WG framework meeti= ng in October

 

 

Hi all,

sorry it has taken me so long t= o put this together. I set up the mailing list, and signed you all up. Plea= se read the minutes

AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences.

 

Lars

 

=3D=3D Attendees =3D=3D<= /p>

* Aravind Gopalakrishnan (AMD)<= /span>

* Anthony Liguori (AWS), Matt W= ilson (AWS)

* Chris Sheperd (Citrix)=

* Demetrios Coulis standing in = for Allan Roberto (CA)

* Greg Lutostanski (Calxeda)

* Konrad Wilk (Oracle)

* Lars Kurth (Chair, Xen Projec= t)

* Will Auld (Intel)

I believe, but am not 100% sure= that

*Harry Hart and Don Slutz (Veri= zon) also stayed at the meeting

 

{{Action|Don}} Please confirm w= hether you were at the meeting and whether Verizon would in fact also parti= cipate in the WG and who the respective company rep would be

 

=3D=3D Agenda =3D=3D

 

* Introductions

* Purpose and Scope of Working = Group

 

:Lars: My view is that the WG p= rovides oversight and guidance for creating a test infrastructure for the X= en Project on behalf of the Xen Community. This is merely providing a stake= in the ground to start a discussion.

 

As such, the group needs to

**Make proposals to the Advisor= y Board for funding, for example

*** What do we need to set up s= uch a system in terms of hosting space, machines, =85

*** Do we need a full-time reso= urce employed by the LF to maintain and evolve the framework, =85

** Make proposals to the commun= ity and work with the community on a solution that works for developers in = the community

*** Without community buy-in an= d creating something that the community actually wants, we won=92t be able = to improve upstream quality

** Make decisions to help boots= trap demo systems, proof of concepts, etc.

** The group will also need to = approve Statement of Works or provide input on hiring contractors/resource

** Influence or set ground rule= s to ensure that the money that the Advisory Board invests delivers value t= o the community and the Advisory Board

** Highlight focus areas for in= vestment: e.g. the group would decide on questions such as

*** Are there focus areas for t= est coverage the WG/AB cares about?

*** Can WG/AB members find reso= urces to cover these areas?

*** If not, do we need to use p= roject funds to seed such areas?

 

:Comment by Matt Wilson (AWS= ): I'd like to suggest that the AB and working groups try to avoid making t= echnology decisions. It makes sense for the WG to decide where to make inve= stments on behalf of Xen Project members (e.g. where to invest through contractors / grants, capital investments, e= tc.). While this might influence technology decisions, it should avoid trou= ble some engineering-by-committee problems.

 

:My proposal would be to go = through the list above and add/remove items and examples. We do not need to= make decisions on items listed above at this meeting (I see these as examp= le to clarify  the scope/purpose/boundaries of the WG). I would then take the output and draft a charter which we can = vote on in the WG subsequently and to put it forward to the Advisory Board = and the wider community.

 

* Status Update and scenarios g= oing forward (Chris Shepherd)

:We probably need to give a qui= ck overview of what we have (there are quite a few new people on the list)<= /span>

 

:Status update:

** The OSU Open Source Lab in p= rinciple agreed to set up a Test-As-A-Service rig for Xen for a limited tim= e

** The proposal is based on Cit= rix=92s internal XenRT system. This would be a proof of concept aimed at in= forming further investment decisions.

** OSSTest is already running (= however on Citrix servers, with limited access by the community) =85 see

*** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/=

*** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/

 

* Meeting Mechanics :

** Meeting cadence/time/etc.

** Mailing lists

 

=3D=3D=3D Introductions =3D=3D= =3D

Please feel free to expand and = ask questions, etc.

* Aravind Gopalakrishnan, works= in the Server OS team at AMD and specialises in RAS features

* Chris Sheperd, leads the Test= Department for XenServer at Citrix

* Will Auld is Performance Arch= itect and  Principal Engineer at Intel, but also works closely with th= e Cloud Virtualization Test Group

* Anthony Liguori is the QEMU p= roject lead at Amazon

* Demetrios Coulis, is product = manager for AppLogic at CA

* Greg Lutostanski is working i= n the Validation team (sorry, may have gotten this wrong) at Calxeda and is= new to Xen

* Konrad Wilk is Software Devel= opment Manager at Oracle, but also Xen maintainer in Linux

* Lars Kurth is the community m= anager for the Xen Project and is volunteering to chair this working group = until it becomes self-sustaining

 

=3D=3D=3D Purpose and Scope of = Working Group  =3D=3D=3D

 

We didn=92t really work through= the items in the list in the agenda. We ended up having a discussion. = ; I didn=92t capture all of it - the discussion got quite engaged =96 and I= was at taking notes. Please feel free to augment on the list and correct me.

 

Lars: We started with the list = of items in the agenda, but realized that maybe we need to take a step back= and check our assumptions.

 

Anthony:

* Raised the point that the int= roduction of KVM autotest (http://www.linux-kvm.org/page/KVM-Autotest) ha= s been problematic. Developers generally tend to write test code if it fits into their development workflow (i.e. t= hey can run tests easily locally on their branch very easily).

* System Test frameworks (such = as Xen OSSTest and XenRT) which are run *after* submission are more = problematic and many devs tend to ignore them

* From Anthony=92s experience, = we also approached the OSSTest / XenRT discussions wrongly. We should *n= ot* just ask developers to write tests, but understand better what test= s framework they would find attractive and provide something which helps them.

 

Lars:

* We can fix the last point on = engaging the community

 

We then covered an angle on wha= t is wrong with OSS test right now

 

Konrad:  stated that OSSTe= st is not well enough maintained and owned right now (IanJ does this in his= spare time).  OSSTest or other systems the AB supports will need to b= e properly owned and funded, otherwise the system would lose trust.

 

Matt commented that he doesn=92= t like the pushgate mechanism in OSSTest (we didn=92t cover this in detail)=

 

On the other hand =85 running C= overity on the code base has been a huge success.

 

Konrad:

* In 2 months we had > 215 b= ug fixes

* the quality of the xen core c= omponents according to coverity is now higher than that in the Linux kernel=

 

This implies that there is a wi= llingness in the community to use tools to improve quality.

 

We then slipped into a discussi= on about goals: Anthony raised a few questions

* How do we determine how we (o= r what we propose) provides value to members and the community

* We would need to define measu= rable success criteria

 

The conclusion we came to (note= that we didn=92t vote) was

* Understand what member compan= ies want to get out of the framework

* Understand what the community= wants to get out of it (as value for companies depends on community buy in= ).

 

{{Action|Lars}} Carefully draft= a mail to the devel lists (after sending to this list for approval) along = the lines of:

* As AB we have resources that = we can use to help the community

* We are only willing to spend = money, if we are confident enough that this helps the community and is like= ly to be adopted

* Here are a few options and wh= at we think would help you [intended to seed the discussion]

* We are looking for volunteers= in the community to work with the WG

 

{{Action|All}} Each member comp= any should provide a statement of what they are trying to achieve, whether = there are any specific test related items, constraints, platforms, goals, e= tc. that they are care about

 

=92=92=92Additional Notes: =92= =92=92

I am also including some notes = and discussions I had outside the WG meeting at the bottom of this section.=

 

I had a conversation with a num= ber of developers after the test talks

* Anthony stated that maybe to = have a test tool based on QEMU that allows people to run some relevant test= s locally would be best (problem: performance). He also raised the point th= at both OSSTest and XenRT are just like autotest. Why do we need yet another new test framework rather than using = something which is out there already?

* Ian Jackson stated that the m= ain issue right now is that OSSTest is not properly resourced and does have= not good coverage. He spends 50% of his time keeping OSSTest running and s= ometimes tracking down hardware issues. So whatever we do,  resourcing needs to be resolved.

* When I talked to Citrix Platf= orm team members, it turned out that only a few have actually used OSSTest = (lack of documentation is the key issue). It=92s too hard to get started wi= th OSSTest right now.

* Another developer stated (sou= rce not disclosed): why don=92t the member companies fund the =93creation o= f tests (regardless of test framework) if they care about quality=94=

* Ian Jackson: there are some i= nteresting properties about OSSTest (e.g. the capability to submit test cod= e with a spec on the fly). [Aside by Lars: this may be close enough to prov= iding the capability to try something locally, if we there also was a capability to test a devs personal git bra= nch with some test code]

* Ian J also stated that he wou= ld look at XenRT, if the code was made available in a Git repo rather than = a tarball. Any issues and discussions could then happen using the normal wa= ys of discussing larger code contributions on xen-devel

 

This clearly shows that we may = have an issue winning the community over.

 

=3D=3D=3D Status Update and sce= narios going forward  =3D=3D=3D

(mostly by Chris Sheperd)

 

http://www= .slideshare.net/xen_com_mgr/xen-summit-taas-and-xenrt-0 shows a status = update

Documentation at http://wiki.xenproject.org/wiki/Category:XenRT

 

Harry Hart: mentioned that Veri= zon tried XenRT but that they got stuck (Aside Lars: the fact that right no= w XenRT is sort of outside the Xen Project community does not lend itself t= o discuss and raise issues about it)

 

The main benefit of XenRT is th= at we inherit a large number of existing tests (including for example perfo= rmance tests and others)

At the time of the meeting, the= intention was to show a demo running at http://osuosl.org/, bu= t OSL had not set this up

 

=91=92=92Update: 8/11/13: Sorry= for dropping the ball on this. We have the systems basically ready to go o= utside of getting public internet plumbed. Are you still in need of them? P= lease let me know!=92=92=92

 

The question now is whether

a) Citrix is willing to spend s= ome time on setting a test environment up (and add support for xl)

b) Whether others on this list = think that a XenRT demo instance is a good idea

 

My question to Chris:

{{Action|Chris}} Determine whet= her Citrix is willing to set up XenRT on http://osuosl.org/ (an= d add support for =93xl=94)

{{Action|Chris}} Let Lance Albe= rtson from http://osuosl.org/ &nb= sp;know

 

Now as we do have a WG, I sugge= st to CC it if the answer to the

 

The other question I have to th= e group (in light of the previous discussion) is:

{{Action|All} Let me know by = =93replying +1=94 to this item whether you feel there is value in setti= ng up a XenRT demo instance on http://osuosl.org/; = =93replying 0=94 means you don=92t care; =93replying -1=93 means you believ= e this is a bad idea (to satisfy Xen Project conventions you would have to = justify why you think this)

 

=3D=3D=3D Meeting Mechaniscs&nb= sp; =3D=3D=3D

 

{{Action|All}  Let me know= what day and time of week you would be available for a monthly call. All p= eople on the list are based in the EU or US (East to West coast), so a 4pm = GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences.

 

--_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_-- --===============6750491874880416836== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework --===============6750491874880416836==-- From wg-test-framework-bounces@lists.xenproject.org Fri Nov 15 08:05:08 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 15 Nov 2013 08:05:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VhEOq-0006Fa-MX for archives@lists.xenproject.org; Fri, 15 Nov 2013 08:05:08 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VhEOo-0006Ew-FG for wg-test-framework@lists.xenproject.org; Fri, 15 Nov 2013 08:05:07 +0000 Received: from [85.158.143.35:33525] by server-1.bemta-4.messagelabs.com id B2/6E-02132-1B5D5825; Fri, 15 Nov 2013 08:05:05 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-14.tower-21.messagelabs.com!1384502694!3384608!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 23619 invoked from network); 15 Nov 2013 08:04:55 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-14.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 15 Nov 2013 08:04:55 -0000 X-IronPort-AV: E=Sophos;i="4.93,705,1378857600"; d="scan'208,217";a="72259526" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 15 Nov 2013 08:04:53 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL03.citrite.net (10.13.107.80) with Microsoft SMTP Server (TLS) id 14.2.342.4; Fri, 15 Nov 2013 03:04:53 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Fri, 15 Nov 2013 09:04:51 +0100 From: Lars Kurth To: "Gopalakrishnan, Aravind" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: AQHO4ZUJbVlsdHNVqUeeF7Fxi5fPz5ol7B+D Date: Fri, 15 Nov 2013 08:04:50 +0000 Message-ID: References: , <4923C2DE085EEB4FAB1D375DD09D0BA61AD3D37C@SATLEXDAG01.amd.com> In-Reply-To: <4923C2DE085EEB4FAB1D375DD09D0BA61AD3D37C@SATLEXDAG01.amd.com> Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.69.46.1] MIME-Version: 1.0 X-DLP: MIA2 Cc: "wg-test-framework@lists.xenproject.org" Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============6750491874880416836==" Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org --===============6750491874880416836== Content-Language: en-GB Content-Type: multipart/alternative; boundary="_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_" --_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Aravaind, thanks for getting back. I have followed up a little about the system test = vs. unit test argument with the developers in Cambridge at least. I think w= hat this comes down to is really a question about workflow. Right now, OSSTest as well as Xen RT require (in my understanding) a) code to be submitted for review b) code to be reviewed and be submitted by the committer to the staging bra= nch c) only then tests are run d) if stuff fails, it has to be taken out and the whole process starts agai= n Whereas what would be really desirable and IMHO attractive to devs is the f= ollowing work-flow a) developer has a well working prototype on their personal git branch some= where b) developer can run a set of interesting (or new) tests on some machines o= n different architectures locally (which is somewhat impractical) c) alternatively developer points a to his branch, plus some interesting te= sts and specifies interesting machines and tests are quickly run on a centr= al test farm (the question is how quickly and smoothly this process would b= e) d) if all works well, code is submitted for review (and test results could = be attached) I think this is what Anthony's argument ultimately comes down to. The quest= ion is whether the second workflow is achievable with something like OSSTes= t and XenRT. Of course the system testing approach is also needed. Regards Lars ________________________________ From: Gopalakrishnan, Aravind [Aravind.Gopalakrishnan@amd.com] Sent: 14 November 2013 23:54 To: Lars Kurth Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting = in October Hi Lars, I am available on all days (except Thursday) between 10am to 12noon And, I am =93+1=94 for a demo instance setup at OSU. I do agree with Anthony=92s comment that XenRT is similar to autotest.. but= AFAICT, autotest=92s xen testing support is mostly non-existent.. Maybe understanding more about the feature-set of XenRT could go some way t= o help figure out if we can just leverage XenRT or need to beef up Xen supp= ort in autotest.. (Apologies for the late reply) Thanks, -Aravind. From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framew= ork-bounces@lists.xenproject.org] On Behalf Of Lars Kurth Sent: Monday, November 11, 2013 6:55 AM To: wg-test-framework@lists.xenproject.org; advisory-board@lists.xenproject= .org Cc: Slutz, Donald Christopher (dslutz@verizon.com); Hart, Harry D (hhart@ve= rizon.com); Coulis, Demetrios (Demetrios.Coulis@ca.com) Subject: [Wg-test-framework] [Minutes] First Test WG framework meeting in O= ctober Hi all, sorry it has taken me so long to put this together. I set up the mailing li= st, and signed you all up. Please read the minutes AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences. Lars =3D=3D Attendees =3D=3D * Aravind Gopalakrishnan (AMD) * Anthony Liguori (AWS), Matt Wilson (AWS) * Chris Sheperd (Citrix) * Demetrios Coulis standing in for Allan Roberto (CA) * Greg Lutostanski (Calxeda) * Konrad Wilk (Oracle) * Lars Kurth (Chair, Xen Project) * Will Auld (Intel) I believe, but am not 100% sure that *Harry Hart and Don Slutz (Verizon) also stayed at the meeting {{Action|Don}} Please confirm whether you were at the meeting and whether V= erizon would in fact also participate in the WG and who the respective comp= any rep would be =3D=3D Agenda =3D=3D * Introductions * Purpose and Scope of Working Group :Lars: My view is that the WG provides oversight and guidance for creating = a test infrastructure for the Xen Project on behalf of the Xen Community. T= his is merely providing a stake in the ground to start a discussion. As such, the group needs to **Make proposals to the Advisory Board for funding, for example *** What do we need to set up such a system in terms of hosting space, mach= ines, =85 *** Do we need a full-time resource employed by the LF to maintain and evol= ve the framework, =85 ** Make proposals to the community and work with the community on a solutio= n that works for developers in the community *** Without community buy-in and creating something that the community actu= ally wants, we won=92t be able to improve upstream quality ** Make decisions to help bootstrap demo systems, proof of concepts, etc. ** The group will also need to approve Statement of Works or provide input = on hiring contractors/resource ** Influence or set ground rules to ensure that the money that the Advisory= Board invests delivers value to the community and the Advisory Board ** Highlight focus areas for investment: e.g. the group would decide on que= stions such as *** Are there focus areas for test coverage the WG/AB cares about? *** Can WG/AB members find resources to cover these areas? *** If not, do we need to use project funds to seed such areas? :Comment by Matt Wilson (AWS): I'd like to suggest that the AB and working = groups try to avoid making technology decisions. It makes sense for the WG = to decide where to make investments on behalf of Xen Project members (e.g. = where to invest through contractors / grants, capital investments, etc.). W= hile this might influence technology decisions, it should avoid trouble som= e engineering-by-committee problems. :My proposal would be to go through the list above and add/remove items and= examples. We do not need to make decisions on items listed above at this m= eeting (I see these as example to clarify the scope/purpose/boundaries of = the WG). I would then take the output and draft a charter which we can vote= on in the WG subsequently and to put it forward to the Advisory Board and = the wider community. * Status Update and scenarios going forward (Chris Shepherd) :We probably need to give a quick overview of what we have (there are quite= a few new people on the list) :Status update: ** The OSU Open Source Lab in principle agreed to set up a Test-As-A-Servic= e rig for Xen for a limited time ** The proposal is based on Citrix=92s internal XenRT system. This would be= a proof of concept aimed at informing further investment decisions. ** OSSTest is already running (however on Citrix servers, with limited acce= ss by the community) =85 see *** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osst= est/ *** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-b= y-step/ * Meeting Mechanics : ** Meeting cadence/time/etc. ** Mailing lists =3D=3D=3D Introductions =3D=3D=3D Please feel free to expand and ask questions, etc. * Aravind Gopalakrishnan, works in the Server OS team at AMD and specialise= s in RAS features * Chris Sheperd, leads the Test Department for XenServer at Citrix * Will Auld is Performance Architect and Principal Engineer at Intel, but = also works closely with the Cloud Virtualization Test Group * Anthony Liguori is the QEMU project lead at Amazon * Demetrios Coulis, is product manager for AppLogic at CA * Greg Lutostanski is working in the Validation team (sorry, may have gotte= n this wrong) at Calxeda and is new to Xen * Konrad Wilk is Software Development Manager at Oracle, but also Xen maint= ainer in Linux * Lars Kurth is the community manager for the Xen Project and is volunteeri= ng to chair this working group until it becomes self-sustaining =3D=3D=3D Purpose and Scope of Working Group =3D=3D=3D We didn=92t really work through the items in the list in the agenda. We end= ed up having a discussion. I didn=92t capture all of it - the discussion g= ot quite engaged =96 and I was at taking notes. Please feel free to augment= on the list and correct me. Lars: We started with the list of items in the agenda, but realized that ma= ybe we need to take a step back and check our assumptions. Anthony: * Raised the point that the introduction of KVM autotest (http://www.linux-= kvm.org/page/KVM-Autotest) has been problematic. Developers generally tend = to write test code if it fits into their development workflow (i.e. they ca= n run tests easily locally on their branch very easily). * System Test frameworks (such as Xen OSSTest and XenRT) which are run *aft= er* submission are more problematic and many devs tend to ignore them * From Anthony=92s experience, we also approached the OSSTest / XenRT discu= ssions wrongly. We should *not* just ask developers to write tests, but und= erstand better what tests framework they would find attractive and provide = something which helps them. Lars: * We can fix the last point on engaging the community We then covered an angle on what is wrong with OSS test right now Konrad: stated that OSSTest is not well enough maintained and owned right = now (IanJ does this in his spare time). OSSTest or other systems the AB su= pports will need to be properly owned and funded, otherwise the system woul= d lose trust. Matt commented that he doesn=92t like the pushgate mechanism in OSSTest (we= didn=92t cover this in detail) On the other hand =85 running Coverity on the code base has been a huge suc= cess. Konrad: * In 2 months we had > 215 bug fixes * the quality of the xen core components according to coverity is now highe= r than that in the Linux kernel This implies that there is a willingness in the community to use tools to i= mprove quality. We then slipped into a discussion about goals: Anthony raised a few questio= ns * How do we determine how we (or what we propose) provides value to members= and the community * We would need to define measurable success criteria The conclusion we came to (note that we didn=92t vote) was * Understand what member companies want to get out of the framework * Understand what the community wants to get out of it (as value for compan= ies depends on community buy in). {{Action|Lars}} Carefully draft a mail to the devel lists (after sending to= this list for approval) along the lines of: * As AB we have resources that we can use to help the community * We are only willing to spend money, if we are confident enough that this = helps the community and is likely to be adopted * Here are a few options and what we think would help you [intended to seed= the discussion] * We are looking for volunteers in the community to work with the WG {{Action|All}} Each member company should provide a statement of what they = are trying to achieve, whether there are any specific test related items, c= onstraints, platforms, goals, etc. that they are care about =92=92=92Additional Notes: =92=92=92 I am also including some notes and discussions I had outside the WG meeting= at the bottom of this section. I had a conversation with a number of developers after the test talks * Anthony stated that maybe to have a test tool based on QEMU that allows p= eople to run some relevant tests locally would be best (problem: performanc= e). He also raised the point that both OSSTest and XenRT are just like auto= test. Why do we need yet another new test framework rather than using somet= hing which is out there already? * Ian Jackson stated that the main issue right now is that OSSTest is not p= roperly resourced and does have not good coverage. He spends 50% of his tim= e keeping OSSTest running and sometimes tracking down hardware issues. So w= hatever we do, resourcing needs to be resolved. * When I talked to Citrix Platform team members, it turned out that only a = few have actually used OSSTest (lack of documentation is the key issue). It= =92s too hard to get started with OSSTest right now. * Another developer stated (source not disclosed): why don=92t the member c= ompanies fund the =93creation of tests (regardless of test framework) if th= ey care about quality=94 * Ian Jackson: there are some interesting properties about OSSTest (e.g. th= e capability to submit test code with a spec on the fly). [Aside by Lars: t= his may be close enough to providing the capability to try something locall= y, if we there also was a capability to test a devs personal git branch wit= h some test code] * Ian J also stated that he would look at XenRT, if the code was made avail= able in a Git repo rather than a tarball. Any issues and discussions could = then happen using the normal ways of discussing larger code contributions o= n xen-devel This clearly shows that we may have an issue winning the community over. =3D=3D=3D Status Update and scenarios going forward =3D=3D=3D (mostly by Chris Sheperd) http://www.slideshare.net/xen_com_mgr/xen-summit-taas-and-xenrt-0 shows a s= tatus update Documentation at http://wiki.xenproject.org/wiki/Category:XenRT Harry Hart: mentioned that Verizon tried XenRT but that they got stuck (Asi= de Lars: the fact that right now XenRT is sort of outside the Xen Project c= ommunity does not lend itself to discuss and raise issues about it) The main benefit of XenRT is that we inherit a large number of existing tes= ts (including for example performance tests and others) At the time of the meeting, the intention was to show a demo running at htt= p://osuosl.org/, but OSL had not set this up =91=92=92Update: 8/11/13: Sorry for dropping the ball on this. We have the = systems basically ready to go outside of getting public internet plumbed. A= re you still in need of them? Please let me know!=92=92=92 The question now is whether a) Citrix is willing to spend some time on setting a test environment up (a= nd add support for xl) b) Whether others on this list think that a XenRT demo instance is a good i= dea My question to Chris: {{Action|Chris}} Determine whether Citrix is willing to set up XenRT on htt= p://osuosl.org/ (and add support for =93xl=94) {{Action|Chris}} Let Lance Albertson from http://osuosl.org/ know Now as we do have a WG, I suggest to CC it if the answer to the The other question I have to the group (in light of the previous discussion= ) is: {{Action|All} Let me know by =93replying +1=94 to this item whether you fee= l there is value in setting up a XenRT demo instance on http://osuosl.org/;= =93replying 0=94 means you don=92t care; =93replying -1=93 means you belie= ve this is a bad idea (to satisfy Xen Project conventions you would have to= justify why you think this) =3D=3D=3D Meeting Mechaniscs =3D=3D=3D {{Action|All} Let me know what day and time of week you would be available= for a monthly call. All people on the list are based in the EU or US (East= to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Ple= ase state 2-3 preferences. --_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable = object[type$=3D"x-shockwave-fla= sh"]:not([classid]),object[type$=3D"futuresplash"]:not([classid]),embed[typ= e$=3D"x-shockwave-flash"],embed[type$=3D"futuresplash"]{display:none !impor= tant}
Aravaind,

thanks for getting back. I have followed up a little about the system = test vs. unit test argument with the developers in Cambridge at least. I th= ink what this comes down to is really a question about workflow.

Right now, OSSTest as well as Xen RT require (in my understanding)
a) code to be submitted for review
b) code to be reviewed and be submitted by the committer to the stagin= g branch
c) only then tests are run
d) if stuff fails, it has to be taken out and the whole process starts= again

Whereas what would be really desirable and IMHO attractive to devs is = the following work-flow
a) developer has a well working prototype on their personal git branch= somewhere
b) developer can run a set of interesting (or new) tests on some machi= nes on different architectures locally (which is somewhat impractical)
c) alternatively developer points a to his branch, plus some interesti= ng tests and specifies interesting machines and tests are quickly run on a = central test farm (the question is how quickly and smoothly this process wo= uld be)
d) if all works well, code is submitted for review (and test results c= ould be attached)

I think this is what Anthony's argument ultimately comes down to. The = question is whether the second workflow is achievable with something like O= SSTest and XenRT. Of course the system testing approach is also needed.

Regards
Lars

From: Gopalakrishnan, Aravind [Aravind.Go= palakrishnan@amd.com]
Sent: 14 November 2013 23:54
To: Lars Kurth
Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework m= eeting in October

Hi Lars,

I am available on all = days (except Thursday) between 10am to 12noon

 

And, I am =93+1=94= for a demo instance setup at OSU.

 

I do agree with Anthon= y=92s comment that XenRT is similar to autotest.. but AFAICT, autotest=92s = xen testing support  is mostly non-existent..

Maybe understanding mo= re about the feature-set of XenRT could go some way to help figure out if w= e can just leverage XenRT or need to beef up Xen support in autotest..

 

(Apologies for the lat= e reply)

 

Thanks,

-Aravind.

 

From: wg-tes= t-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@= lists.xenproject.org] On Behalf Of Lars Kurth
Sent: Monday, November 11, 2013 6:55 AM
To: wg-test-framework@lists.xenproject.org; advisory-board@lists.xen= project.org
Cc: Slutz, Donald Christopher (dslutz@verizon.com); Hart, Harry D (h= hart@verizon.com); Coulis, Demetrios (Demetrios.Coulis@ca.com)
Subject: [Wg-test-framework] [Minutes] First Test WG framework meeti= ng in October

 

 

Hi all,

sorry it has taken me so long t= o put this together. I set up the mailing list, and signed you all up. Plea= se read the minutes

AND let me know what day and time of week you would be available for a mont= hly call. All people on the list are based in the EU or US (East to West co= ast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2= -3 preferences.

 

Lars

 

=3D=3D Attendees =3D=3D<= /p>

* Aravind Gopalakrishnan (AMD)<= /span>

* Anthony Liguori (AWS), Matt W= ilson (AWS)

* Chris Sheperd (Citrix)=

* Demetrios Coulis standing in = for Allan Roberto (CA)

* Greg Lutostanski (Calxeda)

* Konrad Wilk (Oracle)

* Lars Kurth (Chair, Xen Projec= t)

* Will Auld (Intel)

I believe, but am not 100% sure= that

*Harry Hart and Don Slutz (Veri= zon) also stayed at the meeting

 

{{Action|Don}} Please confirm w= hether you were at the meeting and whether Verizon would in fact also parti= cipate in the WG and who the respective company rep would be

 

=3D=3D Agenda =3D=3D

 

* Introductions

* Purpose and Scope of Working = Group

 

:Lars: My view is that the WG p= rovides oversight and guidance for creating a test infrastructure for the X= en Project on behalf of the Xen Community. This is merely providing a stake= in the ground to start a discussion.

 

As such, the group needs to

**Make proposals to the Advisor= y Board for funding, for example

*** What do we need to set up s= uch a system in terms of hosting space, machines, =85

*** Do we need a full-time reso= urce employed by the LF to maintain and evolve the framework, =85

** Make proposals to the commun= ity and work with the community on a solution that works for developers in = the community

*** Without community buy-in an= d creating something that the community actually wants, we won=92t be able = to improve upstream quality

** Make decisions to help boots= trap demo systems, proof of concepts, etc.

** The group will also need to = approve Statement of Works or provide input on hiring contractors/resource

** Influence or set ground rule= s to ensure that the money that the Advisory Board invests delivers value t= o the community and the Advisory Board

** Highlight focus areas for in= vestment: e.g. the group would decide on questions such as

*** Are there focus areas for t= est coverage the WG/AB cares about?

*** Can WG/AB members find reso= urces to cover these areas?

*** If not, do we need to use p= roject funds to seed such areas?

 

:Comment by Matt Wilson (AWS= ): I'd like to suggest that the AB and working groups try to avoid making t= echnology decisions. It makes sense for the WG to decide where to make inve= stments on behalf of Xen Project members (e.g. where to invest through contractors / grants, capital investments, e= tc.). While this might influence technology decisions, it should avoid trou= ble some engineering-by-committee problems.

 

:My proposal would be to go = through the list above and add/remove items and examples. We do not need to= make decisions on items listed above at this meeting (I see these as examp= le to clarify  the scope/purpose/boundaries of the WG). I would then take the output and draft a charter which we can = vote on in the WG subsequently and to put it forward to the Advisory Board = and the wider community.

 

* Status Update and scenarios g= oing forward (Chris Shepherd)

:We probably need to give a qui= ck overview of what we have (there are quite a few new people on the list)<= /span>

 

:Status update:

** The OSU Open Source Lab in p= rinciple agreed to set up a Test-As-A-Service rig for Xen for a limited tim= e

** The proposal is based on Cit= rix=92s internal XenRT system. This would be a proof of concept aimed at in= forming further investment decisions.

** OSSTest is already running (= however on Citrix servers, with limited access by the community) =85 see

*** http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/=

*** http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/

 

* Meeting Mechanics :

** Meeting cadence/time/etc.

** Mailing lists

 

=3D=3D=3D Introductions =3D=3D= =3D

Please feel free to expand and = ask questions, etc.

* Aravind Gopalakrishnan, works= in the Server OS team at AMD and specialises in RAS features

* Chris Sheperd, leads the Test= Department for XenServer at Citrix

* Will Auld is Performance Arch= itect and  Principal Engineer at Intel, but also works closely with th= e Cloud Virtualization Test Group

* Anthony Liguori is the QEMU p= roject lead at Amazon

* Demetrios Coulis, is product = manager for AppLogic at CA

* Greg Lutostanski is working i= n the Validation team (sorry, may have gotten this wrong) at Calxeda and is= new to Xen

* Konrad Wilk is Software Devel= opment Manager at Oracle, but also Xen maintainer in Linux

* Lars Kurth is the community m= anager for the Xen Project and is volunteering to chair this working group = until it becomes self-sustaining

 

=3D=3D=3D Purpose and Scope of = Working Group  =3D=3D=3D

 

We didn=92t really work through= the items in the list in the agenda. We ended up having a discussion. = ; I didn=92t capture all of it - the discussion got quite engaged =96 and I= was at taking notes. Please feel free to augment on the list and correct me.

 

Lars: We started with the list = of items in the agenda, but realized that maybe we need to take a step back= and check our assumptions.

 

Anthony:

* Raised the point that the int= roduction of KVM autotest (http://www.linux-kvm.org/page/KVM-Autotest) ha= s been problematic. Developers generally tend to write test code if it fits into their development workflow (i.e. t= hey can run tests easily locally on their branch very easily).

* System Test frameworks (such = as Xen OSSTest and XenRT) which are run *after* submission are more = problematic and many devs tend to ignore them

* From Anthony=92s experience, = we also approached the OSSTest / XenRT discussions wrongly. We should *n= ot* just ask developers to write tests, but understand better what test= s framework they would find attractive and provide something which helps them.

 

Lars:

* We can fix the last point on = engaging the community

 

We then covered an angle on wha= t is wrong with OSS test right now

 

Konrad:  stated that OSSTe= st is not well enough maintained and owned right now (IanJ does this in his= spare time).  OSSTest or other systems the AB supports will need to b= e properly owned and funded, otherwise the system would lose trust.

 

Matt commented that he doesn=92= t like the pushgate mechanism in OSSTest (we didn=92t cover this in detail)=

 

On the other hand =85 running C= overity on the code base has been a huge success.

 

Konrad:

* In 2 months we had > 215 b= ug fixes

* the quality of the xen core c= omponents according to coverity is now higher than that in the Linux kernel=

 

This implies that there is a wi= llingness in the community to use tools to improve quality.

 

We then slipped into a discussi= on about goals: Anthony raised a few questions

* How do we determine how we (o= r what we propose) provides value to members and the community

* We would need to define measu= rable success criteria

 

The conclusion we came to (note= that we didn=92t vote) was

* Understand what member compan= ies want to get out of the framework

* Understand what the community= wants to get out of it (as value for companies depends on community buy in= ).

 

{{Action|Lars}} Carefully draft= a mail to the devel lists (after sending to this list for approval) along = the lines of:

* As AB we have resources that = we can use to help the community

* We are only willing to spend = money, if we are confident enough that this helps the community and is like= ly to be adopted

* Here are a few options and wh= at we think would help you [intended to seed the discussion]

* We are looking for volunteers= in the community to work with the WG

 

{{Action|All}} Each member comp= any should provide a statement of what they are trying to achieve, whether = there are any specific test related items, constraints, platforms, goals, e= tc. that they are care about

 

=92=92=92Additional Notes: =92= =92=92

I am also including some notes = and discussions I had outside the WG meeting at the bottom of this section.=

 

I had a conversation with a num= ber of developers after the test talks

* Anthony stated that maybe to = have a test tool based on QEMU that allows people to run some relevant test= s locally would be best (problem: performance). He also raised the point th= at both OSSTest and XenRT are just like autotest. Why do we need yet another new test framework rather than using = something which is out there already?

* Ian Jackson stated that the m= ain issue right now is that OSSTest is not properly resourced and does have= not good coverage. He spends 50% of his time keeping OSSTest running and s= ometimes tracking down hardware issues. So whatever we do,  resourcing needs to be resolved.

* When I talked to Citrix Platf= orm team members, it turned out that only a few have actually used OSSTest = (lack of documentation is the key issue). It=92s too hard to get started wi= th OSSTest right now.

* Another developer stated (sou= rce not disclosed): why don=92t the member companies fund the =93creation o= f tests (regardless of test framework) if they care about quality=94=

* Ian Jackson: there are some i= nteresting properties about OSSTest (e.g. the capability to submit test cod= e with a spec on the fly). [Aside by Lars: this may be close enough to prov= iding the capability to try something locally, if we there also was a capability to test a devs personal git bra= nch with some test code]

* Ian J also stated that he wou= ld look at XenRT, if the code was made available in a Git repo rather than = a tarball. Any issues and discussions could then happen using the normal wa= ys of discussing larger code contributions on xen-devel

 

This clearly shows that we may = have an issue winning the community over.

 

=3D=3D=3D Status Update and sce= narios going forward  =3D=3D=3D

(mostly by Chris Sheperd)

 

http://www= .slideshare.net/xen_com_mgr/xen-summit-taas-and-xenrt-0 shows a status = update

Documentation at http://wiki.xenproject.org/wiki/Category:XenRT

 

Harry Hart: mentioned that Veri= zon tried XenRT but that they got stuck (Aside Lars: the fact that right no= w XenRT is sort of outside the Xen Project community does not lend itself t= o discuss and raise issues about it)

 

The main benefit of XenRT is th= at we inherit a large number of existing tests (including for example perfo= rmance tests and others)

At the time of the meeting, the= intention was to show a demo running at http://osuosl.org/, bu= t OSL had not set this up

 

=91=92=92Update: 8/11/13: Sorry= for dropping the ball on this. We have the systems basically ready to go o= utside of getting public internet plumbed. Are you still in need of them? P= lease let me know!=92=92=92

 

The question now is whether

a) Citrix is willing to spend s= ome time on setting a test environment up (and add support for xl)

b) Whether others on this list = think that a XenRT demo instance is a good idea

 

My question to Chris:

{{Action|Chris}} Determine whet= her Citrix is willing to set up XenRT on http://osuosl.org/ (an= d add support for =93xl=94)

{{Action|Chris}} Let Lance Albe= rtson from http://osuosl.org/ &nb= sp;know

 

Now as we do have a WG, I sugge= st to CC it if the answer to the

 

The other question I have to th= e group (in light of the previous discussion) is:

{{Action|All} Let me know by = =93replying +1=94 to this item whether you feel there is value in setti= ng up a XenRT demo instance on http://osuosl.org/; = =93replying 0=94 means you don=92t care; =93replying -1=93 means you believ= e this is a bad idea (to satisfy Xen Project conventions you would have to = justify why you think this)

 

=3D=3D=3D Meeting Mechaniscs&nb= sp; =3D=3D=3D

 

{{Action|All}  Let me know= what day and time of week you would be available for a monthly call. All p= eople on the list are based in the EU or US (East to West coast), so a 4pm = GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences.

 

--_000_B5EC053C78C5D746BD8AC3386F2303C90DC52FAMSPEX01CL01citri_-- --===============6750491874880416836== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework --===============6750491874880416836==-- From wg-test-framework-bounces@lists.xenproject.org Thu Nov 21 17:27:13 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 21 Nov 2013 17:27:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY25-00038q-84 for archives@lists.xenproject.org; Thu, 21 Nov 2013 17:27:13 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY23-00038l-PC for wg-test-framework@lists.xenproject.org; Thu, 21 Nov 2013 17:27:11 +0000 Received: from [85.158.143.35:49054] by server-1.bemta-4.messagelabs.com id 7C/80-02132-F624E825; Thu, 21 Nov 2013 17:27:11 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-16.tower-21.messagelabs.com!1385054822!127090!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 22735 invoked from network); 21 Nov 2013 17:27:03 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-16.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 21 Nov 2013 17:27:03 -0000 X-IronPort-AV: E=Sophos;i="4.93,745,1378857600"; d="scan'208";a="74443134" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 21 Nov 2013 17:27:01 +0000 Received: from AMSPEX01CL03.citrite.net (10.69.46.34) by FTLPEX01CL02.citrite.net (10.13.107.79) with Microsoft SMTP Server (TLS) id 14.2.342.4; Thu, 21 Nov 2013 12:27:01 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL03.citrite.net ([10.69.46.34]) with mapi id 14.02.0342.004; Thu, 21 Nov 2013 18:25:42 +0100 From: Lars Kurth To: "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVg== Date: Thu, 21 Nov 2013 17:25:41 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.168] MIME-Version: 1.0 X-DLP: MIA1 Cc: "Dugger, Donald D \(donald.d.dugger@intel.com\)" , Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Thu Nov 21 17:27:13 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 21 Nov 2013 17:27:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY25-00038q-84 for archives@lists.xenproject.org; Thu, 21 Nov 2013 17:27:13 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY23-00038l-PC for wg-test-framework@lists.xenproject.org; Thu, 21 Nov 2013 17:27:11 +0000 Received: from [85.158.143.35:49054] by server-1.bemta-4.messagelabs.com id 7C/80-02132-F624E825; Thu, 21 Nov 2013 17:27:11 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-16.tower-21.messagelabs.com!1385054822!127090!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 22735 invoked from network); 21 Nov 2013 17:27:03 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-16.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 21 Nov 2013 17:27:03 -0000 X-IronPort-AV: E=Sophos;i="4.93,745,1378857600"; d="scan'208";a="74443134" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 21 Nov 2013 17:27:01 +0000 Received: from AMSPEX01CL03.citrite.net (10.69.46.34) by FTLPEX01CL02.citrite.net (10.13.107.79) with Microsoft SMTP Server (TLS) id 14.2.342.4; Thu, 21 Nov 2013 12:27:01 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL03.citrite.net ([10.69.46.34]) with mapi id 14.02.0342.004; Thu, 21 Nov 2013 18:25:42 +0100 From: Lars Kurth To: "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVg== Date: Thu, 21 Nov 2013 17:25:41 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.168] MIME-Version: 1.0 X-DLP: MIA1 Cc: "Dugger, Donald D \(donald.d.dugger@intel.com\)" , Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Thu Nov 21 17:32:39 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 21 Nov 2013 17:32:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY7L-0003bF-La for archives@lists.xenproject.org; Thu, 21 Nov 2013 17:32:39 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY7K-0003bA-Dr for wg-test-framework@lists.xenproject.org; Thu, 21 Nov 2013 17:32:38 +0000 Received: from [193.109.254.147:11425] by server-7.bemta-14.messagelabs.com id 93/E6-14870-5B34E825; Thu, 21 Nov 2013 17:32:37 +0000 X-Env-Sender: prvs=0300f213d=aliguori@amazon.com X-Msg-Ref: server-6.tower-27.messagelabs.com!1385055154!4339011!1 X-Originating-IP: [207.171.184.29] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 25579 invoked from network); 21 Nov 2013 17:32:36 -0000 Received: from smtp-fw-9102.amazon.com (HELO smtp-fw-9102.amazon.com) (207.171.184.29) by server-6.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 21 Nov 2013 17:32:36 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1385055156; x=1416591156; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=SOU4JTJrxYN+PbCrQ/S6kSqPMrph9bRIrqKFfTgkS/E=; b=CknxvOAy/3e4tzayaOk3i47/tlc8U7A3twQbwceNiClXYPP6eKE5Uw0F IEwnfnfxBdejWwRfALMhtHS3FX3AJMyle5yUEWUeom02QpKN2n/ottmU9 vsF0id41Gkh6PKMGEeThSuGnhMVQk7UVLYdii/Wi6JI2hWTv8IO8pnmkx U=; X-IronPort-AV: E=Sophos;i="4.93,745,1378857600"; d="scan'208";a="37010792" Received: from smtp-in-1002.vdc.amazon.com ([10.118.132.23]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 21 Nov 2013 17:32:32 +0000 Received: from ex10-hub-9003.ant.amazon.com (ex10-hub-9003.ant.amazon.com [10.185.137.132]) by smtp-in-1002.vdc.amazon.com (8.14.7/8.14.7) with ESMTP id rALHWT3S008096 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=OK); Thu, 21 Nov 2013 17:32:31 GMT Received: from EX10-MBX-36008.ant.amazon.com ([fe80::ad76:8a73:f090:3ec]) by ex10-hub-9003.ant.amazon.com ([::1]) with mapi id 14.02.0342.003; Thu, 21 Nov 2013 09:32:23 -0800 From: "Liguori, Anthony" To: Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgAA89y6 Date: Thu, 21 Nov 2013 17:32:22 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.17.1.114] MIME-Version: 1.0 Cc: "Dugger, Donald D \(donald.d.dugger@intel.com\)" , Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org 8am Pacific is difficult for me as that's when I'm commuting. 7am or 9am would be better. Regards, Anthony Liguori ________________________________________ From: wg-test-framework-bounces@lists.xenproject.org [wg-test-framework-bounces@lists.xenproject.org] on behalf of Lars Kurth [lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 9:25 AM To: wg-test-framework@lists.xenproject.org Cc: Dugger, Donald D (donald.d.dugger@intel.com); Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Thu Nov 21 17:32:39 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 21 Nov 2013 17:32:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY7L-0003bF-La for archives@lists.xenproject.org; Thu, 21 Nov 2013 17:32:39 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjY7K-0003bA-Dr for wg-test-framework@lists.xenproject.org; Thu, 21 Nov 2013 17:32:38 +0000 Received: from [193.109.254.147:11425] by server-7.bemta-14.messagelabs.com id 93/E6-14870-5B34E825; Thu, 21 Nov 2013 17:32:37 +0000 X-Env-Sender: prvs=0300f213d=aliguori@amazon.com X-Msg-Ref: server-6.tower-27.messagelabs.com!1385055154!4339011!1 X-Originating-IP: [207.171.184.29] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 25579 invoked from network); 21 Nov 2013 17:32:36 -0000 Received: from smtp-fw-9102.amazon.com (HELO smtp-fw-9102.amazon.com) (207.171.184.29) by server-6.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 21 Nov 2013 17:32:36 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1385055156; x=1416591156; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=SOU4JTJrxYN+PbCrQ/S6kSqPMrph9bRIrqKFfTgkS/E=; b=CknxvOAy/3e4tzayaOk3i47/tlc8U7A3twQbwceNiClXYPP6eKE5Uw0F IEwnfnfxBdejWwRfALMhtHS3FX3AJMyle5yUEWUeom02QpKN2n/ottmU9 vsF0id41Gkh6PKMGEeThSuGnhMVQk7UVLYdii/Wi6JI2hWTv8IO8pnmkx U=; X-IronPort-AV: E=Sophos;i="4.93,745,1378857600"; d="scan'208";a="37010792" Received: from smtp-in-1002.vdc.amazon.com ([10.118.132.23]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 21 Nov 2013 17:32:32 +0000 Received: from ex10-hub-9003.ant.amazon.com (ex10-hub-9003.ant.amazon.com [10.185.137.132]) by smtp-in-1002.vdc.amazon.com (8.14.7/8.14.7) with ESMTP id rALHWT3S008096 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=OK); Thu, 21 Nov 2013 17:32:31 GMT Received: from EX10-MBX-36008.ant.amazon.com ([fe80::ad76:8a73:f090:3ec]) by ex10-hub-9003.ant.amazon.com ([::1]) with mapi id 14.02.0342.003; Thu, 21 Nov 2013 09:32:23 -0800 From: "Liguori, Anthony" To: Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgAA89y6 Date: Thu, 21 Nov 2013 17:32:22 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.17.1.114] MIME-Version: 1.0 Cc: "Dugger, Donald D \(donald.d.dugger@intel.com\)" , Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org 8am Pacific is difficult for me as that's when I'm commuting. 7am or 9am would be better. Regards, Anthony Liguori ________________________________________ From: wg-test-framework-bounces@lists.xenproject.org [wg-test-framework-bounces@lists.xenproject.org] on behalf of Lars Kurth [lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 9:25 AM To: wg-test-framework@lists.xenproject.org Cc: Dugger, Donald D (donald.d.dugger@intel.com); Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Thu Nov 21 18:33:40 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 21 Nov 2013 18:33:40 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjZ4O-0007Ao-J7 for archives@lists.xenproject.org; Thu, 21 Nov 2013 18:33:40 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjZ4N-0007Ab-2b for wg-test-framework@lists.xenproject.org; Thu, 21 Nov 2013 18:33:39 +0000 Received: from [85.158.137.68:59137] by server-2.bemta-3.messagelabs.com id 55/28-17329-2025E825; Thu, 21 Nov 2013 18:33:38 +0000 X-Env-Sender: donald.d.dugger@intel.com X-Msg-Ref: server-11.tower-31.messagelabs.com!1385058816!2094754!1 X-Originating-IP: [134.134.136.24] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjQgPT4gMzkwOTcx\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 23505 invoked from network); 21 Nov 2013 18:33:37 -0000 Received: from mga09.intel.com (HELO mga09.intel.com) (134.134.136.24) by server-11.tower-31.messagelabs.com with SMTP; 21 Nov 2013 18:33:37 -0000 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP; 21 Nov 2013 10:30:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.93,746,1378882800"; d="scan'208";a="412566239" Received: from orsmsx109.amr.corp.intel.com ([10.22.240.7]) by orsmga001.jf.intel.com with ESMTP; 21 Nov 2013 10:33:35 -0800 Received: from orsmsx102.amr.corp.intel.com ([169.254.1.112]) by ORSMSX109.amr.corp.intel.com ([169.254.2.204]) with mapi id 14.03.0123.003; Thu, 21 Nov 2013 10:33:35 -0800 From: "Dugger, Donald D" To: Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDg Date: Thu, 21 Nov 2013 18:33:35 +0000 Message-ID: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.138] MIME-Version: 1.0 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Thu Nov 21 18:33:40 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 21 Nov 2013 18:33:40 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjZ4O-0007Ao-J7 for archives@lists.xenproject.org; Thu, 21 Nov 2013 18:33:40 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjZ4N-0007Ab-2b for wg-test-framework@lists.xenproject.org; Thu, 21 Nov 2013 18:33:39 +0000 Received: from [85.158.137.68:59137] by server-2.bemta-3.messagelabs.com id 55/28-17329-2025E825; Thu, 21 Nov 2013 18:33:38 +0000 X-Env-Sender: donald.d.dugger@intel.com X-Msg-Ref: server-11.tower-31.messagelabs.com!1385058816!2094754!1 X-Originating-IP: [134.134.136.24] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjQgPT4gMzkwOTcx\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 23505 invoked from network); 21 Nov 2013 18:33:37 -0000 Received: from mga09.intel.com (HELO mga09.intel.com) (134.134.136.24) by server-11.tower-31.messagelabs.com with SMTP; 21 Nov 2013 18:33:37 -0000 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP; 21 Nov 2013 10:30:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.93,746,1378882800"; d="scan'208";a="412566239" Received: from orsmsx109.amr.corp.intel.com ([10.22.240.7]) by orsmga001.jf.intel.com with ESMTP; 21 Nov 2013 10:33:35 -0800 Received: from orsmsx102.amr.corp.intel.com ([169.254.1.112]) by ORSMSX109.amr.corp.intel.com ([169.254.2.204]) with mapi id 14.03.0123.003; Thu, 21 Nov 2013 10:33:35 -0800 From: "Dugger, Donald D" To: Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDg Date: Thu, 21 Nov 2013 18:33:35 +0000 Message-ID: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.138] MIME-Version: 1.0 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Fri Nov 22 14:12:21 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 22 Nov 2013 14:12:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjrT3-0005PI-5U for archives@lists.xenproject.org; Fri, 22 Nov 2013 14:12:21 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjrT2-0005P9-FA for wg-test-framework@lists.xenproject.org; Fri, 22 Nov 2013 14:12:20 +0000 Received: from [85.158.139.211:51948] by server-17.bemta-5.messagelabs.com id 2E/46-19152-3466F825; Fri, 22 Nov 2013 14:12:19 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-7.tower-206.messagelabs.com!1385129537!3633203!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 8199 invoked from network); 22 Nov 2013 14:12:18 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-7.tower-206.messagelabs.com with RC4-SHA encrypted SMTP; 22 Nov 2013 14:12:18 -0000 X-IronPort-AV: E=Sophos;i="4.93,752,1378857600"; d="scan'208";a="74758803" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 22 Nov 2013 14:12:16 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL01.citrite.net (10.13.107.78) with Microsoft SMTP Server (TLS) id 14.2.342.4; Fri, 22 Nov 2013 09:12:16 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Fri, 22 Nov 2013 15:11:34 +0100 From: Lars Kurth To: "Dugger, Donald D" , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDgACkR/IA= Date: Fri, 22 Nov 2013 14:11:34 +0000 Message-ID: References: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> In-Reply-To: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.30] MIME-Version: 1.0 X-DLP: MIA2 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Don, 4 PM GMT doesn't work for Anthony, but it seems that 3PM or 5PM would work. Days which we know don't work are Monday and Thursdays. As you are on the West Coast, would 3PM be too early? Lars -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 21 November 2013 18:34 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Fri Nov 22 14:12:21 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 22 Nov 2013 14:12:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjrT3-0005PI-5U for archives@lists.xenproject.org; Fri, 22 Nov 2013 14:12:21 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VjrT2-0005P9-FA for wg-test-framework@lists.xenproject.org; Fri, 22 Nov 2013 14:12:20 +0000 Received: from [85.158.139.211:51948] by server-17.bemta-5.messagelabs.com id 2E/46-19152-3466F825; Fri, 22 Nov 2013 14:12:19 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-7.tower-206.messagelabs.com!1385129537!3633203!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 8199 invoked from network); 22 Nov 2013 14:12:18 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-7.tower-206.messagelabs.com with RC4-SHA encrypted SMTP; 22 Nov 2013 14:12:18 -0000 X-IronPort-AV: E=Sophos;i="4.93,752,1378857600"; d="scan'208";a="74758803" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 22 Nov 2013 14:12:16 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL01.citrite.net (10.13.107.78) with Microsoft SMTP Server (TLS) id 14.2.342.4; Fri, 22 Nov 2013 09:12:16 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Fri, 22 Nov 2013 15:11:34 +0100 From: Lars Kurth To: "Dugger, Donald D" , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDgACkR/IA= Date: Fri, 22 Nov 2013 14:11:34 +0000 Message-ID: References: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> In-Reply-To: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.30] MIME-Version: 1.0 X-DLP: MIA2 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Don, 4 PM GMT doesn't work for Anthony, but it seems that 3PM or 5PM would work. Days which we know don't work are Monday and Thursdays. As you are on the West Coast, would 3PM be too early? Lars -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 21 November 2013 18:34 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Fri Nov 22 17:03:20 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 22 Nov 2013 17:03:20 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vju8W-0006xf-7n for archives@lists.xenproject.org; Fri, 22 Nov 2013 17:03:20 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vju8V-0006xG-Ej; Fri, 22 Nov 2013 17:03:19 +0000 Received: from [193.109.254.147:54191] by server-9.bemta-14.messagelabs.com id FF/13-08751-65E8F825; Fri, 22 Nov 2013 17:03:18 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-7.tower-27.messagelabs.com!1385139795!179145!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 16873 invoked from network); 22 Nov 2013 17:03:16 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-7.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 22 Nov 2013 17:03:16 -0000 X-IronPort-AV: E=Sophos;i="4.93,753,1378857600"; d="scan'208,217";a="74832953" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 22 Nov 2013 17:03:14 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL03.citrite.net (10.13.107.80) with Microsoft SMTP Server (TLS) id 14.2.342.4; Fri, 22 Nov 2013 12:03:14 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Fri, 22 Nov 2013 18:03:12 +0100 From: Lars Kurth To: "wg-test-framework@lists.xenproject.org" Thread-Topic: E-mail related to Test Framework for xen-devel, xen-api and mirage list for review (I do want to send this out before Thanksgiving - ideally on the 26th AM) Thread-Index: Ac7npJxFMK8upToRQrKJraefYFnM8w== Date: Fri, 22 Nov 2013 17:03:12 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.30] MIME-Version: 1.0 X-DLP: MIA1 Cc: "advisory-board@lists.xenproject.org" Subject: [Wg-test-framework] E-mail related to Test Framework for xen-devel, xen-api and mirage list for review (I do want to send this out before Thanksgiving - ideally on the 26th AM) X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============3276154934326554649==" Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org --===============3276154934326554649== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_" --_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Please let me know if you are happy if I send the following mail to the dev= el lists Regards Lars Hi all, you probably have all heard by now that the Xen Project Advisory Board (a g= roup of vendors who provide funds to the Xen Project that are intended to b= e used for the good of the community) recently created the Test Framework W= orking Group. http://wiki.xenproject.org/wiki/AB_WG/Test_Framework contains= more information about the group. The working group had its first meeting = a few weeks ago and one of the actions I had was to kick off a thread on de= velopment lists to figure out what would help the developer community. I was planning to kick off this thread with some questions and options, whi= ch reflect some discussions I had with individuals in the community, variou= s meetings (WG and AB meetings), etc. which I condensed into a picture. This reflects my personal opinion (not a Citrix opinion) and is merely inte= nded to get a discussion going. Feel free to pick it apart: I won't be ups= et. First, I wanted to clear up a few misconceptions that I have heard from a f= ew people: * The Advisory Board has funds that can be used to create an independently = hosted test infrastructure to help the developer community. However, funds = are limited. Thus, it is important that we do what is right for the Xen com= munity in the short term and the longer term. Otherwise, we will burn funds= that could be used to help the Xen community in other ways. * The Test Framework Working Group is made up of people by vendors who have= some experience in testing. * There is no intention to prescribe a test environment that you then have = to use. Advisory Board members made clear to me that they want to make sure= that what we end up with a solution that works for you. * At the Xen Developer Summit two different solutions for system testing we= re presented. The intention was to explain what is there and what we can us= e going forward. A presentation on OSSTest which runs regularly today was g= iven. And one for XenRT, for which there is a plan to get a small 3 box sys= tem up and running that can be used for you to look at. Citrix volunteered = to set this up at its own cost. * Just to be clear: what works for you may be one of these, none of these, = both of them, ... * There may also be different answers in the short and the long run. * At the end of the day, different community members will have different vi= ews. Also the Advisory Board members who provide the funds, will also have = specific interests that they will push for. Thus, in all likelihood, we wil= l have to find a good enough compromise. * Also, the vast majority of Advisory Board members care about the Hypervis= or (and not so much about XAPI and Mirage OS). Thus, it is likely that the = focus of the test system would be the Hypervisor. So let me try and condense some of the arguments and opinions I heard and i= nformation that is around. This list may be incomplete. =3D=3D Work Flow =3D=3D I added this section, because some members of the community and the working= group had prior experience with attempts to introduce a test infrastructur= e for an open source community in the past, and these may not have worked a= s well. I made up some of the terminology. *Local testing*: the basic idea here is for a developer to write their code= and be able to run some tests on their machine locally as part of their de= velopment workflow. We do have same capability to do this with OSSTest (see= http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-s= tep/), but the drawback is that this only works on the developers machine. = This may be good enough. We could "emulate" different environments using QE= MU, but that is likely to be slow. *System testing*: both OSSTest and XenRT are essentially system test enviro= nments, which would be run on code that is already in the Xen mainline (or = a staging branch). This is really great for regression testing on differen= t hardware platforms, but does not match the typical workflow in the commun= ity: a) code developed locally b) code submitted to list for review c) code reviewed and then submitted by the committer to the staging branch/= mainline d) only then tests are run e) if tests fail, the whole process starts again One of the issues here is that the elapsed time between development and the= tests being run. Another issue we have with the current set-up is that OSS= Test is currently hosted at Citrix, which makes difficult for community mem= bers to log into a machine and investigate issues. Admittedly this can fixe= d by having the system test hosted somewhere neutral (some sort of access c= ontrol would still remain). *Test on demand:* this would be a mixture between local testing and system = testing. It can probably be implemented in both OSSTest and XenRT, although= I believe XenRT is a closer to this model. The workflow would look like th= is: a) code developed locally b) developer checks working prototype into their personal git branch (which= would need to be accessible from the web) c) developer submits a job to the test farm which checks out and builds cod= e and runs a set of interesting (or new) tests on some machines of differen= t architectures. New tests would probably need to be on the git branch d) if the test fail, the cycle starts again d) if all works well, code is submitted for review as normal (and test resu= lts could be attached) This would only really work, if the interface into the system is easy to us= e and tests are run fast enough. Access control, etc. would be similar to s= ystem test. IMHO, this would be a nice mid to long-term goal, assuming it c= ould be made to work with the funds we have. =3D=3D OSSTest =3D=3D What runs now and thus easiest to get started on More Info * http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstes= t/ * http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-= step/ * http://www.youtube.com/watch?v=3DJxTFZIwZzJ8 Problems: * Run on Citrix premises (thus general access is an issue) * Ian Jackson is acting as sys-admin in his spare time. But, the Advisory B= oard could provide resource to fix this * Basic test coverage * Not a lot of documentation (which is a bit of a barrier to adoption) Risks * Not well understood =3D=3D XenRT =3D=3D Used by Citrix for XenServer testing. Tarballs have been made available by = Citrix under GPL. But the code has not been put into live repos: my underst= anding is that Citrix would do this, if the Xen community believes this is = valuable. More Info * http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT * http://www.youtube.com/watch?v=3Ds11_Iw7AI_U Problems: * No publicly accessible demo instance (this is being worked on - to be hos= ted on a small test bed at http://osuosl.org/ - work sponsored by Citrix) * Currently does not yet support "xl" (a "xl" connector is being worked on = - sponsored by Citrix) * Code not in yet public repo Potentially Interesting Properties: * Very large test coverage (including performance, security and other tests= ). Most of them should work once an "xl" connector is in place * Been in production at scale for a long time: thus well understood * XenRT has a lot of provisioning functionality and supports a distributed = architecture: aka the ability to manage machines in different locations (da= ta centres). The detail is abstracted away from users. This creates some in= teresting possibilities. For example: ** Hardware Vendors on the Advisory Board could provide hardware to the com= munity on their site (assuming that these can be hosted outside a firewall)= . Some HW vendors on the AB indicated that this would indeed be doable. ** This would open up the opportunity to make available cutting edge or "un= usual" HW for testing to the community. ** It would also mean that machines that would be expensive to ship and hos= t by the project, could be hosted on premise by AB vendors * XenRT has the capability to "inject" some test code on the fly (i.e. the = test code is attached to a job that is submitted). * I checked this with the XenRT devs and the *Test on demand* approach shou= ld be relatively easy to implement, but does not exist. I do not know what of the above would apply to OSSTest. Risks * Complexity * The cost of supporting such a system may be too high * Not in use by the community today * Not clear whether a *local test* version of XenRT is feasible =3D=3D Support and Ownership =3D=3D Whatever solution we go for, needs to be properly funded and looked after. = This is understood and the intention would be for the Xen Project (aka Advi= sory Board) to fund a Linux Foundation employee to do this on behalf of the= Xen Project: this is a bit like Greg KH and others being LF employees work= ing on the kernel. Some vendors on the Advisory Board indicated that provid= ing Colo/hosting space and HW would be possible in principle. =3D=3D Access =3D=3D Any central system, has of course the issue of access control and managing = users. This is obviously a barrier to entry (if we do not have also a local= test mechanism). Am wondering how other FOSS communities handle this. This= should certainly be the job of the Test Framework owner. Best Regards Lars --_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

Please let me know= if you are happy if I send the following mail to the devel lists

Regards=

Lars

 <= /p>

 

Hi all,

you probably have all heard by now that the Xen Project Advisory Board (a g= roup of vendors who provide funds to the Xen Project that are intended to b= e used for the good of the community) recently created the Test Framework W= orking Group. http://wik= i.xenproject.org/wiki/AB_WG/Test_Framework contains more information ab= out the group. The working group had its first meeting a few weeks ago and = one of the actions I had was to kick off a thread on development lists to figure out what would help the develo= per community.

I was planning to kick off this thread with some que= stions and options, which reflect some discussions I had with individuals i= n the community, various meetings (WG and AB meetings), etc. which I conden= sed into a picture.

This reflects my personal opinion (not a Citrix opin= ion) and is merely intended to get a discussion going.  Feel free to p= ick it apart: I won’t be upset.

First, I wanted to clear up a few misconceptions tha= t I have heard from a few people:

* The Advisory Board has funds that can be used to c= reate an independently hosted test infrastructure to help the developer com= munity. However, funds are limited. Thus, it is important that we do what i= s right for the Xen community in the short term and the longer term. Otherwise, we will burn funds that could b= e used to help the Xen community in other ways.
* The Test Framework Working Group is made up of people by vendors who have= some experience in testing.
* There is no intention to prescribe a test environment that you then have = to use. Advisory Board members made clear to me that they want to make sure= that what we end up with a solution that works for you.
* At the Xen Developer Summit two different solutions for system testing we= re presented. The intention was to explain what is there and what we can us= e going forward. A presentation on OSSTest which runs regularly today was g= iven. And one for XenRT, for which there is a plan to get a small 3 box system up and running that can be use= d for you to look at. Citrix volunteered to set this up at its own cost.&nb= sp;
* Just to be clear: what works for you may be one of these, none of these, = both of them, …
* There may also be different answers in the short and the long run.
* At the end of the day, different community members will have different vi= ews. Also the Advisory Board members who provide the funds, will also have = specific interests that they will push for. Thus, in all likelihood, we wil= l have to find a good enough compromise.
* Also, the vast majority of Advisory Board members care about the Hypervis= or (and not so much about XAPI and Mirage OS). Thus, it is likely that the = focus of the test system would be the Hypervisor. 

So let me try and condense some of the arguments and opinions I heard and i= nformation that is around. This list may be incomplete.

=3D=3D Work Flow =3D=3D

I added this section, because some members of the co= mmunity and the working group had prior experience with attempts to introdu= ce a test infrastructure for an open source community in the past, and thes= e may not have worked as well. I made up some of the terminology.

*Local testing*: the basic idea here is for a develo= per to write their code and be able to run some tests on their machine loca= lly as part of their development workflow. We do have same capability to do= this with OSSTest (see http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/), but the drawback is that this only works on the developers machin= e. This may be good enough. We could “emulate” different enviro= nments using QEMU, but that is likely to be slow.

*System testing*: both OSSTest and XenRT are essenti= ally system test environments, which would be run on code that is already i= n the Xen mainline (or a staging branch).  This is really great for re= gression testing on different hardware platforms, but does not match the typical workflow in the community:
a) code developed locally
b) code submitted to list for review
c) code reviewed and then submitted by the committer to the staging branch/= mainline
d) only then tests are run
e) if tests fail, the whole process starts again
One of the issues here is that the elapsed time between development and the= tests being run. Another issue we have with the current set-up is that OSS= Test is currently hosted at Citrix, which makes difficult for community mem= bers to log into a machine and investigate issues. Admittedly this can fixed by having the system test hosted somewhe= re neutral (some sort of access control would still remain).

*Test on demand:* this would be a mixture between lo= cal testing and system testing. It can probably be implemented in both OSST= est and XenRT, although I believe XenRT is a closer to this model. The work= flow would look like this:
a) code developed locally
b) developer checks working prototype into their personal git branch (which= would need to be accessible from the web)
c) developer submits a job to the test farm which checks out and builds cod= e and runs a set of interesting (or new) tests on some machines of differen= t architectures. New tests would probably need to be on the git branch
d) if the test fail, the cycle starts again
d) if all works well, code is submitted for review as normal (and test resu= lts could be attached)
This would only really work, if the interface into the system is easy to us= e and tests are run fast enough. Access control, etc. would be similar to s= ystem test. IMHO, this would be a nice mid to long-term goal, assuming it c= ould be made to work with the funds we have.

=3D=3D OSSTest =3D=3D

What runs now and thus easiest to get started on

More Info
* http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/=
* http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/
* http://www.youtu= be.com/watch?v=3DJxTFZIwZzJ8

Problems:
* Run on Citrix premises (thus general access is an issue)
* Ian Jackson is acting as sys-admin in his spare time. But, the Advisory B= oard could provide resource to fix this
* Basic test coverage
* Not a lot of documentation (which is a bit of a barrier to adoption)=

Risks
* Not well understood

=3D=3D XenRT =3D=3D

Used by Citrix for XenServer testing. Tarballs have = been made available by Citrix under GPL. But the code has not been put into= live repos: my understanding is that Citrix would do this, if the Xen comm= unity believes this is valuable.

More Info
* ht= tp://wiki.xenproject.org/wiki/Getting_Started_with_XenRT
* http://www.youtu= be.com/watch?v=3Ds11_Iw7AI_U

Problems:
* No publicly accessible demo instance (this is being worked on – to = be hosted on a small test bed at  http://osuosl.org/  – work sp= onsored by Citrix)
* Currently does not yet support “xl” (a “xl” conne= ctor is being worked on – sponsored by Citrix)
* Code not in yet public repo

Potentially Interesting Properties:
* Very large test coverage (including performance, security and other tests= ). Most of them should work once an “xl” connector is in place<= br> * Been in production at scale for a long time: thus well understood
* XenRT has a lot of provisioning functionality and supports a distributed = architecture: aka the ability to manage machines in different locations (da= ta centres). The detail is abstracted away from users. This creates some in= teresting possibilities. For example:
** Hardware Vendors on the Advisory Board could provide hardware to the com= munity on their site (assuming that these can be hosted outside a firewall)= . Some HW vendors on the AB indicated that this would indeed be doable.
** This would open up the opportunity to make available cutting edge or = 220;unusual” HW for testing to the community.
** It would also mean that machines that would be expensive to ship and hos= t by the project, could be hosted on premise by AB vendors
* XenRT has the capability to “inject” some test code on the fl= y (i.e. the test code is attached to a job that is submitted).
* I checked this with the XenRT devs and the *Test on demand* approach shou= ld be relatively easy to implement, but does not exist.

I do not know what of the above would apply to OSSTe= st.

Risks
* Complexity
* The cost of supporting such a system may be too high
* Not in use by the community today
* Not clear whether a *local test* version of XenRT is feasible<= /p>

=3D=3D Support and Ownership =3D=3D

Whatever solution we go for, needs to be properly fu= nded and looked after. This is understood and the intention would be for th= e Xen Project (aka Advisory Board) to fund a Linux Foundation employee to d= o this on behalf of the Xen Project: this is a bit like Greg KH and others being LF employees working on the ke= rnel. Some vendors on the Advisory Board indicated that providing Colo/host= ing space and HW would be possible in principle.

=3D=3D Access =3D=3D

Any central system, has of course the issue of acces= s control and managing users. This is obviously a barrier to entry (if we d= o not have also a local test mechanism). Am wondering how other FOSS commun= ities handle this. This should certainly be the job of the Test Framework owner.

Best Regards

Lars

--_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_-- --===============3276154934326554649== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework --===============3276154934326554649==-- From wg-test-framework-bounces@lists.xenproject.org Fri Nov 22 17:03:20 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 22 Nov 2013 17:03:20 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vju8W-0006xf-7n for archives@lists.xenproject.org; Fri, 22 Nov 2013 17:03:20 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vju8V-0006xG-Ej; Fri, 22 Nov 2013 17:03:19 +0000 Received: from [193.109.254.147:54191] by server-9.bemta-14.messagelabs.com id FF/13-08751-65E8F825; Fri, 22 Nov 2013 17:03:18 +0000 X-Env-Sender: lars.kurth@citrix.com X-Msg-Ref: server-7.tower-27.messagelabs.com!1385139795!179145!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 16873 invoked from network); 22 Nov 2013 17:03:16 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-7.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 22 Nov 2013 17:03:16 -0000 X-IronPort-AV: E=Sophos;i="4.93,753,1378857600"; d="scan'208,217";a="74832953" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 22 Nov 2013 17:03:14 +0000 Received: from AMSPEX01CL02.citrite.net (10.69.46.33) by FTLPEX01CL03.citrite.net (10.13.107.80) with Microsoft SMTP Server (TLS) id 14.2.342.4; Fri, 22 Nov 2013 12:03:14 -0500 Received: from AMSPEX01CL01.citrite.net ([169.254.6.125]) by AMSPEX01CL02.citrite.net ([169.254.7.242]) with mapi id 14.02.0342.004; Fri, 22 Nov 2013 18:03:12 +0100 From: Lars Kurth To: "wg-test-framework@lists.xenproject.org" Thread-Topic: E-mail related to Test Framework for xen-devel, xen-api and mirage list for review (I do want to send this out before Thanksgiving - ideally on the 26th AM) Thread-Index: Ac7npJxFMK8upToRQrKJraefYFnM8w== Date: Fri, 22 Nov 2013 17:03:12 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.2.30] MIME-Version: 1.0 X-DLP: MIA1 Cc: "advisory-board@lists.xenproject.org" Subject: [Wg-test-framework] E-mail related to Test Framework for xen-devel, xen-api and mirage list for review (I do want to send this out before Thanksgiving - ideally on the 26th AM) X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============3276154934326554649==" Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org --===============3276154934326554649== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_" --_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Please let me know if you are happy if I send the following mail to the dev= el lists Regards Lars Hi all, you probably have all heard by now that the Xen Project Advisory Board (a g= roup of vendors who provide funds to the Xen Project that are intended to b= e used for the good of the community) recently created the Test Framework W= orking Group. http://wiki.xenproject.org/wiki/AB_WG/Test_Framework contains= more information about the group. The working group had its first meeting = a few weeks ago and one of the actions I had was to kick off a thread on de= velopment lists to figure out what would help the developer community. I was planning to kick off this thread with some questions and options, whi= ch reflect some discussions I had with individuals in the community, variou= s meetings (WG and AB meetings), etc. which I condensed into a picture. This reflects my personal opinion (not a Citrix opinion) and is merely inte= nded to get a discussion going. Feel free to pick it apart: I won't be ups= et. First, I wanted to clear up a few misconceptions that I have heard from a f= ew people: * The Advisory Board has funds that can be used to create an independently = hosted test infrastructure to help the developer community. However, funds = are limited. Thus, it is important that we do what is right for the Xen com= munity in the short term and the longer term. Otherwise, we will burn funds= that could be used to help the Xen community in other ways. * The Test Framework Working Group is made up of people by vendors who have= some experience in testing. * There is no intention to prescribe a test environment that you then have = to use. Advisory Board members made clear to me that they want to make sure= that what we end up with a solution that works for you. * At the Xen Developer Summit two different solutions for system testing we= re presented. The intention was to explain what is there and what we can us= e going forward. A presentation on OSSTest which runs regularly today was g= iven. And one for XenRT, for which there is a plan to get a small 3 box sys= tem up and running that can be used for you to look at. Citrix volunteered = to set this up at its own cost. * Just to be clear: what works for you may be one of these, none of these, = both of them, ... * There may also be different answers in the short and the long run. * At the end of the day, different community members will have different vi= ews. Also the Advisory Board members who provide the funds, will also have = specific interests that they will push for. Thus, in all likelihood, we wil= l have to find a good enough compromise. * Also, the vast majority of Advisory Board members care about the Hypervis= or (and not so much about XAPI and Mirage OS). Thus, it is likely that the = focus of the test system would be the Hypervisor. So let me try and condense some of the arguments and opinions I heard and i= nformation that is around. This list may be incomplete. =3D=3D Work Flow =3D=3D I added this section, because some members of the community and the working= group had prior experience with attempts to introduce a test infrastructur= e for an open source community in the past, and these may not have worked a= s well. I made up some of the terminology. *Local testing*: the basic idea here is for a developer to write their code= and be able to run some tests on their machine locally as part of their de= velopment workflow. We do have same capability to do this with OSSTest (see= http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-s= tep/), but the drawback is that this only works on the developers machine. = This may be good enough. We could "emulate" different environments using QE= MU, but that is likely to be slow. *System testing*: both OSSTest and XenRT are essentially system test enviro= nments, which would be run on code that is already in the Xen mainline (or = a staging branch). This is really great for regression testing on differen= t hardware platforms, but does not match the typical workflow in the commun= ity: a) code developed locally b) code submitted to list for review c) code reviewed and then submitted by the committer to the staging branch/= mainline d) only then tests are run e) if tests fail, the whole process starts again One of the issues here is that the elapsed time between development and the= tests being run. Another issue we have with the current set-up is that OSS= Test is currently hosted at Citrix, which makes difficult for community mem= bers to log into a machine and investigate issues. Admittedly this can fixe= d by having the system test hosted somewhere neutral (some sort of access c= ontrol would still remain). *Test on demand:* this would be a mixture between local testing and system = testing. It can probably be implemented in both OSSTest and XenRT, although= I believe XenRT is a closer to this model. The workflow would look like th= is: a) code developed locally b) developer checks working prototype into their personal git branch (which= would need to be accessible from the web) c) developer submits a job to the test farm which checks out and builds cod= e and runs a set of interesting (or new) tests on some machines of differen= t architectures. New tests would probably need to be on the git branch d) if the test fail, the cycle starts again d) if all works well, code is submitted for review as normal (and test resu= lts could be attached) This would only really work, if the interface into the system is easy to us= e and tests are run fast enough. Access control, etc. would be similar to s= ystem test. IMHO, this would be a nice mid to long-term goal, assuming it c= ould be made to work with the funds we have. =3D=3D OSSTest =3D=3D What runs now and thus easiest to get started on More Info * http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstes= t/ * http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-= step/ * http://www.youtube.com/watch?v=3DJxTFZIwZzJ8 Problems: * Run on Citrix premises (thus general access is an issue) * Ian Jackson is acting as sys-admin in his spare time. But, the Advisory B= oard could provide resource to fix this * Basic test coverage * Not a lot of documentation (which is a bit of a barrier to adoption) Risks * Not well understood =3D=3D XenRT =3D=3D Used by Citrix for XenServer testing. Tarballs have been made available by = Citrix under GPL. But the code has not been put into live repos: my underst= anding is that Citrix would do this, if the Xen community believes this is = valuable. More Info * http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT * http://www.youtube.com/watch?v=3Ds11_Iw7AI_U Problems: * No publicly accessible demo instance (this is being worked on - to be hos= ted on a small test bed at http://osuosl.org/ - work sponsored by Citrix) * Currently does not yet support "xl" (a "xl" connector is being worked on = - sponsored by Citrix) * Code not in yet public repo Potentially Interesting Properties: * Very large test coverage (including performance, security and other tests= ). Most of them should work once an "xl" connector is in place * Been in production at scale for a long time: thus well understood * XenRT has a lot of provisioning functionality and supports a distributed = architecture: aka the ability to manage machines in different locations (da= ta centres). The detail is abstracted away from users. This creates some in= teresting possibilities. For example: ** Hardware Vendors on the Advisory Board could provide hardware to the com= munity on their site (assuming that these can be hosted outside a firewall)= . Some HW vendors on the AB indicated that this would indeed be doable. ** This would open up the opportunity to make available cutting edge or "un= usual" HW for testing to the community. ** It would also mean that machines that would be expensive to ship and hos= t by the project, could be hosted on premise by AB vendors * XenRT has the capability to "inject" some test code on the fly (i.e. the = test code is attached to a job that is submitted). * I checked this with the XenRT devs and the *Test on demand* approach shou= ld be relatively easy to implement, but does not exist. I do not know what of the above would apply to OSSTest. Risks * Complexity * The cost of supporting such a system may be too high * Not in use by the community today * Not clear whether a *local test* version of XenRT is feasible =3D=3D Support and Ownership =3D=3D Whatever solution we go for, needs to be properly funded and looked after. = This is understood and the intention would be for the Xen Project (aka Advi= sory Board) to fund a Linux Foundation employee to do this on behalf of the= Xen Project: this is a bit like Greg KH and others being LF employees work= ing on the kernel. Some vendors on the Advisory Board indicated that provid= ing Colo/hosting space and HW would be possible in principle. =3D=3D Access =3D=3D Any central system, has of course the issue of access control and managing = users. This is obviously a barrier to entry (if we do not have also a local= test mechanism). Am wondering how other FOSS communities handle this. This= should certainly be the job of the Test Framework owner. Best Regards Lars --_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

Please let me know= if you are happy if I send the following mail to the devel lists

Regards=

Lars

 <= /p>

 

Hi all,

you probably have all heard by now that the Xen Project Advisory Board (a g= roup of vendors who provide funds to the Xen Project that are intended to b= e used for the good of the community) recently created the Test Framework W= orking Group. http://wik= i.xenproject.org/wiki/AB_WG/Test_Framework contains more information ab= out the group. The working group had its first meeting a few weeks ago and = one of the actions I had was to kick off a thread on development lists to figure out what would help the develo= per community.

I was planning to kick off this thread with some que= stions and options, which reflect some discussions I had with individuals i= n the community, various meetings (WG and AB meetings), etc. which I conden= sed into a picture.

This reflects my personal opinion (not a Citrix opin= ion) and is merely intended to get a discussion going.  Feel free to p= ick it apart: I won’t be upset.

First, I wanted to clear up a few misconceptions tha= t I have heard from a few people:

* The Advisory Board has funds that can be used to c= reate an independently hosted test infrastructure to help the developer com= munity. However, funds are limited. Thus, it is important that we do what i= s right for the Xen community in the short term and the longer term. Otherwise, we will burn funds that could b= e used to help the Xen community in other ways.
* The Test Framework Working Group is made up of people by vendors who have= some experience in testing.
* There is no intention to prescribe a test environment that you then have = to use. Advisory Board members made clear to me that they want to make sure= that what we end up with a solution that works for you.
* At the Xen Developer Summit two different solutions for system testing we= re presented. The intention was to explain what is there and what we can us= e going forward. A presentation on OSSTest which runs regularly today was g= iven. And one for XenRT, for which there is a plan to get a small 3 box system up and running that can be use= d for you to look at. Citrix volunteered to set this up at its own cost.&nb= sp;
* Just to be clear: what works for you may be one of these, none of these, = both of them, …
* There may also be different answers in the short and the long run.
* At the end of the day, different community members will have different vi= ews. Also the Advisory Board members who provide the funds, will also have = specific interests that they will push for. Thus, in all likelihood, we wil= l have to find a good enough compromise.
* Also, the vast majority of Advisory Board members care about the Hypervis= or (and not so much about XAPI and Mirage OS). Thus, it is likely that the = focus of the test system would be the Hypervisor. 

So let me try and condense some of the arguments and opinions I heard and i= nformation that is around. This list may be incomplete.

=3D=3D Work Flow =3D=3D

I added this section, because some members of the co= mmunity and the working group had prior experience with attempts to introdu= ce a test infrastructure for an open source community in the past, and thes= e may not have worked as well. I made up some of the terminology.

*Local testing*: the basic idea here is for a develo= per to write their code and be able to run some tests on their machine loca= lly as part of their development workflow. We do have same capability to do= this with OSSTest (see http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/), but the drawback is that this only works on the developers machin= e. This may be good enough. We could “emulate” different enviro= nments using QEMU, but that is likely to be slow.

*System testing*: both OSSTest and XenRT are essenti= ally system test environments, which would be run on code that is already i= n the Xen mainline (or a staging branch).  This is really great for re= gression testing on different hardware platforms, but does not match the typical workflow in the community:
a) code developed locally
b) code submitted to list for review
c) code reviewed and then submitted by the committer to the staging branch/= mainline
d) only then tests are run
e) if tests fail, the whole process starts again
One of the issues here is that the elapsed time between development and the= tests being run. Another issue we have with the current set-up is that OSS= Test is currently hosted at Citrix, which makes difficult for community mem= bers to log into a machine and investigate issues. Admittedly this can fixed by having the system test hosted somewhe= re neutral (some sort of access control would still remain).

*Test on demand:* this would be a mixture between lo= cal testing and system testing. It can probably be implemented in both OSST= est and XenRT, although I believe XenRT is a closer to this model. The work= flow would look like this:
a) code developed locally
b) developer checks working prototype into their personal git branch (which= would need to be accessible from the web)
c) developer submits a job to the test farm which checks out and builds cod= e and runs a set of interesting (or new) tests on some machines of differen= t architectures. New tests would probably need to be on the git branch
d) if the test fail, the cycle starts again
d) if all works well, code is submitted for review as normal (and test resu= lts could be attached)
This would only really work, if the interface into the system is easy to us= e and tests are run fast enough. Access control, etc. would be similar to s= ystem test. IMHO, this would be a nice mid to long-term goal, assuming it c= ould be made to work with the funds we have.

=3D=3D OSSTest =3D=3D

What runs now and thus easiest to get started on

More Info
* http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/=
* http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-st= ep/
* http://www.youtu= be.com/watch?v=3DJxTFZIwZzJ8

Problems:
* Run on Citrix premises (thus general access is an issue)
* Ian Jackson is acting as sys-admin in his spare time. But, the Advisory B= oard could provide resource to fix this
* Basic test coverage
* Not a lot of documentation (which is a bit of a barrier to adoption)=

Risks
* Not well understood

=3D=3D XenRT =3D=3D

Used by Citrix for XenServer testing. Tarballs have = been made available by Citrix under GPL. But the code has not been put into= live repos: my understanding is that Citrix would do this, if the Xen comm= unity believes this is valuable.

More Info
* ht= tp://wiki.xenproject.org/wiki/Getting_Started_with_XenRT
* http://www.youtu= be.com/watch?v=3Ds11_Iw7AI_U

Problems:
* No publicly accessible demo instance (this is being worked on – to = be hosted on a small test bed at  http://osuosl.org/  – work sp= onsored by Citrix)
* Currently does not yet support “xl” (a “xl” conne= ctor is being worked on – sponsored by Citrix)
* Code not in yet public repo

Potentially Interesting Properties:
* Very large test coverage (including performance, security and other tests= ). Most of them should work once an “xl” connector is in place<= br> * Been in production at scale for a long time: thus well understood
* XenRT has a lot of provisioning functionality and supports a distributed = architecture: aka the ability to manage machines in different locations (da= ta centres). The detail is abstracted away from users. This creates some in= teresting possibilities. For example:
** Hardware Vendors on the Advisory Board could provide hardware to the com= munity on their site (assuming that these can be hosted outside a firewall)= . Some HW vendors on the AB indicated that this would indeed be doable.
** This would open up the opportunity to make available cutting edge or = 220;unusual” HW for testing to the community.
** It would also mean that machines that would be expensive to ship and hos= t by the project, could be hosted on premise by AB vendors
* XenRT has the capability to “inject” some test code on the fl= y (i.e. the test code is attached to a job that is submitted).
* I checked this with the XenRT devs and the *Test on demand* approach shou= ld be relatively easy to implement, but does not exist.

I do not know what of the above would apply to OSSTe= st.

Risks
* Complexity
* The cost of supporting such a system may be too high
* Not in use by the community today
* Not clear whether a *local test* version of XenRT is feasible<= /p>

=3D=3D Support and Ownership =3D=3D

Whatever solution we go for, needs to be properly fu= nded and looked after. This is understood and the intention would be for th= e Xen Project (aka Advisory Board) to fund a Linux Foundation employee to d= o this on behalf of the Xen Project: this is a bit like Greg KH and others being LF employees working on the ke= rnel. Some vendors on the Advisory Board indicated that providing Colo/host= ing space and HW would be possible in principle.

=3D=3D Access =3D=3D

Any central system, has of course the issue of acces= s control and managing users. This is obviously a barrier to entry (if we d= o not have also a local test mechanism). Am wondering how other FOSS commun= ities handle this. This should certainly be the job of the Test Framework owner.

Best Regards

Lars

--_000_B5EC053C78C5D746BD8AC3386F2303C90DF291AMSPEX01CL01citri_-- --===============3276154934326554649== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework --===============3276154934326554649==-- From wg-test-framework-bounces@lists.xenproject.org Fri Nov 22 17:42:43 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 22 Nov 2013 17:42:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vjukd-0004oh-9l for archives@lists.xenproject.org; Fri, 22 Nov 2013 17:42:43 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vjukb-0004oR-Lm for wg-test-framework@lists.xenproject.org; Fri, 22 Nov 2013 17:42:41 +0000 Received: from [85.158.143.35:12275] by server-3.bemta-4.messagelabs.com id 85/D7-32360-1979F825; Fri, 22 Nov 2013 17:42:41 +0000 X-Env-Sender: donald.d.dugger@intel.com X-Msg-Ref: server-16.tower-21.messagelabs.com!1385142159!420569!1 X-Originating-IP: [192.55.52.88] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTkyLjU1LjUyLjg4ID0+IDM3NDcyNQ==\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 20386 invoked from network); 22 Nov 2013 17:42:40 -0000 Received: from mga01.intel.com (HELO mga01.intel.com) (192.55.52.88) by server-16.tower-21.messagelabs.com with SMTP; 22 Nov 2013 17:42:40 -0000 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP; 22 Nov 2013 09:42:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.93,753,1378882800"; d="scan'208";a="438138991" Received: from orsmsx102.amr.corp.intel.com ([10.22.225.129]) by fmsmga002.fm.intel.com with ESMTP; 22 Nov 2013 09:42:33 -0800 Received: from orsmsx115.amr.corp.intel.com (10.22.240.11) by ORSMSX102.amr.corp.intel.com (10.22.225.129) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 22 Nov 2013 09:42:33 -0800 Received: from orsmsx102.amr.corp.intel.com ([169.254.1.112]) by ORSMSX115.amr.corp.intel.com ([10.22.240.11]) with mapi id 14.03.0123.003; Fri, 22 Nov 2013 09:42:32 -0800 From: "Dugger, Donald D" To: Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDgACkR/IAAB19VkA== Date: Fri, 22 Nov 2013 17:42:32 +0000 Message-ID: <6AF484C0160C61439DE06F17668F3BCB527E5E85@ORSMSX102.amr.corp.intel.com> References: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.138] MIME-Version: 1.0 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Lars- Just to make your life simple :-) No, I'm in Mountain time, 1 hr. later than California. 3PM GMT is fine every day except Tues. 5PM GMT works Tues to Fri. except for the first Thurs. of the month (you have the same conflict on that Thurs.) -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Friday, November 22, 2013 7:12 AM To: Dugger, Donald D; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Don, 4 PM GMT doesn't work for Anthony, but it seems that 3PM or 5PM would work. Days which we know don't work are Monday and Thursdays. As you are on the West Coast, would 3PM be too early? Lars -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 21 November 2013 18:34 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Fri Nov 22 17:42:43 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 22 Nov 2013 17:42:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vjukd-0004oh-9l for archives@lists.xenproject.org; Fri, 22 Nov 2013 17:42:43 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vjukb-0004oR-Lm for wg-test-framework@lists.xenproject.org; Fri, 22 Nov 2013 17:42:41 +0000 Received: from [85.158.143.35:12275] by server-3.bemta-4.messagelabs.com id 85/D7-32360-1979F825; Fri, 22 Nov 2013 17:42:41 +0000 X-Env-Sender: donald.d.dugger@intel.com X-Msg-Ref: server-16.tower-21.messagelabs.com!1385142159!420569!1 X-Originating-IP: [192.55.52.88] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTkyLjU1LjUyLjg4ID0+IDM3NDcyNQ==\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 20386 invoked from network); 22 Nov 2013 17:42:40 -0000 Received: from mga01.intel.com (HELO mga01.intel.com) (192.55.52.88) by server-16.tower-21.messagelabs.com with SMTP; 22 Nov 2013 17:42:40 -0000 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP; 22 Nov 2013 09:42:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.93,753,1378882800"; d="scan'208";a="438138991" Received: from orsmsx102.amr.corp.intel.com ([10.22.225.129]) by fmsmga002.fm.intel.com with ESMTP; 22 Nov 2013 09:42:33 -0800 Received: from orsmsx115.amr.corp.intel.com (10.22.240.11) by ORSMSX102.amr.corp.intel.com (10.22.225.129) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 22 Nov 2013 09:42:33 -0800 Received: from orsmsx102.amr.corp.intel.com ([169.254.1.112]) by ORSMSX115.amr.corp.intel.com ([10.22.240.11]) with mapi id 14.03.0123.003; Fri, 22 Nov 2013 09:42:32 -0800 From: "Dugger, Donald D" To: Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDgACkR/IAAB19VkA== Date: Fri, 22 Nov 2013 17:42:32 +0000 Message-ID: <6AF484C0160C61439DE06F17668F3BCB527E5E85@ORSMSX102.amr.corp.intel.com> References: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.138] MIME-Version: 1.0 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Lars- Just to make your life simple :-) No, I'm in Mountain time, 1 hr. later than California. 3PM GMT is fine every day except Tues. 5PM GMT works Tues to Fri. except for the first Thurs. of the month (you have the same conflict on that Thurs.) -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Friday, November 22, 2013 7:12 AM To: Dugger, Donald D; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Don, 4 PM GMT doesn't work for Anthony, but it seems that 3PM or 5PM would work. Days which we know don't work are Monday and Thursdays. As you are on the West Coast, would 3PM be too early? Lars -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 21 November 2013 18:34 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Wed Nov 27 17:19:00 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 27 Nov 2013 17:19:00 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VlilQ-0005zQ-0n for archives@lists.xenproject.org; Wed, 27 Nov 2013 17:19:00 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VlilO-0005zG-0V for wg-test-framework@lists.xenproject.org; Wed, 27 Nov 2013 17:18:58 +0000 Received: from [193.109.254.147:49889] by server-3.bemta-14.messagelabs.com id D9/C1-11000-18926925; Wed, 27 Nov 2013 17:18:57 +0000 X-Env-Sender: Chris.Shepherd@citrix.com X-Msg-Ref: server-16.tower-27.messagelabs.com!1385572735!1216115!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 1490 invoked from network); 27 Nov 2013 17:18:56 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-16.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 27 Nov 2013 17:18:56 -0000 X-IronPort-AV: E=Sophos;i="4.93,783,1378857600"; d="scan'208";a="76314142" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 27 Nov 2013 17:18:54 +0000 Received: from AMSPEX01CL01.citrite.net (10.69.46.32) by FTLPEX01CL01.citrite.net (10.13.107.78) with Microsoft SMTP Server (TLS) id 14.2.342.4; Wed, 27 Nov 2013 12:18:54 -0500 Received: from AMSPEX01CL02.citrite.net ([169.254.7.18]) by AMSPEX01CL01.citrite.net ([10.69.46.32]) with mapi id 14.02.0342.004; Wed, 27 Nov 2013 18:18:53 +0100 From: Chris Shepherd To: "Dugger, Donald D" , Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDgACkR/IAAB19VkAD6lS6A Date: Wed, 27 Nov 2013 17:18:52 +0000 Message-ID: References: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> <6AF484C0160C61439DE06F17668F3BCB527E5E85@ORSMSX102.amr.corp.intel.com> In-Reply-To: <6AF484C0160C61439DE06F17668F3BCB527E5E85@ORSMSX102.amr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.80.3.96] MIME-Version: 1.0 X-DLP: MIA1 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Lars, Given the options below I propose 3pm GMT on either Wednesday or Thursday. Regards Chris -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 22 November 2013 17:43 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Lars- Just to make your life simple :-) No, I'm in Mountain time, 1 hr. later than California. 3PM GMT is fine every day except Tues. 5PM GMT works Tues to Fri. except for the first Thurs. of the month (you have the same conflict on that Thurs.) -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Friday, November 22, 2013 7:12 AM To: Dugger, Donald D; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Don, 4 PM GMT doesn't work for Anthony, but it seems that 3PM or 5PM would work. Days which we know don't work are Monday and Thursdays. As you are on the West Coast, would 3PM be too early? Lars -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 21 November 2013 18:34 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework From wg-test-framework-bounces@lists.xenproject.org Wed Nov 27 17:19:00 2013 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 27 Nov 2013 17:19:00 +0000 Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VlilQ-0005zQ-0n for archives@lists.xenproject.org; Wed, 27 Nov 2013 17:19:00 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VlilO-0005zG-0V for wg-test-framework@lists.xenproject.org; Wed, 27 Nov 2013 17:18:58 +0000 Received: from [193.109.254.147:49889] by server-3.bemta-14.messagelabs.com id D9/C1-11000-18926925; Wed, 27 Nov 2013 17:18:57 +0000 X-Env-Sender: Chris.Shepherd@citrix.com X-Msg-Ref: server-16.tower-27.messagelabs.com!1385572735!1216115!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 1490 invoked from network); 27 Nov 2013 17:18:56 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-16.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 27 Nov 2013 17:18:56 -0000 X-IronPort-AV: E=Sophos;i="4.93,783,1378857600"; d="scan'208";a="76314142" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 27 Nov 2013 17:18:54 +0000 Received: from AMSPEX01CL01.citrite.net (10.69.46.32) by FTLPEX01CL01.citrite.net (10.13.107.78) with Microsoft SMTP Server (TLS) id 14.2.342.4; Wed, 27 Nov 2013 12:18:54 -0500 Received: from AMSPEX01CL02.citrite.net ([169.254.7.18]) by AMSPEX01CL01.citrite.net ([10.69.46.32]) with mapi id 14.02.0342.004; Wed, 27 Nov 2013 18:18:53 +0100 From: Chris Shepherd To: "Dugger, Donald D" , Lars Kurth , "wg-test-framework@lists.xenproject.org" Thread-Topic: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Thread-Index: Ac7m27TwPmofMqvgRhu54PupAM+JVgADHDDgACkR/IAAB19VkAD6lS6A Date: Wed, 27 Nov 2013 17:18:52 +0000 Message-ID: References: <6AF484C0160C61439DE06F17668F3BCB527E25E7@ORSMSX102.amr.corp.intel.com> <6AF484C0160C61439DE06F17668F3BCB527E5E85@ORSMSX102.amr.corp.intel.com> In-Reply-To: <6AF484C0160C61439DE06F17668F3BCB527E5E85@ORSMSX102.amr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.80.3.96] MIME-Version: 1.0 X-DLP: MIA1 Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October X-BeenThere: wg-test-framework@lists.xenproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen Project Test Framework Working Group List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: wg-test-framework-bounces@lists.xenproject.org Errors-To: wg-test-framework-bounces@lists.xenproject.org Lars, Given the options below I propose 3pm GMT on either Wednesday or Thursday. Regards Chris -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 22 November 2013 17:43 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Lars- Just to make your life simple :-) No, I'm in Mountain time, 1 hr. later than California. 3PM GMT is fine every day except Tues. 5PM GMT works Tues to Fri. except for the first Thurs. of the month (you have the same conflict on that Thurs.) -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Friday, November 22, 2013 7:12 AM To: Dugger, Donald D; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Don, 4 PM GMT doesn't work for Anthony, but it seems that 3PM or 5PM would work. Days which we know don't work are Monday and Thursdays. As you are on the West Coast, would 3PM be too early? Lars -----Original Message----- From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Dugger, Donald D Sent: 21 November 2013 18:34 To: Lars Kurth; wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk Subject: Re: [Wg-test-framework] [Minutes] First Test WG framework meeting in October 4PM GMT every day except Mon. works for me. -- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale Ph: 303/443-3786 -----Original Message----- From: Lars Kurth [mailto:lars.kurth@citrix.com] Sent: Thursday, November 21, 2013 10:26 AM To: wg-test-framework@lists.xenproject.org Cc: Konrad Rzeszutek Wilk; Dugger, Donald D Subject: RE: [Wg-test-framework] [Minutes] First Test WG framework meeting in October Hi all, first I wanted to let you know that Donald D Dugger will take over from Will as Intel representative. Also, I published cleaned up minutes (the original had some typos in it) at http://wiki.xenproject.org/wiki/AB_WG/Test/October_2013_Minutes We don't seem to have an agreement on times and dates for regular meetings. It seems Thursdays are off the table, Wednesdays are also pretty full. Konrad suggested 4pm GMT, which is 8am Pacific and 11am East. Is this too early for people on the west coast? I would suggest the second Tue of each month as a time with either 4PM or 5PM as time slot. Regards Lars Also including Konrad's reply below ... -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: 15 November 2013 19:34 To: Lars Kurth Subject: Re: [Advisory-board] [Minutes] First Test WG framework meeting in October > The other question I have to the group (in light of the previous discussion) is: > {{Action|All} Let me know by "replying +1" to this item whether you feel there is value in setting up a XenRT demo instance on http://osuosl.org/; "replying 0" means you don't care; "replying -1" means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this) +1 > > === Meeting Mechaniscs === > > {{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences. > 4pm _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework _______________________________________________ Wg-test-framework mailing list Wg-test-framework@lists.xenproject.org http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework