Hi Lars,
I am available on all days (except Thursday) between 10am to 12noon
And, I am “+1” for a demo instance setup at OSU.
I do agree with Anthony’s comment that XenRT is similar to autotest.. but AFAICT, autotest’s xen testing support is mostly non-existent..
Maybe understanding more about the feature-set of XenRT could go some way to help figure out if we can just leverage XenRT or need to beef up Xen support in autotest..
(Apologies for the late reply)
Thanks,
-Aravind.
From: wg-test-framework-bounces@xxxxxxxxxxxxxxxxxxxx [mailto:wg-test-framework-bounces@xxxxxxxxxxxxxxxxxxxx]
On Behalf Of Lars Kurth
Sent: Monday, November 11, 2013 6:55 AM
To: wg-test-framework@xxxxxxxxxxxxxxxxxxxx; advisory-board@xxxxxxxxxxxxxxxxxxxx
Cc: Slutz, Donald Christopher (dslutz@xxxxxxxxxxx); Hart, Harry D (hhart@xxxxxxxxxxx); Coulis, Demetrios (Demetrios.Coulis@xxxxxx)
Subject: [Wg-test-framework] [Minutes] First Test WG framework meeting in October
Hi all,
sorry it has taken me so long to put this together. I set up the mailing list, and signed you all up. Please read the minutes
AND let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best. Please state 2-3 preferences.
Lars
== Attendees ==
* Aravind Gopalakrishnan (AMD)
* Anthony Liguori (AWS), Matt Wilson (AWS)
* Chris Sheperd (Citrix)
* Demetrios Coulis standing in for Allan Roberto (CA)
* Greg Lutostanski (Calxeda)
* Konrad Wilk (Oracle)
* Lars Kurth (Chair, Xen Project)
* Will Auld (Intel)
I believe, but am not 100% sure that
*Harry Hart and Don Slutz (Verizon) also stayed at the meeting
{{Action|Don}} Please confirm whether you were at the meeting and whether Verizon would in fact also participate in the WG and who the respective company rep would be
== Agenda ==
* Introductions
* Purpose and Scope of Working Group
:Lars: My view is that the WG provides oversight and guidance for creating a test infrastructure for the Xen Project on behalf of the Xen Community. This is merely providing a stake in the ground to start a discussion.
As such, the group needs to
**Make proposals to the Advisory Board for funding, for example
*** What do we need to set up such a system in terms of hosting space, machines, …
*** Do we need a full-time resource employed by the LF to maintain and evolve the framework, …
** Make proposals to the community and work with the community on a solution that works for developers in the community
*** Without community buy-in and creating something that the community actually wants, we won’t be able to improve upstream quality
** Make decisions to help bootstrap demo systems, proof of concepts, etc.
** The group will also need to approve Statement of Works or provide input on hiring contractors/resource
** Influence or set ground rules to ensure that the money that the Advisory Board invests delivers value to the community and the Advisory Board
** Highlight focus areas for investment: e.g. the group would decide on questions such as
*** Are there focus areas for test coverage the WG/AB cares about?
*** Can WG/AB members find resources to cover these areas?
*** If not, do we need to use project funds to seed such areas?
:Comment by Matt Wilson (AWS): I'd like to suggest that the AB and working groups try to avoid making technology decisions. It makes sense for the WG to decide where to make investments on behalf of Xen Project members
(e.g. where to invest through contractors / grants, capital investments, etc.). While this might influence technology decisions, it should avoid trouble some engineering-by-committee problems.
:My proposal would be to go through the list above and add/remove items and examples. We do not need to make decisions on items listed above at this meeting (I see these as example to clarify the scope/purpose/boundaries
of the WG). I would then take the output and draft a charter which we can vote on in the WG subsequently and to put it forward to the Advisory Board and the wider community.
* Status Update and scenarios going forward (Chris Shepherd)
:We probably need to give a quick overview of what we have (there are quite a few new people on the list)
:Status update:
** The OSU Open Source Lab in principle agreed to set up a Test-As-A-Service rig for Xen for a limited time
** The proposal is based on Citrix’s internal XenRT system. This would be a proof of concept aimed at informing further investment decisions.
** OSSTest is already running (however on Citrix servers, with limited access by the community) … see
***
http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/
***
http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/
* Meeting Mechanics :
** Meeting cadence/time/etc.
** Mailing lists
=== Introductions ===
Please feel free to expand and ask questions, etc.
* Aravind Gopalakrishnan, works in the Server OS team at AMD and specialises in RAS features
* Chris Sheperd, leads the Test Department for XenServer at Citrix
* Will Auld is Performance Architect and Principal Engineer at Intel, but also works closely with the Cloud Virtualization Test Group
* Anthony Liguori is the QEMU project lead at Amazon
* Demetrios Coulis, is product manager for AppLogic at CA
* Greg Lutostanski is working in the Validation team (sorry, may have gotten this wrong) at Calxeda and is new to Xen
* Konrad Wilk is Software Development Manager at Oracle, but also Xen maintainer in Linux
* Lars Kurth is the community manager for the Xen Project and is volunteering to chair this working group until it becomes self-sustaining
=== Purpose and Scope of Working Group ===
We didn’t really work through the items in the list in the agenda. We ended up having a discussion. I didn’t capture all of it - the discussion got quite engaged – and I was at taking notes. Please feel free to augment
on the list and correct me.
Lars: We started with the list of items in the agenda, but realized that maybe we need to take a step back and check our assumptions.
Anthony:
* Raised the point that the introduction of KVM autotest (http://www.linux-kvm.org/page/KVM-Autotest) has been problematic. Developers generally
tend to write test code if it fits into their development workflow (i.e. they can run tests easily locally on their branch very easily).
* System Test frameworks (such as Xen OSSTest and XenRT) which are run *after* submission are more problematic and many devs tend to ignore them
* From Anthony’s experience, we also approached the OSSTest / XenRT discussions wrongly. We should *not* just ask developers to write tests, but understand better what tests framework they would find attractive
and provide something which helps them.
Lars:
* We can fix the last point on engaging the community
We then covered an angle on what is wrong with OSS test right now
Konrad: stated that OSSTest is not well enough maintained and owned right now (IanJ does this in his spare time). OSSTest or other systems the AB supports will need to be properly owned and funded, otherwise the system
would lose trust.
Matt commented that he doesn’t like the pushgate mechanism in OSSTest (we didn’t cover this in detail)
On the other hand … running Coverity on the code base has been a huge success.
Konrad:
* In 2 months we had > 215 bug fixes
* the quality of the xen core components according to coverity is now higher than that in the Linux kernel
This implies that there is a willingness in the community to use tools to improve quality.
We then slipped into a discussion about goals: Anthony raised a few questions
* How do we determine how we (or what we propose) provides value to members and the community
* We would need to define measurable success criteria
The conclusion we came to (note that we didn’t vote) was
* Understand what member companies want to get out of the framework
* Understand what the community wants to get out of it (as value for companies depends on community buy in).
{{Action|Lars}} Carefully draft a mail to the devel lists (after sending to this list for approval) along the lines of:
* As AB we have resources that we can use to help the community
* We are only willing to spend money, if we are confident enough that this helps the community and is likely to be adopted
* Here are a few options and what we think would help you [intended to seed the discussion]
* We are looking for volunteers in the community to work with the WG
{{Action|All}} Each member company should provide a statement of what they are trying to achieve, whether there are any specific test related items, constraints, platforms, goals, etc. that they are care about
’’’Additional Notes: ’’’
I am also including some notes and discussions I had outside the WG meeting at the bottom of this section.
I had a conversation with a number of developers after the test talks
* Anthony stated that maybe to have a test tool based on QEMU that allows people to run some relevant tests locally would be best (problem: performance). He also raised the point that both OSSTest and XenRT are just like
autotest. Why do we need yet another new test framework rather than using something which is out there already?
* Ian Jackson stated that the main issue right now is that OSSTest is not properly resourced and does have not good coverage. He spends 50% of his time keeping OSSTest running and sometimes tracking down hardware issues.
So whatever we do, resourcing needs to be resolved.
* When I talked to Citrix Platform team members, it turned out that only a few have actually used OSSTest (lack of documentation is the key issue). It’s too hard to get started with OSSTest right now.
* Another developer stated (source not disclosed): why don’t the member companies fund the “creation of tests (regardless of test framework) if they care about quality”
* Ian Jackson: there are some interesting properties about OSSTest (e.g. the capability to submit test code with a spec on the fly). [Aside by Lars: this may be close enough to providing the capability to try something
locally, if we there also was a capability to test a devs personal git branch with some test code]
* Ian J also stated that he would look at XenRT, if the code was made available in a Git repo rather than a tarball. Any issues and discussions could then happen using the normal ways of discussing larger code contributions
on xen-devel
This clearly shows that we may have an issue winning the community over.
=== Status Update and scenarios going forward ===
(mostly by Chris Sheperd)
http://www.slideshare.net/xen_com_mgr/xen-summit-taas-and-xenrt-0 shows a status update
Documentation at
http://wiki.xenproject.org/wiki/Category:XenRT
Harry Hart: mentioned that Verizon tried XenRT but that they got stuck (Aside Lars: the fact that right now XenRT is sort of outside the Xen Project community does not lend itself to discuss and raise issues about it)
The main benefit of XenRT is that we inherit a large number of existing tests (including for example performance tests and others)
At the time of the meeting, the intention was to show a demo running at
http://osuosl.org/, but OSL had not set this up
‘’’Update: 8/11/13: Sorry for dropping the ball on this. We have the systems basically ready to go outside of getting public internet plumbed. Are you still in need of them? Please let me know!’’’
The question now is whether
a) Citrix is willing to spend some time on setting a test environment up (and add support for xl)
b) Whether others on this list think that a XenRT demo instance is a good idea
My question to Chris:
{{Action|Chris}} Determine whether Citrix is willing to set up XenRT on
http://osuosl.org/ (and add support for “xl”)
{{Action|Chris}} Let Lance Albertson from
http://osuosl.org/ know
Now as we do have a WG, I suggest to CC it if the answer to the
The other question I have to the group (in light of the previous discussion) is:
{{Action|All} Let me know by “replying +1” to this item whether you feel there is value in setting up a XenRT demo instance on
http://osuosl.org/; “replying 0” means you don’t care; “replying -1“ means you believe this is a bad idea (to satisfy Xen Project conventions you would have to justify why you think this)
=== Meeting Mechaniscs ===
{{Action|All} Let me know what day and time of week you would be available for a monthly call. All people on the list are based in the EU or US (East to West coast), so a 4pm GMT or 5pm GMT slot would probably work best.
Please state 2-3 preferences.
