[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Wg-test-framework] Proposed Agenda for next Test Framework WG meeting



Hi all,

 

please find attached the agenda for the next Test Framework working group meeting.  The meeting is on Feb 13th at 14:00 UTC. Please check whether you have the invite.

 

The current list of WG members can be found here : http://wiki.xenproject.org/wiki/AB_WG/Test_Framework  If your company is on the Advisory Board, and you would like to influence the direction of the Test Framework WG, you can still nominate a representative for the Test Framework WG. Please drop me a line, if you want to join.

 

The good news is that the Advisory Board has approved the budget that we asked for (which is outlined below). The total allocated budget for the Test Framework is USD 190,612.50 (including Linux Foundation charges) – this is a conservative estimate. We may be able to build a test system with less that the allocated budget or build a larger system.

 

Note that Konrad Wilk from Oracle cannot attend the next few WG meetings, and has not found a rep from Oracle to cover for him. I agreed with Konrad, that we will try and use the mailing lists for decisions, reviewing documents, etc. as much as possible and use the meetings to raise concerns and issues. I also added Ian Jackson to this email thread and the mailing list, as he is maintaining and running OSSTest. OSSTest is the existing test framework for the Xen Project which is hosted by Citrix and his input is thus important for this group. Ian also nominated himself as community rep for the WG : we do have to follow the process that is outlined in the e-mail titled “Looking for a volunteer to represent the Xen Project developer community at Test Working”. If there are no other nominees and no objections by the maintainer community, Ian would be confirmed as community representative before the WG meeting.

 

== Agenda ==

 

I would like to review the Strawman of a plan at the meeting (see below). Feedback by e-mail prior to the meeting is desired. If I get feedback before the Test Framework WG meeting I will try to incorporate.

 

At a minimum, I would like to

* Get agreement on some of the basic principles in the Strawman

** The split between short term and long term goals

** Agreement by stake-holders, i.e. Ian Jackson and Chris Shepherd, to provide the information that we need in the short and long term

** Identify a technical leader : in the best case a WG member volunteers and will be confirmed by the group.
*** If nobody steps up, we would set a deadline.

*** If the deadline passes and still nobody steps up, I will need to ask the Advisory Board for advice on how to proceed and whether a shared Test Infrastructure is still viable

** Identify a deputy who will work alongside me on non-technical questions

* As a stretch goal, we may be able to approve sections of the plan, or the entire plan

 

== Strawman of a Plan to move forward ==

 

=== Ownership ===

 

A) We need someone from this group with technical expertise and insight into testing Xen who can lead and drive technical tasks in the WG. I cannot fulfil this role and without it, I do not believe we will get anywhere. I am looking for one or several of the vendors on this group to step up. Given that both OSSTest and XenRT are currently the only viable solutions for the test framework on the table, the reality is that Citrix needs to step up. Unless we throw another solution into the mix.

 

The tasks at hand that the technical leader will need to drive are :

* Describe/specify a viable Hardware set-up that we can afford :
   my suggestion here would be that as a starting point, Ian Jackson and Chris Shepherd work together and work out a minimum environment that works for both OSSTest and XenRT.
   This would allow us to de-couple the “what are we doing in the long term” from all the logistical questions. If necessary (and we have enough budget), we can extend the setup as needed

* Establish concrete requirements for a COLO provider (ideally with a short-list of possible vendors)

* Establish requirements for sysadmin and support

* Establish a plan that lists the main tasks that are needed to get us to a system that is up and running in the short term (I can help with the plan)
* Work with all stake-holders on a long-term plan (I can help with the plan)

B) Ownership of non-technical areas of the WG. I can do this, but would prefer if someone else within the group would act as a deputy and work with me. I travel quite frequently and thus work items may get stuck.

 

The tasks at hand that this WG member will need to drive are :

* Work with the community representative and the developer community to ensure that what we propose is valuable to the Xen project developers and would in fact be used by them

* Handle any contractual issues (e.g. finding and getting a COLO, buying licenses, buying machines) – in other words act as the interface to the Linux Foundation

* Resolving any issues related to sysadmin and support (e.g. hiring a contractor or part-time resource as needed and resolve any practical issues around it)

* Work on proposals for process related questions, such as

** Who can access, how is access managed

** How do we add new HW in future, how do we add extra software configurations

** Etc.

=== Short Term Goal ===

 

I think we should transition the existing OSSTest system to new hardware immediately and then consider how to evolve our test system in the future. We want to end up with a solution that the community can use to test xen.git, right away, at our current level of service or better.  At the moment only OSSTest can do this : there are too many open questions related to XenRT and progress on a XenRT proof of concept has been very slow (this has pretty much stalled since August 2013). However we should choose the infrastructure such that both OSSTest and XenRT can run on it.

 

This approach has several advantages

* It decouples creating the infrastructure from the choice of test system

* It allows us to iron out any logistical and practical issues that owning an independently hosted test infrastructure involves

* It delivers immediate value to the community and shows progress with little disruption. If we stay in limbo, the Xen Project developer community will not write any new tests. Developers will simply wait for a decision.

* It leaves all options open,  while creating a motivation to work on the Long Term Goals and ensure we end up with a solution that works. If we fail to agree and do not deliver on the Long Term Goals, the Xen Project will still have an independent test infrastructure. However the infrastructure may be less aligned with the commercial interests of the Advisory Board. In other words, we create a motivation to resolve the long term goals more quickly than otherwise would exist.

 

=== Long Term Goal ===

 

To ensure that we do the right thing in the long run, I believe we need to take stock of the technical options we have: right now we have a choice of OSSTest and XenRT. There may be others as well, which can of course be proposed.

To figure out what is best in the long run, we need to get a concrete understanding of

* The level of testing that can be achieved with the solutions we have available today (maybe in form of a very coarse list such as “boot a variety of different guests”, … for per solution)

* The Hardware requirements that are needed to do this

* Software requirements (e.g. licenses)

* Highlight any issues, work that needs to be performed, etc. BEFORE the solution can be deployed

* Highlight any concerns that may prevent the Xen Project developer community from adopting each solution and mitigations on how we may address these
  (I can help with this and will work with the community representative)
* And there may be other areas that I am missing


To drive this forward, we need a champion for each proposal who will work with the WG (ideally is a member of it) and who commits to providing the necessary information and if necessary secures resources in their respective organization to ensure that gaps and community concerns can be addressed. It also requires input from the Community rep on the WG.

Best Regards

Lars

 

-------------------------

 

Source: Chris Shepherd, Citrix

Based on operational expertise with XenRT

Item

Item

Cost

Notes

One off costs for hardware

Core infrastructure (filers, etc.)

24,000.00

A pool of servers and shared storage to run the various VMs you need to run the network, monitor infrastructure, pxe boot, dhcp, scheduling etc. Osstest would presumably need something similar.

XenRT 'site' infrastructure

12,000.00

XenRT can be built up into multiple ‘sites’, each consisting of a number of servers (we have sites of 16, 32 and 64 servers). Each site needs its own small pool of servers with shared storage and switches to run the site. For a small XenRT deployment (one site) we could probably share the core infrastructure for the site infrastructure, thereby removing (or at least postponing) this cost.

16 mid range test servers at $4K each

64,000.00

Would only include off-the shelf hardware (no prototypes)
Could be lower spec machines (OSSTest machines are at $1K)
A reasonable approach would be to use a mixture of machines up to $4K, which should get us to more than 16 servers

Total:

100,000.00

Colo cost / month

Cost per rack / month

2,500.00

Inclusive of space, power and cooling

20Mbps internet with high SLA / month

1,000.00

Could reduce this cost very significantly in return for lower SLA

Sysadmin for HW only

500.00

Based on experience with XenRT, we will probably need 1 day per month for routine maintenance, plus time to setup any new equipment.
Don’t have a mechanism to do this (avg hourly rate $30/h)
Conservatively costing this at $500

Total/month:

4,000.00

Remote admin and support of environment

20% of a tools dev /devops type person

This is based on Citrix effort to support Citrix' XenRT lab which consists of well over 200 servers at 2 separate physical sites (US west coast and UK). Includes:
a) General user support e.g. answering usage queries including helping trouble-shoot failed tests (that may be due to infrastructure problems rather than product test)
b) Clearing out files that are taking up disk space
c) Monitoring server & network health and fixing broken machines e.g. server lost BIOS setting, network connectivity etc

I don't know the typical cost for such a profile and of course there is the issue that we cannot employ 20% of a person. An average test engineer salary in the US costs $88000 per annum.

It seems that the overhead for OSSTest is higher (Ian Jackson says he spends 50% of his time looking after fewer machines)

Estimated Total/month:

3,750.00

Conservatively assuming 50% of an engineer for $90K (unloaded) as a ball-park figure for feasibility purpuses only. This item would need to be revised after some more discussion within the Advisory Board.

 

 

_______________________________________________
Wg-test-framework mailing list
Wg-test-framework@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.