Re: [FuSa SIG] Resources related to testing on Xen

[Wei Liu <liuw@xxxxxxxxx>]

CC the right Andrew :-)

On Thu, 20 Jun 2019 at 17:57, Lars Kurth <lars.kurth@xxxxxxxxxx> wrote:
>
> Hi all,
>
>
>
> I had an action to put some resources regarding to testing on Xen together. I 
> also CC’ed people who have an interest in CI systems in Xen and own or have 
> worked on parts of the system. Following this, there is then an action by 
> FuSa SIG people on the TO list to put together a set of questions together 
> and for Lars to arrange a demo/Q&A.
>
>
>
> @Ian, @Andy, @Wei, @Doug: the intention was to look at what we do in Xen now 
> and see whether there are any tie-ins with existing test infrastructure in 
> Xen to build a baseline for more extensive testing for a subset of the Xen 
> codebase.
>
>
>
> Best Regards
>
> Lars
>
>
>
> Xen Source Trees
>
> Commits to Xen trees are made to a staging tree
>
> Changes need to pass our push gate which is controlled by OSSTEST which runs 
> different configurations of Xen on different Hardware.
> Once the tests have passed commits on staging trees are automatically merged 
> to master trees
>
>
>
> GitLab CI (unofficial)
>
> URL: https://gitlab.com/xen-project/xen/pipelines
>
> This is a skunkworks project which was started by Doug, which does the 
> following for every commit to the Xen staging tree
>
> It does build tests for different combinations of Linux distros and compilers 
> (gcc, clang)
> It also does some very basic run-time testing (aka does Xen boot using 
> emulation via QEMU)
> It uses container based images which can be downloaded to reproduce issues 
> locally
> It uses cloud infrastructure to execute jobs using Rackspace (x86) and 
> Packer.io (Arm)
> The Arm support, which we are targeting as part of the SIG is somewhat 
> limited I believe (restricted to Debian)
> @Julien/@Stefano need to confirm what is implemented and what isn’t
> The GitLab CI is not currently gating commits to any Xen trees: in other 
> words, test failures on a staging tree will not automatically prevent merges 
> to master
>
>
>
> There was a plan which has not yet been completed to …
>
> Take patches of xen-devel@, run them on the GitLab CI and have a bot report 
> back to the list before commits are made to staging.
> This would ensure that failures are picked up before code changes are 
> reviewed on xen-devel@
> This would also potentially provide a hook for other tools on proposed 
> changes to the codebase, e.g. coding standard compliance hook
> I am pretty sure that there will be some discussion about this at the 
> Developer Summit
>
>
>
> Presentations
>
> https://xensummit19.sched.com/event/PFWO/will-robots-automate-your-job-away-streamlining-xen-project-contributions-through-ci-doug-goldstein-rackspace
>
>
>
> Relevance to SIG: This infrastructure could potentially be a hook for …
>
> MisraC checking on some configurations – although the Perforce set-up works 
> totally differently and there may be licensing issues
> Other tools we may consider, e.g. if say we reverse engineer a subset of Xen 
> with requirements statements, design documents and additional tests, this 
> would be a good hook to check traceability and completeness
>
>
>
> XTF
>
> URL: http://xenbits.xen.org/docs/xtf/
>
> This is a framework for creating microkernel-based tests, and a suite of 
> tests built using the framework.
>
> It is the closest we can possibly get to Unit-type testing and is also useful 
> for some functional tests, given the close interaction between a Hypervisor, 
> HW and other parts of the system
>
> Most tests are dependent on specific HW and test the presence of security 
> vulnerabilities in the codebase
>
> Note that XTF is run as part of OSSTEST
>
> There is no Arm support, which is an issue
>
>
>
> Presentations
>
> https://www.slideshare.net/xen_com_mgr/xpdds17-xen-test-framework-testing-from-a-guests-perspective-andrew-cooper-citrix
>  / & https://www.youtube.com/watch?v=Jz0_QnZHoFw
>
>
>
> Relevance to SIG:
>
> Using XTF is probably the most feasible route for most testing that we would 
> ever want to do in a safety context
>
>
>
> OSSTEST (official)
>
> OSSTEST primarily runs interoperability tests of different components in a 
> Xen stack and performs common actions, such as migration, updates between 
> versions, … in different environments
>
> OSSTEST does build AND interoperability tests (as such there is some 
> duplication between OSSTEST and the GitLab CI)
>
> OSSTEST runs in our datacentre on lots of different Hardware – as such test 
> runs are fairly long and also expensive)
> OSSTEST gates commits to master
>
>
>
> Presentations
>
> https://www.slideshare.net/xen_com_mgr/slides-28160849 & 
> https://www.youtube.com/watch?v=JxTFZIwZzJ8
> https://www.slideshare.net/xen_com_mgr/slides-38366853 & 
> https://www.youtube.com/watch?v=Fi8S7Jddct4
> https://www.slideshare.net/xen_com_mgr/xpdds17-xen-test-lab-the-installation-and-our-plans-ian-jackson-citrix
> https://www.slideshare.net/xen_com_mgr/xpdds17-osstest-view-inside-a-unique-test-automaton-ian-jackson-citrix
>
>

_______________________________________________
Fusa-sig mailing list
Fusa-sig@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/fusa-sig