[Xen-devel] [XenSummit 2017] Build tools follow up

[Oleksandr Andrushchenko <oleksandr_andrushchenko@xxxxxxxx>]

Hi, all!

This is a follow-up on Xen distribution build systems we saw at the
summit and invitation for sharing thoughts and ways we build our images and
distros. I would like to specifically ask OpenXT project to reply with the
description of their build system so we have clear picture of what is being
developed and used around. And of course if there are other approaches 
to do the
same you are welcome to share those as well.

On our side at EPAM we have developed a distribution which is called 
xt-distro
(xt stays for Xen troops [0]) with the following goals in mind:
1. Make it possible to easily build different images as a single 
distribution
   - Separate images for different domains should be combined into a
     distribution, e.g. set of images/artifacts required to run them as a
     system
   - Xt-distro allows doing so as easy as running bitbake with the 
predefined
     target, e.g. “bitbake xt-image”
2. Easily deal with different BSPs for different platforms using a 
unified way
   with little or no EPAM specific code/scripts
   - We use bitbake and stripped version of Poky/meta-openembedded, so 
there are
     little in-house extensions we added to deal with Yocto based BSPs 
[1], [2]
   - We use bitbake’s well defined scripting language [3], so no yet 
another
     language to learn
   - We use Google repo tool to maintain sets of meta layers as 
manifest files [4]
3. Ability to easily collect component revisions, so any build can be
   reproduced if need be
   - We save commit IDs of every component of the build including 
versions of the
     meta layers
4. Ability to easily customize and tune builds, e.g. URIs, 
versions/commitIDs,
   branches used
   - This is purely done in bitbake’s recipe language
5. Make patching process easy
   - With our extensions you can use bblayers.conf, local.conf, patches 
which
     are part of a meta layer which is usually a manual step
6. Code reuse is a must, e.g. the same set of software must be easily 
built for
   different platforms without copying build scripts of the existing 
components
   - With bitbake’s meta layers we define generic recipes, e.g. 
suitable for all
     platforms [5] (think of it as a library) and tuning meta layers [6]
     (we use product concept here) which define specific 
versions/revisions of the
     components, apply specific patches and add/remove software
7. Cross compilation must be an easy task to do
   - This is easily achieved with Yocto builds and usually not a 
problem for other
     build systems as well (with SDKs)
   - Allows building for x86/ARM and other architectures as well
8. There must be a simple way to share build artifacts of different 
domains between
   each other, for example, DomU’s kernel should be a part of Dom0’s 
rootfs, so
   xl/libxl can access it
   - This is achieved with bitbake’s recipes, so no EPAM extensions
9. Possibility to easily use different build systems for different 
components of
   the system.
   - Bitbake allows you building makefile based projects, CMake etc. 
out of the box,
     so adding firmware, stubdoms made easy
10. Development support
   - We use SDKs built by bitbake so during everyday development you 
are packed with
     all that is needed to re-create build environment (with both host 
and target
     environments, e.g. x86 host tools and ARM cross-compiler)
   - Speed up builds for developers within organization, e.g. we reuse 
downloads and
     build cache between all of the developer’s machines on our network
11. Make the build system suitable for build automation
    - We have created a python script which can easily be used with
      Jenkins or even cron [7]

For example, with xt-distro our development product delivers: Xen image and
policy file, Dom0 kernel and dtb images (Dom0 + DomU), Dom0 rootfs image 
(with
embedded kernels and configuration for DomU), DomU rootfs image. This
effectively means, that the build system can provide you with the 
complete set
of images to run your product/distribution.

We are looking forward for any kind of feedback and will be glad to 
collaborate
on the above.

Thank you,
Xen-troops at EPAM

[0] https://github.com/xen-troops/xt-distro
[1] https://git.yoctoproject.org/cgit/cgit.cgi/poky
[2] http://cgit.openembedded.org/meta-openembedded
[3] 
http://www.yoctoproject.org/docs/2.3/bitbake-user-manual/bitbake-user-manual.html
[4] https://gerrit.googlesource.com/git-repo
[5] https://github.com/xen-troops/meta-xt-images
[6] https://github.com/xen-troops/meta-xt-prod-devel
[7] https://github.com/xen-troops/build-scripts


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel