[Xen-devel] xen.git build system (Re: [HACKATHON] Toolstack session)

[Wei Liu <wei.liu2@xxxxxxxxxx>]

Hi all

I spent some time this morning to work out the details of xen.git build
system.

* How build system works at the moment?
  1. Stubdom.mk.in and Tools.mk.in define FETCHER variable.
  2. m4/fetcher.m4 checks for wget or ftp, which becomes FETCHER.
  3. StdGNU.mk defines GIT. It can be overwritten by setting envar
     when building.
  4. scripts/git-checkout.sh is used to checkout git tree.
  5. Invocation of git-checkout.sh in Makefile, tools/Makefile and
     tools/firmware/Makefile.
  6. Direct invocation of GIT in Makefile, tools/Makefile,
     tools/firmware/Makefile in the subtree force update targets.
  7. stubdom/Makefile and tools/firmware/etherboot/Makefile invoke FETCHER.

* What will be cloned?
  1. mini-os
  2. qemu-trad
  3. qemu-xen -- can be skipped with --with-system-qemu
  4. seabios -- can be skipped with --with-system-seabios
  5. ovmf -- can be skipped with --with-system-ovmf

* What needs to be fetched?
  1. Stubdom needs:
     - newlib
     - zlib
     - libpci
     - lwip
     - gmp
     - polarssl
     - tpmemu
     - ocaml
     - grub
  2. tools/firmware/etherboot needs:
     - ipxe
  A dumb way of dealing with these tarballs might be just to commit them
  in tree. That way we can just eliminate fetcher all together. But I
  think there are people who want a convenient way to replace tarballs
  (how?), so input welcome here.

And some comments on individual items.

On Wed, Apr 20, 2016 at 07:20:33PM +0100, Roger Pau Monné wrote:
> Topics:
> 
>  - Build system
> 
> Wei: what downstream consumers expect from the build system. Xen has a top 
> level makefile that builds everything, by pulling other projects source 
> code. Trying to make it cleaner.
> 
> George: someone recommended to pull grub2 into the Xen build system, but it 
> was seen as too much. Try to remove other pieces that Xen build system pull 
> in in order to perform a build. Package Raisin in a way that includes all 
> the needed dependencies (source).
> 
> Doug: Gentoo/Yocto build system based on the one from Debian. It's not good 
> to pull things from the internet when performing a build. Yocto build system 
> disables network access when performing a build, custom patches are needed 
> in order to fix that. XenServer has to do the same.
> 

I can provide patches to add a --disable-download configure option. That
would basically stub out GIT and FETCHER to be /bin/false (Linux) or
/usr/bin/false (*BSD). The default would still be --enable-download.

Is such big switch good enough as first step?

> George: In general people packaging Xen needs to work around the fact that 
> Xen pulls things from the internet while building
> 
> Doug: this doesn't work for stubdom
> 
> Olaf: it's possible to do the same for stubdom, Suse does it.
> 

The rationale for a --disable-download switch is because it doesn't
affect distros (see reason above).

> Doug: stubdom build always call the git binary, even if the code it's 
> already there.
> 

This can be sorted out.

> Ian: Debian doesn't build stubdom.
> 
> Wei: it's a PITA that the Xen build system downloads stuff while building. 
> Need to solve this.
> 
> Ian: two different things in the build system:
> 
>  - Our build process downloads stuff while building. This annoys packagers, 
> but it's probably important for users that build Xen from source. We cannot 
> force them to download stuff manually.
>  - Release tarballs _still_ need to download more stuff while building 
> (firmware?).
> 
> How to fix:
>  - Everything controlled by the Xen build system, make it clear what will be 
> downloaded, have a target to download the required sources.

What I don't really get is whether this implies a dedicated target to
download components (and recursively search for dependencies) *and*
place them in the right place in xen.git build system. This option
doesn't seem maintainable to me. The anticipation is that  we might add
more stuff in xen.git. We can't track what every package needs and
provide some options to work out where to put those dependencies.

I think having xen.git build system only track top-level components
(such as seabios, ovmf etc) is doable.

>  - Use Raisin or something similar, that pulls in everything.
> 
> George: difficulties building things from outside, like blktap. Requires 
> libxc, which is inside the Xen build system.
> 
> Juergen: add the download section to configure.
> 
> Ian: seems worse than adding it to the makefile.
> 
> Olaf: configure to select what to download/install.
> 

Only top-level component is ok.

Wei.

> Ian: add download options to configure, add a download target to the build 
> makefile. Default to stay the same as it is now.
> 
> Konrad: configure to create a list of what Xen build wants to download.
> 
> Ian: hard to make it declarative.
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel