Re: [Xen-devel] [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, May 23, 2019 at 11:38:57AM +0100, Ian Jackson wrote:
> Roger Pau Monne writes ("[Xen-devel] [PATCH 5/6] osstest: introduce a script 
> to build a FreeBSD package repository"):
> > diff --git a/make-freebsd-flight b/make-freebsd-flight
> > index d3c413b5..fc3d2d83 100755
> > --- a/make-freebsd-flight
> > +++ b/make-freebsd-flight
> > @@ -38,13 +38,15 @@ job_create_build_filter_callback () {
> >  
> >  for arch in "$arches"; do
> >      set_freebsd_runvars
> > -
> >      create_freebsd_build_job build-$arch-freebsd
> >  
> > -    # Create an identical job that's going to use the build output from
> > -    # the previous one.
> > +    # Create a job to build the packages against the new world.
> >      freebsd_runvars="$freebsd_runvars freebsdbuildjob=build-$arch-freebsd \
> >                       recipe_testinstall=true"
> > +    create_freebsd_pkg_build_job build-$arch-freebsd-packages
> > +
> > +    # Create an identical job that's going to use the build output from
> > +    # the previous one.
> >      create_freebsd_build_job build-$arch-freebsd-again
> >  
> >      # Create a Xen build job that's going to use the output from the first
> 
> This looks OK.
> 
> > @@ -768,7 +773,9 @@ proc prepare-build-host-freebsd {} {
> >      global jobinfo
> >      if {[recipe-flag testinstall]} { set broken fail } { set broken broken 
> > }
> >      run-ts $broken host-install(*) ts-freebsd-host-install
> > -    run-ts . host-build-prep ts-build-prep-freebsd
> > +    if {![recipe-flag skipbuildprep]} {
> > +        run-ts . host-build-prep ts-build-prep-freebsd
> 
> What's this for ?  Oh, I see.

The job that creates the package repository cannot use build-prep
because the packages are not yet built.

> I notice that none of your freebsd build jobs pass any share- hostflag
> so they always use a fresh installation.  Is that necessary ?

Hm, I don't think so. build-amd64-xen-freebsd and
build-amd64-freebsd-again could share a host. I need to take a look at
how to do this, I could send this as a separate fix for the existing
jobs.

> >  proc need-hosts/coverity {} { return BUILD_LINUX }
> > diff --git a/ts-freebsd-build-packages b/ts-freebsd-build-packages
> > new file mode 100755
> > index 00000000..9202dd9f
> > --- /dev/null
> > +++ b/ts-freebsd-build-packages
> > @@ -0,0 +1,145 @@
> > +#!/usr/bin/perl -w
> > +# This is part of "osstest", an automated testing framework for Xen.
> > +# Copyright (C) 2019 Citrix Inc.
> ...
> > +# Consumes the following input runvars:
> > +# svnrevision_freebsdports: ports svn revision id to use.
> > +# svntree_freebsdports ports svn tree to fetch the source code from.
> 
> More regular in osstest terms would be
>   tree_freebsdports
>   revision_freebsdports
>   treevcs_freebsdports=svn
> But I guess svn is sufficiently unlike what osstest expects out of a
> vcs that this is not feasible, and it is better to do it this way.

I don't really have an opinion, I somehow assumed that using the same
format might interfere with things like bisection, so I've decided to
pass the git revision using tree_freebsdports &c and the svn revision
using the newly introduced flags.

> > +sub checkout () {
> > +    my $u = URI->new($c{HttpProxy});
> > +    my $host = $u->host;
> > +    my $port = $u->port;
> > +    prepbuilddirs();
> > +
> > +    logm("Checkout ports tree from svn");
> > +    target_cmd_build($ho, 4000, $builddir, <<END);
> > +cd $builddir
> > +rm -rf ports
> > +# svn ignores HTTP_PROXY envvar
> > +svnlite checkout --config-option servers:global:http-proxy-host=$host \\
> > +                 --config-option servers:global:http-proxy-port=$port \\
> > +                 --trust-server-cert \\
> > +                 $r{"svntree_freebsdports"} \\
> > +                 -r $r{"svnrevision_freebsdports"} ports
> 
> Will this work to cache the checkout ?  

I think so? Would https somehow prevent the caching?

> All of this says http but I
> assume it's really https ? 

AFAIK svn uses the http-proxy options for both http and https.

> Typically, https clients expect to do the
> TLS themselves but I think you're using our squid mitm and that's what
> "--trust-server-cert" is doing ?

I can't really remember why I've added this option, but I'm quite
sure  it was failing without it. As you say the proxy is acting as a
mitm, so that's likely why trust-server-cert is required.

> Rather than "--trust-server-cert" which disables TLS's own mitm
> protection it would be rather better to inject the osstest mitm squid
> cert into the testbed, but that may be difficult, and the risk is only
> from internal things between the build (test) box and the proxy.

I can look into this, but at the end of day this is all internal, so
I'm not sure there's a lot of risk here.

> > +sub create_jail() {
> > +    my $src_prefix = $r{"freebsd_distpath"} ||
> > +                     get_stashed("path_freebsddist", 
> > $r{"freebsdbuildjob"});
> > +    my $dst_prefix = "/root/sets";
> 
> Do we need a jail for this ?  We have a whole baremetal OS install
> whose entire purpose is to do this build ...

Yes, that's how the repository package builder (poudriere) works, it
requires a jail to do the package building. In our case it's not so
important, but I assume this is mostly done to always use a clean
install, so that currently installed packages on the system don't
interfere with package building.

> > +logm("FreeBSD packages built successful");
>                                           ^ly :-)

Thanks.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel