Re: [Xen-devel] [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Tue, 2015-10-20 at 11:34 +0100, Ian Campbell wrote:
> On Mon, 2015-10-19 at 12:44 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,
> > -traditional}.git trees"):
> > > We discussed on IRC with you and Stefano and are going to aim to push
> > > this
> > > in the w/c 19 October.
> > 
> > We have decided under the circumstances to postpone this to next week.
> > 
> > It would probably have been possible for me to pick things up from
> > the deployment plan in your mail
> >    [PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu
> > version
> > (trad vs upstream)
> > But it seems better to wait for you to be back.
> 
> That was probably wise, although I'm on vacation next week.
> 
> I thought maybe I'd be able to manage it today given the existence of the
> checklist, but I think actually I'm too fuzzy headed even for that.
> 
> I made some updates to the plan last week, see below.
> 
> Ian.
> 
> 
>  * Today we can already just remove the old staging/qemu-xen-* trees, they
>    are unused (apart from being manually pushed to along with the staging
>    trees, I think).

I have interpreted remove here (and everywhere else) as "move to
~xen/git.single-qemu.deleteme".

I did however delete the corresponding symlinks from /var/xenbits
-www/html/git-http/

>  * Move the two new trees out of people/ianc into the correct places.
>    Ensure git-http etc all works and that Stefano + IanJ have appropriate
>    permissions.

Appropriate permissions == owned by xen:xenmaint and g+s set for the
directories.

Also, in *.git/config:
    [receive]
            denyNonFastForwards = true
            unpackLimit = 10000
    [gc]
            autopacklimit = 25

Copied from xen.git/config. I also copied xen.git/hooks/pre-receive which
prevents pushes to non-staging branches (except by osstest) and prevents
pushes of unannottated tags.

I also in both cases moved hooks/post-update.sample to hooks/post-update,
which runs update-server-info. I then ran this by hand in both repos.

I also updated config/description of new and old trees.

I also edited ~xen/HG/patchbot/versions to add the appropriate lines for
all the new branches. That is I removed them and added:

/home/xen/git qemu-xen.git#master               
xen-changelog@xxxxxxxxxxxxxxxxxxx       xen-devel@xxxxxxxxxxxxxxxxxxx
/home/xen/git qemu-xen.git#stable-4.6           
xen-changelog@xxxxxxxxxxxxxxxxxxx       xen-devel@xxxxxxxxxxxxxxxxxxx
[...]
/home/xen/git qemu-xen.git#stable-4.2           
xen-changelog@xxxxxxxxxxxxxxxxxxx       xen-devel@xxxxxxxxxxxxxxxxxxx
/home/xen/git qemu-xen-traditional.git#master           
xen-changelog@xxxxxxxxxxxxxxxxxxx       xen-devel@xxxxxxxxxxxxxxxxxxx
/home/xen/git qemu-xen-traditional.git#stable-4.6               
xen-changelog@xxxxxxxxxxxxxxxxxxx       xen-devel@xxxxxxxxxxxxxxxxxxx
[...]
/home/xen/git qemu-xen-traditional.git#stable-3.3               
xen-changelog@xxxxxxxxxxxxxxxxxxx       xen-devel@xxxxxxxxxxxxxxxxxxx

and

/home/xen/git qemu-xen.git#staging              xen-staging@xxxxxxxxxxxxxxxxxxx 
        xen-devel@xxxxxxxxxxxxxxxxxxx
/home/xen/git qemu-xen.git#staging-4.6          xen-staging@xxxxxxxxxxxxxxxxxxx 
        xen-devel@xxxxxxxxxxxxxxxxxxx
[...]
/home/xen/git qemu-xen.git#staging-4.2          xen-staging@xxxxxxxxxxxxxxxxxxx 
        xen-devel@xxxxxxxxxxxxxxxxxxx
/home/xen/git qemu-xen-traditional.git#staging          
xen-staging@xxxxxxxxxxxxxxxxxxx         xen-devel@xxxxxxxxxxxxxxxxxxx
/home/xen/git qemu-xen-traditional.git#staging-4.6              
xen-staging@xxxxxxxxxxxxxxxxxxx         xen-devel@xxxxxxxxxxxxxxxxxxx
[...]
/home/xen/git qemu-xen-traditional.git#staging-3.3              
xen-staging@xxxxxxxxxxxxxxxxxxx         xen-devel@xxxxxxxxxxxxxxxxxxx

This follows the pattern for the existing xen.git and old qemu ones, which
I've removed.

I also did moved ~xen/HG/patchbot/qemu-* to ~xen/git.single-qemu.deleteme/
_after_ having made a botched attempt to rename them appropriately for the
new scheme.

To unbotch it I regenerated from scratch for qemu-xen-traditional:

$ for i in master stable-3.{3,4} stable-4.{0,1,2,3,4,5,6} ; do r=$(git 
rev-parse $i); echo "echo $r > 
qemu-xen-traditional--$i.patchbot-reported-heads" ; done
echo bc00cad75d8bcc3ba696992bec219c21db8406aa > 
qemu-xen-traditional--master.patchbot-reported-heads
echo f3115dc6719e4d21c426b9395b2b30e7062b97cf > 
qemu-xen-traditional--stable-3.3.patchbot-reported-heads
echo 62539a7d5974cdc0a448d469cda458761940cd33 > 
qemu-xen-traditional--stable-3.4.patchbot-reported-heads
echo eaa1bd612f50d2f253738ed19e14981e4ede98a5 > 
qemu-xen-traditional--stable-4.0.patchbot-reported-heads
echo 77d9bdb27c4237a007ba93a6f159791eed317abc > 
qemu-xen-traditional--stable-4.1.patchbot-reported-heads
echo 112599882987da1afbbe4c16f6b049065a64f065 > 
qemu-xen-traditional--stable-4.2.patchbot-reported-heads
echo 1e5099d596b6f7a977d4bc040a54edc2a6a3c6a4 > 
qemu-xen-traditional--stable-4.3.patchbot-reported-heads
echo 5ae0569d964ad1a6d8dc781e5566d39210a5d063 > 
qemu-xen-traditional--stable-4.4.patchbot-reported-heads
echo dfe880e8d5fdc863ce6bbcdcaebaf918f8689cc0 > 
qemu-xen-traditional--stable-4.5.patchbot-reported-heads
echo 1c8d43cbdf0fc01a8f05acfbf55b805a83da34bb > 
qemu-xen-traditional--stable-4.6.patchbot-reported-heads

And for qemu-xen:

$ for i in master staging {staging,stable}-4.{2,3,4,5,6} ; do r=$(git rev-parse 
$i); echo "echo $r > qemu-xen--$i.patchbot-reported-heads" ; done
echo 816609b2841297925a223ec377c336360e044ee5 > 
qemu-xen--master.patchbot-reported-heads
echo 816609b2841297925a223ec377c336360e044ee5 > 
qemu-xen--staging.patchbot-reported-heads
echo c17e602ae64fb24405ebd256679ba9a6f5819086 > 
qemu-xen--staging-4.2.patchbot-reported-heads
echo b188780861662e8cf1847ec562799b32bb44f05e > 
qemu-xen--staging-4.3.patchbot-reported-heads
echo 5fe74249f5ab528fe84a26fa60438a6de4c787f0 > 
qemu-xen--staging-4.4.patchbot-reported-heads
echo e5a1bb22cfb307db909dbd3404c48e5bbeb9e66d > 
qemu-xen--staging-4.5.patchbot-reported-heads
echo 980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9 > 
qemu-xen--staging-4.6.patchbot-reported-heads
echo c17e602ae64fb24405ebd256679ba9a6f5819086 > 
qemu-xen--stable-4.2.patchbot-reported-heads
echo b188780861662e8cf1847ec562799b32bb44f05e > 
qemu-xen--stable-4.3.patchbot-reported-heads
echo 5fe74249f5ab528fe84a26fa60438a6de4c787f0 > 
qemu-xen--stable-4.4.patchbot-reported-heads
echo e5a1bb22cfb307db909dbd3404c48e5bbeb9e66d > 
qemu-xen--stable-4.5.patchbot-reported-heads
echo 980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9 > 
qemu-xen--stable-4.6.patchbot-reported-heads

The above was run in the relevant git repo and then the resulting commands
were then run in ~xen/HG/patchbot/.

Previously for qemu-xen it looks like only qemu-upstream-unstable.git
(staging and master) generated these mails, I have deliberately changed
that here so all branches should have mails.

http://lists.xen.org/archives/html/xen-devel/2015-02/msg03547.html and the
various followups were quite useful here.

I manually ran:
git clone git://xenbits.xen.org/qemu-xen.git  
git clone git://xenbits.xen.org/qemu-xen-traditional.git  
git clone http://xenbits.xen.org/git-http/qemu-xen.git qemu-xen.http
git clone http://xenbits.xen.org/git-http/qemu-xen-traditional.git 
qemu-xen-traditional.http

and observed that they seemed to be fetching stuff.

>  * See if there is a way to prevent pushes to the old trees (e.g. a
>    setting in their .git/config file).

I deployed the snippet in <1445424646.9563.86.camel@xxxxxxxxxx> into all
the old trees. I didn't actually try anything until later during ap-push
testing, when I realised that these would stop osstest's ap-push too.
Furthermore the old qemu-xen-trad trees would not be updated to do anything
at all. After discussion with IanJ I remove the hook from the old trad
trees (qemu-xen-unstable.git and qemu-xen-X.Y-testing.git) and replaced the
one in qemu-upstream-* with:
    #!/bin/sh
    set -e
    whoami=`whoami`
    reject () {
            echo >&2 "
    *** push rejected: $1 ***
    "
            exit 1
    }
    if [ "x$whoami" != xosstest ]; then
            reject 'only osstest may push here'
    fi


staging/qemu-upstream-*.git retain the original hook which rejects
everything.


>  * Test osstest patch <1443793989.11707.121.camel@xxxxxxxxxx>. Quoth iwj:
> 
>    * explicitly ap-push and ap-fetch the relevant trees. Perhaps do
>      the ap-push as a user without the appropriate permissions to get
>      a dry run.
> 
>    * Update patch as necessary.

This highlighted a missing } on:

+: ${PUSH_TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-xen.git

With that fixed I ran ap-fetch-version, ap-fetch-version-baseline and ap
-fetch-version-old on a representative set of branches
    qemu-upstream-unstable
    qemu-upstream-4.6-testing
    qemu-upstream-4.2-testing

I skipped ap-fetch-version-baseline-late since that only does anything for
linux-next. There is nothing to test for qemu-xen-traditional since there
is no gate.

Since everything is quiescent at the moment all of the above got the same
answer for a given branch.

I repeated with qemu-mainline and as expected ap-fetch-version returned
something different to the other two (which were the same as each other).

I then for each branch above list above I ran, as osstest on the production
vm:
    ./ap-fetch-version-baseline $branch
    ./ap-push $branch <result of fetch>

For qemu-upstream-unstable this did:
+ git push osstest@xxxxxxxxxxxxxxx:/home/xen/git/qemu-xen.git 
816609b2841297925a223ec377c336360e044ee5:refs/heads/master

For qemu-upstream-4.6-testing it did:
+ git push osstest@xxxxxxxxxxxxxxx:/home/xen/git/qemu-xen.git 
980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9:refs/heads/stable-4.6
+ git push osstest@xxxxxxxxxxxxxxx:/home/xen/git/qemu-upstream-4.6-testing.git 
980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9:refs/heads/master

qemu-upstream-4.2-testing similarly.

For qemu-mainline this did:
+ git push osstest@xxxxxxxxxxxxxxx:/home/xen/git/qemu-xen.git 
26c7be842637ee65a79cd77f96a99c23ddcd90ad:refs/heads/upstream-tested

In all cases the push was "Everything up-to-date" and the paths and
branches were as expected.
   
>  * Push resulting tested osstest patch. Probably force push

Done, by force push.

>  * Once that change is in osstest.git#production Stefano and Ian would
>    switch to pushing to the appropriate staging* branches new trees.
> 
>    osstest will ignore the old staging trees and osstest will update both
>    the new master/stable branches and the old stable trees (but not the
> old
>    qemu-upstream-unstable#master branch).
>    
>  * ASAP after the osstest patch reaches production push the patch
>    <1442486509.18856.166.camel@xxxxxxxxxx> to xen.git#staging.
> 
>    NOTE: s/qemu-xen-traditional\.h/qemu-xen-traditional.git./ on the
> commit
>    message and add Ian J's ack.
>    
>    This will cause the xen-unstable builds to use the new output gate.
> 
>    Until this is done unstable builds will continue using the old
>    master push gate, which is not updated after the osstest update
>    (only stable branches are).

Patch pushed to staging.

>  * Remove the old staging/qemu-upstream-* trees, they are not
>    referenced by anything.

Done (by move aside a discussed above).

The remainder are things which should not happen immediately.

Including a new step:

 * Remove non-stagingÂqemu-upstream-unstable.git andÂqemu-xen-unstable.git
   which can occur once any flights which might use them have been and gone
   (including possible bisect attempts)

>    
>  * At our leisure backport the xen.git change to stable branches,
> probably
>    back as far as 4.2 (when qemu-xen was introduced).
>    
>  * Do stable releases of each of the above.
>  
>  * Drop (one by one or all at once) the push to the legacy stable
> branches
>    from osstest as stable releases are made referencing the new trees.
>    
>  * Consider hiding (or removing) the old output trees from xenbits as
> well.
>    => Not possible with current gitweb setup.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel