[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Gitlab status on older branches (Inc some 4.18 blockers)


  • To: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
  • Date: Wed, 12 Apr 2023 18:08:30 +0100
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BtD/8AvZNPytXou2Hlw9qZJmjNoCMdu898mKJ+GnOoA=; b=gwsoK/Hkm1P3QvpRpAZKKQukUTj0icu/Ag3XReUA+lB4ISC5gp4wKlKr77xTeDm5Gn79s2bnaNIyXJZ+/5a2MUo862aQplFCuD0NvHnE9vavHa8tq7zf1yxodrkILNRpQOlzvpns2PC23Zajc3BCwtV11HKgvVUoXjiHu2i0j4erXbiuiep+RBdaH2qCGj+AKpx8wgqC+jnZ+Er8MVg1+1vuDT1kZ6hwy5tjQ0KER50hFDI6eXSqkr6VrMtpJGymMjyBK0sMpg2xIo8zZM4wwwV0HWSgaWjfbLsZD+8sUtOJiUs+hZTPy+IqKh+AKO9Pq/jOViwPHnYpYKsf0MsNDQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dkHHcB9HciYGK9Ja1WmMAQKJ49R5T4r3mlZVzLOisUg8N2xjdnzg6CE7bkfyNylKtvB8q4GEP6gjF/K1YrAyECJijHRa29azA8e2OD3B9WsTcIPdehR3PArVkMRpHq72F9CEhjZFzeIGMYs7jwLked8hCrg1hWCfkECLo4VRCPulyl14qDCcviB32N6zuMilydG8Xs3R53PQoKNMvkbvCF40YjH+uP+i/YBaagFkyCGcZZiJtkT/khV4ta6uDYnKfEtZHsN6f0Vh7vBPjyPSbdm4XRF6WjgXio1TWcTPuqz0TYPyJsYoFWWqLT8LLx9Ss7CY9pQiTWjiXZCCTeH5fA==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com;
  • Cc: "committers@xxxxxxxxxxxxxx" <committers@xxxxxxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Anthony PERARD <anthony.perard@xxxxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Michal Orzel <Michal.Orzel@xxxxxxx>, Doug Goldstein <cardoe@xxxxxxxxxx>, Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>, Henry Wang <Henry.Wang@xxxxxxx>
  • Delivery-date: Wed, 12 Apr 2023 17:09:01 +0000
  • Ironport-data: A9a23:CGDWOqo5VhRwq+ZC2sQ4OzCsrv9eBmKwZBIvgKrLsJaIsI4StFCzt garIBnVa/bbMWHwcohyPI+zo0ID7cLUzYRhTFFo+yszEXhH9ZuZCYyVIHmrMnLJJKUvbq7FA +Y2MYCccZ9uHhcwgj/3b9ANeFEljfngqoLUUbKCYWYpA1c/Ek/NsDo788YhmIlknNOlNA2Ev NL2sqX3NUSsnjV5KQr40YrawP9UlKm06WNwUmAWP6gR5weCziNNV/rzGInqR5fGatgMdgKFb 76rIIGRpgvx4xorA9W5pbf3GmVirmn6ZFXmZtJ+AsBOszAazsAA+v9T2Mk0MC+7vw6hjdFpo OihgLTrIesf0g8gr8xGO/VQO3kW0aSrY9YrK1Dn2SCY5xWun3cBX5yCpaz5VGEV0r8fPI1Ay RAXACoTQy2jrv+E+7b4aMlsms94L8PUFZxK7xmMzRmBZRonabbqZvySoPpnhnI3jM0IGuvCb c0EbzYpdA7HfxBEJlYQDtQ5gfusgX78NTZfrTp5p4JuuzSVkFM3jeWraYKMEjCJbZw9ckKwj 2TK5WnmRDodM8SS02Gt+XOwnO7f2yj8Xer+EZXhrqMw0Q3KnzZ75Bs+SH27/dewtWSHRYh7d 09N6AEnlpE2+xn+JjX6d1jiyJKehTYMVtwVH+Ak5QWlzqvP/x3fFmUCViRGatEtqIkxXzNC/ lSOmtboGSBiqqa9WX+b7q2Trz65JW4SN2BqTS0OQBYB4tLjiJoulR+JRdFmeIaulcH8Ezz0x zGMrQA9iq8VgMpN0L+0lXjigjmsr5yPaRQn6wH/V3igqAh+YeaNbIWy5V/Wxf1JNoqeQx+Ku 31ss82X9uUHF5yOvC2LXuQWHbut6uqFMTvTmlpmFd8q8DHF016ue5pBpg53IkhBO9wBPzTuZ SfuVRh54ZZSOD6vcvVxaofoU8Ayl/G4TJLiS+zeacdIbt5pbgib8SpyZEmWmWfwjEwrlqJ5M pCeGSqxMUsn5W1c5GLeb48gPXUDnEjSGUu7qUjH8ima
  • Ironport-hdrordr: A9a23:D8EMs62nvJHiy5Op3lPE8QqjBKMkLtp133Aq2lEZdPU1SKGlfq WV954mPHDP+VUssQ4b6LK90cW7L080lqQY3WByB9eftWDd0QOVxepZgrcKrQeAJ8T2zJ856Z td
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

Hello,

I've done various backports to staging-4.14 and later trying to improve
the state of Gitlab testing.

The good news is that 4.16 and 4.17 now pass.  The bad news is that
there are still bugs which need fixing, but lets start with the older
branches.

Also, I was forced to backport an update to SeaBIOS 1.16 to all branches
in order to fix compile failures in build environments we supported at
the time of each of these releases.  I honestly don't know what we were
failing to do testing wise back then, but whatever we missed ought to
have been release blockers.


4.15:
https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/834460832

Individual failure instances:

1) https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/4097232265

This is a -Werror=array-bounds failure in HVMLoader but the same
job/container works in 4.16 and newer, and the underlying code is the
same.  There must be some change in the build environment, but I haven't
worked out what yet.

2) https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/4097232266

This is a -Werror=array-bounds in iPXE.  Probably needs an update like
SeaBIOS too.

3) https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/4097232290

This is a Qemu upstream failure which I do vaguely recall.  Bit it also
means that Xen 4.15 had a dead-on-arrival version of qemu which call
into question a number of our normal release activities.  Probably the
least-bad option is to backport the one fix relevant to this, because
changing the version of qemu in the security-only trees is far riskier
than changing one of the in-guest ROMs.

4) https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/4097232334

I have no idea what's going on here.  If nothing else, we're failing to
collect all the relevant log files from a build and that probably wants
fixing and backporting.

5) https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/4097232324

This isn't so much about the failure, as the fact that the OpenSUSE Leap
tests (which were replaced with tumbleweed in newer versions of Xen)
probably want the same doing to them.  And being marked as
non-blocking.  This is a failure somewhere in the middle of qemu.

But, on top of all of ^, I discovered that we have a majority of tests
being debian/unstable which, when we refreshed to fix the HTTPS issue,
ended up retrofitting a newer-than-at-release-time build environment to
the old trees.

This has come up previously, and not addressed, so I'm now declaring it
a blocker for 4.18.  Only tests against a fixed disto version can be
blocking; those against an unstable distro must be non-blocking, and
most of the currently unstable things should be transformed into their
stable alternative.  For backports, we want to retrofit what
debian/unstable was at the time of release, rather than what it
currently is.

Furthermore, the fixed distros we currently test in staging are old
bordering on obsolete.  Which is not a healthy position to be in as far
as the 4.18 release goes.


4.14:
https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/834461234

6) https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/4097236330

This is the only 4.14 failure I can see which isn't a duplicate of a
4.15 failure, but it is an OpenSUSE leap failure in qemu so perhaps
related to #5


As a general note, we still have too much testing (and/or insufficient
testing resource).  It's very painful waiting 2h for each branch to
complete.  I'm very tempted to trim things down further on staging and
backport the results.

~Andrew



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.