Re: Solving the gitlab-ci git fetch issue, was: [PATCH 3/3] automation: add a QEMU based x86_64 Dom0/DomU test

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 28 Oct 2021, Anthony PERARD wrote:
> On Wed, Oct 27, 2021 at 02:43:46PM -0700, Stefano Stabellini wrote:
> > On Wed, 27 Oct 2021, Anthony PERARD wrote:
> > > > But we do have a severe problem at the moment with external sources: our
> > > > "git clones" keep failing during the build on x86. That is definitely
> > > > something worth improving (see my other email thread on the subject) and
> > > > it is the main problem affecting gitlab-ci at the moment, I keep having
> > > > to restart jobs almost daily to get the overall pipeline to "pass".
> > > > 
> > > > If you have any ideas on how to stop fetching things using "git" from
> > > > external repositories in gitlab-ci that would be fantastic :-)
> > > > The only thing I could think of to "fix it" is moving all external repos
> > > > to gitlab repositories mirrors.
> > > 
> > > I don't think that would work, I've seen the initial clone/fetch of a
> > > job fail as well, so from gitlab. If we could have a cache of those
> > > external resources closer to the runners, that would be better.
> > 
> > You mean like a git repository mirror inside the Rackspace network (the
> > provider of the x86 runner), right? Then we would force the git client
> > to go to the Rackspace mirror instead of directly to the target using
> > "insteadOf".
> 
> That would seems the best to me. If we could install Ian's
> git-cache-proxy that is used in osstest, that would be good I think.
> Having a mirror instead might work too but that would mean figure out
> which repo we would need a mirror of.
> 
> I did try a different alternative a while back, I tried to use gitlab's
> caching capability:
>     automation: Cache sub-project git tree in build jobs
>     
> https://lore.kernel.org/xen-devel/20191219144217.305851-3-anthony.perard@xxxxxxxxxx/
> It mostly works but I'm not sure how useful it is as it seems there is
> 10 computers that would maintain 10 different caches, and most of them
> for a short while.

Interesting!

With the current runners being short-lived, it makes sense only to have
git-cache-proxy on a different always-on host. But even after the
runners are changed to be always running, instead of spawned on demand,
it might be better to have git-cache-proxy on a separate host so that
all runners can use it.  It should be more efficient?