Re: Stats on Xen tarball downloads

[George Dunlap <george.dunlap@xxxxxxxxx>]

On Mon, Feb 19, 2024 at 11:34 PM Elliott Mitchell <ehem+xen@xxxxxxx> wrote:
>
> On Mon, Feb 19, 2024 at 06:01:54PM +0800, George Dunlap wrote:
> >
> > Looking at the *non*-4.18 downloads, nearly all of them have user
> > agents that make it clear they're part of automated build systems:
> > user agents like curl and wget, but also "Go-http-client", "libfetch",
>                    ^^^^     ^^^^
>
> I reject this claim.  `curl` or `wget` could be part of an interactive
> operation.  Telling a browser to copy a URL into the paste buffer, then
> using `wget`/`curl` is entirely possible.  I may be the outlier, but I
> routinely do this.

It's not just the user agent; there are certain statistical
regularities that make me think it's automated.  e.g., a specific
version of curl always downloading a specific version of the tarball,
the tar.gz and the tar.gz.sig being downloaded exactly the same time
distance apart.  There certainly *are* manual wget / curl invocations,
but the majority of them look to me like they're part of automated
systems.

(And the "Go-http-client" instances are kind of fascinating to me --
someone wrote something on golang to download the Xen tarball?  And
always 4.15.1?  And it's being run from both NH, USA and from Finland,
and a handful of other places that seem unrelated?  What project is
this?)

> > It's not really clear to me why we'd be getting 300-ish people
> > downloading the Xen 4.18.0 tarball, 2/3 of which are on Windows.  But
> > then I'm also not sure why someone would *fake* hundreds of downloads
> > a week from unique IP addresses; and in particular, if you were going
> > to fake hundreds of downloads a week, I'm not sure why you'd only fake
> > the most recent release.
>
> Remember the browser wars?  At one point many sites were looking for
> IE/Windows and sending back error messages without those.  Getting the
> tarball on Windows doesn't seem too likely, faking the browser was
> pretty common for a while.

Right, which is why I wanted to look more into the rest of the data to
see if I could get a feel for it.  There are very few Windows user
agents for the other versions; the handful of browser agents for
non-4.18.0 tarballs look very normal and unix-y.  So the question is,
why would you fake loads of downloads for Chrome / Firefox / Edge on
Windows *only* for 4.18.0?

I agree that none of the current explanations make a lot of sense; but
I continue to believe that the "We have loads of actual humans
downloading the 4.18.0 tarball via browsers, even on Windows" is the
least-bad fit.  (Feel free to propose others, though.)

 -George