Re: Golang Xen packages and the golang packaging system

[George Dunlap <George.Dunlap@xxxxxxxxxx>]

> On Apr 24, 2020, at 5:04 AM, Nick Rosbrook <rosbrookn@xxxxxxxxx> wrote:
> 
> On Thu, Apr 23, 2020 at 1:22 PM George Dunlap <George.Dunlap@xxxxxxxxxx> 
> wrote:
>> 
>> 
>>> On Apr 23, 2020, at 12:49 PM, George Dunlap <george.dunlap@xxxxxxxxxx> 
>>> wrote:
>>> 
>>> 
>>> 
>>>> On Apr 23, 2020, at 12:27 PM, Ian Jackson <ian.jackson@xxxxxxxxxx> wrote:
>>>> 
>>>> Ian Jackson writes ("Re: Golang Xen packages and the golang packaging 
>>>> system"):
>>>>> This is quite unpleasant.  In particular, it makes a git tree out of
>>>>> output files.  What will we do when someone sends us patches to the
>>>>> bindings ?
>>>> 
>>>> Also, anyone who redistributes your proposed golang package is
>>>> violating our licence unless they ship a copy of xen.git[1] too, since
>>>> the golang package is not source code.
>>>> 
>>>> [1] Technically, a copy of the relevant parts will do.
>>> 
>>> The “relevant parts” would primarily be gengotypes.py, right?  Oh, and I 
>>> guess libxl_test.idl and friends.  libxl_test.idl isn’t included in the 
>>> distribution either.
>>> 
>>> I’m not an expert in the golang build system, but they generally seem to be 
>>> trying to keep the functionality simple (which of course, means if you want 
>>> to do anything non-basic, it’s incredibly complicated or completely 
>>> impossible).
>>> 
>>> There’s a command, `go generate`, which we could use to run gengotypes.py 
>>> to generate the appropriate files.  But I’m not sure how to use that in a 
>>> practical way for this sort of package: it might end up that people wanting 
>>> to use the package would need to manually clone it, then manually run `go 
>>> generate` before manually building the package.
>>> 
>>> Checking in the generated files means that someone can simply add 
>>> `golang.xenproject.org/xenlight` as a dependency (perhaps with a specific 
>>> version tag, like v4.14), and everything Just Works.
>>> 
>>> Nick may have some ideas on how to use the golang build system more 
>>> effectively.
>> 
>> So, the following seems to work quite well actually:
>> 
>> mkdir vendor
>> ln -s vendor/golang.xenproject.org 
>> /usr/share/gocode/src/golang.xenproject.org
>> echo “golang.xenproject.org/xenlight” >> vendor/modules.txt
>> go build -mod=vendor
>> 
>> Using the above method, (say) redctl.git would build exactly the same on Xen 
>> 4.14 as on Xen 4.15 (assuming redctl wasn’t using anything not available in 
>> 4.14).
>> 
>> I’m inclined to say we should start with just telling people to do that, and 
>> look at doing something else if we discover that’s not suitable for some 
>> reason.
> 
> If it's not viable to create another repo for the xenlight package, I
> think we should should just initialize the go module, i.e. go.mod, at
> xen.git/tools/golang. The downside is that tags cannot be independent
> from the rest of xen.git, so users need to have `require <module
> path>/xenlight@RELEASE-4.14.0` in their go.mod, but at least its `go
> get`-able. And, this does not fetch the entire git tree.
> 
> This would also mean that we actually track the generated code (which
> isn't really a big deal IMO, it's expected that people track their
> generated gRPC code, for example).

Yes, I was playing with this yesterday and it seems to work OK.

The thing I didn’t necessarily like about this was that suppose you had a 
public project that used the xenlight bindings, and you upgraded to Xen 4.15, 
but some of your users hadn’t.  If you updated this to RELEASE-4.15.0, then all 
your downstreams would stop working, even if you weren’t using any 
functionality specific to Xen 4.15.

But I suppose what that would really mean is that:
1) We should make sure that xenlight@RELEASE-$V works on > $V as well
2) Projects depending on the bindings should use the oldest version of the Xen 
bindings suitable for their use case.

Both of those are probably reasonable.

Another issue that happens with checking in generated code is that the idl 
changes and nobody re-generates the code.  We’d probably want an osstest check 
that would refuse to push from staging -> master if re-running the code 
generator produced a different output.  (But that has its own annoyances: it 
seems that different versions of python sort things in different orders, so I 
often have to throw away spurious changes to the generated files because our 
two versions of python seem to order some things differently.)

BTW the separate repo isn’t off the table.  But there were some things other 
Ian pointed out:

1. The GPL requires that you provide the “preferred form for modification” to 
all the code.  I’m not sure this has been adjudicated in court, but there’s a 
strong argument that *generated* code doesn’t match that criteria: that to 
satisfy the GPL you’d need to include libxl_types.idl, idl.py, gengotypes.py, 
and a Makefile suitable for tying them all together.  (Not that the generation 
needs to be run with `go build`, but that ideally the infrastructure would be 
there so that it *could* be run.)

2. Ian was concerned with how someone using the bindings would submit a patch 
upstream.  Suppose someone cloned our “bindings” repo, made some changes so 
that it worked for them, then wanted to submit the patch upstream.  How would 
they do that?

I suppose even if we do a “deep link” into our actual repo, it won’t 
necessarily be obvious to people how to modify things and submit patches.  We’d 
probably want a how-to.

Anyway, do you want to submit a patch adding a `go.mod` in the appropriate 
place?  I’ve always had a hard time figuring out how go.mod actually works; 
there seems to be no *manual*, only *howtos*.

 -George