Re: Golang Xen packages and the golang packaging system

[Nick Rosbrook <rosbrookn@xxxxxxxxx>]

On Fri, Apr 24, 2020 at 7:26 AM George Dunlap <George.Dunlap@xxxxxxxxxx> wrote:
>
>
>
> > On Apr 24, 2020, at 5:04 AM, Nick Rosbrook <rosbrookn@xxxxxxxxx> wrote:
> >
> > On Thu, Apr 23, 2020 at 1:22 PM George Dunlap <George.Dunlap@xxxxxxxxxx> 
> > wrote:
> >>
> >>
> >>> On Apr 23, 2020, at 12:49 PM, George Dunlap <george.dunlap@xxxxxxxxxx> 
> >>> wrote:
> >>>
> >>>
> >>>
> >>>> On Apr 23, 2020, at 12:27 PM, Ian Jackson <ian.jackson@xxxxxxxxxx> wrote:
> >>>>
> >>>> Ian Jackson writes ("Re: Golang Xen packages and the golang packaging 
> >>>> system"):
> >>>>> This is quite unpleasant.  In particular, it makes a git tree out of
> >>>>> output files.  What will we do when someone sends us patches to the
> >>>>> bindings ?
> >>>>
> >>>> Also, anyone who redistributes your proposed golang package is
> >>>> violating our licence unless they ship a copy of xen.git[1] too, since
> >>>> the golang package is not source code.
> >>>>
> >>>> [1] Technically, a copy of the relevant parts will do.
> >>>
> >>> The “relevant parts” would primarily be gengotypes.py, right?  Oh, and I 
> >>> guess libxl_test.idl and friends.  libxl_test.idl isn’t included in the 
> >>> distribution either.
> >>>
> >>> I’m not an expert in the golang build system, but they generally seem to 
> >>> be trying to keep the functionality simple (which of course, means if you 
> >>> want to do anything non-basic, it’s incredibly complicated or completely 
> >>> impossible).
> >>>
> >>> There’s a command, `go generate`, which we could use to run gengotypes.py 
> >>> to generate the appropriate files.  But I’m not sure how to use that in a 
> >>> practical way for this sort of package: it might end up that people 
> >>> wanting to use the package would need to manually clone it, then manually 
> >>> run `go generate` before manually building the package.
> >>>
> >>> Checking in the generated files means that someone can simply add 
> >>> `golang.xenproject.org/xenlight` as a dependency (perhaps with a specific 
> >>> version tag, like v4.14), and everything Just Works.
> >>>
> >>> Nick may have some ideas on how to use the golang build system more 
> >>> effectively.
> >>
> >> So, the following seems to work quite well actually:
> >>
> >> mkdir vendor
> >> ln -s vendor/golang.xenproject.org 
> >> /usr/share/gocode/src/golang.xenproject.org
> >> echo “golang.xenproject.org/xenlight” >> vendor/modules.txt
> >> go build -mod=vendor
> >>
> >> Using the above method, (say) redctl.git would build exactly the same on 
> >> Xen 4.14 as on Xen 4.15 (assuming redctl wasn’t using anything not 
> >> available in 4.14).
> >>
> >> I’m inclined to say we should start with just telling people to do that, 
> >> and look at doing something else if we discover that’s not suitable for 
> >> some reason.
> >
> > If it's not viable to create another repo for the xenlight package, I
> > think we should should just initialize the go module, i.e. go.mod, at
> > xen.git/tools/golang. The downside is that tags cannot be independent
> > from the rest of xen.git, so users need to have `require <module
> > path>/xenlight@RELEASE-4.14.0` in their go.mod, but at least its `go
> > get`-able. And, this does not fetch the entire git tree.
> >
> > This would also mean that we actually track the generated code (which
> > isn't really a big deal IMO, it's expected that people track their
> > generated gRPC code, for example).
>
> Yes, I was playing with this yesterday and it seems to work OK.
>
> The thing I didn’t necessarily like about this was that suppose you had a 
> public project that used the xenlight bindings, and you upgraded to Xen 4.15, 
> but some of your users hadn’t.  If you updated this to RELEASE-4.15.0, then 
> all your downstreams would stop working, even if you weren’t using any 
> functionality specific to Xen 4.15.

The go.mod is really just a way of specifying dependencies. Definig a
module for xenlight does not take away the ability of downstreams to
do `go build -mod=vendor`, where their preferred version of xenlight
is vendored.

> But I suppose what that would really mean is that:
> 1) We should make sure that xenlight@RELEASE-$V works on > $V as well
> 2) Projects depending on the bindings should use the oldest version of the 
> Xen bindings suitable for their use case.
>
> Both of those are probably reasonable.

I agree.

> Another issue that happens with checking in generated code is that the idl 
> changes and nobody re-generates the code.  We’d probably want an osstest 
> check that would refuse to push from staging -> master if re-running the code 
> generator produced a different output.  (But that has its own annoyances: it 
> seems that different versions of python sort things in different orders, so I 
> often have to throw away spurious changes to the generated files because our 
> two versions of python seem to order some things differently.)

Yeah, I have noticed that on my machine as well. I guess we could just
sort the cases by value or alphabetically before writing the switch
statements. Have you seen the re-ordering happen in other places too?

[...]

> Anyway, do you want to submit a patch adding a `go.mod` in the appropriate 
> place?  I’ve always had a hard time figuring out how go.mod actually works; 
> there seems to be no *manual*, only *howtos*.

Yes, I can do that. I'm going to keep tinkering with the remote import
path and module proxy so that users could still just do `import
"golang.xenproject.org/xenlight"`, but if I can't get something like
that working I'll define the module "canonically."

Thanks,
-NR