Re: [PATCH] releases: use newer compression methods for tarballs

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 15/07/2025 7:33 am, Jan Beulich wrote:
> Other projects have long switched to xz and/or lzip.
>
> In the mktarball script, don't rely on the tar used supporting the -J
> (xz) or --lzip (lzip) options.
>
> Signed-off-by: Jan Beulich <jbeulich@xxxxxxxx>
> ---
> Question (to the release technicians) is whether this would be adding
> undue overhead, primarily as a result of anything done outside of the
> build system.

I'm happy with this.

>
> It's unclear to me why we have git create a tarball, extract that, just
> to then make a tarball again (without any special options, like to
> override owner or timestamps;

That's because in c5be91eb8140 you deleted the intermediate step of
merging qemus


> in this context I notice that tarballs
> created by Andrew have file ownership of andrew/andrew, while ones made
> by Julien use root/root).

Ownership where exactly?  the tarball itself (which will be down to
accounts on downloads.xenproject.org) or the tarball contents itself?

>
> Without passing -9, I observe lzip to compress worse than xz; the win
> of passing -9 to xz isn't overly big anyway (about 100k, compared to
> about 250k with lzip).

As these are created once and downloaded many times, we should always do
max compression.  Even if it takes minutes extra to create, that's still
a win overall.

>
> lzip, unlike the other two tools, doesn't really show a progress
> indicator with -v. Merely having final statistics may make the use of
> the option here questionable.

I can't say I find any of the stats relevant.

> --- a/docs/process/xen-release-management.pandoc
> +++ b/docs/process/xen-release-management.pandoc
> @@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
>  https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
>  
>  For your convenience there is also a tarball at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
>  
>  And the signature is at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
>  
>  Please send bug reports and test reports to xen-devel@xxxxxxxxxxxxxxxxxxxx.
>  When sending bug reports, please CC relevant maintainers and me
> --- a/tools/misc/mktarball
> +++ b/tools/misc/mktarball
> @@ -31,4 +31,14 @@ git_archive_into $xen_root $tdir/xen-$de
>  
>  GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>  
> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
> +if [ -n "$(command -v xz)" ]
> +then
> +  tar c -C $tdir xen-$desc | $(command -v xz) -v -9 
> >$xen_root/dist/xen-$desc.tar.xz
> +fi
> +
> +if [ -n "$(command -v lzip)" ]
> +then
> +  tar c -C $tdir xen-$desc | $(command -v lzip) -v -9 
> >$xen_root/dist/xen-$desc.tar.lz
> +fi
> +
> +echo "Source tarball in $xen_root/dist/xen-$desc".tar.[glx]z

If we're deciding to use multiple compressions, they want to not be
optional here.  I'd far rather have a reminder to install the package,
than for it to simply be omitted.

We don't want to be re-tar-ing now that the qemu's are gone, so I think
the structure wants to end up as:

git archive --format=tar HEAD > tmp.tar
gzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.gz &
zx -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.xz &
lzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.lz &
wait

Might as well use the multiple cores better...

~Andrew