git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] archive: support gzipped tar files

From
René Scharfe <rene.scharfe@lsrfire.ath.cx>
Date
Jun 14, 2011, 19:39 UTC
Message-ID
<4DF7B90B.9050802@lsrfire.ath.cx>
In-Reply-To
<20110614181821.GA32685@sigill.intra.peff.net>
Am 14.06.2011 20:18, schrieb Jeff King:
Show 9 quoted lines
> git-archive already supports the creation of tar files. For
> local cases, one can simply pipe the output to gzip, and
> having git-archive do the gzip is a minor convenience.
> 
> However, when running git-archive against a remote site,
> having the remote side do the compression can save
> considerable bandwidth. Service providers could always wrap
> git-archive to provide that functionality, but this makes it
> much simpler.

That's a good point and one that was overlooked when this topic came up earlier (see http://kerneltrap.org/mailarchive/git/2009/9/10/11507 and http://kerneltrap.org/mailarchive/git/2009/9/11/11577). That implementation was ... heavier than yours, but it also avoided an unnecessary level of buffering. I wonder if it makes a measurable difference, though.

Show 76 quoted lines
> Creating gzipped archives is of course more expensive than
> regular tar archives; however, the amount of work should be
> comparable to that of creating a zip file, which is already
> possible. So there should be no new security implications
> with respect to creating load on a remote server.
> 
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  Documentation/git-archive.txt |   17 +++++++++++++++--
>  archive-tar.c                 |   27 +++++++++++++++++++++++++++
>  archive.c                     |    1 +
>  archive.h                     |    1 +
>  builtin/archive.c             |    6 ++++++
>  t/t5000-tar-tree.sh           |   26 ++++++++++++++++++++++++++
>  6 files changed, 76 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
> index 9c750e2..963bec4 100644
> --- a/Documentation/git-archive.txt
> +++ b/Documentation/git-archive.txt
> @@ -34,10 +34,11 @@ OPTIONS
>  -------
>  
>  --format=<fmt>::
> -	Format of the resulting archive: 'tar' or 'zip'. If this option
> +	Format of the resulting archive: 'tar', 'tgz', or 'zip'. If this option
>  	is not given, and the output file is specified, the format is
>  	inferred from the filename if possible (e.g. writing to "foo.zip"
> -	makes the output to be in the zip format). Otherwise the output
> +	creates the output in the zip format; "foo.tgz" or "foo.tar.gz"
> +	creates the output in the tgz format). Otherwise the output
>  	format is `tar`.
>  
>  -l::
> @@ -89,6 +90,12 @@ zip
>  	Highest and slowest compression level.  You can specify any
>  	number from 1 to 9 to adjust compression speed and ratio.
>  
> +tgz
> +~~~
> +-9::
> +	Highest and slowest compression level. You can specify any
> +	number from 1 to 9 to adjust compression speed and ratio.
> +
>  
>  CONFIGURATION
>  -------------
> @@ -133,6 +140,12 @@ git archive --format=tar --prefix=git-1.4.0/ v1.4.0 | gzip >git-1.4.0.tar.gz::
>  
>  	Create a compressed tarball for v1.4.0 release.
>  
> +git archive --prefix=git-1.4.0/ -o git-1.4.0.tar.gz v1.4.0
> +
> +	Same as above, except that we use the internal gzip. Note that
> +	the output format is inferred by the extension of the output
> +	file.
> +
>  git archive --format=tar --prefix=git-1.4.0/ v1.4.0{caret}\{tree\} | gzip >git-1.4.0.tar.gz::
>  
>  	Create a compressed tarball for v1.4.0 release, but without a
> diff --git a/archive-tar.c b/archive-tar.c
> index b1aea87..86c8aa9 100644
> --- a/archive-tar.c
> +++ b/archive-tar.c
> @@ -260,3 +260,30 @@ int write_tar_archive(struct archiver_args *args)
>  	output = output_write;
>  	return write_tar_archive_internal(args);
>  }
> +
> +static gzFile gz_file;
> +static void output_gz(const char *buf, unsigned long len)
> +{
> +	if (!gzwrite(gz_file, buf, len))
> +		die("unable to write compressed stream: %s",
> +		    gzerror(gz_file, NULL));
> +}

Does this do the right things when faced with interrupted writes or truncated pipes? I ask because the earlier attempt had a gzwrite_or_die() which did that, but I don't know anymore if that is strictly needed. Oh, and bridging the gap between unsigned long and int was certainly another reason for the existence of this function.

Previous: Jeff KingNext: Jeff King
Message 5 of 56 in “archive: factor out write phase of tar format”
  1. 1/2 archive: factor out write phase of tar formatJeff King, Jun 14, 2011
  2. 2/2 archive: support gzipped tar filesJeff King, Jun 14, 2011
  3. J.H.Jun 14, 2011
  4. Jeff KingJun 14, 2011
  5. René ScharfeJun 14, 2011
  6. Jeff KingJun 14, 2011
  7. Jeff KingJun 14, 2011
  8. 0/7 user-configurable git-archive output formatsJeff King, Jun 15, 2011
  9. 1/7 archive: reorder option parsing and config readingJeff King, Jun 15, 2011
  10. 2/7 archive: add user-configurable tar-filter infrastructureJeff King, Jun 15, 2011
  11. Junio C HamanoJun 15, 2011
  12. Jeff KingJun 16, 2011
  13. 3/7 archive: support user tar-filters via --formatJeff King, Jun 15, 2011
  14. 4/7 archive: advertise user tar-filters in --listJeff King, Jun 15, 2011
  15. 5/7 archive: refactor format-guessing from filenameJeff King, Jun 15, 2011
  16. Junio C HamanoJun 15, 2011
  17. Jeff KingJun 16, 2011
  18. 6/7 archive: match extensions from user-configured formatsJeff King, Jun 15, 2011
  19. 7/7 archive: provide builtin .tar.gz filterJeff King, Jun 15, 2011
  20. Junio C HamanoJun 15, 2011
  21. Junio C HamanoJun 15, 2011
  22. Jeff KingJun 16, 2011
  23. Junio C HamanoJun 16, 2011
  24. Jeff KingJun 16, 2011
  25. Chris WebbJun 16, 2011
  26. Jeff KingJun 16, 2011
  27. Junio C HamanoJun 16, 2011
  28. Jeff KingJun 16, 2011
  29. John SzakmeisterJun 16, 2011
  30. Junio C HamanoJun 16, 2011
  31. Jeff KingJun 16, 2011
  32. René ScharfeJun 18, 2011
  33. Jakub NarebskiJun 18, 2011
  34. Junio C HamanoJun 20, 2011
  35. 0/9 configurable tar compressorsJeff King, Jun 22, 2011
  36. 1/9 archive: reorder option parsing and config readingJeff King, Jun 22, 2011
  37. 2/9 archive-tar: don't reload default config optionsJeff King, Jun 22, 2011
  38. 3/9 archive: refactor list of archive formatsJeff King, Jun 22, 2011
  39. Thiago FarinaJun 23, 2011
  40. Jeff KingJun 23, 2011
  41. 4/9 archive: pass archiver struct to write_archive callbackJeff King, Jun 22, 2011
  42. 5/9 archive: move file extension format-guessing lowerJeff King, Jun 22, 2011
  43. 6/9 archive: refactor file extension format-guessingJeff King, Jun 22, 2011
  44. 7/9 archive: implement configurable tar filtersJeff King, Jun 22, 2011
  45. Jeff KingJun 22, 2011
  46. René ScharfeJun 22, 2011
  47. Jeff KingJun 22, 2011
  48. 8/9 archive: provide builtin .tar.gz filterJeff King, Jun 22, 2011
  49. 9/9 upload-archive: allow user to turn off filtersJeff King, Jun 22, 2011
  50. Jeff KingJun 22, 2011
  51. Jeff KingJun 21, 2011
  52. René ScharfeJun 18, 2011
  53. Junio C HamanoJun 14, 2011
  54. Jeff KingJun 14, 2011
  55. Miles BaderJun 14, 2011
  56. Jeff KingJun 15, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.