git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] archive: support gzipped tar files

From
Jeff King <peff@github.com>
Date
Jun 14, 2011, 20:45 UTC
Message-ID
<20110614204521.GA12776@sigill.intra.peff.net>
In-Reply-To
<20110614201433.GB1567@sigill.intra.peff.net>
On Tue, Jun 14, 2011 at 04:14:33PM -0400, Jeff King wrote:
Show 5 quoted lines
> Yeah, I am unsurprised that doing it in a single process is actually
> slower. I do think because of the remote issue that we should provide
> something like this. But we could implement it by piping to an external
> gzip. That would make us just slightly less portable, but would give us
> the multi-processor speedup, or even allow using something like pigz.

So here's a relatively quick implementation of the pipe idea. It just handles .tar.gz, but it would be trivial to do bz2 or other formats, as long as they can act as a stdio filter.

The gzip path is not configurable at all. Probably it should read the path and arguments from the config file. In fact, we could even allow arbitrary config like:

  [tarfilter "tgz"]
    command = gzip -c
    extension = tgz
    extension = tar.gz

which also solves the "don't advertise in --list if you don't have it installed problem". At the same time, that is a lot to have to configure for somebody who is not providing remote service and just wants:

  git archive -o HEAD foo.tar.gz
to work out of the box.

I think we could probably allow arbitrary config, but provide a few sane, common defaults like gzip and bz2 unless the user specifically turns them off at build time.

---
 archive-tar.c       |   45 +++++++++++++++++++++++++++++++++++++++++++++
 archive.c           |    1 +
 archive.h           |    1 +
 builtin/archive.c   |    6 ++++++
 t/t5000-tar-tree.sh |   26 ++++++++++++++++++++++++++
 5 files changed, 79 insertions(+), 0 deletions(-)
diff --git a/archive-tar.c b/archive-tar.c
index cee06ce..a77d605 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -4,6 +4,7 @@
 #include "cache.h"
 #include "tar.h"
 #include "archive.h"
+#include "run-command.h"
 
 #define RECORDSIZE	(512)
 #define BLOCKSIZE	(RECORDSIZE * 20)
@@ -248,3 +249,47 @@ int write_tar_archive(struct archiver_args *args)
 		write_trailer();
 	return err;
 }
+
+static int write_tar_to_filter(struct archiver_args *args, const char **argv)
+{
+	struct child_process filter;
+	int r;
+
+	memset(&filter, 0, sizeof(filter));
+	filter.argv = argv;
+	filter.in = -1;
+
+	if (start_command(&filter) < 0)
+		die_errno("unable to start '%s' filter", argv[0]);
+	close(1);
+	if (dup2(filter.in, 1) < 0)
+		die_errno("unable to redirect descriptor");
+	close(filter.in);
+
+	r = write_tar_archive(args);
+
+	close(1);
+	if (finish_command(&filter) != 0)
+		die("'%s' filter reported error", argv[0]);
+
+	return r;
+}
+
+int write_tgz_archive(struct archiver_args *args)
+{
+	char compression[4];
+	const char *argv[] = {
+		"gzip",
+		"-c",
+		NULL, /* compression level */
+		NULL
+	};
+
+	if (args->compression_level >= 0) {
+		snprintf(compression, sizeof(compression),
+			 "-%d", args->compression_level);
+		argv[2] = compression;
+	}
+
+	return write_tar_to_filter(args, argv);
+}
diff --git a/archive.c b/archive.c
index 42f2d2f..6073a8d 100644
--- a/archive.c
+++ b/archive.c
@@ -23,6 +23,7 @@ static const struct archiver {
 } archivers[] = {
 	{ "tar", write_tar_archive },
 	{ "zip", write_zip_archive, USES_ZLIB_COMPRESSION },
+	{ "tgz", write_tgz_archive, USES_ZLIB_COMPRESSION },
 };
 
 static void format_subst(const struct commit *commit,
diff --git a/archive.h b/archive.h
index 038ac35..c1bf72e 100644
--- a/archive.h
+++ b/archive.h
@@ -23,6 +23,7 @@ typedef int (*write_archive_entry_fn_t)(struct archiver_args *args, const unsign
  */
 extern int write_tar_archive(struct archiver_args *);
 extern int write_zip_archive(struct archiver_args *);
+extern int write_tgz_archive(struct archiver_args *);
 
 extern int write_archive_entries(struct archiver_args *args, write_archive_entry_fn_t write_entry);
 extern int write_archive(int argc, const char **argv, const char *prefix, int setup_prefix);
diff --git a/builtin/archive.c b/builtin/archive.c
index b14eaba..4f60af5 100644
--- a/builtin/archive.c
+++ b/builtin/archive.c
@@ -71,6 +71,12 @@ static const char *format_from_name(const char *filename)
 	ext++;
 	if (!strcasecmp(ext, "zip"))
 		return "--format=zip";
+	if (!strcasecmp(ext, "tgz"))
+		return "--format=tgz";
+	if (!strcasecmp(ext, "gz") &&
+	    ext - 4 >= filename &&
+	    !strcasecmp(ext - 4, "tar.gz"))
+		return "--format=tgz";
 	return NULL;
 }
 
diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh
index cff1b3e..faf2784 100755
--- a/t/t5000-tar-tree.sh
+++ b/t/t5000-tar-tree.sh
@@ -26,6 +26,7 @@ commit id embedding:
 
 . ./test-lib.sh
 UNZIP=${UNZIP:-unzip}
+GUNZIP=${GUNZIP:-gunzip}
 
 SUBSTFORMAT=%H%n
 
@@ -252,4 +253,29 @@ test_expect_success 'git-archive --prefix=olde-' '
 	test -f h/olde-a/bin/sh
 '
 
+test_expect_success 'git archive --format=tgz' '
+	git archive --format=tgz HEAD >e.tgz
+'
+
+test_expect_success 'infer tgz from .tgz filename' '
+	git archive --output=e1.tgz HEAD &&
+	test_cmp e.tgz e1.tgz
+'
+
+test_expect_success 'infer tgz from .tar.gz filename' '
+	git archive --output=e2.tar.gz HEAD &&
+	test_cmp e.tgz e2.tar.gz
+'
+
+if $GUNZIP --version >/dev/null 2>&1; then
+	test_set_prereq GUNZIP
+else
+	say "Skipping tgz tests because gunzip was not found"
+fi
+
+test_expect_success GUNZIP 'extract tgz file' '
+	gunzip -c <e.tgz >e.tar &&
+	test_cmp b.tar e.tar
+'
+
 test_done

-- 
1.7.6.rc1.4.g49204
Previous: Jeff KingNext: Jeff King
Message 7 of 56 in “archive: factor out write phase of tar format”
  1. 1/2 archive: factor out write phase of tar formatJeff King, Jun 14, 2011
  2. 2/2 archive: support gzipped tar filesJeff King, Jun 14, 2011
  3. J.H.Jun 14, 2011
  4. Jeff KingJun 14, 2011
  5. René ScharfeJun 14, 2011
  6. Jeff KingJun 14, 2011
  7. Jeff KingJun 14, 2011
  8. 0/7 user-configurable git-archive output formatsJeff King, Jun 15, 2011
  9. 1/7 archive: reorder option parsing and config readingJeff King, Jun 15, 2011
  10. 2/7 archive: add user-configurable tar-filter infrastructureJeff King, Jun 15, 2011
  11. Junio C HamanoJun 15, 2011
  12. Jeff KingJun 16, 2011
  13. 3/7 archive: support user tar-filters via --formatJeff King, Jun 15, 2011
  14. 4/7 archive: advertise user tar-filters in --listJeff King, Jun 15, 2011
  15. 5/7 archive: refactor format-guessing from filenameJeff King, Jun 15, 2011
  16. Junio C HamanoJun 15, 2011
  17. Jeff KingJun 16, 2011
  18. 6/7 archive: match extensions from user-configured formatsJeff King, Jun 15, 2011
  19. 7/7 archive: provide builtin .tar.gz filterJeff King, Jun 15, 2011
  20. Junio C HamanoJun 15, 2011
  21. Junio C HamanoJun 15, 2011
  22. Jeff KingJun 16, 2011
  23. Junio C HamanoJun 16, 2011
  24. Jeff KingJun 16, 2011
  25. Chris WebbJun 16, 2011
  26. Jeff KingJun 16, 2011
  27. Junio C HamanoJun 16, 2011
  28. Jeff KingJun 16, 2011
  29. John SzakmeisterJun 16, 2011
  30. Junio C HamanoJun 16, 2011
  31. Jeff KingJun 16, 2011
  32. René ScharfeJun 18, 2011
  33. Jakub NarebskiJun 18, 2011
  34. Junio C HamanoJun 20, 2011
  35. 0/9 configurable tar compressorsJeff King, Jun 22, 2011
  36. 1/9 archive: reorder option parsing and config readingJeff King, Jun 22, 2011
  37. 2/9 archive-tar: don't reload default config optionsJeff King, Jun 22, 2011
  38. 3/9 archive: refactor list of archive formatsJeff King, Jun 22, 2011
  39. Thiago FarinaJun 23, 2011
  40. Jeff KingJun 23, 2011
  41. 4/9 archive: pass archiver struct to write_archive callbackJeff King, Jun 22, 2011
  42. 5/9 archive: move file extension format-guessing lowerJeff King, Jun 22, 2011
  43. 6/9 archive: refactor file extension format-guessingJeff King, Jun 22, 2011
  44. 7/9 archive: implement configurable tar filtersJeff King, Jun 22, 2011
  45. Jeff KingJun 22, 2011
  46. René ScharfeJun 22, 2011
  47. Jeff KingJun 22, 2011
  48. 8/9 archive: provide builtin .tar.gz filterJeff King, Jun 22, 2011
  49. 9/9 upload-archive: allow user to turn off filtersJeff King, Jun 22, 2011
  50. Jeff KingJun 22, 2011
  51. Jeff KingJun 21, 2011
  52. René ScharfeJun 18, 2011
  53. Junio C HamanoJun 14, 2011
  54. Jeff KingJun 14, 2011
  55. Miles BaderJun 14, 2011
  56. Jeff KingJun 15, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.