git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 3/5] archive-tar: add internal gzip implementation

From
René Scharfe <l.s.r@web.de>
Date
Jun 12, 2022, 06:08 UTC
Message-ID
<75e76d0f-2ab0-ebf7-4a4e-7a6e0fca0b1a@web.de>
In-Reply-To
<217a2f4d-4fc2-aaed-f5c2-1b7e134b046d@web.de>

Git uses zlib for its own object store, but calls gzip when creating tgz archives. Add an option to perform the gzip compression for the latter using zlib, without depending on the external gzip binary.

Plug it in by making write_block a function pointer and switching to a compressing variant if the filter command has the magic value "git archive gzip". Does that indirection slow down tar creation? Not really, at least not in this test:

$ hyperfine -w3 -L rev HEAD,origin/main -p 'git checkout {rev} && make' \
'./git -C ../linux archive --format=tar HEAD # {rev}'
Benchmark #1: ./git -C ../linux archive --format=tar HEAD # HEAD
  Time (mean ± σ):      4.044 s ±  0.007 s    [User: 3.901 s, System: 0.137 s]
  Range (min … max):    4.038 s …  4.059 s    10 runs
Benchmark #2: ./git -C ../linux archive --format=tar HEAD # origin/main
  Time (mean ± σ):      4.047 s ±  0.009 s    [User: 3.903 s, System: 0.138 s]
  Range (min … max):    4.038 s …  4.066 s    10 runs
How does tgz creation perform?
$ hyperfine -w3 -L command 'gzip -cn','git archive gzip' \
'./git -c tar.tgz.command="{command}" -C ../linux archive --format=tgz HEAD'
Benchmark #1: ./git -c tar.tgz.command="gzip -cn" -C ../linux archive --format=tgz HEAD
  Time (mean ± σ):     20.404 s ±  0.006 s    [User: 23.943 s, System: 0.401 s]
  Range (min … max):   20.395 s … 20.414 s    10 runs
Benchmark #2: ./git -c tar.tgz.command="git archive gzip" -C ../linux archive --format=tgz HEAD
  Time (mean ± σ):     23.807 s ±  0.023 s    [User: 23.655 s, System: 0.145 s]
  Range (min … max):   23.782 s … 23.857 s    10 runs
Summary
  './git -c tar.tgz.command="gzip -cn" -C ../linux archive --format=tgz HEAD' ran
    1.17 ± 0.00 times faster than './git -c tar.tgz.command="git archive gzip" -C ../linux archive --format=tgz HEAD'

So the internal implementation takes 17% longer on the Linux repo, but uses 2% less CPU time. That's because the external gzip can run in parallel on its own processor, while the internal one works sequentially and avoids the inter-process communication overhead.

What are the benefits? Only an internal sequential implementation can offer this eco mode, and it allows avoiding the gzip(1) requirement.

This implementation uses the helper functions from our zlib.c instead of the convenient gz* functions from zlib, because the latter doesn't give the control over the generated gzip header that the next patch requires.

Original-patch-by: Rohit Ashiwal <rohit.ashiwal265@gmail.com>
Signed-off-by: René Scharfe <l.s.r@web.de>
---
 Documentation/git-archive.txt |  3 ++-
 archive-tar.c                 | 45 ++++++++++++++++++++++++++++++++++-
 t/t5000-tar-tree.sh           | 16 +++++++++++++
 3 files changed, 62 insertions(+), 2 deletions(-)
diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
index 56989a2f34..5b017c2bdc 100644
--- a/Documentation/git-archive.txt
+++ b/Documentation/git-archive.txt
@@ -148,7 +148,8 @@ tar.<format>.command::
 	format is given.
 +
 The "tar.gz" and "tgz" formats are defined automatically and default to
-`gzip -cn`. You may override them with custom commands.
+`gzip -cn`. You may override them with custom commands. An internal gzip
+implementation can be used by specifying the value `git archive gzip`.

 tar.<format>.remote::
 	If true, enable `<format>` for use by remote clients via
diff --git a/archive-tar.c b/archive-tar.c
index 4e6a3deb80..53d0ef685c 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -38,11 +38,13 @@ static int write_tar_filter_archive(const struct archiver *ar,
 #define USTAR_MAX_MTIME 077777777777ULL
 #endif

-static void write_block(const void *buf)
+static void tar_write_block(const void *buf)
 {
 	write_or_die(1, buf, BLOCKSIZE);
 }

+static void (*write_block)(const void *) = tar_write_block;
+
 /* writes out the whole block, but only if it is full */
 static void write_if_needed(void)
 {
@@ -430,6 +432,34 @@ static int write_tar_archive(const struct archiver *ar,
 	return err;
 }

+static git_zstream gzstream;
+static unsigned char outbuf[16384];
+
+static void tgz_deflate(int flush)
+{
+	while (gzstream.avail_in || flush == Z_FINISH) {
+		int status = git_deflate(&gzstream, flush);
+		if (!gzstream.avail_out || status == Z_STREAM_END) {
+			write_or_die(1, outbuf, gzstream.next_out - outbuf);
+			gzstream.next_out = outbuf;
+			gzstream.avail_out = sizeof(outbuf);
+			if (status == Z_STREAM_END)
+				break;
+		}
+		if (status != Z_OK && status != Z_BUF_ERROR)
+			die(_("deflate error (%d)"), status);
+	}
+}
+
+static void tgz_write_block(const void *data)
+{
+	gzstream.next_in = (void *)data;
+	gzstream.avail_in = BLOCKSIZE;
+	tgz_deflate(Z_NO_FLUSH);
+}
+
+static const char internal_gzip_command[] = "git archive gzip";
+
 static int write_tar_filter_archive(const struct archiver *ar,
 				    struct archiver_args *args)
 {
@@ -440,6 +470,19 @@ static int write_tar_filter_archive(const struct archiver *ar,
 	if (!ar->filter_command)
 		BUG("tar-filter archiver called with no filter defined");

+	if (!strcmp(ar->filter_command, internal_gzip_command)) {
+		write_block = tgz_write_block;
+		git_deflate_init_gzip(&gzstream, args->compression_level);
+		gzstream.next_out = outbuf;
+		gzstream.avail_out = sizeof(outbuf);
+
+		r = write_tar_archive(ar, args);
+
+		tgz_deflate(Z_FINISH);
+		git_deflate_end(&gzstream);
+		return r;
+	}
+
 	strbuf_addstr(&cmd, ar->filter_command);
 	if (args->compression_level >= 0)
 		strbuf_addf(&cmd, " -%d", args->compression_level);
diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh
index 7f8d2ab0a7..9ac0ec67fe 100755
--- a/t/t5000-tar-tree.sh
+++ b/t/t5000-tar-tree.sh
@@ -374,6 +374,22 @@ test_expect_success GZIP 'remote tar.gz can be disabled' '
 		>remote.tar.gz
 '

+test_expect_success 'git archive --format=tgz (internal gzip)' '
+	test_config tar.tgz.command "git archive gzip" &&
+	git archive --format=tgz HEAD >internal_gzip.tgz
+'
+
+test_expect_success 'git archive --format=tar.gz (internal gzip)' '
+	test_config tar.tar.gz.command "git archive gzip" &&
+	git archive --format=tar.gz HEAD >internal_gzip.tar.gz &&
+	test_cmp_bin internal_gzip.tgz internal_gzip.tar.gz
+'
+
+test_expect_success GZIP 'extract tgz file (internal gzip)' '
+	gzip -d -c <internal_gzip.tgz >internal_gzip.tar &&
+	test_cmp_bin b.tar internal_gzip.tar
+'
+
 test_expect_success 'archive and :(glob)' '
 	git archive -v HEAD -- ":(glob)**/sh" >/dev/null 2>actual &&
 	cat >expect <<EOF &&
--
2.36.1
Previous: René ScharfeNext: Junio C Hamano
Message 39 of 74 in “Avoid spawning gzip in git archive”
  1. 0/2 Avoid spawning gzip in git archiveJohannes Schindelin via GitGitGadget, Apr 12, 2019
  2. 1/2 archive: replace write_or_die() calls with write_block_or_die()Rohit Ashiwal via GitGitGadget, Apr 12, 2019
  3. Jeff KingApr 13, 2019
  4. Junio C HamanoApr 13, 2019
  5. Rohit AshiwalApr 14, 2019
  6. Johannes SchindelinApr 26, 2019
  7. Junio C HamanoApr 26, 2019
  8. Johannes SchindelinApr 29, 2019
  9. Jeff KingMay 1, 2019
  10. René ScharfeMay 2, 2019
  11. Junio C HamanoMay 5, 2019
  12. Jeff KingMay 6, 2019
  13. Rohit AshiwalApr 14, 2019
  14. Junio C HamanoApr 14, 2019
  15. Johannes SchindelinApr 26, 2019
  16. Jeff KingMay 1, 2019
  17. 2/2 archive: avoid spawning `gzip`Rohit Ashiwal via GitGitGadget, Apr 12, 2019
  18. Jeff KingApr 13, 2019
  19. René ScharfeApr 13, 2019
  20. Jeff KingApr 15, 2019
  21. Johannes SchindelinApr 26, 2019
  22. René ScharfeApr 27, 2019
  23. René ScharfeApr 27, 2019
  24. Johannes SchindelinApr 29, 2019
  25. René ScharfeMay 1, 2019
  26. Jeff KingMay 1, 2019
  27. René ScharfeJun 10, 2019
  28. Jeff KingJun 13, 2019
  29. brian m. carlsonApr 13, 2019
  30. Jeff KingApr 15, 2019
  31. Johannes SchindelinApr 26, 2019
  32. Ævar Arnfjörð BjarmasonMay 2, 2019
  33. Johannes SchindelinMay 3, 2019
  34. Jeff KingMay 3, 2019
  35. Johannes SchindelinApr 26, 2019
  36. 0/5 Avoid spawning gzip in git archiveRené Scharfe, Jun 12, 2022
  37. 1/5 archive: rename archiver data field to filter_commandRené Scharfe, Jun 12, 2022
  38. 2/5 archive-tar: factor out write_block()René Scharfe, Jun 12, 2022
  39. 3/5 archive-tar: add internal gzip implementationRené Scharfe, Jun 12, 2022
  40. Junio C HamanoJun 13, 2022
  41. 4/5 archive-tar: use OS_CODE 3 (Unix) for internal gzipRené Scharfe, Jun 12, 2022
  42. 5/5 archive-tar: use internal gzip by defaultRené Scharfe, Jun 12, 2022
  43. Junio C HamanoJun 13, 2022
  44. Johannes SchindelinJun 14, 2022
  45. René ScharfeJun 14, 2022
  46. René ScharfeJun 14, 2022
  47. Johannes SchindelinJun 14, 2022
  48. René ScharfeJun 14, 2022
  49. Junio C HamanoJun 15, 2022
  50. Johannes SchindelinJun 14, 2022
  51. René ScharfeJun 14, 2022
  52. Johannes SchindelinJun 30, 2022
  53. Johannes SchindelinJul 1, 2022
  54. Jeff KingJul 1, 2022
  55. Junio C HamanoJul 1, 2022
  56. 0/6 Avoid spawning gzip in git archiveRené Scharfe, Jun 15, 2022
  57. 1/6 archive: update format documentationRené Scharfe, Jun 15, 2022
  58. 2/6 archive: rename archiver data field to filter_commandRené Scharfe, Jun 15, 2022
  59. 3/6 archive-tar: factor out write_block()René Scharfe, Jun 15, 2022
  60. 4/6 archive-tar: add internal gzip implementationRené Scharfe, Jun 15, 2022
  61. Ævar Arnfjörð BjarmasonJun 15, 2022
  62. René ScharfeJun 16, 2022
  63. Ævar Arnfjörð BjarmasonJun 24, 2022
  64. René ScharfeJun 24, 2022
  65. 5/6 archive-tar: use OS_CODE 3 (Unix) for internal gzipRené Scharfe, Jun 15, 2022
  66. 6/6 archive-tar: use internal gzip by defaultRené Scharfe, Jun 15, 2022
  67. René ScharfeMay 2, 2019
  68. René ScharfeMay 2, 2019
  69. Johannes SchindelinMay 8, 2019
  70. Jeff KingMay 8, 2019
  71. Johannes SchindelinMay 9, 2019
  72. Jeff KingMay 9, 2019
  73. René ScharfeMay 10, 2019
  74. Jeff KingMay 10, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.