git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] archive: avoid spawning `gzip`

From
René Scharfe <l.s.r@web.de>
Date
Apr 27, 2019, 17:39 UTC
Message-ID
<45afd432-9e45-ea76-aa1b-e8cd1264e3a0@web.de>
In-Reply-To
<f6f32bc0-109c-e0eb-f7d2-9e46647f260c@web.de>
Am 27.04.19 um 11:59 schrieb René Scharfe:> Am 26.04.19 um 16:51 schrieb Johannes Schindelin:
Show 15 quoted lines
>>
>> On Mon, 15 Apr 2019, Jeff King wrote:
>>
>>> On Sun, Apr 14, 2019 at 12:01:10AM +0200, René Scharfe wrote:
>>>
>>>> Doing compression in its own thread may be a good idea.
>>>
>>> Yeah. It might even make the patch simpler, since I'd expect it to be
>>> implemented with start_async() and a descriptor, making it look just
>>> like a gzip pipe to the caller. :)
>>
>> Sadly, it does not really look like it is simpler.
>
> I have to agree -- at least I was unable to pull off the stdout
> plumbing trick.

The simplest solution is of course to not touch the archive code. The patch below makes that possible:

Benchmark #1: ~/src/git/git -c tar.tgz.command=~/src/git/git-gzip archive --format=tgz HEAD >/dev/null
  Time (mean ± σ):     17.256 s ±  0.299 s    [User: 20.380 s, System: 0.294 s]
  Range (min … max):   16.940 s … 17.804 s    10 runs
Curious to see how it looks like on other systems and platforms.
And perhaps the buffer size needs to be tuned.
-- >8 --
Subject: [PATCH] add git gzip

Add a cheap gzip lookalike based on zlib for systems that don't have (or want) the real thing. It can be used e.g. to generate tgz files using git archive and its configuration options tar.tgz.command and tar.tar.gz.command, without any other external dependency.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 .gitignore       |  1 +
 Makefile         |  1 +
 builtin.h        |  1 +
 builtin/gzip.c   | 64 ++++++++++++++++++++++++++++++++++++++++++++++++
 command-list.txt |  1 +
 git.c            |  1 +
 6 files changed, 69 insertions(+)
 create mode 100644 builtin/gzip.c
diff --git a/.gitignore b/.gitignore
index 44c74402c8..e550868219 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,6 +71,7 @@
 /git-gc
 /git-get-tar-commit-id
 /git-grep
+/git-gzip
 /git-hash-object
 /git-help
 /git-http-backend
diff --git a/Makefile b/Makefile
index 9f1b6e8926..2b34f1a4aa 100644
--- a/Makefile
+++ b/Makefile
@@ -1075,6 +1075,7 @@ BUILTIN_OBJS += builtin/fsck.o
 BUILTIN_OBJS += builtin/gc.o
 BUILTIN_OBJS += builtin/get-tar-commit-id.o
 BUILTIN_OBJS += builtin/grep.o
+BUILTIN_OBJS += builtin/gzip.o
 BUILTIN_OBJS += builtin/hash-object.o
 BUILTIN_OBJS += builtin/help.o
 BUILTIN_OBJS += builtin/index-pack.o
diff --git a/builtin.h b/builtin.h
index b78ab6e30b..abc34cc9d0 100644
--- a/builtin.h
+++ b/builtin.h
@@ -170,6 +170,7 @@ extern int cmd_fsck(int argc, const char **argv, const char *prefix);
 extern int cmd_gc(int argc, const char **argv, const char *prefix);
 extern int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);
 extern int cmd_grep(int argc, const char **argv, const char *prefix);
+extern int cmd_gzip(int argc, const char **argv, const char *prefix);
 extern int cmd_hash_object(int argc, const char **argv, const char *prefix);
 extern int cmd_help(int argc, const char **argv, const char *prefix);
 extern int cmd_index_pack(int argc, const char **argv, const char *prefix);
diff --git a/builtin/gzip.c b/builtin/gzip.c
new file mode 100644
index 0000000000..90a98c44ce
--- /dev/null
+++ b/builtin/gzip.c
@@ -0,0 +1,64 @@
+#include "cache.h"
+#include "builtin.h"
+#include "parse-options.h"
+
+static const char * const gzip_usage[] = {
+	N_("git gzip [-NUM]"),
+	NULL
+};
+
+static int level_callback(const struct option *opt, const char *arg, int unset)
+{
+	int *levelp = opt->value;
+	int value;
+	const char *endp;
+
+	if (unset)
+		BUG("switch -NUM cannot be negated");
+
+	value = strtol(arg, (char **)&endp, 10);
+	if (*endp)
+		BUG("switch -NUM cannot be non-numeric");
+
+	*levelp = value;
+	return 0;
+}
+
+#define BUFFERSIZE (64 * 1024)
+
+int cmd_gzip(int argc, const char **argv, const char *prefix)
+{
+	gzFile gz;
+	int level = Z_DEFAULT_COMPRESSION;
+	struct option options[] = {
+		OPT_NUMBER_CALLBACK(&level, N_("compression level"),
+				    level_callback),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, prefix, options, gzip_usage, 0);
+	if (argc > 0)
+		usage_with_options(gzip_usage, options);
+
+	gz = gzdopen(1, "wb");
+	if (!gz)
+		die(_("unable to gzdopen stdout"));
+
+	if (gzsetparams(gz, level, Z_DEFAULT_STRATEGY) != Z_OK)
+		die(_("unable to set compression level %d"), level);
+
+	for (;;) {
+		char buf[BUFFERSIZE];
+		ssize_t read_bytes = xread(0, buf, sizeof(buf));
+		if (read_bytes < 0)
+			die_errno(_("unable to read from stdin"));
+		if (read_bytes == 0)
+			break;
+		if (gzwrite(gz, buf, read_bytes) != read_bytes)
+			die(_("gzwrite failed"));
+	}
+
+	if (gzclose(gz) != Z_OK)
+		die(_("gzclose failed"));
+	return 0;
+}
diff --git a/command-list.txt b/command-list.txt
index 3a9af104b5..755848842c 100644
--- a/command-list.txt
+++ b/command-list.txt
@@ -99,6 +99,7 @@ git-gc                                  mainporcelain
 git-get-tar-commit-id                   plumbinginterrogators
 git-grep                                mainporcelain           info
 git-gui                                 mainporcelain
+git-gzip                                purehelpers
 git-hash-object                         plumbingmanipulators
 git-help                                ancillaryinterrogators          complete
 git-http-backend                        synchingrepositories
diff --git a/git.c b/git.c
index 50da125c60..48f7fc6c56 100644
--- a/git.c
+++ b/git.c
@@ -510,6 +510,7 @@ static struct cmd_struct commands[] = {
 	{ "gc", cmd_gc, RUN_SETUP },
 	{ "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT },
 	{ "grep", cmd_grep, RUN_SETUP_GENTLY },
+	{ "gzip", cmd_gzip },
 	{ "hash-object", cmd_hash_object },
 	{ "help", cmd_help },
 	{ "index-pack", cmd_index_pack, RUN_SETUP_GENTLY | NO_PARSEOPT },
--
2.21.0
Previous: René ScharfeNext: Johannes Schindelin
Message 23 of 74 in “Avoid spawning gzip in git archive”
  1. 0/2 Avoid spawning gzip in git archiveJohannes Schindelin via GitGitGadget, Apr 12, 2019
  2. 1/2 archive: replace write_or_die() calls with write_block_or_die()Rohit Ashiwal via GitGitGadget, Apr 12, 2019
  3. Jeff KingApr 13, 2019
  4. Junio C HamanoApr 13, 2019
  5. Rohit AshiwalApr 14, 2019
  6. Johannes SchindelinApr 26, 2019
  7. Junio C HamanoApr 26, 2019
  8. Johannes SchindelinApr 29, 2019
  9. Jeff KingMay 1, 2019
  10. René ScharfeMay 2, 2019
  11. Junio C HamanoMay 5, 2019
  12. Jeff KingMay 6, 2019
  13. Rohit AshiwalApr 14, 2019
  14. Junio C HamanoApr 14, 2019
  15. Johannes SchindelinApr 26, 2019
  16. Jeff KingMay 1, 2019
  17. 2/2 archive: avoid spawning `gzip`Rohit Ashiwal via GitGitGadget, Apr 12, 2019
  18. Jeff KingApr 13, 2019
  19. René ScharfeApr 13, 2019
  20. Jeff KingApr 15, 2019
  21. Johannes SchindelinApr 26, 2019
  22. René ScharfeApr 27, 2019
  23. René ScharfeApr 27, 2019
  24. Johannes SchindelinApr 29, 2019
  25. René ScharfeMay 1, 2019
  26. Jeff KingMay 1, 2019
  27. René ScharfeJun 10, 2019
  28. Jeff KingJun 13, 2019
  29. brian m. carlsonApr 13, 2019
  30. Jeff KingApr 15, 2019
  31. Johannes SchindelinApr 26, 2019
  32. Ævar Arnfjörð BjarmasonMay 2, 2019
  33. Johannes SchindelinMay 3, 2019
  34. Jeff KingMay 3, 2019
  35. Johannes SchindelinApr 26, 2019
  36. 0/5 Avoid spawning gzip in git archiveRené Scharfe, Jun 12, 2022
  37. 1/5 archive: rename archiver data field to filter_commandRené Scharfe, Jun 12, 2022
  38. 2/5 archive-tar: factor out write_block()René Scharfe, Jun 12, 2022
  39. 3/5 archive-tar: add internal gzip implementationRené Scharfe, Jun 12, 2022
  40. Junio C HamanoJun 13, 2022
  41. 4/5 archive-tar: use OS_CODE 3 (Unix) for internal gzipRené Scharfe, Jun 12, 2022
  42. 5/5 archive-tar: use internal gzip by defaultRené Scharfe, Jun 12, 2022
  43. Junio C HamanoJun 13, 2022
  44. Johannes SchindelinJun 14, 2022
  45. René ScharfeJun 14, 2022
  46. René ScharfeJun 14, 2022
  47. Johannes SchindelinJun 14, 2022
  48. René ScharfeJun 14, 2022
  49. Junio C HamanoJun 15, 2022
  50. Johannes SchindelinJun 14, 2022
  51. René ScharfeJun 14, 2022
  52. Johannes SchindelinJun 30, 2022
  53. Johannes SchindelinJul 1, 2022
  54. Jeff KingJul 1, 2022
  55. Junio C HamanoJul 1, 2022
  56. 0/6 Avoid spawning gzip in git archiveRené Scharfe, Jun 15, 2022
  57. 1/6 archive: update format documentationRené Scharfe, Jun 15, 2022
  58. 2/6 archive: rename archiver data field to filter_commandRené Scharfe, Jun 15, 2022
  59. 3/6 archive-tar: factor out write_block()René Scharfe, Jun 15, 2022
  60. 4/6 archive-tar: add internal gzip implementationRené Scharfe, Jun 15, 2022
  61. Ævar Arnfjörð BjarmasonJun 15, 2022
  62. René ScharfeJun 16, 2022
  63. Ævar Arnfjörð BjarmasonJun 24, 2022
  64. René ScharfeJun 24, 2022
  65. 5/6 archive-tar: use OS_CODE 3 (Unix) for internal gzipRené Scharfe, Jun 15, 2022
  66. 6/6 archive-tar: use internal gzip by defaultRené Scharfe, Jun 15, 2022
  67. René ScharfeMay 2, 2019
  68. René ScharfeMay 2, 2019
  69. Johannes SchindelinMay 8, 2019
  70. Jeff KingMay 8, 2019
  71. Johannes SchindelinMay 9, 2019
  72. Jeff KingMay 9, 2019
  73. René ScharfeMay 10, 2019
  74. Jeff KingMay 10, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.