{"thread":{"id":"50928","subject":"[PATCH 0/2] Avoid spawning gzip in git archive","startedAt":"2019-04-12T23:04:42Z","lastAt":"2022-07-01T17:47:37Z","messageCount":74,"participants":["Johannes Schindelin via GitGitGadget","Rohit Ashiwal via GitGitGadget","Jeff King","Junio C Hamano","René Scharfe","brian m. carlson","Rohit Ashiwal","Johannes Schindelin","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"373769","messageId":"pull.145.git.gitgitgadget@gmail.com","threadId":"50928","inReplyTo":null,"subject":"[PATCH 0/2] Avoid spawning gzip in git archive","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-12T23:04:38Z","receivedAt":"2019-04-12T23:04:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"When creating .tar.gz archives with git archive, we let gzip handle the\ncompression part. But that is not even necessary, as we already require zlib\n(to compress our loose objects). It is also unfortunate, as it requires gzip \nto be in the PATH (which is not the case e.g. with MinGit for Windows, which\ntries to bundle just the bare minimum of files to make Git work\nnon-interactively, for use with 3rd-party applications requiring Git).\n\nThis patch series resolves this conundrum by teaching git archive the trick\nto gzip-compress in-process.\n\nRohit Ashiwal (2):\n archive: replace write_or_die() calls with write_block_or_die()\n archive: avoid spawning `gzip`\n\n archive-tar.c | 54 ++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 41 insertions(+), 13 deletions(-)\n\n\nbase-commit: 8104ec994ea3849a968b4667d072fedd1e688642\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-145%2Fdscho%2Fdont-spawn-gzip-in-archive-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-145/dscho/dont-spawn-gzip-in-archive-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/145\n-- \ngitgitgadget\n"},{"id":"373770","messageId":"7a9525a78a7b7b237150b9264cf675a4a0b37267.1555110278.git.gitgitgadget@gmail.com","threadId":"50928","inReplyTo":"pull.145.git.gitgitgadget@gmail.com","subject":"[PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Rohit Ashiwal via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-12T23:04:39Z","receivedAt":"2019-04-12T23:04:43Z","isPatch":true,"sender":{"key":"rohit.ashiwal265@gmail.com","avatar":"https://avatars.githubusercontent.com/u/31043830?v=4"},"body":"From: Rohit Ashiwal \n\nMinGit for Windows comes without `gzip` bundled inside, git-archive uses\n`gzip -cn` to compress tar files but for this to work, gzip needs to be\npresent on the host system.\n\nIn the next commit, we will change the gzip compression so that we no\nlonger spawn `gzip` but let zlib perform the compression in the same\nprocess instead.\n\nIn preparation for this, we consolidate all the block writes into a\nsingle function.\n\nThis closes https://github.com/git-for-windows/git/issues/1970\n\nSigned-off-by: Rohit Ashiwal \nSigned-off-by: Johannes Schindelin \n---\n archive-tar.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 4aabd566fb..ba37dad27c 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -17,6 +17,8 @@ static unsigned long offset;\n \n static int tar_umask = 002;\n \n+static gzFile gzip;\n+\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args);\n \n@@ -38,11 +40,21 @@ static int write_tar_filter_archive(const struct archiver *ar,\n #define USTAR_MAX_MTIME 077777777777ULL\n #endif\n \n+/* writes out the whole block, or dies if fails */\n+static void write_block_or_die(const char *block) {\n+\tif (gzip) {\n+\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n+\t\t\tdie(_(\"gzwrite failed\"));\n+\t} else {\n+\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t}\n+}\n+\n /* writes out the whole block, but only if it is full */\n static void write_if_needed(void)\n {\n \tif (offset == BLOCKSIZE) {\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_block_or_die(block);\n \t\toffset = 0;\n \t}\n }\n@@ -66,7 +78,7 @@ static void do_write_blocked(const void *data, unsigned long size)\n \t\twrite_if_needed();\n \t}\n \twhile (size >= BLOCKSIZE) {\n-\t\twrite_or_die(1, buf, BLOCKSIZE);\n+\t\twrite_block_or_die(buf);\n \t\tsize -= BLOCKSIZE;\n \t\tbuf += BLOCKSIZE;\n \t}\n@@ -101,10 +113,10 @@ static void write_trailer(void)\n {\n \tint tail = BLOCKSIZE - offset;\n \tmemset(block + offset, 0, tail);\n-\twrite_or_die(1, block, BLOCKSIZE);\n+\twrite_block_or_die(block);\n \tif (tail < 2 * RECORDSIZE) {\n \t\tmemset(block, 0, offset);\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_block_or_die(block);\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"373771","messageId":"44d5371ae6808ec40e8f52c3dc258a85c878b27e.1555110278.git.gitgitgadget@gmail.com","threadId":"50928","inReplyTo":"pull.145.git.gitgitgadget@gmail.com","subject":"[PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Rohit Ashiwal via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-12T23:04:40Z","receivedAt":"2019-04-12T23:04:44Z","isPatch":true,"sender":{"key":"rohit.ashiwal265@gmail.com","avatar":"https://avatars.githubusercontent.com/u/31043830?v=4"},"body":"From: Rohit Ashiwal \n\nAs we already link to the zlib library, we can perform the compression\nwithout even requiring gzip on the host machine.\n\nSigned-off-by: Rohit Ashiwal \nSigned-off-by: Johannes Schindelin \n---\n archive-tar.c | 34 +++++++++++++++++++++++++---------\n 1 file changed, 25 insertions(+), 9 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex ba37dad27c..5979ed14b7 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -466,18 +466,34 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tfilter.use_shell = 1;\n \tfilter.in = -1;\n \n-\tif (start_command(&filter) < 0)\n-\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n-\tclose(1);\n-\tif (dup2(filter.in, 1) < 0)\n-\t\tdie_errno(_(\"unable to redirect descriptor\"));\n-\tclose(filter.in);\n+\tif (!strcmp(\"gzip -cn\", ar->data)) {\n+\t\tchar outmode[4] = \"wb\\0\";\n+\n+\t\tif (args->compression_level >= 0 && args->compression_level <= 9)\n+\t\t\toutmode[2] = '0' + args->compression_level;\n+\n+\t\tgzip = gzdopen(fileno(stdout), outmode);\n+\t\tif (!gzip)\n+\t\t\tdie(_(\"Could not gzdopen stdout\"));\n+\t} else {\n+\t\tif (start_command(&filter) < 0)\n+\t\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n+\t\tclose(1);\n+\t\tif (dup2(filter.in, 1) < 0)\n+\t\t\tdie_errno(_(\"unable to redirect descriptor\"));\n+\t\tclose(filter.in);\n+\t}\n \n \tr = write_tar_archive(ar, args);\n \n-\tclose(1);\n-\tif (finish_command(&filter) != 0)\n-\t\tdie(_(\"'%s' filter reported error\"), argv[0]);\n+\tif (gzip) {\n+\t\tif (gzclose(gzip) != Z_OK)\n+\t\t\tdie(_(\"gzclose failed\"));\n+\t} else {\n+\t\tclose(1);\n+\t\tif (finish_command(&filter) != 0)\n+\t\t\tdie(_(\"'%s' filter reported error\"), argv[0]);\n+\t}\n \n \tstrbuf_release(&cmd);\n \treturn r;\n-- \ngitgitgadget\n"},{"id":"373774","messageId":"20190413013451.GB2040@sigill.intra.peff.net","threadId":"50928","inReplyTo":"7a9525a78a7b7b237150b9264cf675a4a0b37267.1555110278.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-04-13T01:34:52Z","receivedAt":"2019-04-13T01:34:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 12, 2019 at 04:04:39PM -0700, Rohit Ashiwal via GitGitGadget wrote:\n\n> From: Rohit Ashiwal \n> \n> MinGit for Windows comes without `gzip` bundled inside, git-archive uses\n> `gzip -cn` to compress tar files but for this to work, gzip needs to be\n> present on the host system.\n> \n> In the next commit, we will change the gzip compression so that we no\n> longer spawn `gzip` but let zlib perform the compression in the same\n> process instead.\n> \n> In preparation for this, we consolidate all the block writes into a\n> single function.\n\nSounds like a good preparatory step. This part confused me, though:\n\n> @@ -38,11 +40,21 @@ static int write_tar_filter_archive(const struct archiver *ar,\n> #define USTAR_MAX_MTIME 077777777777ULL\n> #endif\n> \n> +/* writes out the whole block, or dies if fails */\n> +static void write_block_or_die(const char *block) {\n> +\tif (gzip) {\n> +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n> +\t\t\tdie(_(\"gzwrite failed\"));\n> +\t} else {\n> +\t\twrite_or_die(1, block, BLOCKSIZE);\n> +\t}\n> +}\n\nWhat is gzwrite()? At first I thought this was an out-of-sequence bit of\nthe series, but it turns out that this is a zlib.h interface. So the\nidea (I think) is that here we introduce a \"gzip\" variable that is\nalways false, and this first conditional arm is effectively dead code.\nAnd then in a later patch we'd set up \"gzip\" and it would become\nnot-dead.\n\nI think it would be less confusing if this just factored out\nwrite_block_or_die(), which starts as a thin wrapper and then grows the\ngzip parts in the next patch.\n\nA few nits on the code itself:\n\n> +static gzFile gzip;\n> [...]\n> + if (gzip) {\n\nIs it OK for us to ask about the truthiness of this opaque type? That\nworks if it's really a pointer behind the scenes, but it seems like it\nwould be equally OK for zlib to declare it as a struct.\n\nIt looks OK in my version of zlib, and that library tends to be fairly\nconservative so I wouldn't be surprised if it was that way back to the\nbeginning and remains that way for eternity. But it feels like a bad\npattern.\n\n> +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n\nThis cast is interesting. All of the matching write_or_die() calls are\npromoting it to a size_t, which is also unsigned.\n\nBLOCKSIZE is a constant. Should we be defining it with a \"U\" in the first place?\n\nI doubt it matters much either way from a correctness perspective. I\njust wonder when I see a cast like that if we're going to get unexpected\ntruncation or similar.\n\n-Peff\n"},{"id":"373775","messageId":"20190413015102.GC2040@sigill.intra.peff.net","threadId":"50928","inReplyTo":"44d5371ae6808ec40e8f52c3dc258a85c878b27e.1555110278.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-04-13T01:51:02Z","receivedAt":"2019-04-13T01:51:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 12, 2019 at 04:04:40PM -0700, Rohit Ashiwal via GitGitGadget wrote:\n\n> From: Rohit Ashiwal \n> \n> As we already link to the zlib library, we can perform the compression\n> without even requiring gzip on the host machine.\n\nVery cool. It's nice to drop a dependency, and this should be a bit more\nefficient, too.\n\n> diff --git a/archive-tar.c b/archive-tar.c\n> index ba37dad27c..5979ed14b7 100644\n> --- a/archive-tar.c\n> +++ b/archive-tar.c\n> @@ -466,18 +466,34 @@ static int write_tar_filter_archive(const struct archiver *ar,\n> \tfilter.use_shell = 1;\n> \tfilter.in = -1;\n> \n> -\tif (start_command(&filter) < 0)\n> -\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n> -\tclose(1);\n> -\tif (dup2(filter.in, 1) < 0)\n> -\t\tdie_errno(_(\"unable to redirect descriptor\"));\n> -\tclose(filter.in);\n> +\tif (!strcmp(\"gzip -cn\", ar->data)) {\n\nI wondered how you were going to kick this in, since users can define\narbitrary filters. I think it's kind of neat to automagically convert\n\"gzip -cn\" (which also happens to be the default). But I think we should\nmention that in the Documentation, in case somebody tries to use a\ncustom version of gzip and wonders why it isn't kicking in.\n\nLikewise, it might make sense in the tests to put a poison gzip in the\n$PATH so that we can be sure we're using our internal code, and not just\ncalling out to gzip (on platforms that have it, of course).\n\nThe alternative is that we could use a special token like \":zlib\" or\nsomething to indicate that the internal implementation should be used\n(and then tweak the baked-in default, too). That might be less\nsurprising for users, but most people would still get the benefit since\nthey'd be using the default config.\n\n> +\t\tchar outmode[4] = \"wb\\0\";\n\nThis looks sufficiently magical that it might merit a comment. I had to\nlook in the zlib header file to learn that this is just a normal\nstdio-style mode. But we can't just do:\n\n gzip = gzdopen(fd, \"wb\");\n\nbecause we want to (maybe) append a compression level. It's also\nslightly confusing that it explicitly includes a NUL, but later:\n\n> +\t\tif (args->compression_level >= 0 && args->compression_level <= 9)\n> +\t\t\toutmode[2] = '0' + args->compression_level;\n\nwe may overwrite that and assume that outmode[3] is also a NUL. Which it\nis, because of how C initialization works. But that means we also do not\nneed the \"\\0\" in the initializer.\n\nDropping that may make it slightly less jarring (any time I see a\nbackslash escape in an initializer, I assume I'm in for some binary\ntrickery, but this turns out to be much more mundane).\n\nI'd also consider just using a strbuf:\n\n struct strbuf outmode = STRBUF_INIT;\n\n strbuf_addstr(&outmode, \"wb\");\n if (args->compression_level >= 0 && args->compression_level <= 9)\n\tstrbuf_addch(&outmode, '0' + args->compression_level);\n\nThat's overkill in a sense, but it saves us having to deal with\nmanually-counted offsets, and this code is only run once per program\ninvocation, so the efficiency shouldn't matter.\n\n> +\t\tgzip = gzdopen(fileno(stdout), outmode);\n> +\t\tif (!gzip)\n> +\t\t\tdie(_(\"Could not gzdopen stdout\"));\n\nIs there a way to get a more specific error from zlib? I'm less\nconcerned about gzdopen here (which should never fail), and more about\nthe writing and closing steps. I don't see anything good for gzdopen(),\nbut...\n\n> +\tif (gzip) {\n> +\t\tif (gzclose(gzip) != Z_OK)\n> +\t\t\tdie(_(\"gzclose failed\"));\n\n...according to zlib.h, here the returned int is meaningful. And if\nZ_ERRNO, we should probably use die_errno() to give a better message.\n\n> [...]\n\nThat was a lot of little nits, but the overall shape of the patch looks\ngood to me (and I think the goal is obviously good). Thanks for working\non it.\n\n-Peff\n"},{"id":"373781","messageId":"xmqqzhouwizg.fsf@gitster-ct.c.googlers.com","threadId":"50928","inReplyTo":"20190413013451.GB2040@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-13T05:51:31Z","receivedAt":"2019-04-13T05:51:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> +/* writes out the whole block, or dies if fails */\n>> +static void write_block_or_die(const char *block) {\n>> +\tif (gzip) {\n>> +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n>> +\t\t\tdie(_(\"gzwrite failed\"));\n>> +\t} else {\n>> +\t\twrite_or_die(1, block, BLOCKSIZE);\n>> +\t}\n>> +}\n\nI agree everything you said you your two review messages.\n\nOne thing you did not mention but I found disturbing was that this\ndoes not take size argument but hardcodes BLOCKSIZE. If the patch\nwere removing use of BLOCKSIZE in its callers (because everybody\nuses the same constant), this would not have bothered me, but as the\ncaller passes BLOCKSIZE to all its callees except this one, I found\nthat the interface optimizes for a wrong thing (i.e. reducing\none-time pain of writing this single patch of having to repeat\nBLOCKSIZE in all calls to this function). This function should be\nupdated to take the size_t and have its caller(s) pass BLOCKSIZE.\n\nThanks for a review, and thanks Rohit for starting to get rid of\nexternal dependency on gzip binary.\n"},{"id":"373804","messageId":"8ef2164c-1d44-33bf-ea8a-49fa0b5c8abf@web.de","threadId":"50928","inReplyTo":"20190413015102.GC2040@sigill.intra.peff.net","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-04-13T22:01:10Z","receivedAt":"2019-04-13T22:01:50Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.04.2019 um 03:51 schrieb Jeff King:\n> On Fri, Apr 12, 2019 at 04:04:40PM -0700, Rohit Ashiwal via GitGitGadget wrote:\n>\n>> From: Rohit Ashiwal \n>>\n>> As we already link to the zlib library, we can perform the compression\n>> without even requiring gzip on the host machine.\n>\n> Very cool. It's nice to drop a dependency, and this should be a bit more\n> efficient, too.\n\nGetting rid of dependencies is good, and using zlib is the obvious way to\ngenerate .tgz files. Last time I tried something like that, a separate gzip\nprocess was faster, though -- at least on Linux [1]. How does this one\nfare?\n\nDoing compression in its own thread may be a good idea.\n\nRené\n\n\n[1] http://public-inbox.org/git/4AAAC8CE.8020302@lsrfire.ath.cx/\n"},{"id":"373805","messageId":"20190413221646.GL12419@genre.crustytoothpaste.net","threadId":"50928","inReplyTo":"20190413015102.GC2040@sigill.intra.peff.net","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-04-13T22:16:46Z","receivedAt":"2019-04-13T22:17:06Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Apr 12, 2019 at 09:51:02PM -0400, Jeff King wrote:\n> I wondered how you were going to kick this in, since users can define\n> arbitrary filters. I think it's kind of neat to automagically convert\n> \"gzip -cn\" (which also happens to be the default). But I think we should\n> mention that in the Documentation, in case somebody tries to use a\n> custom version of gzip and wonders why it isn't kicking in.\n> \n> Likewise, it might make sense in the tests to put a poison gzip in the\n> $PATH so that we can be sure we're using our internal code, and not just\n> calling out to gzip (on platforms that have it, of course).\n> \n> The alternative is that we could use a special token like \":zlib\" or\n> something to indicate that the internal implementation should be used\n> (and then tweak the baked-in default, too). That might be less\n> surprising for users, but most people would still get the benefit since\n> they'd be using the default config.\n\nI agree that a special value (or NULL, if that's possible) would be\nnicer here. That way, if someone does specify a custom gzip, we honor\nit, and it serves to document the code better. For example, if someone\nsymlinked pigz to gzip and used \"gzip -cn\", then they might not get the\nparallelization benefits they expected.\n\nI'm fine overall with the idea of bringing the compression into the\nbinary using zlib, provided that we preserve the \"-n\" behavior\n(producing reproducible archives).\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"373812","messageId":"20190414043409.9547-1-rohit.ashiwal265@gmail.com","threadId":"50928","inReplyTo":"20190413013451.GB2040@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Rohit Ashiwal","fromEmail":"rohit.ashiwal265@gmail.com","sentAt":"2019-04-14T04:34:09Z","receivedAt":"2019-04-14T04:38:08Z","isPatch":true,"sender":{"key":"rohit.ashiwal265@gmail.com","avatar":"https://avatars.githubusercontent.com/u/31043830?v=4"},"body":"Hey Peff!\n\nOn 2019-04-13 1:34 UTC Jeff King wrote:\n\n> What is gzwrite()?\n> [...]\n> I think it would be less confusing if this just factored out\n> write_block_or_die(), which starts as a thin wrapper and then grows the\n> gzip parts in the next patch.\n\nYou are right, it might appear to someone as a bit confusing, but I feel\nlike, this is the right commit to put it.\n\n> Is it OK for us to ask about the truthiness of this opaque type? That\n> works if it's really a pointer behind the scenes, but it seems like it\n> would be equally OK for zlib to declare it as a struct.\n\nIt would be perfectly sane on zlib's part to make gzFile a struct, and if\nso happens, I'll be there to refactor the code.\n\nRegards\nRohit\n\n"},{"id":"373813","messageId":"20190414043602.9729-1-rohit.ashiwal265@gmail.com","threadId":"50928","inReplyTo":"xmqqzhouwizg.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Rohit Ashiwal","fromEmail":"rohit.ashiwal265@gmail.com","sentAt":"2019-04-14T04:36:02Z","receivedAt":"2019-04-14T04:38:08Z","isPatch":true,"sender":{"key":"rohit.ashiwal265@gmail.com","avatar":"https://avatars.githubusercontent.com/u/31043830?v=4"},"body":"Hey jch!\n\nI'll change the signature of the function in next revision.\n\nThanks\nRohit\n\n"},{"id":"373817","messageId":"xmqq7ebwx4e4.fsf@gitster-ct.c.googlers.com","threadId":"50928","inReplyTo":"20190414043409.9547-1-rohit.ashiwal265@gmail.com","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-14T10:33:39Z","receivedAt":"2019-04-14T10:33:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Rohit Ashiwal writes:\n\n> On 2019-04-13 1:34 UTC Jeff King wrote:\n>\n>> What is gzwrite()?\n>> [...]\n>> I think it would be less confusing if this just factored out\n>> write_block_or_die(), which starts as a thin wrapper and then grows the\n>> gzip parts in the next patch.\n>\n> You are right, it might appear to someone as a bit confusing, but I feel\n> like, this is the right commit to put it.\n\nOften, the original author is the worst judge about the patch series\norganization, because s/he has been staring at his or her own\npatches too long and knows too much about them. Unless the author\nis very experienced and is good at pretending to be the first-time\nreader when proofreading his or her own patch, that is.\n\nFWIW, I tend to agree with Peff that the organization would become\nmuch easier to follow with \"first refactor without new feature, and\nin gzip related step add gzip thing\".\n\n>> Is it OK for us to ask about the truthiness of this opaque type? That\n>> works if it's really a pointer behind the scenes, but it seems like it\n>> would be equally OK for zlib to declare it as a struct.\n\nOr a small integer indexing into an internal array the library keeps\ntrack of ;-) At that point, truthiness would be completely gone, and\nthe compiler would not help catching \"if (opaque)\" as a syntax error\n(if the library implements the opaque thing as a structure, then we\nwill be saved).\n\n> It would be perfectly sane on zlib's part to make gzFile a struct, and if\n> so happens, I'll be there to refactor the code.\n\nWe do not trust any single developer enough with \"I'll do so when\nneeded\"---in practice, it will often be done by somebody else, and\nmore importantly, we would want anybody to be able to take things\nover, instead of relying on any one \"indispensable contributor\".\n\nIf it is reasonable to expect that things can easily be broken by an\nexternal factor, I'd prefer to see us being defensive from day one.\n\n"},{"id":"373913","messageId":"20190415213556.GB28128@sigill.intra.peff.net","threadId":"50928","inReplyTo":"8ef2164c-1d44-33bf-ea8a-49fa0b5c8abf@web.de","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-04-15T21:35:56Z","receivedAt":"2019-04-15T21:36:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Apr 14, 2019 at 12:01:10AM +0200, René Scharfe wrote:\n\n> >> As we already link to the zlib library, we can perform the compression\n> >> without even requiring gzip on the host machine.\n> >\n> > Very cool. It's nice to drop a dependency, and this should be a bit more\n> > efficient, too.\n> \n> Getting rid of dependencies is good, and using zlib is the obvious way to\n> generate .tgz files. Last time I tried something like that, a separate gzip\n> process was faster, though -- at least on Linux [1]. How does this one\n> fare?\n\nI'd expect a separate gzip to be faster in wall-clock time for a\nmulti-core machine, but overall consume more CPU. I'm slightly surprised\nthat your timings show that it actually wins on total CPU, too.\n\nHere are best-of-five times for \"git archive --format=tar.gz HEAD\" on\nlinux.git (the machine is a quad-core):\n\n [before, separate gzip]\n real\t0m21.501s\n user\t0m26.148s\n sys\t0m0.619s\n\n [after, internal gzwrite]\n real\t0m25.156s\n user\t0m25.059s\n sys\t0m0.096s\n\nwhich does show what I expect (longer overall, but less total CPU).\n\nWhich one you prefer depends on your situation, of course. A user on a\nworkstation with multiple cores probably cares most about end-to-end\nlatency and using all of their available horsepower. A server hosting\nrepositories and receiving many unrelated requests probably cares more\nabout total CPU (though the differences there are small enough that it\nmay not even be worth having a config knob to un-parallelize it).\n\n> Doing compression in its own thread may be a good idea.\n\nYeah. It might even make the patch simpler, since I'd expect it to be\nimplemented with start_async() and a descriptor, making it look just\nlike a gzip pipe to the caller. :)\n\n-Peff\n"},{"id":"373914","messageId":"20190415213659.GC28128@sigill.intra.peff.net","threadId":"50928","inReplyTo":"20190413221646.GL12419@genre.crustytoothpaste.net","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-04-15T21:36:59Z","receivedAt":"2019-04-15T21:37:02Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Apr 13, 2019 at 10:16:46PM +0000, brian m. carlson wrote:\n\n> > The alternative is that we could use a special token like \":zlib\" or\n> > something to indicate that the internal implementation should be used\n> > (and then tweak the baked-in default, too). That might be less\n> > surprising for users, but most people would still get the benefit since\n> > they'd be using the default config.\n> \n> I agree that a special value (or NULL, if that's possible) would be\n> nicer here. That way, if someone does specify a custom gzip, we honor\n> it, and it serves to document the code better. For example, if someone\n> symlinked pigz to gzip and used \"gzip -cn\", then they might not get the\n> parallelization benefits they expected.\n\nThanks for spelling that out. I had a vague feeling somebody might be\nsurprised, but I didn't know if people actually did stuff like\nsymlinking pigz to gzip (though it makes perfect sense to do so).\n\n> I'm fine overall with the idea of bringing the compression into the\n> binary using zlib, provided that we preserve the \"-n\" behavior\n> (producing reproducible archives).\n\nI just assumed that gzwrite() would have the \"-n\" behavior, but it's\ndefinitely worth double-checking.\n\n-Peff\n"},{"id":"374559","messageId":"nycvar.QRO.7.76.6.1904261026070.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"20190413013451.GB2040@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-04-26T14:28:12Z","receivedAt":"2019-04-26T14:28:29Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 12 Apr 2019, Jeff King wrote:\n\n> On Fri, Apr 12, 2019 at 04:04:39PM -0700, Rohit Ashiwal via GitGitGadget wrote:\n>\n> > From: Rohit Ashiwal \n> >\n> > MinGit for Windows comes without `gzip` bundled inside, git-archive uses\n> > `gzip -cn` to compress tar files but for this to work, gzip needs to be\n> > present on the host system.\n> >\n> > In the next commit, we will change the gzip compression so that we no\n> > longer spawn `gzip` but let zlib perform the compression in the same\n> > process instead.\n> >\n> > In preparation for this, we consolidate all the block writes into a\n> > single function.\n>\n> Sounds like a good preparatory step. This part confused me, though:\n>\n> > @@ -38,11 +40,21 @@ static int write_tar_filter_archive(const struct archiver *ar,\n> > #define USTAR_MAX_MTIME 077777777777ULL\n> > #endif\n> >\n> > +/* writes out the whole block, or dies if fails */\n> > +static void write_block_or_die(const char *block) {\n> > +\tif (gzip) {\n> > +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n> > +\t\t\tdie(_(\"gzwrite failed\"));\n> > +\t} else {\n> > +\t\twrite_or_die(1, block, BLOCKSIZE);\n> > +\t}\n> > +}\n>\n> What is gzwrite()? At first I thought this was an out-of-sequence bit of\n> the series, but it turns out that this is a zlib.h interface. So the\n> idea (I think) is that here we introduce a \"gzip\" variable that is\n> always false, and this first conditional arm is effectively dead code.\n> And then in a later patch we'd set up \"gzip\" and it would become\n> not-dead.\n>\n> I think it would be less confusing if this just factored out\n> write_block_or_die(), which starts as a thin wrapper and then grows the\n> gzip parts in the next patch.\n\nYes, I missed this in my pre-submission review. Sorry about that!\n\n> A few nits on the code itself:\n>\n> > +static gzFile gzip;\n> > [...]\n> > + if (gzip) {\n>\n> Is it OK for us to ask about the truthiness of this opaque type? That\n> works if it's really a pointer behind the scenes, but it seems like it\n> would be equally OK for zlib to declare it as a struct.\n>\n> It looks OK in my version of zlib, and that library tends to be fairly\n> conservative so I wouldn't be surprised if it was that way back to the\n> beginning and remains that way for eternity. But it feels like a bad\n> pattern.\n\nIt is even part of the public API that `gzFile` is `typedef`'d to a\npointer. So I think in the interest of simplicity, I'll leave it at that\n(but I'll mention this in the commit message).\n\n> > +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n>\n> This cast is interesting. All of the matching write_or_die() calls are\n> promoting it to a size_t, which is also unsigned.\n>\n> BLOCKSIZE is a constant. Should we be defining it with a \"U\" in the\n> first place?\n\nYep, good idea.\n\nCiao,\nDscho\n"},{"id":"374560","messageId":"nycvar.QRO.7.76.6.1904261028220.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"xmqqzhouwizg.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-04-26T14:29:45Z","receivedAt":"2019-04-26T14:29:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Sat, 13 Apr 2019, Junio C Hamano wrote:\n\n> Jeff King writes:\n>\n> >> +/* writes out the whole block, or dies if fails */\n> >> +static void write_block_or_die(const char *block) {\n> >> +\tif (gzip) {\n> >> +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n> >> +\t\t\tdie(_(\"gzwrite failed\"));\n> >> +\t} else {\n> >> +\t\twrite_or_die(1, block, BLOCKSIZE);\n> >> +\t}\n> >> +}\n>\n> I agree everything you said you your two review messages.\n>\n> One thing you did not mention but I found disturbing was that this\n> does not take size argument but hardcodes BLOCKSIZE.\n\nThat is very much on purpose, as this code really is specific to the `tar`\nfile format, which has a fixed, well-defined block size. It would make it\neasier to introduce a bug if that was a parameter.\n\nCiao,\nDscho\n"},{"id":"374563","messageId":"nycvar.QRO.7.76.6.1904261035090.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"20190413015102.GC2040@sigill.intra.peff.net","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-04-26T14:47:04Z","receivedAt":"2019-04-26T14:47:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Fri, 12 Apr 2019, Jeff King wrote:\n\n> On Fri, Apr 12, 2019 at 04:04:40PM -0700, Rohit Ashiwal via GitGitGadget wrote:\n>\n> > From: Rohit Ashiwal \n> >\n> > As we already link to the zlib library, we can perform the compression\n> > without even requiring gzip on the host machine.\n>\n> Very cool. It's nice to drop a dependency, and this should be a bit more\n> efficient, too.\n\nSadly, no, as René intuited and as your testing shows: there seems to be a\n~15% penalty for compressing in the same thread as producing the data to\nbe compressed.\n\nGiven that it reduces the number of dependencies, and given that it might\nbe better to rely on the external command `pigz -cn` if speed is really a\nmatter, I still think it makes sense to switch the default, though.\n\n> > diff --git a/archive-tar.c b/archive-tar.c\n> > index ba37dad27c..5979ed14b7 100644\n> > --- a/archive-tar.c\n> > +++ b/archive-tar.c\n> > @@ -466,18 +466,34 @@ static int write_tar_filter_archive(const struct archiver *ar,\n> > \tfilter.use_shell = 1;\n> > \tfilter.in = -1;\n> >\n> > -\tif (start_command(&filter) < 0)\n> > -\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n> > -\tclose(1);\n> > -\tif (dup2(filter.in, 1) < 0)\n> > -\t\tdie_errno(_(\"unable to redirect descriptor\"));\n> > -\tclose(filter.in);\n> > +\tif (!strcmp(\"gzip -cn\", ar->data)) {\n>\n> I wondered how you were going to kick this in, since users can define\n> arbitrary filters. I think it's kind of neat to automagically convert\n> \"gzip -cn\" (which also happens to be the default). But I think we should\n> mention that in the Documentation, in case somebody tries to use a\n> custom version of gzip and wonders why it isn't kicking in.\n>\n> Likewise, it might make sense in the tests to put a poison gzip in the\n> $PATH so that we can be sure we're using our internal code, and not just\n> calling out to gzip (on platforms that have it, of course).\n>\n> The alternative is that we could use a special token like \":zlib\" or\n> something to indicate that the internal implementation should be used\n> (and then tweak the baked-in default, too). That might be less\n> surprising for users, but most people would still get the benefit since\n> they'd be using the default config.\n\nI went with `:zlib`.\n\n> > +\t\tchar outmode[4] = \"wb\\0\";\n>\n> This looks sufficiently magical that it might merit a comment. I had to\n> look in the zlib header file to learn that this is just a normal\n> stdio-style mode. But we can't just do:\n>\n> gzip = gzdopen(fd, \"wb\");\n>\n> because we want to (maybe) append a compression level. It's also\n> slightly confusing that it explicitly includes a NUL, but later:\n>\n> > +\t\tif (args->compression_level >= 0 && args->compression_level <= 9)\n> > +\t\t\toutmode[2] = '0' + args->compression_level;\n>\n> we may overwrite that and assume that outmode[3] is also a NUL. Which it\n> is, because of how C initialization works. But that means we also do not\n> need the \"\\0\" in the initializer.\n>\n> Dropping that may make it slightly less jarring (any time I see a\n> backslash escape in an initializer, I assume I'm in for some binary\n> trickery, but this turns out to be much more mundane).\n>\n> I'd also consider just using a strbuf:\n>\n> struct strbuf outmode = STRBUF_INIT;\n>\n> strbuf_addstr(&outmode, \"wb\");\n> if (args->compression_level >= 0 && args->compression_level <= 9)\n> \tstrbuf_addch(&outmode, '0' + args->compression_level);\n>\n> That's overkill in a sense, but it saves us having to deal with\n> manually-counted offsets, and this code is only run once per program\n> invocation, so the efficiency shouldn't matter.\n\nI'll change that, too, as it seems that `pigz` allows compression levels\nhigher than 9, in which case we need `strbuf_addf()` anyway. I will not\nadjust the condition `<= 9`, of course, as zlib is still limited that way.\n\n> > +\t\tgzip = gzdopen(fileno(stdout), outmode);\n> > +\t\tif (!gzip)\n> > +\t\t\tdie(_(\"Could not gzdopen stdout\"));\n>\n> Is there a way to get a more specific error from zlib? I'm less\n> concerned about gzdopen here (which should never fail), and more about\n> the writing and closing steps. I don't see anything good for gzdopen(),\n> but...\n\nSadly, I did not find anything there.\n\n> > +\tif (gzip) {\n> > +\t\tif (gzclose(gzip) != Z_OK)\n> > +\t\t\tdie(_(\"gzclose failed\"));\n>\n> ...according to zlib.h, here the returned int is meaningful. And if\n> Z_ERRNO, we should probably use die_errno() to give a better message.\n\nOkay.\n\nThanks,\nDscho\n"},{"id":"374564","messageId":"nycvar.QRO.7.76.6.1904261047560.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"20190415213556.GB28128@sigill.intra.peff.net","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-04-26T14:51:02Z","receivedAt":"2019-04-26T14:51:15Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Mon, 15 Apr 2019, Jeff King wrote:\n\n> On Sun, Apr 14, 2019 at 12:01:10AM +0200, René Scharfe wrote:\n>\n> > >> As we already link to the zlib library, we can perform the compression\n> > >> without even requiring gzip on the host machine.\n> > >\n> > > Very cool. It's nice to drop a dependency, and this should be a bit more\n> > > efficient, too.\n> >\n> > Getting rid of dependencies is good, and using zlib is the obvious way to\n> > generate .tgz files. Last time I tried something like that, a separate gzip\n> > process was faster, though -- at least on Linux [1]. How does this one\n> > fare?\n>\n> I'd expect a separate gzip to be faster in wall-clock time for a\n> multi-core machine, but overall consume more CPU. I'm slightly surprised\n> that your timings show that it actually wins on total CPU, too.\n\nIf performance is really a concern, you'll be much better off using `pigz`\nthan `gzip`.\n\n> Here are best-of-five times for \"git archive --format=tar.gz HEAD\" on\n> linux.git (the machine is a quad-core):\n>\n> [before, separate gzip]\n> real\t0m21.501s\n> user\t0m26.148s\n> sys\t0m0.619s\n>\n> [after, internal gzwrite]\n> real\t0m25.156s\n> user\t0m25.059s\n> sys\t0m0.096s\n>\n> which does show what I expect (longer overall, but less total CPU).\n>\n> Which one you prefer depends on your situation, of course. A user on a\n> workstation with multiple cores probably cares most about end-to-end\n> latency and using all of their available horsepower. A server hosting\n> repositories and receiving many unrelated requests probably cares more\n> about total CPU (though the differences there are small enough that it\n> may not even be worth having a config knob to un-parallelize it).\n\nI am a bit sad that this is so noticeable. Nevertheless, I think that\ndropping the dependency is worth it, in particular given that `gzip` is\nnot exactly fast to begin with (you really should switch to `pigz` or to a\nfaster compression if you are interested in speed).\n\n> > Doing compression in its own thread may be a good idea.\n>\n> Yeah. It might even make the patch simpler, since I'd expect it to be\n> implemented with start_async() and a descriptor, making it look just\n> like a gzip pipe to the caller. :)\n\nSadly, it does not really look like it is simpler.\n\nAnd when going into the direction of multi-threaded compression anyway,\nthe `pigz` trick of compressing 32kB chunks in parallel is an interesting\nidea, too.\n\nAll of this, however, is outside of the purview of this (still relatively\nsimple) patch series.\n\nCiao,\nDscho\n"},{"id":"374565","messageId":"nycvar.QRO.7.76.6.1904261051310.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"20190413221646.GL12419@genre.crustytoothpaste.net","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-04-26T14:54:04Z","receivedAt":"2019-04-26T14:54:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi brian,\n\nOn Sat, 13 Apr 2019, brian m. carlson wrote:\n\n> On Fri, Apr 12, 2019 at 09:51:02PM -0400, Jeff King wrote:\n> > I wondered how you were going to kick this in, since users can define\n> > arbitrary filters. I think it's kind of neat to automagically convert\n> > \"gzip -cn\" (which also happens to be the default). But I think we should\n> > mention that in the Documentation, in case somebody tries to use a\n> > custom version of gzip and wonders why it isn't kicking in.\n> >\n> > Likewise, it might make sense in the tests to put a poison gzip in the\n> > $PATH so that we can be sure we're using our internal code, and not just\n> > calling out to gzip (on platforms that have it, of course).\n> >\n> > The alternative is that we could use a special token like \":zlib\" or\n> > something to indicate that the internal implementation should be used\n> > (and then tweak the baked-in default, too). That might be less\n> > surprising for users, but most people would still get the benefit since\n> > they'd be using the default config.\n>\n> I agree that a special value (or NULL, if that's possible) would be\n> nicer here. That way, if someone does specify a custom gzip, we honor\n> it, and it serves to document the code better. For example, if someone\n> symlinked pigz to gzip and used \"gzip -cn\", then they might not get the\n> parallelization benefits they expected.\n\nI went with `:zlib`. The `NULL` value would not really work, as there is\nno way to specify that via `archive.tgz.command`.\n\nAbout the symlinked thing: I do not really want to care to support such\nhacks. If you want a different compressor than the default (which can\nchange), you should specify it specifically.\n\n> I'm fine overall with the idea of bringing the compression into the\n> binary using zlib, provided that we preserve the \"-n\" behavior\n> (producing reproducible archives).\n\nThanks for voicing this concern. I had a look at zlib's source code, and\nit looks like it requires an extra function call (that we don't call) to\nmake the resulting file non-reproducible. In other words, it has the\nopposite default behavior from `gzip`.\n\nCiao,\nDscho\n"},{"id":"374584","messageId":"xmqqd0l8tjph.fsf@gitster-ct.c.googlers.com","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1904261028220.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-26T23:44:58Z","receivedAt":"2019-04-26T23:45:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> >> +/* writes out the whole block, or dies if fails */\n>> >> +static void write_block_or_die(const char *block) {\n>> >> +\tif (gzip) {\n>> >> +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n>> >> +\t\t\tdie(_(\"gzwrite failed\"));\n>> >> +\t} else {\n>> >> +\t\twrite_or_die(1, block, BLOCKSIZE);\n>> >> +\t}\n>> >> +}\n>>\n>> I agree everything you said you your two review messages.\n>>\n>> One thing you did not mention but I found disturbing was that this\n>> does not take size argument but hardcodes BLOCKSIZE.\n>\n> That is very much on purpose, as this code really is specific to the `tar`\n> file format, which has a fixed, well-defined block size. It would make it\n> easier to introduce a bug if that was a parameter.\n\nI am not so sure for two reasons.\n\nOne is that its caller is full of BLOCKSIZE constants passed as\nparameters (instead of calling a specialized function that hardcodes\nthe BLOCKSIZE without taking it as a parameter), and this being a\nfile-scope static, it does not really matter with respect to an\naccidental bug of mistakenly changing BLOCKSIZE either in the caller\nor callee.\n\nAnother is that I am not sure how your \"fixed format\" argument\nmeshes with the \"-b blocksize\" parameter to affect the tar/pax\noutput. The format may be fixed, but it is parameterized. If\nwe ever need to grow the ability to take \"-b\", having the knowledge\nthat our current code is limited to the fixed BLOCKSIZE in a single\nfunction (i.e. the caller of this function , not the callee) would \nbe less error prone.\n\nThese two are in addition to the uniformity of the abstraction\nconcerns I raised in my original review comment.\n\nSo, sorry, I do not think your response makes much sense.\n\nThanks.\n\n"},{"id":"374591","messageId":"f6f32bc0-109c-e0eb-f7d2-9e46647f260c@web.de","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1904261047560.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-04-27T09:59:09Z","receivedAt":"2019-04-27T09:59:46Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 26.04.19 um 16:51 schrieb Johannes Schindelin:> Hi Peff,\n>\n> On Mon, 15 Apr 2019, Jeff King wrote:\n>\n>> On Sun, Apr 14, 2019 at 12:01:10AM +0200, René Scharfe wrote:\n>>\n>>>>> As we already link to the zlib library, we can perform the compression\n>>>>> without even requiring gzip on the host machine.\n>>>>\n>>>> Very cool. It's nice to drop a dependency, and this should be a bit more\n>>>> efficient, too.\n>>>\n>>> Getting rid of dependencies is good, and using zlib is the obvious way to\n>>> generate .tgz files. Last time I tried something like that, a separate gzip\n>>> process was faster, though -- at least on Linux [1]. How does this one\n>>> fare?\n>>\n>> I'd expect a separate gzip to be faster in wall-clock time for a\n>> multi-core machine, but overall consume more CPU. I'm slightly surprised\n>> that your timings show that it actually wins on total CPU, too.\n\nMy initial expectation back then was that moving data between processes\nis costly and that compressing in-process would improve the overall\nperformance. Your expectation is more in line with what I then actually\nsaw. The difference in total CPU time wasn't that big, perhaps just\nnoise.\n\n> If performance is really a concern, you'll be much better off using `pigz`\n> than `gzip`.\n\nPerformance is always a concern, but on the other hand I didn't see any\ncomplaints about slow archiving so far.\n\n>> Here are best-of-five times for \"git archive --format=tar.gz HEAD\" on\n>> linux.git (the machine is a quad-core):\n>>\n>> [before, separate gzip]\n>> real\t0m21.501s\n>> user\t0m26.148s\n>> sys\t0m0.619s\n>>\n>> [after, internal gzwrite]\n>> real\t0m25.156s\n>> user\t0m25.059s\n>> sys\t0m0.096s\n>>\n>> which does show what I expect (longer overall, but less total CPU).\n\nI get similar numbers with hyperfine:\n\nBenchmark #1: git archive --format=tar.gz HEAD >/dev/null\n Time (mean ± σ): 16.683 s ± 0.451 s [User: 20.230 s, System: 0.375 s]\n Range (min … max): 16.308 s … 17.852 s 10 runs\n\nBenchmark #2: ~/src/git/git-archive --format=tar.gz HEAD >/dev/null\n Time (mean ± σ): 19.898 s ± 0.228 s [User: 19.825 s, System: 0.073 s]\n Range (min … max): 19.627 s … 20.355 s 10 runs\n\nBenchmark #3: git archive --format=zip HEAD >/dev/null\n Time (mean ± σ): 16.449 s ± 0.075 s [User: 16.340 s, System: 0.109 s]\n Range (min … max): 16.326 s … 16.611 s 10 runs\n\n#1 is git v2.21.0, #2 is with the two patches applied, #3 is v2.21.0\nagain, but with zip output, just to put things into perspective.\n\n>> Which one you prefer depends on your situation, of course. A user on a\n>> workstation with multiple cores probably cares most about end-to-end\n>> latency and using all of their available horsepower. A server hosting\n>> repositories and receiving many unrelated requests probably cares more\n>> about total CPU (though the differences there are small enough that it\n>> may not even be worth having a config knob to un-parallelize it).\n>\n> I am a bit sad that this is so noticeable. Nevertheless, I think that\n> dropping the dependency is worth it, in particular given that `gzip` is\n> not exactly fast to begin with (you really should switch to `pigz` or to a\n> faster compression if you are interested in speed).\n\nWe could import pigz verbatim, it's just 11K LOCs total. :)\n\n>>> Doing compression in its own thread may be a good idea.\n>>\n>> Yeah. It might even make the patch simpler, since I'd expect it to be\n>> implemented with start_async() and a descriptor, making it look just\n>> like a gzip pipe to the caller. :)\n>\n> Sadly, it does not really look like it is simpler.\n\nI have to agree -- at least I was unable to pull off the stdout\nplumbing trick. Is there a way? But it doesn't look too bad, and\nthe performance is closer to using the real gzip:\n\nBenchmark #1: ~/src/git/git-archive --format=tar.gz HEAD >/dev/null\n Time (mean ± σ): 17.300 s ± 0.198 s [User: 20.825 s, System: 0.356 s]\n Range (min … max): 17.042 s … 17.638 s 10 runs\n\nThis is with the following patch:\n\n---\n archive-tar.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 59 insertions(+), 4 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 3e53aac1e6..c889b84c2c 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -38,11 +38,13 @@ static int write_tar_filter_archive(const struct archiver *ar,\n #define USTAR_MAX_MTIME 077777777777ULL\n #endif\n\n+static int out_fd = 1;\n+\n /* writes out the whole block, but only if it is full */\n static void write_if_needed(void)\n {\n \tif (offset == BLOCKSIZE) {\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_or_die(out_fd, block, BLOCKSIZE);\n \t\toffset = 0;\n \t}\n }\n@@ -66,7 +68,7 @@ static void do_write_blocked(const void *data, unsigned long size)\n \t\twrite_if_needed();\n \t}\n \twhile (size >= BLOCKSIZE) {\n-\t\twrite_or_die(1, buf, BLOCKSIZE);\n+\t\twrite_or_die(out_fd, buf, BLOCKSIZE);\n \t\tsize -= BLOCKSIZE;\n \t\tbuf += BLOCKSIZE;\n \t}\n@@ -101,10 +103,10 @@ static void write_trailer(void)\n {\n \tint tail = BLOCKSIZE - offset;\n \tmemset(block + offset, 0, tail);\n-\twrite_or_die(1, block, BLOCKSIZE);\n+\twrite_or_die(out_fd, block, BLOCKSIZE);\n \tif (tail < 2 * RECORDSIZE) {\n \t\tmemset(block, 0, offset);\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_or_die(out_fd, block, BLOCKSIZE);\n \t}\n }\n\n@@ -434,6 +436,56 @@ static int write_tar_archive(const struct archiver *ar,\n \treturn err;\n }\n\n+static int internal_gzip(int in, int out, void *data)\n+{\n+\tint *levelp = data;\n+\tgzFile gzip = gzdopen(1, \"wb\");\n+\tif (!gzip)\n+\t\tdie(_(\"gzdopen failed\"));\n+\tif (gzsetparams(gzip, *levelp, Z_DEFAULT_STRATEGY) != Z_OK)\n+\t\tdie(_(\"unable to set compression level\"));\n+\n+\tfor (;;) {\n+\t\tchar buf[BLOCKSIZE];\n+\t\tssize_t read = xread(in, buf, sizeof(buf));\n+\t\tif (read < 0)\n+\t\t\tdie_errno(_(\"read failed\"));\n+\t\tif (read == 0)\n+\t\t\tbreak;\n+\t\tif (gzwrite(gzip, buf, read) != read)\n+\t\t\tdie(_(\"gzwrite failed\"));\n+\t}\n+\n+\tif (gzclose(gzip) != Z_OK)\n+\t\tdie(_(\"gzclose failed\"));\n+\tclose(in);\n+\treturn 0;\n+}\n+\n+static int write_tar_gzip_archive(const struct archiver *ar,\n+\t\t\t\t struct archiver_args *args)\n+{\n+\tstruct async filter;\n+\tint r;\n+\n+\tmemset(&filter, 0, sizeof(filter));\n+\tfilter.proc = internal_gzip;\n+\tfilter.data = &args->compression_level;\n+\tfilter.in = -1;\n+\n+\tif (start_async(&filter))\n+\t\tdie(_(\"unable to fork off internal gzip\"));\n+\tout_fd = filter.in;\n+\n+\tr = write_tar_archive(ar, args);\n+\n+\tclose(out_fd);\n+\tif (finish_async(&filter))\n+\t\tdie(_(\"error in internal gzip\"));\n+\n+\treturn r;\n+}\n+\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args)\n {\n@@ -445,6 +497,9 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!ar->data)\n \t\tBUG(\"tar-filter archiver called with no filter defined\");\n\n+\tif (!strcmp(ar->data, \"gzip -cn\"))\n+\t\treturn write_tar_gzip_archive(ar, args);\n+\n \tstrbuf_addstr(&cmd, ar->data);\n \tif (args->compression_level >= 0)\n \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\n--\n2.21.0\n"},{"id":"374602","messageId":"45afd432-9e45-ea76-aa1b-e8cd1264e3a0@web.de","threadId":"50928","inReplyTo":"f6f32bc0-109c-e0eb-f7d2-9e46647f260c@web.de","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-04-27T17:39:27Z","receivedAt":"2019-04-27T17:40:02Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 27.04.19 um 11:59 schrieb René Scharfe:> Am 26.04.19 um 16:51 schrieb Johannes Schindelin:\n>>\n>> On Mon, 15 Apr 2019, Jeff King wrote:\n>>\n>>> On Sun, Apr 14, 2019 at 12:01:10AM +0200, René Scharfe wrote:\n>>>\n>>>> Doing compression in its own thread may be a good idea.\n>>>\n>>> Yeah. It might even make the patch simpler, since I'd expect it to be\n>>> implemented with start_async() and a descriptor, making it look just\n>>> like a gzip pipe to the caller. :)\n>>\n>> Sadly, it does not really look like it is simpler.\n>\n> I have to agree -- at least I was unable to pull off the stdout\n> plumbing trick.\n\nThe simplest solution is of course to not touch the archive code. The\npatch below makes that possible:\n\nBenchmark #1: ~/src/git/git -c tar.tgz.command=~/src/git/git-gzip archive --format=tgz HEAD >/dev/null\n Time (mean ± σ): 17.256 s ± 0.299 s [User: 20.380 s, System: 0.294 s]\n Range (min … max): 16.940 s … 17.804 s 10 runs\n\nCurious to see how it looks like on other systems and platforms.\n\nAnd perhaps the buffer size needs to be tuned.\n\n-- >8 --\nSubject: [PATCH] add git gzip\n\nAdd a cheap gzip lookalike based on zlib for systems that don't have\n(or want) the real thing. It can be used e.g. to generate tgz files\nusing git archive and its configuration options tar.tgz.command and\ntar.tar.gz.command, without any other external dependency.\n\nSigned-off-by: Rene Scharfe \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/gzip.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 6 files changed, 69 insertions(+)\n create mode 100644 builtin/gzip.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 44c74402c8..e550868219 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -71,6 +71,7 @@\n /git-gc\n /git-get-tar-commit-id\n /git-grep\n+/git-gzip\n /git-hash-object\n /git-help\n /git-http-backend\ndiff --git a/Makefile b/Makefile\nindex 9f1b6e8926..2b34f1a4aa 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1075,6 +1075,7 @@ BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\n+BUILTIN_OBJS += builtin/gzip.o\n BUILTIN_OBJS += builtin/hash-object.o\n BUILTIN_OBJS += builtin/help.o\n BUILTIN_OBJS += builtin/index-pack.o\ndiff --git a/builtin.h b/builtin.h\nindex b78ab6e30b..abc34cc9d0 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -170,6 +170,7 @@ extern int cmd_fsck(int argc, const char **argv, const char *prefix);\n extern int cmd_gc(int argc, const char **argv, const char *prefix);\n extern int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n extern int cmd_grep(int argc, const char **argv, const char *prefix);\n+extern int cmd_gzip(int argc, const char **argv, const char *prefix);\n extern int cmd_hash_object(int argc, const char **argv, const char *prefix);\n extern int cmd_help(int argc, const char **argv, const char *prefix);\n extern int cmd_index_pack(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/gzip.c b/builtin/gzip.c\nnew file mode 100644\nindex 0000000000..90a98c44ce\n--- /dev/null\n+++ b/builtin/gzip.c\n@@ -0,0 +1,64 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+\n+static const char * const gzip_usage[] = {\n+\tN_(\"git gzip [-NUM]\"),\n+\tNULL\n+};\n+\n+static int level_callback(const struct option *opt, const char *arg, int unset)\n+{\n+\tint *levelp = opt->value;\n+\tint value;\n+\tconst char *endp;\n+\n+\tif (unset)\n+\t\tBUG(\"switch -NUM cannot be negated\");\n+\n+\tvalue = strtol(arg, (char **)&endp, 10);\n+\tif (*endp)\n+\t\tBUG(\"switch -NUM cannot be non-numeric\");\n+\n+\t*levelp = value;\n+\treturn 0;\n+}\n+\n+#define BUFFERSIZE (64 * 1024)\n+\n+int cmd_gzip(int argc, const char **argv, const char *prefix)\n+{\n+\tgzFile gz;\n+\tint level = Z_DEFAULT_COMPRESSION;\n+\tstruct option options[] = {\n+\t\tOPT_NUMBER_CALLBACK(&level, N_(\"compression level\"),\n+\t\t\t\t level_callback),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, gzip_usage, 0);\n+\tif (argc > 0)\n+\t\tusage_with_options(gzip_usage, options);\n+\n+\tgz = gzdopen(1, \"wb\");\n+\tif (!gz)\n+\t\tdie(_(\"unable to gzdopen stdout\"));\n+\n+\tif (gzsetparams(gz, level, Z_DEFAULT_STRATEGY) != Z_OK)\n+\t\tdie(_(\"unable to set compression level %d\"), level);\n+\n+\tfor (;;) {\n+\t\tchar buf[BUFFERSIZE];\n+\t\tssize_t read_bytes = xread(0, buf, sizeof(buf));\n+\t\tif (read_bytes < 0)\n+\t\t\tdie_errno(_(\"unable to read from stdin\"));\n+\t\tif (read_bytes == 0)\n+\t\t\tbreak;\n+\t\tif (gzwrite(gz, buf, read_bytes) != read_bytes)\n+\t\t\tdie(_(\"gzwrite failed\"));\n+\t}\n+\n+\tif (gzclose(gz) != Z_OK)\n+\t\tdie(_(\"gzclose failed\"));\n+\treturn 0;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 3a9af104b5..755848842c 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -99,6 +99,7 @@ git-gc mainporcelain\n git-get-tar-commit-id plumbinginterrogators\n git-grep mainporcelain info\n git-gui mainporcelain\n+git-gzip purehelpers\n git-hash-object plumbingmanipulators\n git-help ancillaryinterrogators complete\n git-http-backend synchingrepositories\ndiff --git a/git.c b/git.c\nindex 50da125c60..48f7fc6c56 100644\n--- a/git.c\n+++ b/git.c\n@@ -510,6 +510,7 @@ static struct cmd_struct commands[] = {\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"gzip\", cmd_gzip },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY | NO_PARSEOPT },\n--\n2.21.0\n"},{"id":"374667","messageId":"nycvar.QRO.7.76.6.1904291720120.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"45afd432-9e45-ea76-aa1b-e8cd1264e3a0@web.de","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-04-29T21:25:39Z","receivedAt":"2019-04-29T21:25:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi René,\n\n\nOn Sat, 27 Apr 2019, René Scharfe wrote:\n\n> Am 27.04.19 um 11:59 schrieb René Scharfe:> Am 26.04.19 um 16:51 schrieb\n> Johannes Schindelin:\n> >>\n> >> On Mon, 15 Apr 2019, Jeff King wrote:\n> >>\n> >>> On Sun, Apr 14, 2019 at 12:01:10AM +0200, René Scharfe wrote:\n> >>>\n> >>>> Doing compression in its own thread may be a good idea.\n> >>>\n> >>> Yeah. It might even make the patch simpler, since I'd expect it to\n> >>> be implemented with start_async() and a descriptor, making it look\n> >>> just like a gzip pipe to the caller. :)\n> >>\n> >> Sadly, it does not really look like it is simpler.\n> >\n> > I have to agree -- at least I was unable to pull off the stdout\n> > plumbing trick.\n>\n> The simplest solution is of course to not touch the archive code.\n\nWe could do that, of course, and we could avoid adding a new command that\nwe have to support for eternity by introducing a command mode for `git\narchive` instead (think: `git archive --gzip -9`), and marking that\ncommand mode clearly as an internal implementation detail.\n\nBut since the performance is still not quite on par with `gzip`, I would\nactually rather not, and really, just punt on that one, stating that\npeople interested in higher performance should use `pigz`.\n\nAnd who knows, maybe nobody will complain at all about the performance?\nIt's not like `gzip` is really, really fast (IIRC LZO blows gzip out of\nthe water, speed-wise).\n\nAnd if we get \"bug\" reports about this, we\n\n1) have a very easy workaround:\n\n\tgit config --global archive.tgz.command 'gzip -cn'\n\n2) could always implement a pigz-like multi-threading solution.\n\nI strongly expect a YAGNI here, though.\n\nCiao,\nDscho\n"},{"id":"374668","messageId":"nycvar.QRO.7.76.6.1904291732370.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"xmqqd0l8tjph.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-04-29T21:32:50Z","receivedAt":"2019-04-29T21:33:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Sat, 27 Apr 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> >> >> +/* writes out the whole block, or dies if fails */\n> >> >> +static void write_block_or_die(const char *block) {\n> >> >> +\tif (gzip) {\n> >> >> +\t\tif (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n> >> >> +\t\t\tdie(_(\"gzwrite failed\"));\n> >> >> +\t} else {\n> >> >> +\t\twrite_or_die(1, block, BLOCKSIZE);\n> >> >> +\t}\n> >> >> +}\n> >>\n> >> I agree everything you said you your two review messages.\n> >>\n> >> One thing you did not mention but I found disturbing was that this\n> >> does not take size argument but hardcodes BLOCKSIZE.\n> >\n> > That is very much on purpose, as this code really is specific to the `tar`\n> > file format, which has a fixed, well-defined block size. It would make it\n> > easier to introduce a bug if that was a parameter.\n>\n> I am not so sure for two reasons.\n>\n> One is that its caller is full of BLOCKSIZE constants passed as\n> parameters (instead of calling a specialized function that hardcodes\n> the BLOCKSIZE without taking it as a parameter), and this being a\n> file-scope static, it does not really matter with respect to an\n> accidental bug of mistakenly changing BLOCKSIZE either in the caller\n> or callee.\n\nI guess I can try to find some time next week to clean up those callers.\nBut honestly, I do not really think that this cleanup falls squarely into\nthe goal of this here patch series.\n\n> Another is that I am not sure how your \"fixed format\" argument\n> meshes with the \"-b blocksize\" parameter to affect the tar/pax\n> output. The format may be fixed, but it is parameterized. If\n> we ever need to grow the ability to take \"-b\", having the knowledge\n> that our current code is limited to the fixed BLOCKSIZE in a single\n> function (i.e. the caller of this function , not the callee) would\n> be less error prone.\n\nThis argument would hold a lot more water if the following lines were not\npart of archive-tar.c:\n\n\t#define RECORDSIZE (512)\n\t#define BLOCKSIZE (RECORDSIZE * 20)\n\n\tstatic char block[BLOCKSIZE];\n\nIf you can tell me how the `-b` (run-time) parameter can affect the\n(compile-time) `BLOCKSIZE` constant, maybe I can start to understand your\nconcern.\n\n:-)\n\nCiao,\nDscho\n\nP.S.: I just looked, and I do not even see a `-b` option of `git archive`,\nso I suspect that you talked about the generic tar file format? I was not\ntalking about each and every implementation of the tar file format here, I\nwas talking about the tar file format that Git generates.\n"},{"id":"374768","messageId":"05e97774-5dd1-7224-940f-e50558118d93@web.de","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1904291720120.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-05-01T17:45:05Z","receivedAt":"2019-05-01T17:45:44Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Hello Dscho,\n\nAm 29.04.19 um 23:25 schrieb Johannes Schindelin:\n> On Sat, 27 Apr 2019, René Scharfe wrote:\n>> The simplest solution is of course to not touch the archive code.\n>\n> We could do that, of course, and we could avoid adding a new command that\n> we have to support for eternity by introducing a command mode for `git\n> archive` instead (think: `git archive --gzip -9`), and marking that\n> command mode clearly as an internal implementation detail.\n\nadding gzip as the 142nd git command and 18th pure helper *would* be a\nbit embarrassing, in particular for a command that's not directly\nrelated to version control and readily available on all platforms.\nExposing it as a (hidden?) archive sub-command might be better.\n\n> But since the performance is still not quite on par with `gzip`, I would\n> actually rather not, and really, just punt on that one, stating that\n> people interested in higher performance should use `pigz`.\n\nHere are my performance numbers for generating .tar.gz files again:\n\nmaster, using gzip(1):\n Time (mean ± σ): 16.683 s ± 0.451 s [User: 20.230 s, System: 0.375 s]\n Range (min … max): 16.308 s … 17.852 s 10 runs\n\nusing zlib sequentially:\n Time (mean ± σ): 19.898 s ± 0.228 s [User: 19.825 s, System: 0.073 s]\n Range (min … max): 19.627 s … 20.355 s 10 runs\n\nusing zlib asynchronously:\n Time (mean ± σ): 17.300 s ± 0.198 s [User: 20.825 s, System: 0.356 s]\n Range (min … max): 17.042 s … 17.638 s 10 runs\n\nusing a gzip-lookalike:\n Time (mean ± σ): 17.256 s ± 0.299 s [User: 20.380 s, System: 0.294 s]\n Range (min … max): 16.940 s … 17.804 s 10 runs\n\nThe last two have comparable system time, ca. 1% more user time and\nca. 5% longer duration. The second one has much better system time\nand 2% less user time and 19% longer duration. Hmm.\n\n> And who knows, maybe nobody will complain at all about the performance?\n\nProbably. And popular tarballs would be cached anyway, I guess.\n\nSo I'll send comments on your series later this week.\n\nRené\n"},{"id":"374769","messageId":"20190501180752.GA4109@sigill.intra.peff.net","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1904261026070.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-01T18:07:52Z","receivedAt":"2019-05-01T18:07:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 26, 2019 at 10:28:12AM -0400, Johannes Schindelin wrote:\n\n> > > +static gzFile gzip;\n> > > [...]\n> > > + if (gzip) {\n> >\n> > Is it OK for us to ask about the truthiness of this opaque type? That\n> > works if it's really a pointer behind the scenes, but it seems like it\n> > would be equally OK for zlib to declare it as a struct.\n> >\n> > It looks OK in my version of zlib, and that library tends to be fairly\n> > conservative so I wouldn't be surprised if it was that way back to the\n> > beginning and remains that way for eternity. But it feels like a bad\n> > pattern.\n> \n> It is even part of the public API that `gzFile` is `typedef`'d to a\n> pointer. So I think in the interest of simplicity, I'll leave it at that\n> (but I'll mention this in the commit message).\n\nI think that's probably OK. My biggest concern is that we'd notice if\nour assumption changes, but I think modern compilers would generally\ncomplain about checking a tautological truth value.\n\n-Peff\n"},{"id":"374770","messageId":"20190501180936.GB4109@sigill.intra.peff.net","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1904291732370.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-01T18:09:37Z","receivedAt":"2019-05-01T18:09:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 29, 2019 at 05:32:50PM -0400, Johannes Schindelin wrote:\n\n> > Another is that I am not sure how your \"fixed format\" argument\n> > meshes with the \"-b blocksize\" parameter to affect the tar/pax\n> > output. The format may be fixed, but it is parameterized. If\n> > we ever need to grow the ability to take \"-b\", having the knowledge\n> > that our current code is limited to the fixed BLOCKSIZE in a single\n> > function (i.e. the caller of this function , not the callee) would\n> > be less error prone.\n> \n> This argument would hold a lot more water if the following lines were not\n> part of archive-tar.c:\n> \n> \t#define RECORDSIZE (512)\n> \t#define BLOCKSIZE (RECORDSIZE * 20)\n> \n> \tstatic char block[BLOCKSIZE];\n> \n> If you can tell me how the `-b` (run-time) parameter can affect the\n> (compile-time) `BLOCKSIZE` constant, maybe I can start to understand your\n> concern.\n\nFWIW, I agree with you here. These patches are not making anything worse\n(and may even make them better, since we'd probably need to swap out the\nBLOCKSIZE constant for a run-time \"blocksize\" variable in fewer places).\n\n-Peff\n"},{"id":"374772","messageId":"20190501181807.GC4109@sigill.intra.peff.net","threadId":"50928","inReplyTo":"05e97774-5dd1-7224-940f-e50558118d93@web.de","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-01T18:18:07Z","receivedAt":"2019-05-01T18:18:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 01, 2019 at 07:45:05PM +0200, René Scharfe wrote:\n\n> > But since the performance is still not quite on par with `gzip`, I would\n> > actually rather not, and really, just punt on that one, stating that\n> > people interested in higher performance should use `pigz`.\n> \n> Here are my performance numbers for generating .tar.gz files again:\n> \n> master, using gzip(1):\n> Time (mean ± σ): 16.683 s ± 0.451 s [User: 20.230 s, System: 0.375 s]\n> Range (min … max): 16.308 s … 17.852 s 10 runs\n> \n> using zlib sequentially:\n> Time (mean ± σ): 19.898 s ± 0.228 s [User: 19.825 s, System: 0.073 s]\n> Range (min … max): 19.627 s … 20.355 s 10 runs\n> \n> using zlib asynchronously:\n> Time (mean ± σ): 17.300 s ± 0.198 s [User: 20.825 s, System: 0.356 s]\n> Range (min … max): 17.042 s … 17.638 s 10 runs\n> \n> using a gzip-lookalike:\n> Time (mean ± σ): 17.256 s ± 0.299 s [User: 20.380 s, System: 0.294 s]\n> Range (min … max): 16.940 s … 17.804 s 10 runs\n> \n> The last two have comparable system time, ca. 1% more user time and\n> ca. 5% longer duration. The second one has much better system time\n> and 2% less user time and 19% longer duration. Hmm.\n\nI think the start_async() one seems like a good option. It reclaims most\nof the (wall-clock) performance, isn't very much code, and doesn't leave\nany ugly user-visible traces.\n\nI'd be fine to see it come later, though, on top of the patches Dscho is\nsending. Even though changing to sequential zlib is technically a change\nin behavior, the existing behavior wasn't really planned. And given the\nwall-clock versus CPU time tradeoff, it's not entirely clear that one\nsolution is better than the other.\n\n> > And who knows, maybe nobody will complain at all about the performance?\n> \n> Probably. And popular tarballs would be cached anyway, I guess.\n\nAt GitHub we certainly do cache the git-archive output. We'd also be\njust fine with the sequential solution. We generally turn down\npack.threads to 1, and keep our CPUs busy by serving multiple users\nanyway.\n\nSo whatever has the lowest overall CPU time is generally preferable, but\nthe times are close enough that I don't think we'd care much either way\n(and it's probably not worth having a config option or similar).\n\n-Peff\n"},{"id":"374842","messageId":"87ftpwip5m.fsf@evledraar.gmail.com","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1904261051310.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-02T20:20:53Z","receivedAt":"2019-05-02T20:20:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 26 2019, Johannes Schindelin wrote:\n\n> Hi brian,\n>\n> On Sat, 13 Apr 2019, brian m. carlson wrote:\n>\n>> On Fri, Apr 12, 2019 at 09:51:02PM -0400, Jeff King wrote:\n>> > I wondered how you were going to kick this in, since users can define\n>> > arbitrary filters. I think it's kind of neat to automagically convert\n>> > \"gzip -cn\" (which also happens to be the default). But I think we should\n>> > mention that in the Documentation, in case somebody tries to use a\n>> > custom version of gzip and wonders why it isn't kicking in.\n>> >\n>> > Likewise, it might make sense in the tests to put a poison gzip in the\n>> > $PATH so that we can be sure we're using our internal code, and not just\n>> > calling out to gzip (on platforms that have it, of course).\n>> >\n>> > The alternative is that we could use a special token like \":zlib\" or\n>> > something to indicate that the internal implementation should be used\n>> > (and then tweak the baked-in default, too). That might be less\n>> > surprising for users, but most people would still get the benefit since\n>> > they'd be using the default config.\n>>\n>> I agree that a special value (or NULL, if that's possible) would be\n>> nicer here. That way, if someone does specify a custom gzip, we honor\n>> it, and it serves to document the code better. For example, if someone\n>> symlinked pigz to gzip and used \"gzip -cn\", then they might not get the\n>> parallelization benefits they expected.\n>\n> I went with `:zlib`. The `NULL` value would not really work, as there is\n> no way to specify that via `archive.tgz.command`.\n>\n> About the symlinked thing: I do not really want to care to support such\n> hacks.\n\nIt's the standard way by which a lot of systems do this, e.g. on my\nDebian box:\n\n $ find /{,s}bin /usr/{,s}bin -type l -exec file {} \\;|grep /etc/alternatives|wc -l\n 108\n\nTo write this E-Mail I'm invoking one such symlink :)\n\n> If you want a different compressor than the default (which can\n> change), you should specify it specifically.\n\nYou might want to do so system-wide, or for each program at a time.\n\nI don't care about this for gzip myself, just pointing out it *is* a\nthing people use.\n\n>> I'm fine overall with the idea of bringing the compression into the\n>> binary using zlib, provided that we preserve the \"-n\" behavior\n>> (producing reproducible archives).\n>\n> Thanks for voicing this concern. I had a look at zlib's source code, and\n> it looks like it requires an extra function call (that we don't call) to\n> make the resulting file non-reproducible. In other words, it has the\n> opposite default behavior from `gzip`.\n\nJust commenting on the overall thread: I like René's \"new built-in\"\npatch best.\n\nYou mentioned \"new command that we have to support for eternity\". I\nthink calling it \"git gzip\" is a bad idea. We'd make it \"git\narchive--gzip\" or \"git archive--helper\", and we could hide building it\nbehind some compat flag.\n\nThen we'd carry no if/else internal/external code, and the portability\nissue that started this would be addressed, no?\n\nAs a bonus we could also drop the \"GZIP\" prereq from the test suite\nentirely and just put that \"gzip\" in $PATH for the purposes of the\ntests.\n\nI spied on your yet-to-be-submitted patches and you could drop GZIP from\nthe \"git archive\" tests, but we'd still need it in\nt/t5562-http-backend-content-length.sh, but not if we had a \"gzip\"\ncompat helper.\n\nThere's also a long-standing bug/misfeature in git-archive that I wonder\nabout: When you combine --format with --remote you can only generate\ne.g. tar.gz if the remote is OK with it, if it says no you can't even if\nit supports \"tar\" and you could do the \"gz\" part locally. Would such a\npatch be harder with :zlib than if we always just spewed out to external\n\"gzip\" after satisfying some criteria?\n"},{"id":"374843","messageId":"927f1e99-baa3-2d8a-cb11-0aedef6adc5c@web.de","threadId":"50928","inReplyTo":"4ea94a8784876c3a19e387537edd81a957fc692c.1556321244.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/4] archive: optionally use zlib directly for gzip compression","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-05-02T20:29:26Z","receivedAt":"2019-05-02T20:29:57Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 27.04.19 um 01:27 schrieb Rohit Ashiwal via GitGitGadget:\n> From: Rohit Ashiwal \n>\n> As we already link to the zlib library, we can perform the compression\n> without even requiring gzip on the host machine.\n>\n> Note: the `-n` flag that `git archive` passed to `gzip` wants to ensure\n> that a reproducible file is written, i.e. no filename or mtime will be\n> recorded in the compressed output. This is already the default for\n> zlib's `gzopen()` function (if the file name or mtime should be\n> recorded, the `deflateSetHeader()` function would have to be called\n> instead).\n>\n> Note also that the `gzFile` datatype is defined as a pointer in\n> `zlib.h`, i.e. we can rely on the fact that it can be `NULL`.\n>\n> At this point, this new mode is hidden behind the pseudo command\n> `:zlib`: assign this magic string to the `archive.tgz.command` config\n> setting to enable it.\n\nTechnically the patch emits the gzip format using the gz* functions.\nRaw zlib output with deflate* would be slightly different. So I'd\nrather use \"gzip\" instead of \"zlib\" in the magic string.\n\nAnd I'm not sure about the colon as the only magic marker. Perhaps\nthrow in a \"git \" or \"git-\" instead or in addition?\n\n> @@ -459,18 +464,40 @@ static int write_tar_filter_archive(const struct archiver *ar,\n> \tfilter.use_shell = 1;\n> \tfilter.in = -1;\n>\n> -\tif (start_command(&filter) < 0)\n> -\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n> -\tclose(1);\n> -\tif (dup2(filter.in, 1) < 0)\n> -\t\tdie_errno(_(\"unable to redirect descriptor\"));\n> -\tclose(filter.in);\n> +\tif (!strcmp(\":zlib\", ar->data)) {\n> +\t\tstruct strbuf mode = STRBUF_INIT;\n> +\n> +\t\tstrbuf_addstr(&mode, \"wb\");\n> +\n> +\t\tif (args->compression_level >= 0 && args->compression_level <= 9)\n> +\t\t\tstrbuf_addf(&mode, \"%d\", args->compression_level);\n\nUsing gzsetparams() to set the compression level numerically after gzdopen()\ninstead of baking it into the mode string feels cleaner.\n\n> +\n> +\t\tgzip = gzdopen(fileno(stdout), mode.buf);\n> +\t\tif (!gzip)\n> +\t\t\tdie(_(\"Could not gzdopen stdout\"));\n> +\t\tstrbuf_release(&mode);\n> +\t} else {\n> +\t\tif (start_command(&filter) < 0)\n> +\t\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n> +\t\tclose(1);\n> +\t\tif (dup2(filter.in, 1) < 0)\n> +\t\t\tdie_errno(_(\"unable to redirect descriptor\"));\n> +\t\tclose(filter.in);\n> +\t}\n>\n> \tr = write_tar_archive(ar, args);\n>\n> -\tclose(1);\n> -\tif (finish_command(&filter) != 0)\n> -\t\tdie(_(\"'%s' filter reported error\"), argv[0]);\n> +\tif (gzip) {\n> +\t\tint ret = gzclose(gzip);\n> +\t\tif (ret == Z_ERRNO)\n> +\t\t\tdie_errno(_(\"gzclose failed\"));\n> +\t\telse if (ret != Z_OK)\n> +\t\t\tdie(_(\"gzclose failed (%d)\"), ret);\n> +\t} else {\n> +\t\tclose(1);\n> +\t\tif (finish_command(&filter) != 0)\n> +\t\t\tdie(_(\"'%s' filter reported error\"), argv[0]);\n> +\t}\n>\n> \tstrbuf_release(&cmd);\n> \treturn r;\n>\n"},{"id":"374844","messageId":"be339f04-33e0-ede1-dbc2-340d7fb6694f@web.de","threadId":"50928","inReplyTo":"20190501180936.GB4109@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-05-02T20:29:55Z","receivedAt":"2019-05-02T20:30:08Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 01.05.19 um 20:09 schrieb Jeff King:\n> On Mon, Apr 29, 2019 at 05:32:50PM -0400, Johannes Schindelin wrote:\n>\n>>> Another is that I am not sure how your \"fixed format\" argument\n>>> meshes with the \"-b blocksize\" parameter to affect the tar/pax\n>>> output. The format may be fixed, but it is parameterized. If\n>>> we ever need to grow the ability to take \"-b\", having the knowledge\n>>> that our current code is limited to the fixed BLOCKSIZE in a single\n>>> function (i.e. the caller of this function , not the callee) would\n>>> be less error prone.\n>>\n>> This argument would hold a lot more water if the following lines were not\n>> part of archive-tar.c:\n>>\n>> \t#define RECORDSIZE (512)\n>> \t#define BLOCKSIZE (RECORDSIZE * 20)\n>>\n>> \tstatic char block[BLOCKSIZE];\n>>\n>> If you can tell me how the `-b` (run-time) parameter can affect the\n>> (compile-time) `BLOCKSIZE` constant, maybe I can start to understand your\n>> concern.\n>\n> FWIW, I agree with you here. These patches are not making anything worse\n> (and may even make them better, since we'd probably need to swap out the\n> BLOCKSIZE constant for a run-time \"blocksize\" variable in fewer places).\n\nThe block size is mostly relevant for writing tar archives to magnetic\ntapes. You can do that with git archive and a tape drive that supports\nthe blocking factor 20, which is the default for GNU tar and thus should\nbe quite common. You may get higher performance with a higher blocking\nfactor, if supported.\n\nBut so far this didn't come up on the mailing list, and I'd be surprised\nif people really wrote snapshots of git archives directly to tape. So\nI'm not too worried about this define ever becoming a user-settable\noption. Sealing the constant into a function a bit feels dirty, though.\nMixing code and data makes the code more brittle.\n\nAnother example of that is the hard-coded file descriptor in the same\nfunction, by the way. It's a lot of busywork to undo in order to gain\nthe ability to write to some other fd, for the questionable convenience\nof not having to pass that parameter along the call chain. My bad.\n\nBut anyway, I worry more about the fact that blocking is not needed when\ngzip'ing; gzwrite can be fed pieces of any size, not just 20 KB chunks.\nThe tar writer just needs to round up the archive size to a multiple of\n20 KB and pad with NUL bytes at the end, in order to produce the same\nuncompressed output as non-compressing tar.\n\nIf we'd wanted to be tape-friendly, then we'd have to block the gzip'ed\noutput instead of the uncompressed tar file, but I'm not suggesting\ndoing that.\n\nNote to self: I wonder if moving the blocking part out into an\nasynchronous function could simplify the code.\n\nRené\n"},{"id":"374845","messageId":"d960966d-a7d4-dc3e-ca0a-70e9e5d1abf3@web.de","threadId":"50928","inReplyTo":"ac2b2488a1b42b3caf8a84594c48eca796748e59.1556321244.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/4] archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-05-02T20:30:11Z","receivedAt":"2019-05-02T20:30:30Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 27.04.19 um 01:27 schrieb Johannes Schindelin via GitGitGadget:\n> From: Johannes Schindelin \n>\n> They really are unsigned, and we are using e.g. BLOCKSIZE as `size_t`\n> parameter to pass to `write_or_die()`.\n\nTrue, but the compiler converts that value correctly to size_t without\ncomplaint already, doesn't it? What am I missing?\n\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> archive-tar.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/archive-tar.c b/archive-tar.c\n> index af9ea70733..be06c8b205 100644\n> --- a/archive-tar.c\n> +++ b/archive-tar.c\n> @@ -9,7 +9,7 @@\n> #include \"streaming.h\"\n> #include \"run-command.h\"\n>\n> -#define RECORDSIZE\t(512)\n> +#define RECORDSIZE\t(512u)\n> #define BLOCKSIZE\t(RECORDSIZE * 20)\n>\n> static char block[BLOCKSIZE];\n>\n"},{"id":"374891","messageId":"nycvar.QRO.7.76.6.1905031437590.45@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"87ftpwip5m.fsf@evledraar.gmail.com","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-05-03T20:49:17Z","receivedAt":"2019-05-03T20:49:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Thu, 2 May 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Apr 26 2019, Johannes Schindelin wrote:\n>\n> > On Sat, 13 Apr 2019, brian m. carlson wrote:\n> >\n> >> On Fri, Apr 12, 2019 at 09:51:02PM -0400, Jeff King wrote:\n> >> > I wondered how you were going to kick this in, since users can\n> >> > define arbitrary filters. I think it's kind of neat to\n> >> > automagically convert \"gzip -cn\" (which also happens to be the\n> >> > default). But I think we should mention that in the Documentation,\n> >> > in case somebody tries to use a custom version of gzip and wonders\n> >> > why it isn't kicking in.\n> >> >\n> >> > Likewise, it might make sense in the tests to put a poison gzip in\n> >> > the $PATH so that we can be sure we're using our internal code, and\n> >> > not just calling out to gzip (on platforms that have it, of\n> >> > course).\n> >> >\n> >> > The alternative is that we could use a special token like \":zlib\"\n> >> > or something to indicate that the internal implementation should be\n> >> > used (and then tweak the baked-in default, too). That might be less\n> >> > surprising for users, but most people would still get the benefit\n> >> > since they'd be using the default config.\n> >>\n> >> I agree that a special value (or NULL, if that's possible) would be\n> >> nicer here. That way, if someone does specify a custom gzip, we honor\n> >> it, and it serves to document the code better. For example, if\n> >> someone symlinked pigz to gzip and used \"gzip -cn\", then they might\n> >> not get the parallelization benefits they expected.\n> >\n> > I went with `:zlib`. The `NULL` value would not really work, as there\n> > is no way to specify that via `archive.tgz.command`.\n> >\n> > About the symlinked thing: I do not really want to care to support\n> > such hacks.\n>\n> It's the standard way by which a lot of systems do this, e.g. on my\n> Debian box:\n>\n> $ find /{,s}bin /usr/{,s}bin -type l -exec file {} \\;|grep /etc/alternatives|wc -l\n> 108\n>\n> To write this E-Mail I'm invoking one such symlink :)\n\nI am well aware of the way Debian-based systems handle alternatives, and I\nmyself also use something similar to write this E-Mail (but it is not a\nsymlink, it is a Git alias).\n\nBut that's not the hack that I was talking about.\n\nThe hack I meant was: if you symlink `gzip` to `pigz` in your `PATH` *and\nthen expect `git archive --format=tgz` to pick that up*.\n\nAs far as I am concerned, the fact that `git archive --format=tgz` spawns\n`gzip` to perform the compression is an implementation detail, and not\nsomething that users should feel they can rely on.\n\n> > If you want a different compressor than the default (which can\n> > change), you should specify it specifically.\n>\n> You might want to do so system-wide, or for each program at a time.\n>\n> I don't care about this for gzip myself, just pointing out it *is* a\n> thing people use.\n\nSure.\n\n> >> I'm fine overall with the idea of bringing the compression into the\n> >> binary using zlib, provided that we preserve the \"-n\" behavior\n> >> (producing reproducible archives).\n> >\n> > Thanks for voicing this concern. I had a look at zlib's source code,\n> > and it looks like it requires an extra function call (that we don't\n> > call) to make the resulting file non-reproducible. In other words, it\n> > has the opposite default behavior from `gzip`.\n>\n> Just commenting on the overall thread: I like René's \"new built-in\"\n> patch best.\n\nI guess we now have to diverging votes: yours for the `git archive --gzip`\n\"built-in\" and Peff's for the async code ;-)\n\n> You mentioned \"new command that we have to support for eternity\". I\n> think calling it \"git gzip\" is a bad idea. We'd make it \"git\n> archive--gzip\" or \"git archive--helper\", and we could hide building it\n> behind some compat flag.\n>\n> Then we'd carry no if/else internal/external code, and the portability\n> issue that started this would be addressed, no?\n\nSure.\n\nThe async version would leave the door wide open for implementing pigz'\ntrick to multi-thread the compression, though.\n\n> As a bonus we could also drop the \"GZIP\" prereq from the test suite\n> entirely and just put that \"gzip\" in $PATH for the purposes of the\n> tests.\n>\n> I spied on your yet-to-be-submitted patches and you could drop GZIP from\n> the \"git archive\" tests, but we'd still need it in\n> t/t5562-http-backend-content-length.sh, but not if we had a \"gzip\"\n> compat helper.\n\nWe need it at least once for *decompressing* the `--format=tgz` output in\norder to compare it to the `--format=tar` output. Besides, I think it is\nreally important to keep the test that verifies that the output is correct\n(i.e. that gzip can decompress it).\n\n> There's also a long-standing bug/misfeature in git-archive that I wonder\n> about: When you combine --format with --remote you can only generate\n> e.g. tar.gz if the remote is OK with it, if it says no you can't even if\n> it supports \"tar\" and you could do the \"gz\" part locally. Would such a\n> patch be harder with :zlib than if we always just spewed out to external\n> \"gzip\" after satisfying some criteria?\n\nI think it would be precisely the same: you'd still use the same \"filter\"\ncode path.\n\nCiao,\nDscho\n"},{"id":"374893","messageId":"20190503205259.GB17551@sigill.intra.peff.net","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1905031437590.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-03T20:52:59Z","receivedAt":"2019-05-03T20:53:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 03, 2019 at 10:49:17PM +0200, Johannes Schindelin wrote:\n\n> I am well aware of the way Debian-based systems handle alternatives, and I\n> myself also use something similar to write this E-Mail (but it is not a\n> symlink, it is a Git alias).\n> \n> But that's not the hack that I was talking about.\n> \n> The hack I meant was: if you symlink `gzip` to `pigz` in your `PATH` *and\n> then expect `git archive --format=tgz` to pick that up*.\n> \n> As far as I am concerned, the fact that `git archive --format=tgz` spawns\n> `gzip` to perform the compression is an implementation detail, and not\n> something that users should feel they can rely on.\n\nI'd agree with you more if we didn't document a user-facing config\nvariable that claims to run \"gzip\" from the system.\n\n> > Just commenting on the overall thread: I like René's \"new built-in\"\n> > patch best.\n> \n> I guess we now have to diverging votes: yours for the `git archive --gzip`\n> \"built-in\" and Peff's for the async code ;-)\n\nFor the record, I am fine with any of the solutions (including just\ndoing the single-thread bit you already have and letting René do what he\nlikes on top).\n\n-Peff\n"},{"id":"374904","messageId":"xmqqsgttsc9k.fsf@gitster-ct.c.googlers.com","threadId":"50928","inReplyTo":"20190501180936.GB4109@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-05T05:25:59Z","receivedAt":"2019-05-05T05:26:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> FWIW, I agree with you here. These patches are not making anything worse\n> (and may even make them better, since we'd probably need to swap out the\n> BLOCKSIZE constant for a run-time \"blocksize\" variable in fewer places).\n\nIt's just that leaving the interface uneven is an easy way to\nintroduce an unnecessary bug, e.g.\n\n\t-type function(args) {\n\t+type function(args, size_t blocksize) {\n\t\tdecls;\n\t-\thelper_one(BLOCKSIZE, other, args);\n\t+\thelper_one(blocksize, other, args);\n\t\thelper_two(its, args);\n\t-\thelper_three(BLOCKSIZE, even, more, args);\n\t+\thelper_three(blocksize, even, more, args);\n\t }\n\nwhen this caller is away from the implementation of helper_two()\nthat hardcodes the assumption that this callchain only uses\nBLOCKSIZE and in an implicit way.\n\nAnd that can easily be avoided by defensively making helper_two() to\ntake BLOCKSIZE as an argument as everybody else in the caller does.\n\nI do not actually care too deeply, though. Hopefully whoever adds\n\"-b\" would be careful enough to follow all callchain, and at least\nlook at all the callees that are file-scope static, and the one I\nhave trouble with _is_ a file-scope static.\n\nOr maybe nobody does \"-b\", in which case this ticking time bomb will\nnot trigger, so we'd be OK.\n\n\n"},{"id":"374936","messageId":"20190506050726.GA30003@sigill.intra.peff.net","threadId":"50928","inReplyTo":"xmqqsgttsc9k.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/2] archive: replace write_or_die() calls with write_block_or_die()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-06T05:07:26Z","receivedAt":"2019-05-06T05:07:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, May 05, 2019 at 02:25:59PM +0900, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > FWIW, I agree with you here. These patches are not making anything worse\n> > (and may even make them better, since we'd probably need to swap out the\n> > BLOCKSIZE constant for a run-time \"blocksize\" variable in fewer places).\n> \n> It's just that leaving the interface uneven is an easy way to\n> introduce an unnecessary bug, e.g.\n> \n> \t-type function(args) {\n> \t+type function(args, size_t blocksize) {\n> \t\tdecls;\n> \t-\thelper_one(BLOCKSIZE, other, args);\n> \t+\thelper_one(blocksize, other, args);\n> \t\thelper_two(its, args);\n> \t-\thelper_three(BLOCKSIZE, even, more, args);\n> \t+\thelper_three(blocksize, even, more, args);\n> \t }\n> \n> when this caller is away from the implementation of helper_two()\n> that hardcodes the assumption that this callchain only uses\n> BLOCKSIZE and in an implicit way.\n> \n> And that can easily be avoided by defensively making helper_two() to\n> take BLOCKSIZE as an argument as everybody else in the caller does.\n> \n> I do not actually care too deeply, though. Hopefully whoever adds\n> \"-b\" would be careful enough to follow all callchain, and at least\n> look at all the callees that are file-scope static, and the one I\n> have trouble with _is_ a file-scope static.\n\nRight, my assumption was that the first step in the conversion would be\nsomebody doing s/BLOCKSIZE/global_blocksize_variable/. But that is just\na guess.\n\n> Or maybe nobody does \"-b\", in which case this ticking time bomb will\n> not trigger, so we'd be OK.\n\nYes. I suspect we're probably going down an unproductive tangent. :)\n\n-Peff\n"},{"id":"375138","messageId":"nycvar.QRO.7.76.6.1905081334260.44@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"d960966d-a7d4-dc3e-ca0a-70e9e5d1abf3@web.de","subject":"Re: [PATCH v2 2/4] archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-05-08T11:45:25Z","receivedAt":"2019-05-08T11:45:46Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi René,\n\nOn Thu, 2 May 2019, René Scharfe wrote:\n\n> Am 27.04.19 um 01:27 schrieb Johannes Schindelin via GitGitGadget:\n> > From: Johannes Schindelin \n> >\n> > They really are unsigned, and we are using e.g. BLOCKSIZE as `size_t`\n> > parameter to pass to `write_or_die()`.\n>\n> True, but the compiler converts that value correctly to size_t without\n> complaint already, doesn't it? What am I missing?\n\nAre you talking about a specific compiler? It sure sounds as if you did.\n\nI really do not want to fall into the \"you can build Git with *any*\ncompiler, as long as that compiler happens to be GCC, oh, and as long it\nis version X\" trap.\n\nWe *already* rely on GCC's optimization in way too many places for my\nliking, e.g. when we adapted the `hasheq()` code *specifically* to make\nGCC's particular optimization strategies to kick in.\n\nOr the way we defined the `SWAP()` macro: it depends on GCC's ability to\nsee through the veil and out-guess the code, deducing its intent rather\nthan what it *says* (\"Do As I Want, Not As I Say\", anyone?). We *do* want\nto swap registers when possible (instead of forcing register variables to\nbe written to memory just for the sake of being swapped, as our code says\nrather explicitly).\n\nEssentially, we build a cruise ship of a dependency on GCC here. Which\nshould not make anybody happy (except maybe the GCC folks).\n\nLet's not make things worse.\n\nCiao,\nDscho\n"},{"id":"375193","messageId":"20190508230420.GC19990@sigill.intra.peff.net","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1905081334260.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 2/4] archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-08T23:04:20Z","receivedAt":"2019-05-08T23:04:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 08, 2019 at 01:45:25PM +0200, Johannes Schindelin wrote:\n\n> Hi René,\n> \n> On Thu, 2 May 2019, René Scharfe wrote:\n> \n> > Am 27.04.19 um 01:27 schrieb Johannes Schindelin via GitGitGadget:\n> > > From: Johannes Schindelin \n> > >\n> > > They really are unsigned, and we are using e.g. BLOCKSIZE as `size_t`\n> > > parameter to pass to `write_or_die()`.\n> >\n> > True, but the compiler converts that value correctly to size_t without\n> > complaint already, doesn't it? What am I missing?\n> \n> Are you talking about a specific compiler? It sure sounds as if you did.\n> \n> I really do not want to fall into the \"you can build Git with *any*\n> compiler, as long as that compiler happens to be GCC, oh, and as long it\n> is version X\" trap.\n\nI don't this this has anything to do with gcc. The point is that we\nalready have this line:\n\n write_or_die(fd, buf, BLOCKSIZE);\n\nwhich does not cast and nobody has complained, even though the signed\nconstant is implicitly converted to a size_t. So adding another line\nlike:\n\n gzwrite(gzip, block, BLOCKSIZE);\n\nwould in theory be treated the same (gzwrite takes an \"unsigned\").\n\nThe conversion from signed to unsigned is well defined in ANSI C, and\nI'd expect a compiler to either complain about neither or both (and the\nlatter probably with warnings like -Wconversion cranked up).\n\nBut of course if you have data otherwise, we can revise that. Was the\ncast added out of caution, or to squelch a compiler warning?\n\n-Peff\n"},{"id":"375226","messageId":"nycvar.QRO.7.76.6.1905091605070.44@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"20190508230420.GC19990@sigill.intra.peff.net","subject":"Re: [PATCH v2 2/4] archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-05-09T14:06:22Z","receivedAt":"2019-05-09T14:07:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Wed, 8 May 2019, Jeff King wrote:\n\n> On Wed, May 08, 2019 at 01:45:25PM +0200, Johannes Schindelin wrote:\n>\n> > Hi René,\n> >\n> > On Thu, 2 May 2019, René Scharfe wrote:\n> >\n> > > Am 27.04.19 um 01:27 schrieb Johannes Schindelin via GitGitGadget:\n> > > > From: Johannes Schindelin \n> > > >\n> > > > They really are unsigned, and we are using e.g. BLOCKSIZE as `size_t`\n> > > > parameter to pass to `write_or_die()`.\n> > >\n> > > True, but the compiler converts that value correctly to size_t without\n> > > complaint already, doesn't it? What am I missing?\n> >\n> > Are you talking about a specific compiler? It sure sounds as if you did.\n> >\n> > I really do not want to fall into the \"you can build Git with *any*\n> > compiler, as long as that compiler happens to be GCC, oh, and as long it\n> > is version X\" trap.\n>\n> I don't this this has anything to do with gcc. The point is that we\n> already have this line:\n>\n> write_or_die(fd, buf, BLOCKSIZE);\n>\n> which does not cast and nobody has complained,\n\nI mistook this part of your reply in\nhttps://public-inbox.org/git/20190413013451.GB2040@sigill.intra.peff.net/\nas precisely such a complaint:\n\n\tBLOCKSIZE is a constant. Should we be defining it with a \"U\" in\n\tthe first place?\n\nThanks,\nDscho\n\n> even though the signed\n> constant is implicitly converted to a size_t. So adding another line\n> like:\n>\n> gzwrite(gzip, block, BLOCKSIZE);\n>\n> would in theory be treated the same (gzwrite takes an \"unsigned\").\n>\n> The conversion from signed to unsigned is well defined in ANSI C, and\n> I'd expect a compiler to either complain about neither or both (and the\n> latter probably with warnings like -Wconversion cranked up).\n>\n> But of course if you have data otherwise, we can revise that. Was the\n> cast added out of caution, or to squelch a compiler warning?\n>\n> -Peff\n>\n"},{"id":"375249","messageId":"20190509183855.GA28107@sigill.intra.peff.net","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.1905091605070.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 2/4] archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-09T18:38:55Z","receivedAt":"2019-05-09T18:38:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 09, 2019 at 04:06:22PM +0200, Johannes Schindelin wrote:\n\n> > I don't this this has anything to do with gcc. The point is that we\n> > already have this line:\n> >\n> > write_or_die(fd, buf, BLOCKSIZE);\n> >\n> > which does not cast and nobody has complained,\n> \n> I mistook this part of your reply in\n> https://public-inbox.org/git/20190413013451.GB2040@sigill.intra.peff.net/\n> as precisely such a complaint:\n> \n> \tBLOCKSIZE is a constant. Should we be defining it with a \"U\" in\n> \tthe first place?\n\nAh, sorry to introduce confusion. I mostly meant \"if we need to cast,\nwhy not just define as unsigned in the first place?\". But I think René\nwas pointing out that we do not even need to cast, and I am fine with\nthat approach.\n\nI do dream of a world where we do not have a bunch of implicit\nconversions (both signedness but also truncation) in our code base, and\ncan compile cleanly with -Wconversion We know that this case is\nperfectly fine, but I am sure there are many that are not. However, I'm\nnot sure if we'll ever get there, and in the meantime I don't think it's\nworth worrying too much about individual cases like this.\n\n-Peff\n"},{"id":"375306","messageId":"f380b182-633f-f347-ed2e-f90548d7f3c9@web.de","threadId":"50928","inReplyTo":"20190509183855.GA28107@sigill.intra.peff.net","subject":"Re: [PATCH v2 2/4] archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-05-10T17:18:44Z","receivedAt":"2019-05-10T17:19:24Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 09.05.19 um 20:38 schrieb Jeff King:\n> I do dream of a world where we do not have a bunch of implicit\n> conversions (both signedness but also truncation) in our code base, and\n> can compile cleanly with -Wconversion We know that this case is\n> perfectly fine, but I am sure there are many that are not. However, I'm\n> not sure if we'll ever get there, and in the meantime I don't think it's\n> worth worrying too much about individual cases like this.\n\nHere's a rough take on how to silence that warning for archive-tar.c using\nGCC 8.3. Some of the changes are worth polishing and submitting. Some\nare silly. The one for regexec_buf() is scary; I don't see a clean way of\ndealing with that size_t to int conversion.\n\n---\n archive-tar.c | 54 +++++++++++++++++++++++++++++++----------------\n cache.h | 10 ++++-----\n git-compat-util.h | 21 +++++++++++++++---\n hash.h | 2 +-\n strbuf.h | 2 +-\n 5 files changed, 61 insertions(+), 28 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 3e53aac1e6..bfd91782ab 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -15,7 +15,7 @@\n static char block[BLOCKSIZE];\n static unsigned long offset;\n\n-static int tar_umask = 002;\n+static mode_t tar_umask = 002;\n\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args);\n@@ -99,7 +99,7 @@ static void write_blocked(const void *data, unsigned long size)\n */\n static void write_trailer(void)\n {\n-\tint tail = BLOCKSIZE - offset;\n+\tsize_t tail = BLOCKSIZE - offset;\n \tmemset(block + offset, 0, tail);\n \twrite_or_die(1, block, BLOCKSIZE);\n \tif (tail < 2 * RECORDSIZE) {\n@@ -127,12 +127,13 @@ static int stream_blocked(const struct object_id *oid)\n \t\treadlen = read_istream(st, buf, sizeof(buf));\n \t\tif (readlen <= 0)\n \t\t\tbreak;\n-\t\tdo_write_blocked(buf, readlen);\n+\t\tdo_write_blocked(buf, (size_t)readlen);\n \t}\n \tclose_istream(st);\n-\tif (!readlen)\n-\t\tfinish_record();\n-\treturn readlen;\n+\tif (readlen < 0)\n+\t\treturn -1;\n+\tfinish_record();\n+\treturn 0;\n }\n\n /*\n@@ -142,9 +143,9 @@ static int stream_blocked(const struct object_id *oid)\n * string and appends it to a struct strbuf.\n */\n static void strbuf_append_ext_header(struct strbuf *sb, const char *keyword,\n-\t\t\t\t const char *value, unsigned int valuelen)\n+\t\t\t\t const char *value, size_t valuelen)\n {\n-\tint len, tmp;\n+\tsize_t len, tmp;\n\n \t/* \"%u %s=%s\\n\" */\n \tlen = 1 + 1 + strlen(keyword) + 1 + valuelen + 1;\n@@ -152,7 +153,7 @@ static void strbuf_append_ext_header(struct strbuf *sb, const char *keyword,\n \t\tlen++;\n\n \tstrbuf_grow(sb, len);\n-\tstrbuf_addf(sb, \"%u %s=\", len, keyword);\n+\tstrbuf_addf(sb, \"%\"PRIuMAX\" %s=\", (uintmax_t)len, keyword);\n \tstrbuf_add(sb, value, valuelen);\n \tstrbuf_addch(sb, '\\n');\n }\n@@ -168,7 +169,9 @@ static void strbuf_append_ext_header_uint(struct strbuf *sb,\n \tint len;\n\n \tlen = xsnprintf(buf, sizeof(buf), \"%\"PRIuMAX, value);\n-\tstrbuf_append_ext_header(sb, keyword, buf, len);\n+\tif (len < 0)\n+\t\tBUG(\"unable to convert %\"PRIuMAX\" to decimal\", value);\n+\tstrbuf_append_ext_header(sb, keyword, buf, (size_t)len);\n }\n\n static unsigned int ustar_header_chksum(const struct ustar_header *header)\n@@ -177,7 +180,7 @@ static unsigned int ustar_header_chksum(const struct ustar_header *header)\n \tunsigned int chksum = 0;\n \twhile (p < (const unsigned char *)header->chksum)\n \t\tchksum += *p++;\n-\tchksum += sizeof(header->chksum) * ' ';\n+\tchksum += (unsigned int)sizeof(header->chksum) * ' ';\n \tp += sizeof(header->chksum);\n \twhile (p < (const unsigned char *)header + sizeof(struct ustar_header))\n \t\tchksum += *p++;\n@@ -355,12 +358,14 @@ static void write_global_extended_header(struct archiver_args *args)\n }\n\n static struct archiver **tar_filters;\n-static int nr_tar_filters;\n-static int alloc_tar_filters;\n+static size_t nr_tar_filters;\n+static size_t alloc_tar_filters;\n\n-static struct archiver *find_tar_filter(const char *name, int len)\n+static struct archiver *find_tar_filter(const char *name, size_t len)\n {\n \tint i;\n+\tif (len < 1)\n+\t\treturn NULL;\n \tfor (i = 0; i < nr_tar_filters; i++) {\n \t\tstruct archiver *ar = tar_filters[i];\n \t\tif (!strncmp(ar->name, name, len) && !ar->name[len])\n@@ -369,14 +374,27 @@ static struct archiver *find_tar_filter(const char *name, int len)\n \treturn NULL;\n }\n\n+static int parse_config_key2(const char *var, const char *section,\n+\t\t\t const char **subsection, size_t *subsection_len,\n+\t\t\t const char **key)\n+{\n+\tint rc, len;\n+\n+\trc = parse_config_key(var, section, subsection, &len, key);\n+\tif (!rc && len < 0)\n+\t\treturn -1;\n+\t*subsection_len = (size_t)len;\n+\treturn rc;\n+}\n+\n static int tar_filter_config(const char *var, const char *value, void *data)\n {\n \tstruct archiver *ar;\n \tconst char *name;\n \tconst char *type;\n-\tint namelen;\n+\tsize_t namelen;\n\n-\tif (parse_config_key(var, \"tar\", &name, &namelen, &type) < 0 || !name)\n+\tif (parse_config_key2(var, \"tar\", &name, &namelen, &type) < 0 || !name)\n \t\treturn 0;\n\n \tar = find_tar_filter(name, namelen);\n@@ -400,7 +418,7 @@ static int tar_filter_config(const char *var, const char *value, void *data)\n \t\tif (git_config_bool(var, value))\n \t\t\tar->flags |= ARCHIVER_REMOTE;\n \t\telse\n-\t\t\tar->flags &= ~ARCHIVER_REMOTE;\n+\t\t\tar->flags &= ~(unsigned int)ARCHIVER_REMOTE;\n \t\treturn 0;\n \t}\n\n@@ -414,7 +432,7 @@ static int git_tar_config(const char *var, const char *value, void *cb)\n \t\t\ttar_umask = umask(0);\n \t\t\tumask(tar_umask);\n \t\t} else {\n-\t\t\ttar_umask = git_config_int(var, value);\n+\t\t\ttar_umask = (mode_t)git_config_ulong(var, value);\n \t\t}\n \t\treturn 0;\n \t}\ndiff --git a/cache.h b/cache.h\nindex 67cc2e1806..a791034260 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -241,7 +241,7 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \t\t\t\t const struct cache_entry *src)\n {\n \tunsigned int state = dst->ce_flags & CE_HASHED;\n-\tint mem_pool_allocated = dst->mem_pool_allocated;\n+\tunsigned int mem_pool_allocated = dst->mem_pool_allocated;\n\n \t/* Don't copy hash chain and name */\n \tmemcpy(&dst->ce_stat_data, &src->ce_stat_data,\n@@ -249,7 +249,7 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \t\t\toffsetof(struct cache_entry, ce_stat_data));\n\n \t/* Restore the hash state */\n-\tdst->ce_flags = (dst->ce_flags & ~CE_HASHED) | state;\n+\tdst->ce_flags = (dst->ce_flags & ~(unsigned int)CE_HASHED) | state;\n\n \t/* Restore the mem_pool_allocated flag */\n \tdst->mem_pool_allocated = mem_pool_allocated;\n@@ -1314,7 +1314,7 @@ extern int check_and_freshen_file(const char *fn, int freshen);\n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\n {\n-\treturn hexval_table[c];\n+\treturn (unsigned int)hexval_table[c];\n }\n\n /*\n@@ -1323,8 +1323,8 @@ static inline unsigned int hexval(unsigned char c)\n */\n static inline int hex2chr(const char *s)\n {\n-\tunsigned int val = hexval(s[0]);\n-\treturn (val & ~0xf) ? val : (val << 4) | hexval(s[1]);\n+\tint val = hexval_table[(unsigned char)s[0]];\n+\treturn (val < 0) ? val : (val << 4) | hexval_table[(unsigned char)s[1]];\n }\n\n /* Convert to/from hex/sha1 representation */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 4386b3e1c8..cf33e84c96 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1068,7 +1068,7 @@ static inline int strtoul_ui(char const *s, int base, unsigned int *result)\n \tul = strtoul(s, &p, base);\n \tif (errno || *p || p == s || (unsigned int) ul != ul)\n \t\treturn -1;\n-\t*result = ul;\n+\t*result = (unsigned int)ul;\n \treturn 0;\n }\n\n@@ -1081,7 +1081,7 @@ static inline int strtol_i(char const *s, int base, int *result)\n \tul = strtol(s, &p, base);\n \tif (errno || *p || p == s || (int) ul != ul)\n \t\treturn -1;\n-\t*result = ul;\n+\t*result = (int)ul;\n \treturn 0;\n }\n\n@@ -1119,7 +1119,22 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n {\n \tassert(nmatch > 0 && pmatch);\n \tpmatch[0].rm_so = 0;\n-\tpmatch[0].rm_eo = size;\n+\tpmatch[0].rm_eo = (regoff_t)size;\n+\tif (pmatch[0].rm_eo != size) {\n+\t\tif (((regoff_t)-1) < 0) {\n+\t\t\tif (sizeof(regoff_t) == sizeof(int))\n+\t\t\t\tpmatch[0].rm_eo = (regoff_t)INT_MAX;\n+\t\t\telse if (sizeof(regoff_t) == sizeof(long))\n+\t\t\t\tpmatch[0].rm_eo = (regoff_t)LONG_MAX;\n+\t\t\telse\n+\t\t\t\tdie(\"unable to determine maximum value of regoff_t\");\n+\t\t} else {\n+\t\t\tpmatch[0].rm_eo = (regoff_t)-1;\n+\t\t}\n+\t\twarning(\"buffer too big (%\"PRIuMAX\"), \"\n+\t\t\t\"will search only the first %\"PRIuMAX\" bytes\",\n+\t\t\t(uintmax_t)size, (uintmax_t)pmatch[0].rm_eo);\n+\t}\n \treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n }\n\ndiff --git a/hash.h b/hash.h\nindex 661c9f2281..7056f89eb4 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -134,7 +134,7 @@ int hash_algo_by_id(uint32_t format_id);\n /* Identical, except based on the length. */\n int hash_algo_by_length(int len);\n /* Identical, except for a pointer to struct git_hash_algo. */\n-static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n+static inline ptrdiff_t hash_algo_by_ptr(const struct git_hash_algo *p)\n {\n \treturn p - hash_algos;\n }\ndiff --git a/strbuf.h b/strbuf.h\nindex c8d98dfb95..30659f2d5d 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -225,7 +225,7 @@ int strbuf_cmp(const struct strbuf *first, const struct strbuf *second);\n /**\n * Add a single character to the buffer.\n */\n-static inline void strbuf_addch(struct strbuf *sb, int c)\n+static inline void strbuf_addch(struct strbuf *sb, char c)\n {\n \tif (!strbuf_avail(sb))\n \t\tstrbuf_grow(sb, 1);\n--\n2.21.0\n"},{"id":"375325","messageId":"20190510212039.GA20767@sigill.intra.peff.net","threadId":"50928","inReplyTo":"f380b182-633f-f347-ed2e-f90548d7f3c9@web.de","subject":"Re: [PATCH v2 2/4] archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-10T21:20:39Z","receivedAt":"2019-05-10T21:20:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 10, 2019 at 07:18:44PM +0200, René Scharfe wrote:\n\n> Am 09.05.19 um 20:38 schrieb Jeff King:\n> > I do dream of a world where we do not have a bunch of implicit\n> > conversions (both signedness but also truncation) in our code base, and\n> > can compile cleanly with -Wconversion We know that this case is\n> > perfectly fine, but I am sure there are many that are not. However, I'm\n> > not sure if we'll ever get there, and in the meantime I don't think it's\n> > worth worrying too much about individual cases like this.\n> \n> Here's a rough take on how to silence that warning for archive-tar.c using\n> GCC 8.3. Some of the changes are worth polishing and submitting. Some\n> are silly. The one for regexec_buf() is scary; I don't see a clean way of\n> dealing with that size_t to int conversion.\n\nThis is actually slightly less tedious than I had imagined it to be, but\nstill pretty bad. I dunno. If somebody wants to tackle it, I do think it\nwould make the world a better place. But I'm not sure if it is worth the\neffort involved.\n\n> static void write_trailer(void)\n> {\n> -\tint tail = BLOCKSIZE - offset;\n> +\tsize_t tail = BLOCKSIZE - offset;\n\nThese kinds of int/size_t conversions are the ones I think are the most\nvaluable (because the size_t's are often used to allocate or access\narrays, and truncated or negative values there can cause other security\nproblems). _Most_ of them are harmless, of course, but it's hard to\nseparate the important ones from the mundane.\n\n> @@ -414,7 +432,7 @@ static int git_tar_config(const char *var, const char *value, void *cb)\n> \t\t\ttar_umask = umask(0);\n> \t\t\tumask(tar_umask);\n> \t\t} else {\n> -\t\t\ttar_umask = git_config_int(var, value);\n> +\t\t\ttar_umask = (mode_t)git_config_ulong(var, value);\n> \t\t}\n\nIt's nice that the cast here shuts up the compiler, and I agree it is\nnot likely to be a problem in this instance. But we'd probably want some\nkind of \"safe cast\" helper. To some degree, if you put 2^64-1 in your\n\"umask\" value you get what you deserve, but it would be nice if we could\ndetect such nonsense (less for this case, but more for others where we\ndo cast).\n\n> @@ -1119,7 +1119,22 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n> {\n> \tassert(nmatch > 0 && pmatch);\n> \tpmatch[0].rm_so = 0;\n> -\tpmatch[0].rm_eo = size;\n> +\tpmatch[0].rm_eo = (regoff_t)size;\n> +\tif (pmatch[0].rm_eo != size) {\n> +\t\tif (((regoff_t)-1) < 0) {\n> +\t\t\tif (sizeof(regoff_t) == sizeof(int))\n> +\t\t\t\tpmatch[0].rm_eo = (regoff_t)INT_MAX;\n> +\t\t\telse if (sizeof(regoff_t) == sizeof(long))\n> +\t\t\t\tpmatch[0].rm_eo = (regoff_t)LONG_MAX;\n> +\t\t\telse\n> +\t\t\t\tdie(\"unable to determine maximum value of regoff_t\");\n> +\t\t} else {\n> +\t\t\tpmatch[0].rm_eo = (regoff_t)-1;\n> +\t\t}\n> +\t\twarning(\"buffer too big (%\"PRIuMAX\"), \"\n> +\t\t\t\"will search only the first %\"PRIuMAX\" bytes\",\n> +\t\t\t(uintmax_t)size, (uintmax_t)pmatch[0].rm_eo);\n> +\t}\n> \treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n> }\n\nI think a helper could make things less awful here, too. Our xsize_t()\nis sort of like this, but of course it dies. But I think it would be\npossible to write a macro to let you do:\n\n if (ASSIGN_CAST(pmatch[0].rm_eo, size))\n\twarning(...);\n\nThis is definitely a rabbit-hole that I've been afraid to go down. :)\n\n-Peff\n"},{"id":"376921","messageId":"c00a062a-4f01-4754-3429-e7bb2a26aac1@web.de","threadId":"50928","inReplyTo":"20190501181807.GC4109@sigill.intra.peff.net","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-10T10:44:54Z","receivedAt":"2019-06-10T10:45:31Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 01.05.19 um 20:18 schrieb Jeff King:\n> On Wed, May 01, 2019 at 07:45:05PM +0200, René Scharfe wrote:\n>\n>>> But since the performance is still not quite on par with `gzip`, I would\n>>> actually rather not, and really, just punt on that one, stating that\n>>> people interested in higher performance should use `pigz`.\n>>\n>> Here are my performance numbers for generating .tar.gz files again:\n\nOK, tried one more version, with pthreads (patch at the end). Also\nredid all measurements for better comparability; everything is faster\nnow for some reason (perhaps due to a compiler update? clang version\n7.0.1-8 now):\n\nmaster, using gzip(1):\nBenchmark #1: git archive --format=tgz HEAD\n Time (mean ± σ): 15.697 s ± 0.246 s [User: 19.213 s, System: 0.386 s]\n Range (min … max): 15.405 s … 16.103 s 10 runs\n\nusing zlib sequentially:\nBenchmark #1: git archive --format=tgz HEAD\n Time (mean ± σ): 19.191 s ± 0.408 s [User: 19.091 s, System: 0.100 s]\n Range (min … max): 18.802 s … 19.877 s 10 runs\n\nusing a gzip-lookalike:\nBenchmark #1: git archive --format=tgz HEAD\n Time (mean ± σ): 16.289 s ± 0.218 s [User: 19.485 s, System: 0.337 s]\n Range (min … max): 16.020 s … 16.555 s 10 runs\n\nusing zlib with start_async:\nBenchmark #1: git archive --format=tgz HEAD\n Time (mean ± σ): 16.516 s ± 0.334 s [User: 20.282 s, System: 0.383 s]\n Range (min … max): 16.166 s … 17.283 s 10 runs\n\nusing zlib in a separate thread (that's the new one):\nBenchmark #1: git archive --format=tgz HEAD\n Time (mean ± σ): 16.310 s ± 0.237 s [User: 20.075 s, System: 0.173 s]\n Range (min … max): 15.983 s … 16.790 s 10 runs\n\n> I think the start_async() one seems like a good option. It reclaims most\n> of the (wall-clock) performance, isn't very much code, and doesn't leave\n> any ugly user-visible traces.\n\nThe pthreads numbers look a bit better still. The patch is huge though,\nbecause it duplicates almost everything. It was easier that way; a real\npatch series would extract functions that can be used both with static\nand allocated headers first, and keep everything in archive-tar.c.\n\n> I'd be fine to see it come later, though, on top of the patches Dscho is\n> sending. Even though changing to sequential zlib is technically a change\n> in behavior, the existing behavior wasn't really planned. And given the\n> wall-clock versus CPU time tradeoff, it's not entirely clear that one\n> solution is better than the other.\n\nThe current behavior is not an accident; the synchronous method was\nrejected in 2009 because it was slower [1]. Redid the measurements\nwith v1.6.5-rc0 and the old patch [2], but they would only compile with\ngcc (Debian 8.3.0-6) for me, so it's not directly comparable to the\nnumbers above:\n\nv1.6.5-rc0:\nBenchmark #1: ../git/git-archive HEAD | gzip\n Time (mean ± σ): 16.051 s ± 0.486 s [User: 19.514 s, System: 0.341 s]\n Range (min … max): 15.416 s … 17.001 s 10 runs\n\nv1.6.5-rc0 + [2]:\nBenchmark #1: ../git/git-archive --format=tar.gz HEAD\n Time (mean ± σ): 19.684 s ± 0.374 s [User: 19.601 s, System: 0.060 s]\n Range (min … max): 19.082 s … 20.177 s 10 runs\n\nUser time is still slightly higher, but the difference is in the noise.\n\n[1] http://public-inbox.org/git/4AAAC8CE.8020302@lsrfire.ath.cx/\n[2] http://public-inbox.org/git/4AA97B61.6030301@lsrfire.ath.cx/\n\n>>> And who knows, maybe nobody will complain at all about the performance?\n>>\n>> Probably. And popular tarballs would be cached anyway, I guess.\n>\n> At GitHub we certainly do cache the git-archive output. We'd also be\n> just fine with the sequential solution. We generally turn down\n> pack.threads to 1, and keep our CPUs busy by serving multiple users\n> anyway.\n>\n> So whatever has the lowest overall CPU time is generally preferable, but\n> the times are close enough that I don't think we'd care much either way\n> (and it's probably not worth having a config option or similar).\n\nMoving back to 2009 and reducing the number of utilized cores both feels\nweird, but the sequential solution *is* the most obvious, easiest and\n(by a narrow margin) lightest one if gzip(1) is not an option anymore.\n\nAnyway, the threading patch:\n\n---\n Makefile | 1 +\n archive-tar.c | 11 +-\n archive-tgz.c | 452 ++++++++++++++++++++++++++++++++++++++++++++++++++\n archive.h | 4 +\n 4 files changed, 465 insertions(+), 3 deletions(-)\n create mode 100644 archive-tgz.c\n\ndiff --git a/Makefile b/Makefile\nindex 8a7e235352..ed649ac18d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -834,6 +834,7 @@ LIB_OBJS += alloc.o\n LIB_OBJS += apply.o\n LIB_OBJS += archive.o\n LIB_OBJS += archive-tar.o\n+LIB_OBJS += archive-tgz.o\n LIB_OBJS += archive-zip.o\n LIB_OBJS += argv-array.o\n LIB_OBJS += attr.o\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 3e53aac1e6..929eb58235 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -15,7 +15,9 @@\n static char block[BLOCKSIZE];\n static unsigned long offset;\n\n-static int tar_umask = 002;\n+int tar_umask = 002;\n+\n+static const char internal_gzip[] = \"git archive gzip\";\n\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args);\n@@ -445,6 +447,9 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!ar->data)\n \t\tBUG(\"tar-filter archiver called with no filter defined\");\n\n+\tif (!strcmp(ar->data, internal_gzip))\n+\t\treturn write_tgz_archive(ar, args);\n+\n \tstrbuf_addstr(&cmd, ar->data);\n \tif (args->compression_level >= 0)\n \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\n@@ -483,9 +488,9 @@ void init_tar_archiver(void)\n \tint i;\n \tregister_archiver(&tar_archiver);\n\n-\ttar_filter_config(\"tar.tgz.command\", \"gzip -cn\", NULL);\n+\ttar_filter_config(\"tar.tgz.command\", internal_gzip, NULL);\n \ttar_filter_config(\"tar.tgz.remote\", \"true\", NULL);\n-\ttar_filter_config(\"tar.tar.gz.command\", \"gzip -cn\", NULL);\n+\ttar_filter_config(\"tar.tar.gz.command\", internal_gzip, NULL);\n \ttar_filter_config(\"tar.tar.gz.remote\", \"true\", NULL);\n \tgit_config(git_tar_config, NULL);\n \tfor (i = 0; i < nr_tar_filters; i++) {\ndiff --git a/archive-tgz.c b/archive-tgz.c\nnew file mode 100644\nindex 0000000000..ae219e1cc0\n--- /dev/null\n+++ b/archive-tgz.c\n@@ -0,0 +1,452 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"tar.h\"\n+#include \"archive.h\"\n+#include \"object-store.h\"\n+#include \"streaming.h\"\n+\n+#define RECORDSIZE\t(512)\n+#define BLOCKSIZE\t(RECORDSIZE * 20)\n+\n+static gzFile gzip;\n+static size_t offset;\n+\n+/*\n+ * This is the max value that a ustar size header can specify, as it is fixed\n+ * at 11 octal digits. POSIX specifies that we switch to extended headers at\n+ * this size.\n+ *\n+ * Likewise for the mtime (which happens to use a buffer of the same size).\n+ */\n+#if ULONG_MAX == 0xFFFFFFFF\n+#define USTAR_MAX_SIZE ULONG_MAX\n+#else\n+#define USTAR_MAX_SIZE 077777777777UL\n+#endif\n+#if TIME_MAX == 0xFFFFFFFF\n+#define USTAR_MAX_MTIME TIME_MAX\n+#else\n+#define USTAR_MAX_MTIME 077777777777ULL\n+#endif\n+\n+static void tgz_write(const void *data, size_t size)\n+{\n+\tconst char *p = data;\n+\twhile (size) {\n+\t\tsize_t to_write = size;\n+\t\tif (to_write > UINT_MAX)\n+\t\t\tto_write = UINT_MAX;\n+\t\tif (gzwrite(gzip, p, to_write) != to_write)\n+\t\t\tdie(_(\"gzwrite failed\"));\n+\t\tp += to_write;\n+\t\tsize -= to_write;\n+\t\toffset = (offset + to_write) % BLOCKSIZE;\n+\t}\n+}\n+\n+static void tgz_finish_record(void)\n+{\n+\tsize_t tail = offset % RECORDSIZE;\n+\tif (tail) {\n+\t\tsize_t to_seek = RECORDSIZE - tail;\n+\t\tif (gzseek(gzip, to_seek, SEEK_CUR) < 0)\n+\t\t\tdie(_(\"gzseek failed\"));\n+\t\toffset = (offset + to_seek) % BLOCKSIZE;\n+\t}\n+}\n+\n+static void tgz_write_trailer(void)\n+{\n+\tsize_t to_seek = BLOCKSIZE - offset;\n+\tif (to_seek < 2 * RECORDSIZE)\n+\t\tto_seek += BLOCKSIZE;\n+\tif (gzseek(gzip, to_seek, SEEK_CUR) < 0)\n+\t\t die(_(\"gzseek failed\"));\n+\tif (gzflush(gzip, Z_FINISH) != Z_OK)\n+\t\tdie(_(\"gzflush failed\"));\n+}\n+\n+struct work_item {\n+\tvoid *buffer;\n+\tsize_t size;\n+\tint finish_record;\n+};\n+\n+#define TODO_SIZE 64\n+struct work_item todo[TODO_SIZE];\n+static int todo_start;\n+static int todo_end;\n+static int todo_done;\n+static int all_work_added;\n+static pthread_mutex_t tar_mutex;\n+static pthread_t thread;\n+\n+static void tar_lock(void)\n+{\n+\tpthread_mutex_lock(&tar_mutex);\n+}\n+\n+static void tar_unlock(void)\n+{\n+\tpthread_mutex_unlock(&tar_mutex);\n+}\n+\n+static pthread_cond_t cond_add;\n+static pthread_cond_t cond_write;\n+static pthread_cond_t cond_result;\n+\n+static void add_work(void *buffer, size_t size, int finish_record)\n+{\n+\ttar_lock();\n+\n+\twhile ((todo_end + 1) % ARRAY_SIZE(todo) == todo_done)\n+\t\tpthread_cond_wait(&cond_write, &tar_mutex);\n+\n+\ttodo[todo_end].buffer = buffer;\n+\ttodo[todo_end].size = size;\n+\ttodo[todo_end].finish_record = finish_record;\n+\n+\ttodo_end = (todo_end + 1) % ARRAY_SIZE(todo);\n+\n+\tpthread_cond_signal(&cond_add);\n+\ttar_unlock();\n+}\n+\n+static struct work_item *get_work(void)\n+{\n+\tstruct work_item *ret = NULL;\n+\n+\ttar_lock();\n+\twhile (todo_start == todo_end && !all_work_added)\n+\t\tpthread_cond_wait(&cond_add, &tar_mutex);\n+\n+\tif (todo_start != todo_end || !all_work_added) {\n+\t\tret = &todo[todo_start];\n+\t\ttodo_start = (todo_start + 1) % ARRAY_SIZE(todo);\n+\t}\n+\ttar_unlock();\n+\treturn ret;\n+}\n+\n+static void work_done(void)\n+{\n+\ttar_lock();\n+\ttodo_done = (todo_done + 1) % ARRAY_SIZE(todo);\n+\tpthread_cond_signal(&cond_write);\n+\n+\tif (all_work_added && todo_done == todo_end)\n+\t\tpthread_cond_signal(&cond_result);\n+\ttar_unlock();\n+}\n+\n+static void *run(void *arg)\n+{\n+\tfor (;;) {\n+\t\tstruct work_item *w = get_work();\n+\t\tif (!w)\n+\t\t\tbreak;\n+\t\ttgz_write(w->buffer, w->size);\n+\t\tfree(w->buffer);\n+\t\tif (w->finish_record)\n+\t\t\ttgz_finish_record();\n+\t\twork_done();\n+\t}\n+\treturn NULL;\n+}\n+\n+static void start_output_thread(void)\n+{\n+\tint err;\n+\n+\tpthread_mutex_init(&tar_mutex, NULL);\n+\tpthread_cond_init(&cond_add, NULL);\n+\tpthread_cond_init(&cond_write, NULL);\n+\tpthread_cond_init(&cond_result, NULL);\n+\n+\tmemset(todo, 0, sizeof(todo));\n+\n+\terr = pthread_create(&thread, NULL, run, NULL);\n+\tif (err)\n+\t\tdie(_(\"failed to create thread: %s\"), strerror(err));\n+}\n+\n+static void wait_for_output_thread(void)\n+{\n+\ttar_lock();\n+\tall_work_added = 1;\n+\n+\twhile (todo_done != todo_end)\n+\t\tpthread_cond_wait(&cond_result, &tar_mutex);\n+\n+\tpthread_cond_broadcast(&cond_add);\n+\ttar_unlock();\n+\n+\tpthread_join(thread, NULL);\n+\n+\tpthread_mutex_destroy(&tar_mutex);\n+\tpthread_cond_destroy(&cond_add);\n+\tpthread_cond_destroy(&cond_write);\n+\tpthread_cond_destroy(&cond_result);\n+}\n+\n+static int stream_blob(const struct object_id *oid)\n+{\n+\tstruct git_istream *st;\n+\tenum object_type type;\n+\tunsigned long sz;\n+\tssize_t readlen;\n+\tsize_t chunk_size = BLOCKSIZE * 10;\n+\n+\tst = open_istream(oid, &type, &sz, NULL);\n+\tif (!st)\n+\t\treturn error(_(\"cannot stream blob %s\"), oid_to_hex(oid));\n+\tfor (;;) {\n+\t\tchar *buf = xmalloc(chunk_size);\n+\t\treadlen = read_istream(st, buf, chunk_size);\n+\t\tif (readlen <= 0)\n+\t\t\tbreak;\n+\t\tsz -= readlen;\n+\t\tadd_work(buf, readlen, !sz);\n+\t}\n+\tclose_istream(st);\n+\treturn readlen;\n+}\n+\n+/*\n+ * pax extended header records have the format \"%u %s=%s\\n\". %u contains\n+ * the size of the whole string (including the %u), the first %s is the\n+ * keyword, the second one is the value. This function constructs such a\n+ * string and appends it to a struct strbuf.\n+ */\n+static void strbuf_append_ext_header(struct strbuf *sb, const char *keyword,\n+\t\t\t\t const char *value, unsigned int valuelen)\n+{\n+\tint len, tmp;\n+\n+\t/* \"%u %s=%s\\n\" */\n+\tlen = 1 + 1 + strlen(keyword) + 1 + valuelen + 1;\n+\tfor (tmp = len; tmp > 9; tmp /= 10)\n+\t\tlen++;\n+\n+\tstrbuf_grow(sb, len);\n+\tstrbuf_addf(sb, \"%u %s=\", len, keyword);\n+\tstrbuf_add(sb, value, valuelen);\n+\tstrbuf_addch(sb, '\\n');\n+}\n+\n+/*\n+ * Like strbuf_append_ext_header, but for numeric values.\n+ */\n+static void strbuf_append_ext_header_uint(struct strbuf *sb,\n+\t\t\t\t\t const char *keyword,\n+\t\t\t\t\t uintmax_t value)\n+{\n+\tchar buf[40]; /* big enough for 2^128 in decimal, plus NUL */\n+\tint len;\n+\n+\tlen = xsnprintf(buf, sizeof(buf), \"%\"PRIuMAX, value);\n+\tstrbuf_append_ext_header(sb, keyword, buf, len);\n+}\n+\n+static unsigned int ustar_header_chksum(const struct ustar_header *header)\n+{\n+\tconst unsigned char *p = (const unsigned char *)header;\n+\tunsigned int chksum = 0;\n+\twhile (p < (const unsigned char *)header->chksum)\n+\t\tchksum += *p++;\n+\tchksum += sizeof(header->chksum) * ' ';\n+\tp += sizeof(header->chksum);\n+\twhile (p < (const unsigned char *)header + sizeof(struct ustar_header))\n+\t\tchksum += *p++;\n+\treturn chksum;\n+}\n+\n+static size_t get_path_prefix(const char *path, size_t pathlen, size_t maxlen)\n+{\n+\tsize_t i = pathlen;\n+\tif (i > 1 && path[i - 1] == '/')\n+\t\ti--;\n+\tif (i > maxlen)\n+\t\ti = maxlen;\n+\tdo {\n+\t\ti--;\n+\t} while (i > 0 && path[i] != '/');\n+\treturn i;\n+}\n+\n+static void prepare_header(struct archiver_args *args,\n+\t\t\t struct ustar_header *header,\n+\t\t\t unsigned int mode, unsigned long size)\n+{\n+\txsnprintf(header->mode, sizeof(header->mode), \"%07o\", mode & 07777);\n+\txsnprintf(header->size, sizeof(header->size), \"%011\"PRIoMAX , S_ISREG(mode) ? (uintmax_t)size : (uintmax_t)0);\n+\txsnprintf(header->mtime, sizeof(header->mtime), \"%011lo\", (unsigned long) args->time);\n+\n+\txsnprintf(header->uid, sizeof(header->uid), \"%07o\", 0);\n+\txsnprintf(header->gid, sizeof(header->gid), \"%07o\", 0);\n+\tstrlcpy(header->uname, \"root\", sizeof(header->uname));\n+\tstrlcpy(header->gname, \"root\", sizeof(header->gname));\n+\txsnprintf(header->devmajor, sizeof(header->devmajor), \"%07o\", 0);\n+\txsnprintf(header->devminor, sizeof(header->devminor), \"%07o\", 0);\n+\n+\tmemcpy(header->magic, \"ustar\", 6);\n+\tmemcpy(header->version, \"00\", 2);\n+\n+\txsnprintf(header->chksum, sizeof(header->chksum), \"%07o\", ustar_header_chksum(header));\n+}\n+\n+static void write_extended_header(struct archiver_args *args,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t struct strbuf *extended_header)\n+{\n+\tsize_t size;\n+\tchar *buffer = strbuf_detach(extended_header, &size);\n+\tstruct ustar_header *header = xcalloc(1, sizeof(*header));\n+\tunsigned int mode;\n+\t*header->typeflag = TYPEFLAG_EXT_HEADER;\n+\tmode = 0100666;\n+\txsnprintf(header->name, sizeof(header->name), \"%s.paxheader\",\n+\t\t oid_to_hex(oid));\n+\tprepare_header(args, header, mode, size);\n+\tadd_work(header, sizeof(*header), 1);\n+\tadd_work(buffer, size, 1);\n+}\n+\n+static int write_tar_entry(struct archiver_args *args,\n+\t\t\t const struct object_id *oid,\n+\t\t\t const char *path, size_t pathlen,\n+\t\t\t unsigned int mode)\n+{\n+\tstruct ustar_header *header = xcalloc(1, sizeof(*header));\n+\tstruct strbuf ext_header = STRBUF_INIT;\n+\tunsigned int old_mode = mode;\n+\tunsigned long size, size_in_header;\n+\tvoid *buffer;\n+\tint err = 0;\n+\n+\tif (S_ISDIR(mode) || S_ISGITLINK(mode)) {\n+\t\t*header->typeflag = TYPEFLAG_DIR;\n+\t\tmode = (mode | 0777) & ~tar_umask;\n+\t} else if (S_ISLNK(mode)) {\n+\t\t*header->typeflag = TYPEFLAG_LNK;\n+\t\tmode |= 0777;\n+\t} else if (S_ISREG(mode)) {\n+\t\t*header->typeflag = TYPEFLAG_REG;\n+\t\tmode = (mode | ((mode & 0100) ? 0777 : 0666)) & ~tar_umask;\n+\t} else {\n+\t\treturn error(_(\"unsupported file mode: 0%o (SHA1: %s)\"),\n+\t\t\t mode, oid_to_hex(oid));\n+\t}\n+\tif (pathlen > sizeof(header->name)) {\n+\t\tsize_t plen = get_path_prefix(path, pathlen,\n+\t\t\t\t\t sizeof(header->prefix));\n+\t\tsize_t rest = pathlen - plen - 1;\n+\t\tif (plen > 0 && rest <= sizeof(header->name)) {\n+\t\t\tmemcpy(header->prefix, path, plen);\n+\t\t\tmemcpy(header->name, path + plen + 1, rest);\n+\t\t} else {\n+\t\t\txsnprintf(header->name, sizeof(header->name), \"%s.data\",\n+\t\t\t\t oid_to_hex(oid));\n+\t\t\tstrbuf_append_ext_header(&ext_header, \"path\",\n+\t\t\t\t\t\t path, pathlen);\n+\t\t}\n+\t} else\n+\t\tmemcpy(header->name, path, pathlen);\n+\n+\tif (S_ISREG(mode) && !args->convert &&\n+\t oid_object_info(args->repo, oid, &size) == OBJ_BLOB &&\n+\t size > big_file_threshold)\n+\t\tbuffer = NULL;\n+\telse if (S_ISLNK(mode) || S_ISREG(mode)) {\n+\t\tenum object_type type;\n+\t\tbuffer = object_file_to_archive(args, path, oid, old_mode, &type, &size);\n+\t\tif (!buffer)\n+\t\t\treturn error(_(\"cannot read %s\"), oid_to_hex(oid));\n+\t} else {\n+\t\tbuffer = NULL;\n+\t\tsize = 0;\n+\t}\n+\n+\tif (S_ISLNK(mode)) {\n+\t\tif (size > sizeof(header->linkname)) {\n+\t\t\txsnprintf(header->linkname, sizeof(header->linkname),\n+\t\t\t\t \"see %s.paxheader\", oid_to_hex(oid));\n+\t\t\tstrbuf_append_ext_header(&ext_header, \"linkpath\",\n+\t\t\t\t\t\t buffer, size);\n+\t\t} else\n+\t\t\tmemcpy(header->linkname, buffer, size);\n+\t}\n+\n+\tsize_in_header = size;\n+\tif (S_ISREG(mode) && size > USTAR_MAX_SIZE) {\n+\t\tsize_in_header = 0;\n+\t\tstrbuf_append_ext_header_uint(&ext_header, \"size\", size);\n+\t}\n+\n+\tprepare_header(args, header, mode, size_in_header);\n+\n+\tif (ext_header.len > 0) {\n+\t\twrite_extended_header(args, oid, &ext_header);\n+\t}\n+\tadd_work(header, sizeof(*header), 1);\n+\tif (S_ISREG(mode) && size > 0) {\n+\t\tif (buffer)\n+\t\t\tadd_work(buffer, size, 1);\n+\t\telse\n+\t\t\terr = stream_blob(oid);\n+\t} else\n+\t\tfree(buffer);\n+\treturn err;\n+}\n+\n+static void write_global_extended_header(struct archiver_args *args)\n+{\n+\tconst struct object_id *oid = args->commit_oid;\n+\tstruct strbuf ext_header = STRBUF_INIT;\n+\tstruct ustar_header *header = xcalloc(1, sizeof(*header));\n+\tunsigned int mode;\n+\tsize_t size;\n+\tchar *buffer;\n+\n+\tif (oid)\n+\t\tstrbuf_append_ext_header(&ext_header, \"comment\",\n+\t\t\t\t\t oid_to_hex(oid),\n+\t\t\t\t\t the_hash_algo->hexsz);\n+\tif (args->time > USTAR_MAX_MTIME) {\n+\t\tstrbuf_append_ext_header_uint(&ext_header, \"mtime\",\n+\t\t\t\t\t args->time);\n+\t\targs->time = USTAR_MAX_MTIME;\n+\t}\n+\n+\tif (!ext_header.len)\n+\t\treturn;\n+\n+\tbuffer = strbuf_detach(&ext_header, &size);\n+\t*header->typeflag = TYPEFLAG_GLOBAL_HEADER;\n+\tmode = 0100666;\n+\txsnprintf(header->name, sizeof(header->name), \"pax_global_header\");\n+\tprepare_header(args, header, mode, size);\n+\tadd_work(header, sizeof(*header), 1);\n+\tadd_work(buffer, size, 1);\n+}\n+\n+int write_tgz_archive(const struct archiver *ar, struct archiver_args *args)\n+{\n+\tint level = args->compression_level;\n+\tint err = 0;\n+\n+\tgzip = gzdopen(1, \"wb\");\n+\tif (!gzip)\n+\t\treturn error(_(\"gzdopen failed\"));\n+\tif (gzsetparams(gzip, level, Z_DEFAULT_STRATEGY) != Z_OK)\n+\t\treturn error(_(\"unable to set compression level %d\"), level);\n+\n+\tstart_output_thread();\n+\twrite_global_extended_header(args);\n+\terr = write_archive_entries(args, write_tar_entry);\n+\tif (err)\n+\t\treturn err;\n+\twait_for_output_thread();\n+\ttgz_write_trailer();\n+\treturn err;\n+}\ndiff --git a/archive.h b/archive.h\nindex e60e3dd31c..b00afa1a9f 100644\n--- a/archive.h\n+++ b/archive.h\n@@ -45,6 +45,10 @@ void init_tar_archiver(void);\n void init_zip_archiver(void);\n void init_archivers(void);\n\n+int tar_umask;\n+\n+int write_tgz_archive(const struct archiver *ar, struct archiver_args *args);\n+\n typedef int (*write_archive_entry_fn_t)(struct archiver_args *args,\n \t\t\t\t\tconst struct object_id *oid,\n \t\t\t\t\tconst char *path, size_t pathlen,\n--\n2.22.0\n"},{"id":"377179","messageId":"20190613191630.GA30854@sigill.intra.peff.net","threadId":"50928","inReplyTo":"c00a062a-4f01-4754-3429-e7bb2a26aac1@web.de","subject":"Re: [PATCH 2/2] archive: avoid spawning `gzip`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-13T19:16:30Z","receivedAt":"2019-06-13T19:16:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 10, 2019 at 12:44:54PM +0200, René Scharfe wrote:\n\n> Am 01.05.19 um 20:18 schrieb Jeff King:\n> > On Wed, May 01, 2019 at 07:45:05PM +0200, René Scharfe wrote:\n> >\n> >>> But since the performance is still not quite on par with `gzip`, I would\n> >>> actually rather not, and really, just punt on that one, stating that\n> >>> people interested in higher performance should use `pigz`.\n> >>\n> >> Here are my performance numbers for generating .tar.gz files again:\n> \n> OK, tried one more version, with pthreads (patch at the end). Also\n> redid all measurements for better comparability; everything is faster\n> now for some reason (perhaps due to a compiler update? clang version\n> 7.0.1-8 now):\n\nHmm. Interesting that using pthreads is still slower than just shelling\nout to gzip:\n\n> master, using gzip(1):\n> Benchmark #1: git archive --format=tgz HEAD\n> Time (mean ± σ): 15.697 s ± 0.246 s [User: 19.213 s, System: 0.386 s]\n> Range (min … max): 15.405 s … 16.103 s 10 runs\n> [...]\n> using zlib in a separate thread (that's the new one):\n> Benchmark #1: git archive --format=tgz HEAD\n> Time (mean ± σ): 16.310 s ± 0.237 s [User: 20.075 s, System: 0.173 s]\n> Range (min … max): 15.983 s … 16.790 s 10 runs\n\nI wonder if zlib is just slower. Or if the cost of context switching\nis somehow higher than just dumping big chunks over a pipe. In\nparticular, our gzip-alike is still faster than pthreads:\n\n> using a gzip-lookalike:\n> Benchmark #1: git archive --format=tgz HEAD\n> Time (mean ± σ): 16.289 s ± 0.218 s [User: 19.485 s, System: 0.337 s]\n> Range (min … max): 16.020 s … 16.555 s 10 runs\n\nthough it looks like the timings do overlap.\n\n> > At GitHub we certainly do cache the git-archive output. We'd also be\n> > just fine with the sequential solution. We generally turn down\n> > pack.threads to 1, and keep our CPUs busy by serving multiple users\n> > anyway.\n> >\n> > So whatever has the lowest overall CPU time is generally preferable, but\n> > the times are close enough that I don't think we'd care much either way\n> > (and it's probably not worth having a config option or similar).\n> \n> Moving back to 2009 and reducing the number of utilized cores both feels\n> weird, but the sequential solution *is* the most obvious, easiest and\n> (by a narrow margin) lightest one if gzip(1) is not an option anymore.\n\nIt sounds like we resolved to give the \"internal gzip\" its own name\n(whether it's a gzip-alike command, or a special name we recognize to\ntrigger the internal code). So maybe we could continue to default to\n\"gzip -cn\", but platforms could do otherwise when shipping gzip there is\na pain (i.e. Windows, but maybe also anybody else who wants to set\nNO_EXTERNAL_GZIP or detect it from autoconf).\n\n-Peff\n"},{"id":"457071","messageId":"217a2f4d-4fc2-aaed-f5c2-1b7e134b046d@web.de","threadId":"50928","inReplyTo":"pull.145.git.gitgitgadget@gmail.com","subject":"[PATCH v3 0/5] Avoid spawning gzip in git archive","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-12T06:00:36Z","receivedAt":"2022-06-12T06:01:28Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"It's been a while, let's try again.\n\nChanges:\n- Use our own zlib helpers instead of the gz* functions of zlib,\n- ... which allows us to set the OS_CODE header consistently.\n- Pseudo-command \"git archive gzip\" to select the internal\n implementation in config.\n- Use a function pointer to plug in the internal gzip.\n- Tests.\n- Discuss performance in commit message.\n\n archive: rename archiver data field to filter_command\n archive-tar: factor out write_block()\n archive-tar: add internal gzip implementation\n archive-tar: use OS_CODE 3 (Unix) for internal gzip\n archive-tar: use internal gzip by default\n\n Documentation/git-archive.txt | 3 +-\n archive-tar.c | 79 ++++++++++++++++++++++++++++++-----\n archive.h | 2 +-\n t/t5000-tar-tree.sh | 28 ++++++++++---\n 4 files changed, 93 insertions(+), 19 deletions(-)\n\n--\n2.36.1\n"},{"id":"457072","messageId":"35db4457-1f86-ce05-baeb-51be57393bb4@web.de","threadId":"50928","inReplyTo":"217a2f4d-4fc2-aaed-f5c2-1b7e134b046d@web.de","subject":"[PATCH v3 1/5] archive: rename archiver data field to filter_command","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-12T06:03:58Z","receivedAt":"2022-06-12T06:04:12Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"The void pointer \"data\" in struct archiver is only used to store filter\ncommands to pass tar archives to, like gzip. Rename it accordingly and\nalso turn it into a char pointer to document the fact that it's a string\nreference.\n\nSigned-off-by: René Scharfe \n---\n archive-tar.c | 10 +++++-----\n archive.h | 2 +-\n 2 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 042feb66d2..2717e34a1d 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -383,8 +383,8 @@ static int tar_filter_config(const char *var, const char *value, void *data)\n \tif (!strcmp(type, \"command\")) {\n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n-\t\tfree(ar->data);\n-\t\tar->data = xstrdup(value);\n+\t\tfree(ar->filter_command);\n+\t\tar->filter_command = xstrdup(value);\n \t\treturn 0;\n \t}\n \tif (!strcmp(type, \"remote\")) {\n@@ -432,10 +432,10 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tstruct child_process filter = CHILD_PROCESS_INIT;\n \tint r;\n\n-\tif (!ar->data)\n+\tif (!ar->filter_command)\n \t\tBUG(\"tar-filter archiver called with no filter defined\");\n\n-\tstrbuf_addstr(&cmd, ar->data);\n+\tstrbuf_addstr(&cmd, ar->filter_command);\n \tif (args->compression_level >= 0)\n \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\n\n@@ -478,7 +478,7 @@ void init_tar_archiver(void)\n \tgit_config(git_tar_config, NULL);\n \tfor (i = 0; i < nr_tar_filters; i++) {\n \t\t/* omit any filters that never had a command configured */\n-\t\tif (tar_filters[i]->data)\n+\t\tif (tar_filters[i]->filter_command)\n \t\t\tregister_archiver(tar_filters[i]);\n \t}\n }\ndiff --git a/archive.h b/archive.h\nindex 49fab71aaf..08bed3ed3a 100644\n--- a/archive.h\n+++ b/archive.h\n@@ -43,7 +43,7 @@ struct archiver {\n \tconst char *name;\n \tint (*write_archive)(const struct archiver *, struct archiver_args *);\n \tunsigned flags;\n-\tvoid *data;\n+\tchar *filter_command;\n };\n void register_archiver(struct archiver *);\n\n--\n2.36.1\n"},{"id":"457073","messageId":"0c7a4261-47e6-941f-bd0c-4ecc646bb124@web.de","threadId":"50928","inReplyTo":"217a2f4d-4fc2-aaed-f5c2-1b7e134b046d@web.de","subject":"[PATCH v3 2/5] archive-tar: factor out write_block()","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-12T06:05:36Z","receivedAt":"2022-06-12T06:06:06Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"All tar archive writes have the same size and are done to the same file\ndescriptor. Move them to a common function, write_block(), to reduce\ncode duplication and make it easy to change the destination.\n\nOriginal-patch-by: Rohit Ashiwal \nSigned-off-by: René Scharfe \n---\n archive-tar.c | 13 +++++++++----\n 1 file changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 2717e34a1d..4e6a3deb80 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -38,11 +38,16 @@ static int write_tar_filter_archive(const struct archiver *ar,\n #define USTAR_MAX_MTIME 077777777777ULL\n #endif\n\n+static void write_block(const void *buf)\n+{\n+\twrite_or_die(1, buf, BLOCKSIZE);\n+}\n+\n /* writes out the whole block, but only if it is full */\n static void write_if_needed(void)\n {\n \tif (offset == BLOCKSIZE) {\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_block(block);\n \t\toffset = 0;\n \t}\n }\n@@ -66,7 +71,7 @@ static void do_write_blocked(const void *data, unsigned long size)\n \t\twrite_if_needed();\n \t}\n \twhile (size >= BLOCKSIZE) {\n-\t\twrite_or_die(1, buf, BLOCKSIZE);\n+\t\twrite_block(buf);\n \t\tsize -= BLOCKSIZE;\n \t\tbuf += BLOCKSIZE;\n \t}\n@@ -101,10 +106,10 @@ static void write_trailer(void)\n {\n \tint tail = BLOCKSIZE - offset;\n \tmemset(block + offset, 0, tail);\n-\twrite_or_die(1, block, BLOCKSIZE);\n+\twrite_block(block);\n \tif (tail < 2 * RECORDSIZE) {\n \t\tmemset(block, 0, offset);\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_block(block);\n \t}\n }\n\n--\n2.36.1\n"},{"id":"457074","messageId":"75e76d0f-2ab0-ebf7-4a4e-7a6e0fca0b1a@web.de","threadId":"50928","inReplyTo":"217a2f4d-4fc2-aaed-f5c2-1b7e134b046d@web.de","subject":"[PATCH v3 3/5] archive-tar: add internal gzip implementation","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-12T06:08:35Z","receivedAt":"2022-06-12T06:09:00Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Git uses zlib for its own object store, but calls gzip when creating tgz\narchives. Add an option to perform the gzip compression for the latter\nusing zlib, without depending on the external gzip binary.\n\nPlug it in by making write_block a function pointer and switching to a\ncompressing variant if the filter command has the magic value \"git\narchive gzip\". Does that indirection slow down tar creation? Not\nreally, at least not in this test:\n\n$ hyperfine -w3 -L rev HEAD,origin/main -p 'git checkout {rev} && make' \\\n'./git -C ../linux archive --format=tar HEAD # {rev}'\nBenchmark #1: ./git -C ../linux archive --format=tar HEAD # HEAD\n Time (mean ± σ): 4.044 s ± 0.007 s [User: 3.901 s, System: 0.137 s]\n Range (min … max): 4.038 s … 4.059 s 10 runs\n\nBenchmark #2: ./git -C ../linux archive --format=tar HEAD # origin/main\n Time (mean ± σ): 4.047 s ± 0.009 s [User: 3.903 s, System: 0.138 s]\n Range (min … max): 4.038 s … 4.066 s 10 runs\n\nHow does tgz creation perform?\n\n$ hyperfine -w3 -L command 'gzip -cn','git archive gzip' \\\n'./git -c tar.tgz.command=\"{command}\" -C ../linux archive --format=tgz HEAD'\nBenchmark #1: ./git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD\n Time (mean ± σ): 20.404 s ± 0.006 s [User: 23.943 s, System: 0.401 s]\n Range (min … max): 20.395 s … 20.414 s 10 runs\n\nBenchmark #2: ./git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD\n Time (mean ± σ): 23.807 s ± 0.023 s [User: 23.655 s, System: 0.145 s]\n Range (min … max): 23.782 s … 23.857 s 10 runs\n\nSummary\n './git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD' ran\n 1.17 ± 0.00 times faster than './git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD'\n\nSo the internal implementation takes 17% longer on the Linux repo, but\nuses 2% less CPU time. That's because the external gzip can run in\nparallel on its own processor, while the internal one works sequentially\nand avoids the inter-process communication overhead.\n\nWhat are the benefits? Only an internal sequential implementation can\noffer this eco mode, and it allows avoiding the gzip(1) requirement.\n\nThis implementation uses the helper functions from our zlib.c instead of\nthe convenient gz* functions from zlib, because the latter doesn't give\nthe control over the generated gzip header that the next patch requires.\n\nOriginal-patch-by: Rohit Ashiwal \nSigned-off-by: René Scharfe \n---\n Documentation/git-archive.txt | 3 ++-\n archive-tar.c | 45 ++++++++++++++++++++++++++++++++++-\n t/t5000-tar-tree.sh | 16 +++++++++++++\n 3 files changed, 62 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\nindex 56989a2f34..5b017c2bdc 100644\n--- a/Documentation/git-archive.txt\n+++ b/Documentation/git-archive.txt\n@@ -148,7 +148,8 @@ tar..command::\n \tformat is given.\n +\n The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n-`gzip -cn`. You may override them with custom commands.\n+`gzip -cn`. You may override them with custom commands. An internal gzip\n+implementation can be used by specifying the value `git archive gzip`.\n\n tar..remote::\n \tIf true, enable `` for use by remote clients via\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 4e6a3deb80..53d0ef685c 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -38,11 +38,13 @@ static int write_tar_filter_archive(const struct archiver *ar,\n #define USTAR_MAX_MTIME 077777777777ULL\n #endif\n\n-static void write_block(const void *buf)\n+static void tar_write_block(const void *buf)\n {\n \twrite_or_die(1, buf, BLOCKSIZE);\n }\n\n+static void (*write_block)(const void *) = tar_write_block;\n+\n /* writes out the whole block, but only if it is full */\n static void write_if_needed(void)\n {\n@@ -430,6 +432,34 @@ static int write_tar_archive(const struct archiver *ar,\n \treturn err;\n }\n\n+static git_zstream gzstream;\n+static unsigned char outbuf[16384];\n+\n+static void tgz_deflate(int flush)\n+{\n+\twhile (gzstream.avail_in || flush == Z_FINISH) {\n+\t\tint status = git_deflate(&gzstream, flush);\n+\t\tif (!gzstream.avail_out || status == Z_STREAM_END) {\n+\t\t\twrite_or_die(1, outbuf, gzstream.next_out - outbuf);\n+\t\t\tgzstream.next_out = outbuf;\n+\t\t\tgzstream.avail_out = sizeof(outbuf);\n+\t\t\tif (status == Z_STREAM_END)\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR)\n+\t\t\tdie(_(\"deflate error (%d)\"), status);\n+\t}\n+}\n+\n+static void tgz_write_block(const void *data)\n+{\n+\tgzstream.next_in = (void *)data;\n+\tgzstream.avail_in = BLOCKSIZE;\n+\ttgz_deflate(Z_NO_FLUSH);\n+}\n+\n+static const char internal_gzip_command[] = \"git archive gzip\";\n+\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args)\n {\n@@ -440,6 +470,19 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!ar->filter_command)\n \t\tBUG(\"tar-filter archiver called with no filter defined\");\n\n+\tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n+\t\twrite_block = tgz_write_block;\n+\t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n+\t\tgzstream.next_out = outbuf;\n+\t\tgzstream.avail_out = sizeof(outbuf);\n+\n+\t\tr = write_tar_archive(ar, args);\n+\n+\t\ttgz_deflate(Z_FINISH);\n+\t\tgit_deflate_end(&gzstream);\n+\t\treturn r;\n+\t}\n+\n \tstrbuf_addstr(&cmd, ar->filter_command);\n \tif (args->compression_level >= 0)\n \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\ndiff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh\nindex 7f8d2ab0a7..9ac0ec67fe 100755\n--- a/t/t5000-tar-tree.sh\n+++ b/t/t5000-tar-tree.sh\n@@ -374,6 +374,22 @@ test_expect_success GZIP 'remote tar.gz can be disabled' '\n \t\t>remote.tar.gz\n '\n\n+test_expect_success 'git archive --format=tgz (internal gzip)' '\n+\ttest_config tar.tgz.command \"git archive gzip\" &&\n+\tgit archive --format=tgz HEAD >internal_gzip.tgz\n+'\n+\n+test_expect_success 'git archive --format=tar.gz (internal gzip)' '\n+\ttest_config tar.tar.gz.command \"git archive gzip\" &&\n+\tgit archive --format=tar.gz HEAD >internal_gzip.tar.gz &&\n+\ttest_cmp_bin internal_gzip.tgz internal_gzip.tar.gz\n+'\n+\n+test_expect_success GZIP 'extract tgz file (internal gzip)' '\n+\tgzip -d -c internal_gzip.tar &&\n+\ttest_cmp_bin b.tar internal_gzip.tar\n+'\n+\n test_expect_success 'archive and :(glob)' '\n \tgit archive -v HEAD -- \":(glob)**/sh\" >/dev/null 2>actual &&\n \tcat >expect <\n---\nPerhaps makes sense for remote-curl as well (out of scope of this\nseries)?\n\n archive-tar.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 53d0ef685c..bf7e321e0e 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -460,6 +460,14 @@ static void tgz_write_block(const void *data)\n\n static const char internal_gzip_command[] = \"git archive gzip\";\n\n+static void tgz_set_os(git_zstream *strm, int os)\n+{\n+#if ZLIB_VERNUM >= 0x1221\n+\tstruct gz_header_s gzhead = { .os = os };\n+\tdeflateSetHeader(&strm->z, &gzhead);\n+#endif\n+}\n+\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args)\n {\n@@ -473,6 +481,7 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n \t\twrite_block = tgz_write_block;\n \t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n+\t\ttgz_set_os(&gzstream, 3); /* Unix, for reproducibility */\n \t\tgzstream.next_out = outbuf;\n \t\tgzstream.avail_out = sizeof(outbuf);\n\n--\n2.36.1\n"},{"id":"457076","messageId":"d9e75b24-c351-e226-011d-5a5cc2e1c858@web.de","threadId":"50928","inReplyTo":"217a2f4d-4fc2-aaed-f5c2-1b7e134b046d@web.de","subject":"[PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-12T06:19:05Z","receivedAt":"2022-06-12T06:19:25Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Drop the dependency on gzip(1) and use our internal implementation to\ncreate tar.gz and tgz files.\n\nSigned-off-by: René Scharfe \n---\n Documentation/git-archive.txt | 4 ++--\n archive-tar.c | 4 ++--\n t/t5000-tar-tree.sh | 32 ++++++++++++++++----------------\n 3 files changed, 20 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\nindex 5b017c2bdc..9de12896fc 100644\n--- a/Documentation/git-archive.txt\n+++ b/Documentation/git-archive.txt\n@@ -148,8 +148,8 @@ tar..command::\n \tformat is given.\n +\n The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n-`gzip -cn`. You may override them with custom commands. An internal gzip\n-implementation can be used by specifying the value `git archive gzip`.\n+the magic value `git archive gzip`, which invokes an internal\n+implementation of gzip. You may override them with custom commands.\n\n tar..remote::\n \tIf true, enable `` for use by remote clients via\ndiff --git a/archive-tar.c b/archive-tar.c\nindex bf7e321e0e..60669eb7b9 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -528,9 +528,9 @@ void init_tar_archiver(void)\n \tint i;\n \tregister_archiver(&tar_archiver);\n\n-\ttar_filter_config(\"tar.tgz.command\", \"gzip -cn\", NULL);\n+\ttar_filter_config(\"tar.tgz.command\", internal_gzip_command, NULL);\n \ttar_filter_config(\"tar.tgz.remote\", \"true\", NULL);\n-\ttar_filter_config(\"tar.tar.gz.command\", \"gzip -cn\", NULL);\n+\ttar_filter_config(\"tar.tar.gz.command\", internal_gzip_command, NULL);\n \ttar_filter_config(\"tar.tar.gz.remote\", \"true\", NULL);\n \tgit_config(git_tar_config, NULL);\n \tfor (i = 0; i < nr_tar_filters; i++) {\ndiff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh\nindex 9ac0ec67fe..1a68e89a55 100755\n--- a/t/t5000-tar-tree.sh\n+++ b/t/t5000-tar-tree.sh\n@@ -339,21 +339,21 @@ test_expect_success 'only enabled filters are available remotely' '\n \ttest_cmp_bin remote.bar config.bar\n '\n\n-test_expect_success GZIP 'git archive --format=tgz' '\n+test_expect_success 'git archive --format=tgz' '\n \tgit archive --format=tgz HEAD >j.tgz\n '\n\n-test_expect_success GZIP 'git archive --format=tar.gz' '\n+test_expect_success 'git archive --format=tar.gz' '\n \tgit archive --format=tar.gz HEAD >j1.tar.gz &&\n \ttest_cmp_bin j.tgz j1.tar.gz\n '\n\n-test_expect_success GZIP 'infer tgz from .tgz filename' '\n+test_expect_success 'infer tgz from .tgz filename' '\n \tgit archive --output=j2.tgz HEAD &&\n \ttest_cmp_bin j.tgz j2.tgz\n '\n\n-test_expect_success GZIP 'infer tgz from .tar.gz filename' '\n+test_expect_success 'infer tgz from .tar.gz filename' '\n \tgit archive --output=j3.tar.gz HEAD &&\n \ttest_cmp_bin j.tgz j3.tar.gz\n '\n@@ -363,31 +363,31 @@ test_expect_success GZIP 'extract tgz file' '\n \ttest_cmp_bin b.tar j.tar\n '\n\n-test_expect_success GZIP 'remote tar.gz is allowed by default' '\n+test_expect_success 'remote tar.gz is allowed by default' '\n \tgit archive --remote=. --format=tar.gz HEAD >remote.tar.gz &&\n \ttest_cmp_bin j.tgz remote.tar.gz\n '\n\n-test_expect_success GZIP 'remote tar.gz can be disabled' '\n+test_expect_success 'remote tar.gz can be disabled' '\n \tgit config tar.tar.gz.remote false &&\n \ttest_must_fail git archive --remote=. --format=tar.gz HEAD \\\n \t\t>remote.tar.gz\n '\n\n-test_expect_success 'git archive --format=tgz (internal gzip)' '\n-\ttest_config tar.tgz.command \"git archive gzip\" &&\n-\tgit archive --format=tgz HEAD >internal_gzip.tgz\n+test_expect_success GZIP 'git archive --format=tgz (external gzip)' '\n+\ttest_config tar.tgz.command \"gzip -cn\" &&\n+\tgit archive --format=tgz HEAD >external_gzip.tgz\n '\n\n-test_expect_success 'git archive --format=tar.gz (internal gzip)' '\n-\ttest_config tar.tar.gz.command \"git archive gzip\" &&\n-\tgit archive --format=tar.gz HEAD >internal_gzip.tar.gz &&\n-\ttest_cmp_bin internal_gzip.tgz internal_gzip.tar.gz\n+test_expect_success GZIP 'git archive --format=tar.gz (external gzip)' '\n+\ttest_config tar.tar.gz.command \"gzip -cn\" &&\n+\tgit archive --format=tar.gz HEAD >external_gzip.tar.gz &&\n+\ttest_cmp_bin external_gzip.tgz external_gzip.tar.gz\n '\n\n-test_expect_success GZIP 'extract tgz file (internal gzip)' '\n-\tgzip -d -c internal_gzip.tar &&\n-\ttest_cmp_bin b.tar internal_gzip.tar\n+test_expect_success GZIP 'extract tgz file (external gzip)' '\n+\tgzip -d -c external_gzip.tar &&\n+\ttest_cmp_bin b.tar external_gzip.tar\n '\n\n test_expect_success 'archive and :(glob)' '\n--\n2.36.1\n"},{"id":"457119","messageId":"xmqqilp45qhq.fsf@gitster.g","threadId":"50928","inReplyTo":"75e76d0f-2ab0-ebf7-4a4e-7a6e0fca0b1a@web.de","subject":"Re: [PATCH v3 3/5] archive-tar: add internal gzip implementation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-13T19:10:25Z","receivedAt":"2022-06-13T20:26:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> @@ -148,7 +148,8 @@ tar..command::\n> \tformat is given.\n> +\n> The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n> -`gzip -cn`. You may override them with custom commands.\n> +`gzip -cn`. You may override them with custom commands. An internal gzip\n> +implementation can be used by specifying the value `git archive gzip`.\n>\n\nThe new sentence didn't click and I was lost in figuring out what is\nset to which value and where, before looking at a test the patch\nadds.\n\nI think it is not entirely a fault of this patch, but the badness is\nalready in the original. I wouldn't have been as confused if it\nwere like so:\n\n The \"tar.gz\" and \"tgz\" formats are defined automatically to use\n `gzip -cn` as the command by default. An internal gzip\n implementation can be used by specifying the value `git archive\n gzip` for these two formats.\n\nfor example.\n\nThanks.\n\n"},{"id":"457131","messageId":"xmqqk09k449y.fsf@gitster.g","threadId":"50928","inReplyTo":"d9e75b24-c351-e226-011d-5a5cc2e1c858@web.de","subject":"Re: [PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-13T21:55:37Z","receivedAt":"2022-06-13T21:55:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> -test_expect_success GZIP 'git archive --format=tar.gz' '\n> +test_expect_success 'git archive --format=tar.gz' '\n> \tgit archive --format=tar.gz HEAD >j1.tar.gz &&\n> \ttest_cmp_bin j.tgz j1.tar.gz\n> '\n\nCuriously, this breaks for me. It is understandable if we are not\nproducing byte-for-byte identical output with internal gzip.\n\nWith the following hack I can force the step pass, so it seems that\nthe two invocations of internal gzip are not emitting identical\nresult for the tar stream taken out of HEAD^{tree} object?\n\ndiff --git c/t/t5000-tar-tree.sh w/t/t5000-tar-tree.sh\nindex 1a68e89a55..c0a2cb92d4 100755\n--- c/t/t5000-tar-tree.sh\n+++ w/t/t5000-tar-tree.sh\n@@ -340,14 +340,16 @@ test_expect_success 'only enabled filters are available remotely' '\n '\n \n test_expect_success 'git archive --format=tgz' '\n-\tgit archive --format=tgz HEAD >j.tgz\n+\tgit -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD >j.tgz\n '\n \n test_expect_success 'git archive --format=tar.gz' '\n-\tgit archive --format=tar.gz HEAD >j1.tar.gz &&\n+\tgit -c tar.tar.gz.command=\"gzip -cn\" archive --format=tar.gz HEAD >j1.tar.gz &&\n \ttest_cmp_bin j.tgz j1.tar.gz\n '\n \n+exit\n+\n test_expect_success 'infer tgz from .tgz filename' '\n \tgit archive --output=j2.tgz HEAD &&\n \ttest_cmp_bin j.tgz j2.tgz\n\n"},{"id":"457181","messageId":"nycvar.QRO.7.76.6.2206141109270.353@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"xmqqk09k449y.fsf@gitster.g","subject":"Re: [PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-14T11:27:37Z","receivedAt":"2022-06-14T11:28:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 13 Jun 2022, Junio C Hamano wrote:\n\n> René Scharfe writes:\n>\n> > -test_expect_success GZIP 'git archive --format=tar.gz' '\n> > +test_expect_success 'git archive --format=tar.gz' '\n> > \tgit archive --format=tar.gz HEAD >j1.tar.gz &&\n> > \ttest_cmp_bin j.tgz j1.tar.gz\n> > '\n>\n> Curiously, this breaks for me. It is understandable if we are not\n> producing byte-for-byte identical output with internal gzip.\n\nIndeed, I can reproduce this, too. In particular, `j.tgz` and `j1.tar.gz`\ndiffer like this in my test run:\n\n-00000000 1f 8b 08 1a 00 2e ca 09 00 03 04 00 89 45 fc 83 |.............E..|\n+00000000 1f 8b 08 1a 00 35 2a 10 00 03 04 00 89 45 fc 83 |.....5*......E..|\n\nand\n\n-00000010 7d fc 00 f1 d0 ec b7 63 8c 30 cc 9b e6 db b6 6d |}......c.0.....m|\n+00000010 7d fc 00 54 ff ec b7 63 8c 30 cc 9b e6 db b6 6d |}..T...c.0.....m|\n\nAccording to https://datatracker.ietf.org/doc/html/rfc1952#page-5, the\ndifference in the first line is the mtime. For reference, this is the\nversion with `git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz\nHEAD`:\n\n00000000 1f 8b 08 00 00 00 00 00 00 03 ec b7 63 8c 30 cc |............c.0.|\n\nIn other words, `gzip` forces the `mtim` member to all zeros, which makes\nsense.\n\nThe recorded mtimes are a bit funny, according to\nhttps://wolf-tungsten.github.io/gzip-analyzer/, they are 1975-03-17\n00:36:32 and 1978-08-05 22:45:36, respectively...\n\nAnd the mtime actually changes all the time.\n\nWhat's even more funny: if I comment out the `deflateSetHeader()`, the\nmtime header field is left at all-zeros. This is on Ubuntu 18.04 with\nzlib1g 1:1.2.11.dfsg-0ubuntu2.\n\nSo I dug in a bit deeper and what do you know, the `deflateHeader()`\nfunction is implemented like this\n(https://github.com/madler/zlib/blob/21767c654d31/deflate.c#L557-L565):\n\n\tint ZEXPORT deflateSetHeader (strm, head)\n\t z_streamp strm;\n\t gz_headerp head;\n\t{\n\t if (deflateStateCheck(strm) || strm->state->wrap != 2)\n\t\treturn Z_STREAM_ERROR;\n\t strm->state->gzhead = head;\n\t return Z_OK;\n\t}\n\nNow, the caller is implemented like this:\n\n\tstatic void tgz_set_os(git_zstream *strm, int os)\n\t{\n\t#if ZLIB_VERNUM >= 0x1221\n\t\tstruct gz_header_s gzhead = { .os = os };\n\t\tdeflateSetHeader(&strm->z, &gzhead);\n\t#endif\n\t}\n\nThe biggest problem is not that the return value of `deflateSetHeader()`\nis ignored. The biggest problem is that it passes the address of a heap\nvariable to the `deflateSetHeader()` function, which then stores it away\nin another struct that lives beyond the point when we return from\n`tgz_set_os()`.\n\nIn other words, this is the very issue I pointed out as GCC not catching:\nhttps://lore.kernel.org/git/nycvar.QRO.7.76.6.2205272235220.349@tvgsbejvaqbjf.bet/\n\nThe solution is to move the heap variable back into a scope that matches\nthe lifetime of the compression:\n\n-- snip --\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 60669eb7b9c..3d77e0f7509 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -460,17 +460,12 @@ static void tgz_write_block(const void *data)\n\n static const char internal_gzip_command[] = \"git archive gzip\";\n\n-static void tgz_set_os(git_zstream *strm, int os)\n-{\n-#if ZLIB_VERNUM >= 0x1221\n-\tstruct gz_header_s gzhead = { .os = os };\n-\tdeflateSetHeader(&strm->z, &gzhead);\n-#endif\n-}\n-\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args)\n {\n+#if ZLIB_VERNUM >= 0x1221\n+\tstruct gz_header_s gzhead = { .os = 3 }; /* Unix, for reproducibility */\n+#endif\n \tstruct strbuf cmd = STRBUF_INIT;\n \tstruct child_process filter = CHILD_PROCESS_INIT;\n \tint r;\n@@ -481,7 +476,10 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n \t\twrite_block = tgz_write_block;\n \t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n-\t\ttgz_set_os(&gzstream, 3); /* Unix, for reproducibility */\n+#if ZLIB_VERNUM >= 0x1221\n+\t\tif (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK)\n+\t\t\tBUG(\"deflateSetHeader() called too late\");\n+#endif\n \t\tgzstream.next_out = outbuf;\n \t\tgzstream.avail_out = sizeof(outbuf);\n\n-- snap --\n\nWith this, the test passes for me.\n\nRené, would you mind squashing this into your patch series?\n\nThank you,\nDscho\n"},{"id":"457182","messageId":"nycvar.QRO.7.76.6.2206141043150.353@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"217a2f4d-4fc2-aaed-f5c2-1b7e134b046d@web.de","subject":"Re: [PATCH v3 0/5] Avoid spawning gzip in git archive","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-14T11:28:06Z","receivedAt":"2022-06-14T11:29:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi René,\n\nOn Sun, 12 Jun 2022, René Scharfe wrote:\n\n> It's been a while, let's try again.\n\nThank you for picking this up again!\n\n> Changes:\n> - Use our own zlib helpers instead of the gz* functions of zlib,\n> - ... which allows us to set the OS_CODE header consistently.\n> - Pseudo-command \"git archive gzip\" to select the internal\n> implementation in config.\n> - Use a function pointer to plug in the internal gzip.\n> - Tests.\n> - Discuss performance in commit message.\n\nMakes sense. Here is the range-diff:\n\n-- snip --\n-: ----------- > 1: 9847267888e archive: rename archiver data field to filter_command\n1: 7d50f52e490 ! 2: a98ef655af9 archive: factor out writing blocks into a separate function\n @@\n ## Metadata ##\n -Author: Rohit Ashiwal \n +Author: René Scharfe \n\n ## Commit message ##\n - archive: factor out writing blocks into a separate function\n + archive-tar: factor out write_block()\n\n - The `git archive --format=tgz` command spawns `gzip` to perform the\n - actual compression. However, the MinGit flavor of Git for Windows\n - comes without `gzip` bundled inside.\n + All tar archive writes have the same size and are done to the same file\n + descriptor. Move them to a common function, write_block(), to reduce\n + code duplication and make it easy to change the destination.\n\n - To help with that, we will teach `git archive` to let zlib perform the\n - actual compression.\n -\n - In preparation for this, we consolidate all the block writes into the\n - function `write_block_or_die()`.\n -\n - Note: .tar files have a well-defined, fixed block size. For that reason,\n - it does not make any sense to pass anything but a fully-populated,\n - full-length block to the `write_block_or_die()` function, and we can\n - save ourselves some future trouble by not even allowing to pass an\n - incorrect `size` parameter to it.\n -\n - Signed-off-by: Rohit Ashiwal \n - Signed-off-by: Johannes Schindelin \n + Original-patch-by: Rohit Ashiwal \n + Signed-off-by: René Scharfe \n + Signed-off-by: Junio C Hamano \n\n ## archive-tar.c ##\n @@ archive-tar.c: static int write_tar_filter_archive(const struct archiver *ar,\n #define USTAR_MAX_MTIME 077777777777ULL\n #endif\n\n -+/* writes out the whole block, or dies if fails */\n -+static void write_block_or_die(const char *block) {\n -+\twrite_or_die(1, block, BLOCKSIZE);\n ++static void write_block(const void *buf)\n ++{\n ++\twrite_or_die(1, buf, BLOCKSIZE);\n +}\n +\n /* writes out the whole block, but only if it is full */\n @@ archive-tar.c: static int write_tar_filter_archive(const struct archiver *ar,\n {\n \tif (offset == BLOCKSIZE) {\n -\t\twrite_or_die(1, block, BLOCKSIZE);\n -+\t\twrite_block_or_die(block);\n ++\t\twrite_block(block);\n \t\toffset = 0;\n \t}\n }\n @@ archive-tar.c: static void do_write_blocked(const void *data, unsigned long size\n \t}\n \twhile (size >= BLOCKSIZE) {\n -\t\twrite_or_die(1, buf, BLOCKSIZE);\n -+\t\twrite_block_or_die(buf);\n ++\t\twrite_block(buf);\n \t\tsize -= BLOCKSIZE;\n \t\tbuf += BLOCKSIZE;\n \t}\n @@ archive-tar.c: static void write_trailer(void)\n \tint tail = BLOCKSIZE - offset;\n \tmemset(block + offset, 0, tail);\n -\twrite_or_die(1, block, BLOCKSIZE);\n -+\twrite_block_or_die(block);\n ++\twrite_block(block);\n \tif (tail < 2 * RECORDSIZE) {\n \t\tmemset(block, 0, offset);\n -\t\twrite_or_die(1, block, BLOCKSIZE);\n -+\t\twrite_block_or_die(block);\n ++\t\twrite_block(block);\n \t}\n }\n\n2: ac2b2488a1b < -: ----------- archive-tar: mark RECORDSIZE/BLOCKSIZE as unsigned\n3: 4ea94a87848 ! 3: 5e3c0d79589 archive: optionally use zlib directly for gzip compression\n @@\n ## Metadata ##\n -Author: Rohit Ashiwal \n +Author: René Scharfe \n\n ## Commit message ##\n - archive: optionally use zlib directly for gzip compression\n + archive-tar: add internal gzip implementation\n\n - As we already link to the zlib library, we can perform the compression\n - without even requiring gzip on the host machine.\n + Git uses zlib for its own object store, but calls gzip when creating tgz\n + archives. Add an option to perform the gzip compression for the latter\n + using zlib, without depending on the external gzip binary.\n\n - Note: the `-n` flag that `git archive` passed to `gzip` wants to ensure\n - that a reproducible file is written, i.e. no filename or mtime will be\n - recorded in the compressed output. This is already the default for\n - zlib's `gzopen()` function (if the file name or mtime should be\n - recorded, the `deflateSetHeader()` function would have to be called\n - instead).\n + Plug it in by making write_block a function pointer and switching to a\n + compressing variant if the filter command has the magic value \"git\n + archive gzip\". Does that indirection slow down tar creation? Not\n + really, at least not in this test:\n\n - Note also that the `gzFile` datatype is defined as a pointer in\n - `zlib.h`, i.e. we can rely on the fact that it can be `NULL`.\n + $ hyperfine -w3 -L rev HEAD,origin/main -p 'git checkout {rev} && make' \\\n + './git -C ../linux archive --format=tar HEAD # {rev}'\n + Benchmark #1: ./git -C ../linux archive --format=tar HEAD # HEAD\n + Time (mean ± σ): 4.044 s ± 0.007 s [User: 3.901 s, System: 0.137 s]\n + Range (min … max): 4.038 s … 4.059 s 10 runs\n\n - At this point, this new mode is hidden behind the pseudo command\n - `:zlib`: assign this magic string to the `archive.tgz.command` config\n - setting to enable it.\n + Benchmark #2: ./git -C ../linux archive --format=tar HEAD # origin/main\n + Time (mean ± σ): 4.047 s ± 0.009 s [User: 3.903 s, System: 0.138 s]\n + Range (min … max): 4.038 s … 4.066 s 10 runs\n\n - Signed-off-by: Rohit Ashiwal \n - Signed-off-by: Johannes Schindelin \n + How does tgz creation perform?\n\n - ## archive-tar.c ##\n -@@ archive-tar.c: static unsigned long offset;\n -\n - static int tar_umask = 002;\n -\n -+static gzFile gzip;\n -+\n - static int write_tar_filter_archive(const struct archiver *ar,\n - \t\t\t\t struct archiver_args *args);\n + $ hyperfine -w3 -L command 'gzip -cn','git archive gzip' \\\n + './git -c tar.tgz.command=\"{command}\" -C ../linux archive --format=tgz HEAD'\n + Benchmark #1: ./git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD\n + Time (mean ± σ): 20.404 s ± 0.006 s [User: 23.943 s, System: 0.401 s]\n + Range (min … max): 20.395 s … 20.414 s 10 runs\n +\n + Benchmark #2: ./git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD\n + Time (mean ± σ): 23.807 s ± 0.023 s [User: 23.655 s, System: 0.145 s]\n + Range (min … max): 23.782 s … 23.857 s 10 runs\n +\n + Summary\n + './git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD' ran\n + 1.17 ± 0.00 times faster than './git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD'\n +\n + So the internal implementation takes 17% longer on the Linux repo, but\n + uses 2% less CPU time. That's because the external gzip can run in\n + parallel on its own processor, while the internal one works sequentially\n + and avoids the inter-process communication overhead.\n +\n + What are the benefits? Only an internal sequential implementation can\n + offer this eco mode, and it allows avoiding the gzip(1) requirement.\n +\n + This implementation uses the helper functions from our zlib.c instead of\n + the convenient gz* functions from zlib, because the latter doesn't give\n + the control over the generated gzip header that the next patch requires.\n +\n + Original-patch-by: Rohit Ashiwal \n + Signed-off-by: René Scharfe \n + Signed-off-by: Junio C Hamano \n +\n + ## Documentation/git-archive.txt ##\n +@@ Documentation/git-archive.txt: tar..command::\n + \tformat is given.\n + +\n + The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n +-`gzip -cn`. You may override them with custom commands.\n ++`gzip -cn`. You may override them with custom commands. An internal gzip\n ++implementation can be used by specifying the value `git archive gzip`.\n\n + tar..remote::\n + \tIf true, enable `` for use by remote clients via\n +\n + ## archive-tar.c ##\n @@ archive-tar.c: static int write_tar_filter_archive(const struct archiver *ar,\n + #define USTAR_MAX_MTIME 077777777777ULL\n + #endif\n\n - /* writes out the whole block, or dies if fails */\n - static void write_block_or_die(const char *block) {\n --\twrite_or_die(1, block, BLOCKSIZE);\n -+\tif (!gzip)\n -+\t\twrite_or_die(1, block, BLOCKSIZE);\n -+\telse if (gzwrite(gzip, block, (unsigned) BLOCKSIZE) != BLOCKSIZE)\n -+\t\tdie(_(\"gzwrite failed\"));\n +-static void write_block(const void *buf)\n ++static void tar_write_block(const void *buf)\n + {\n + \twrite_or_die(1, buf, BLOCKSIZE);\n }\n\n ++static void (*write_block)(const void *) = tar_write_block;\n ++\n /* writes out the whole block, but only if it is full */\n -@@ archive-tar.c: static int write_tar_filter_archive(const struct archiver *ar,\n - \tfilter.use_shell = 1;\n - \tfilter.in = -1;\n + static void write_if_needed(void)\n + {\n +@@ archive-tar.c: static int write_tar_archive(const struct archiver *ar,\n + \treturn err;\n + }\n\n --\tif (start_command(&filter) < 0)\n --\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n --\tclose(1);\n --\tif (dup2(filter.in, 1) < 0)\n --\t\tdie_errno(_(\"unable to redirect descriptor\"));\n --\tclose(filter.in);\n -+\tif (!strcmp(\":zlib\", ar->data)) {\n -+\t\tstruct strbuf mode = STRBUF_INIT;\n ++static git_zstream gzstream;\n ++static unsigned char outbuf[16384];\n +\n -+\t\tstrbuf_addstr(&mode, \"wb\");\n ++static void tgz_deflate(int flush)\n ++{\n ++\twhile (gzstream.avail_in || flush == Z_FINISH) {\n ++\t\tint status = git_deflate(&gzstream, flush);\n ++\t\tif (!gzstream.avail_out || status == Z_STREAM_END) {\n ++\t\t\twrite_or_die(1, outbuf, gzstream.next_out - outbuf);\n ++\t\t\tgzstream.next_out = outbuf;\n ++\t\t\tgzstream.avail_out = sizeof(outbuf);\n ++\t\t\tif (status == Z_STREAM_END)\n ++\t\t\t\tbreak;\n ++\t\t}\n ++\t\tif (status != Z_OK && status != Z_BUF_ERROR)\n ++\t\t\tdie(_(\"deflate error (%d)\"), status);\n ++\t}\n ++}\n +\n -+\t\tif (args->compression_level >= 0 && args->compression_level <= 9)\n -+\t\t\tstrbuf_addf(&mode, \"%d\", args->compression_level);\n ++static void tgz_write_block(const void *data)\n ++{\n ++\tgzstream.next_in = (void *)data;\n ++\tgzstream.avail_in = BLOCKSIZE;\n ++\ttgz_deflate(Z_NO_FLUSH);\n ++}\n +\n -+\t\tgzip = gzdopen(fileno(stdout), mode.buf);\n -+\t\tif (!gzip)\n -+\t\t\tdie(_(\"Could not gzdopen stdout\"));\n -+\t\tstrbuf_release(&mode);\n -+\t} else {\n -+\t\tif (start_command(&filter) < 0)\n -+\t\t\tdie_errno(_(\"unable to start '%s' filter\"), argv[0]);\n -+\t\tclose(1);\n -+\t\tif (dup2(filter.in, 1) < 0)\n -+\t\t\tdie_errno(_(\"unable to redirect descriptor\"));\n -+\t\tclose(filter.in);\n -+\t}\n -\n - \tr = write_tar_archive(ar, args);\n ++static const char internal_gzip_command[] = \"git archive gzip\";\n ++\n + static int write_tar_filter_archive(const struct archiver *ar,\n + \t\t\t\t struct archiver_args *args)\n + {\n +@@ archive-tar.c: static int write_tar_filter_archive(const struct archiver *ar,\n + \tif (!ar->filter_command)\n + \t\tBUG(\"tar-filter archiver called with no filter defined\");\n\n --\tclose(1);\n --\tif (finish_command(&filter) != 0)\n --\t\tdie(_(\"'%s' filter reported error\"), argv[0]);\n -+\tif (gzip) {\n -+\t\tint ret = gzclose(gzip);\n -+\t\tif (ret == Z_ERRNO)\n -+\t\t\tdie_errno(_(\"gzclose failed\"));\n -+\t\telse if (ret != Z_OK)\n -+\t\t\tdie(_(\"gzclose failed (%d)\"), ret);\n -+\t} else {\n -+\t\tclose(1);\n -+\t\tif (finish_command(&filter) != 0)\n -+\t\t\tdie(_(\"'%s' filter reported error\"), argv[0]);\n ++\tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n ++\t\twrite_block = tgz_write_block;\n ++\t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n ++\t\tgzstream.next_out = outbuf;\n ++\t\tgzstream.avail_out = sizeof(outbuf);\n ++\n ++\t\tr = write_tar_archive(ar, args);\n ++\n ++\t\ttgz_deflate(Z_FINISH);\n ++\t\tgit_deflate_end(&gzstream);\n ++\t\treturn r;\n +\t}\n ++\n + \tstrbuf_addstr(&cmd, ar->filter_command);\n + \tif (args->compression_level >= 0)\n + \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\n +\n + ## t/t5000-tar-tree.sh ##\n +@@ t/t5000-tar-tree.sh: test_expect_success GZIP 'remote tar.gz can be disabled' '\n + \t\t>remote.tar.gz\n + '\n\n - \tstrbuf_release(&cmd);\n - \treturn r;\n ++test_expect_success 'git archive --format=tgz (internal gzip)' '\n ++\ttest_config tar.tgz.command \"git archive gzip\" &&\n ++\tgit archive --format=tgz HEAD >internal_gzip.tgz\n ++'\n ++\n ++test_expect_success 'git archive --format=tar.gz (internal gzip)' '\n ++\ttest_config tar.tar.gz.command \"git archive gzip\" &&\n ++\tgit archive --format=tar.gz HEAD >internal_gzip.tar.gz &&\n ++\ttest_cmp_bin internal_gzip.tgz internal_gzip.tar.gz\n ++'\n ++\n ++test_expect_success GZIP 'extract tgz file (internal gzip)' '\n ++\tgzip -d -c internal_gzip.tar &&\n ++\ttest_cmp_bin b.tar internal_gzip.tar\n ++'\n ++\n + test_expect_success 'archive and :(glob)' '\n + \tgit archive -v HEAD -- \":(glob)**/sh\" >/dev/null 2>actual &&\n + \tcat >expect < 4: af27bea4fc3 archive-tar: use OS_CODE 3 (Unix) for internal gzip\n4: 0e5826e3f25 ! 5: 62038b8e911 archive: use the internal zlib-based gzip compression by default\n @@\n ## Metadata ##\n -Author: Johannes Schindelin \n +Author: René Scharfe \n\n ## Commit message ##\n - archive: use the internal zlib-based gzip compression by default\n + archive-tar: use internal gzip by default\n\n - We just introduced support for compressing `.tar.gz` archives in the\n - `git archive` process itself, using zlib directly instead of spawning\n - `gzip`.\n + Drop the dependency on gzip(1) and use our internal implementation to\n + create tar.gz and tgz files.\n\n - While this takes less CPU time overall, on multi-core machines, this is\n - slightly slower in terms of wall clock time (it seems to be in the\n - ballpark of 15%).\n -\n - It does reduce the number of dependencies by one, though, which makes it\n - desirable to turn that mode on by default.\n -\n - Changing the default benefits most notably the MinGit flavor of Git for\n - Windows (which intends to support 3rd-party applications that want to\n - use Git and want to bundle a minimal set of files for that purpose, i.e.\n - stripping out all non-essential files such as interactive commands,\n - Perl, and yes, also `gzip`).\n -\n - We also can now remove the `GZIP` prerequisite from quite a number of\n - test cases in `t/t5000-tar-tree.sh`.\n -\n - This closes https://github.com/git-for-windows/git/issues/1970\n -\n - Signed-off-by: Johannes Schindelin \n + Signed-off-by: René Scharfe \n + Signed-off-by: Junio C Hamano \n\n ## Documentation/git-archive.txt ##\n @@ Documentation/git-archive.txt: tar..command::\n \tformat is given.\n +\n The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n --`gzip -cn`. You may override them with custom commands.\n -+`:zlib`, triggering an in-process gzip compression. You may override\n -+them with custom commands, e.g. `gzip -cn` or `pigz -cn`.\n +-`gzip -cn`. You may override them with custom commands. An internal gzip\n +-implementation can be used by specifying the value `git archive gzip`.\n ++the magic value `git archive gzip`, which invokes an internal\n ++implementation of gzip. You may override them with custom commands.\n\n tar..remote::\n \tIf true, enable `` for use by remote clients via\n @@ archive-tar.c: void init_tar_archiver(void)\n \tregister_archiver(&tar_archiver);\n\n -\ttar_filter_config(\"tar.tgz.command\", \"gzip -cn\", NULL);\n -+\ttar_filter_config(\"tar.tgz.command\", \":zlib\", NULL);\n ++\ttar_filter_config(\"tar.tgz.command\", internal_gzip_command, NULL);\n \ttar_filter_config(\"tar.tgz.remote\", \"true\", NULL);\n -\ttar_filter_config(\"tar.tar.gz.command\", \"gzip -cn\", NULL);\n -+\ttar_filter_config(\"tar.tar.gz.command\", \":zlib\", NULL);\n ++\ttar_filter_config(\"tar.tar.gz.command\", internal_gzip_command, NULL);\n \ttar_filter_config(\"tar.tar.gz.remote\", \"true\", NULL);\n \tgit_config(git_tar_config, NULL);\n \tfor (i = 0; i < nr_tar_filters; i++) {\n @@ t/t5000-tar-tree.sh: test_expect_success 'only enabled filters are available rem\n \tgit archive --output=j3.tar.gz HEAD &&\n \ttest_cmp_bin j.tgz j3.tar.gz\n '\n -\n -+test_expect_success 'use `archive.tgz.command=:zlib` explicitly' '\n -+\tgit -c archive.tgz.command=:zlib archive --output=j4.tgz HEAD &&\n -+\ttest_cmp_bin j.tgz j4.tgz\n -+'\n -+\n - test_expect_success GZIP 'extract tgz file' '\n - \tgzip -d -c j.tar &&\n +@@ t/t5000-tar-tree.sh: test_expect_success GZIP 'extract tgz file' '\n \ttest_cmp_bin b.tar j.tar\n '\n\n @@ t/t5000-tar-tree.sh: test_expect_success 'only enabled filters are available rem\n \tgit config tar.tar.gz.remote false &&\n \ttest_must_fail git archive --remote=. --format=tar.gz HEAD \\\n \t\t>remote.tar.gz\n + '\n +\n +-test_expect_success 'git archive --format=tgz (internal gzip)' '\n +-\ttest_config tar.tgz.command \"git archive gzip\" &&\n +-\tgit archive --format=tgz HEAD >internal_gzip.tgz\n ++test_expect_success GZIP 'git archive --format=tgz (external gzip)' '\n ++\ttest_config tar.tgz.command \"gzip -cn\" &&\n ++\tgit archive --format=tgz HEAD >external_gzip.tgz\n + '\n +\n +-test_expect_success 'git archive --format=tar.gz (internal gzip)' '\n +-\ttest_config tar.tar.gz.command \"git archive gzip\" &&\n +-\tgit archive --format=tar.gz HEAD >internal_gzip.tar.gz &&\n +-\ttest_cmp_bin internal_gzip.tgz internal_gzip.tar.gz\n ++test_expect_success GZIP 'git archive --format=tar.gz (external gzip)' '\n ++\ttest_config tar.tar.gz.command \"gzip -cn\" &&\n ++\tgit archive --format=tar.gz HEAD >external_gzip.tar.gz &&\n ++\ttest_cmp_bin external_gzip.tgz external_gzip.tar.gz\n + '\n +\n +-test_expect_success GZIP 'extract tgz file (internal gzip)' '\n +-\tgzip -d -c internal_gzip.tar &&\n +-\ttest_cmp_bin b.tar internal_gzip.tar\n ++test_expect_success GZIP 'extract tgz file (external gzip)' '\n ++\tgzip -d -c external_gzip.tar &&\n ++\ttest_cmp_bin b.tar external_gzip.tar\n + '\n +\n + test_expect_success 'archive and :(glob)' '\n-- snap --\n\nAll of these changes look sensible to me, and the performance\nimplications, while at first glance unfavorable because wallclock time\nincreases even as CPU time decreases, are actually quite good. As Peff\nsaid in\nhttps://lore.kernel.org/git/20190501181807.GC4109@sigill.intra.peff.net/t/#u\n\n> [...] whatever has the lowest overall CPU time is generally preferable\n> [...]\n\nBy the way, the main reason why I did not work more is that in\nhttp://madler.net/pipermail/zlib-devel_madler.net/2019-December/003308.html,\nMark Adler (the zlib maintainer) announced that...\n\n> [...] There are many well-tested performance improvements in zlib\n> waiting in the wings that will be incorporated over the next several\n> months. [...]\n\nThis was in December 2019. And now it's June 2022 and I kind of wonder\nwhether those promised improvements will still come.\n\nIn the meantime, however, a viable alternative seems to have cropped up:\nhttps://github.com/zlib-ng/zlib-ng. Essentially, it looks as if it is what\nzlib should have become after above-quoted announcement.\n\nIn particular the CPU intrinsics support (think MMX, SSE2/3, etc) seem to\nbe very interesting and I would not be completely surprised if building\nGit with your patches and linking against zlib-ng would paint a very\nfavorable picture not only in terms of CPU time but also in terms of\nwallclock time. Sadly, I have not been able to set aside time to look into\nthat angle, but maybe I can peak your interest?\n\nThanks,\nDscho\n"},{"id":"457198","messageId":"28f6ec2a-1d94-b29a-4bfd-6a9e74c8edbf@web.de","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.2206141109270.353@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-14T15:47:40Z","receivedAt":"2022-06-14T15:48:20Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 14.06.22 um 13:27 schrieb Johannes Schindelin:\n> Hi Junio,\n>\n> On Mon, 13 Jun 2022, Junio C Hamano wrote:\n>\n>> René Scharfe writes:\n>>\n>>> -test_expect_success GZIP 'git archive --format=tar.gz' '\n>>> +test_expect_success 'git archive --format=tar.gz' '\n>>> \tgit archive --format=tar.gz HEAD >j1.tar.gz &&\n>>> \ttest_cmp_bin j.tgz j1.tar.gz\n>>> '\n>>\n>> Curiously, this breaks for me. It is understandable if we are not\n>> producing byte-for-byte identical output with internal gzip.\n\nMakes sense in retrospect, there's no reason the output of gzip(1) and\nzlib would have to be the same exactly. It just happened to be so on my\nplatform, so the tests deceptively passed for me. I think we simply\nhave to drop those that try to compare compressed files made by\ndifferent tools -- we can still check if their content can be extracted\nand matches.\n\n> Indeed, I can reproduce this, too. In particular, `j.tgz` and `j1.tar.gz`\n> differ like this in my test run:\n>\n> -00000000 1f 8b 08 1a 00 2e ca 09 00 03 04 00 89 45 fc 83 |.............E..|\n> +00000000 1f 8b 08 1a 00 35 2a 10 00 03 04 00 89 45 fc 83 |.....5*......E..|\n>\n> and\n>\n> -00000010 7d fc 00 f1 d0 ec b7 63 8c 30 cc 9b e6 db b6 6d |}......c.0.....m|\n> +00000010 7d fc 00 54 ff ec b7 63 8c 30 cc 9b e6 db b6 6d |}..T...c.0.....m|\n>\n> According to https://datatracker.ietf.org/doc/html/rfc1952#page-5, the\n> difference in the first line is the mtime. For reference, this is the\n> version with `git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz\n> HEAD`:\n>\n> 00000000 1f 8b 08 00 00 00 00 00 00 03 ec b7 63 8c 30 cc |............c.0.|\n>\n> In other words, `gzip` forces the `mtim` member to all zeros, which makes\n> sense.\n\nAnd that's what zlib does as well for me:\n\n $ ./git-archive --format=tgz HEAD | hexdump -C | head -1\n 00000000 1f 8b 08 00 00 00 00 00 00 03 ec bd 59 73 1c 49 |..........?Ys.I|\n\n>\n> The recorded mtimes are a bit funny, according to\n> https://wolf-tungsten.github.io/gzip-analyzer/, they are 1975-03-17\n> 00:36:32 and 1978-08-05 22:45:36, respectively...\n>\n> And the mtime actually changes all the time.\n>\n> What's even more funny: if I comment out the `deflateSetHeader()`, the\n> mtime header field is left at all-zeros. This is on Ubuntu 18.04 with\n> zlib1g 1:1.2.11.dfsg-0ubuntu2.\n>\n> So I dug in a bit deeper and what do you know, the `deflateHeader()`\n> function is implemented like this\n> (https://github.com/madler/zlib/blob/21767c654d31/deflate.c#L557-L565):\n>\n> \tint ZEXPORT deflateSetHeader (strm, head)\n> \t z_streamp strm;\n> \t gz_headerp head;\n> \t{\n> \t if (deflateStateCheck(strm) || strm->state->wrap != 2)\n> \t\treturn Z_STREAM_ERROR;\n> \t strm->state->gzhead = head;\n> \t return Z_OK;\n> \t}\n>\n> Now, the caller is implemented like this:\n>\n> \tstatic void tgz_set_os(git_zstream *strm, int os)\n> \t{\n> \t#if ZLIB_VERNUM >= 0x1221\n> \t\tstruct gz_header_s gzhead = { .os = os };\n> \t\tdeflateSetHeader(&strm->z, &gzhead);\n> \t#endif\n> \t}\n>\n> The biggest problem is not that the return value of `deflateSetHeader()`\n> is ignored. The biggest problem is that it passes the address of a heap\n> variable to the `deflateSetHeader()` function, which then stores it away\n> in another struct that lives beyond the point when we return from\n> `tgz_set_os()`.\n\nAh, you mean the address of an automatic variable on the stack, but I\nget it. D'oh!\n\n>\n> In other words, this is the very issue I pointed out as GCC not catching:\n> https://lore.kernel.org/git/nycvar.QRO.7.76.6.2205272235220.349@tvgsbejvaqbjf.bet/\n>\n> The solution is to move the heap variable back into a scope that matches\n> the lifetime of the compression:\n>\n> -- snip --\n> diff --git a/archive-tar.c b/archive-tar.c\n> index 60669eb7b9c..3d77e0f7509 100644\n> --- a/archive-tar.c\n> +++ b/archive-tar.c\n> @@ -460,17 +460,12 @@ static void tgz_write_block(const void *data)\n>\n> static const char internal_gzip_command[] = \"git archive gzip\";\n>\n> -static void tgz_set_os(git_zstream *strm, int os)\n> -{\n> -#if ZLIB_VERNUM >= 0x1221\n> -\tstruct gz_header_s gzhead = { .os = os };\n> -\tdeflateSetHeader(&strm->z, &gzhead);\n> -#endif\n> -}\n> -\n> static int write_tar_filter_archive(const struct archiver *ar,\n> \t\t\t\t struct archiver_args *args)\n> {\n> +#if ZLIB_VERNUM >= 0x1221\n> +\tstruct gz_header_s gzhead = { .os = 3 }; /* Unix, for reproducibility */\n> +#endif\n> \tstruct strbuf cmd = STRBUF_INIT;\n> \tstruct child_process filter = CHILD_PROCESS_INIT;\n> \tint r;\n> @@ -481,7 +476,10 @@ static int write_tar_filter_archive(const struct archiver *ar,\n> \tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n> \t\twrite_block = tgz_write_block;\n> \t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n> -\t\ttgz_set_os(&gzstream, 3); /* Unix, for reproducibility */\n> +#if ZLIB_VERNUM >= 0x1221\n> +\t\tif (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK)\n> +\t\t\tBUG(\"deflateSetHeader() called too late\");\n> +#endif\n> \t\tgzstream.next_out = outbuf;\n> \t\tgzstream.avail_out = sizeof(outbuf);\n>\n> -- snap --\n\nGood find, thank you! A shorter solution would be to make gzhead static.\n\n>\n> With this, the test passes for me.\n>\n> René, would you mind squashing this into your patch series?\n>\n> Thank you,\n> Dscho\n"},{"id":"457200","messageId":"c67bd455-c643-43e4-3770-7dd891e57b4e@web.de","threadId":"50928","inReplyTo":"28f6ec2a-1d94-b29a-4bfd-6a9e74c8edbf@web.de","subject":"Re: [PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-14T15:56:10Z","receivedAt":"2022-06-14T15:56:38Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 14.06.22 um 17:47 schrieb René Scharfe:\n> Am 14.06.22 um 13:27 schrieb Johannes Schindelin:\n>> Hi Junio,\n>>\n>> On Mon, 13 Jun 2022, Junio C Hamano wrote:\n>>\n>>> René Scharfe writes:\n>>>\n>>>> -test_expect_success GZIP 'git archive --format=tar.gz' '\n>>>> +test_expect_success 'git archive --format=tar.gz' '\n>>>> \tgit archive --format=tar.gz HEAD >j1.tar.gz &&\n>>>> \ttest_cmp_bin j.tgz j1.tar.gz\n>>>> '\n>>>\n>>> Curiously, this breaks for me. It is understandable if we are not\n>>> producing byte-for-byte identical output with internal gzip.\n>\n> Makes sense in retrospect, there's no reason the output of gzip(1) and\n> zlib would have to be the same exactly. It just happened to be so on my\n> platform, so the tests deceptively passed for me. I think we simply\n> have to drop those that try to compare compressed files made by\n> different tools -- we can still check if their content can be extracted\n> and matches.\n\nI have to take that back, I was confused -- the tests are fine. There are no\ncomparisons between gzip-generated and zlib-generated files. There's just\nthe gzhead use-after-return error that Dscho discovered.\n\nRené\n"},{"id":"457202","messageId":"nycvar.QRO.7.76.6.2206141802310.353@tvgsbejvaqbjf.bet","threadId":"50928","inReplyTo":"28f6ec2a-1d94-b29a-4bfd-6a9e74c8edbf@web.de","subject":"Re: [PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-14T16:29:55Z","receivedAt":"2022-06-14T16:30:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi René,\n\nOn Tue, 14 Jun 2022, René Scharfe wrote:\n\n> Am 14.06.22 um 13:27 schrieb Johannes Schindelin:\n>\n> > The solution is to move the heap variable back into a scope that matches\n> > the lifetime of the compression:\n> >\n> > -- snip --\n> > diff --git a/archive-tar.c b/archive-tar.c\n> > index 60669eb7b9c..3d77e0f7509 100644\n> > --- a/archive-tar.c\n> > +++ b/archive-tar.c\n> > @@ -460,17 +460,12 @@ static void tgz_write_block(const void *data)\n> >\n> > static const char internal_gzip_command[] = \"git archive gzip\";\n> >\n> > -static void tgz_set_os(git_zstream *strm, int os)\n> > -{\n> > -#if ZLIB_VERNUM >= 0x1221\n> > -\tstruct gz_header_s gzhead = { .os = os };\n> > -\tdeflateSetHeader(&strm->z, &gzhead);\n> > -#endif\n> > -}\n> > -\n> > static int write_tar_filter_archive(const struct archiver *ar,\n> > \t\t\t\t struct archiver_args *args)\n> > {\n> > +#if ZLIB_VERNUM >= 0x1221\n> > +\tstruct gz_header_s gzhead = { .os = 3 }; /* Unix, for reproducibility */\n> > +#endif\n> > \tstruct strbuf cmd = STRBUF_INIT;\n> > \tstruct child_process filter = CHILD_PROCESS_INIT;\n> > \tint r;\n> > @@ -481,7 +476,10 @@ static int write_tar_filter_archive(const struct archiver *ar,\n> > \tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n> > \t\twrite_block = tgz_write_block;\n> > \t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n> > -\t\ttgz_set_os(&gzstream, 3); /* Unix, for reproducibility */\n> > +#if ZLIB_VERNUM >= 0x1221\n> > +\t\tif (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK)\n> > +\t\t\tBUG(\"deflateSetHeader() called too late\");\n> > +#endif\n> > \t\tgzstream.next_out = outbuf;\n> > \t\tgzstream.avail_out = sizeof(outbuf);\n> >\n> > -- snap --\n>\n> Good find, thank you! A shorter solution would be to make gzhead static.\n\nI should have said that I had considered this, but decided against it\nbecause it would introduce yet another issue: it would render the code\nneedlessly un-multi-threadable. And that can be avoided _really_ easily.\n\nCiao,\nDscho\n"},{"id":"457215","messageId":"4b4da5b3-7898-e97b-af74-a6874c8cb7e2@web.de","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.2206141802310.353@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-14T20:04:59Z","receivedAt":"2022-06-14T20:05:35Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 14.06.22 um 18:29 schrieb Johannes Schindelin:\n> Hi René,\n>\n> On Tue, 14 Jun 2022, René Scharfe wrote:\n>\n>> Am 14.06.22 um 13:27 schrieb Johannes Schindelin:\n>>\n>>> The solution is to move the heap variable back into a scope that matches\n>>> the lifetime of the compression:\n>>>\n>>> -- snip --\n>>> diff --git a/archive-tar.c b/archive-tar.c\n>>> index 60669eb7b9c..3d77e0f7509 100644\n>>> --- a/archive-tar.c\n>>> +++ b/archive-tar.c\n>>> @@ -460,17 +460,12 @@ static void tgz_write_block(const void *data)\n>>>\n>>> static const char internal_gzip_command[] = \"git archive gzip\";\n>>>\n>>> -static void tgz_set_os(git_zstream *strm, int os)\n>>> -{\n>>> -#if ZLIB_VERNUM >= 0x1221\n>>> -\tstruct gz_header_s gzhead = { .os = os };\n>>> -\tdeflateSetHeader(&strm->z, &gzhead);\n>>> -#endif\n>>> -}\n>>> -\n>>> static int write_tar_filter_archive(const struct archiver *ar,\n>>> \t\t\t\t struct archiver_args *args)\n>>> {\n>>> +#if ZLIB_VERNUM >= 0x1221\n>>> +\tstruct gz_header_s gzhead = { .os = 3 }; /* Unix, for reproducibility */\n>>> +#endif\n>>> \tstruct strbuf cmd = STRBUF_INIT;\n>>> \tstruct child_process filter = CHILD_PROCESS_INIT;\n>>> \tint r;\n>>> @@ -481,7 +476,10 @@ static int write_tar_filter_archive(const struct archiver *ar,\n>>> \tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n>>> \t\twrite_block = tgz_write_block;\n>>> \t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n>>> -\t\ttgz_set_os(&gzstream, 3); /* Unix, for reproducibility */\n>>> +#if ZLIB_VERNUM >= 0x1221\n>>> +\t\tif (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK)\n>>> +\t\t\tBUG(\"deflateSetHeader() called too late\");\n>>> +#endif\n>>> \t\tgzstream.next_out = outbuf;\n>>> \t\tgzstream.avail_out = sizeof(outbuf);\n>>>\n>>> -- snap --\n>>\n>> Good find, thank you! A shorter solution would be to make gzhead static.\n>\n> I should have said that I had considered this, but decided against it\n> because it would introduce yet another issue: it would render the code\n> needlessly un-multi-threadable. And that can be avoided _really_ easily.\n\narchive-tar.c (and archive-zip.c) use other static variables, so a\nstatic gzhead won't break or block anything in this regard. There was\nno interest in running it in parallel threads so far AFAIK, and it's\nhard for me to imagine the usefulness of creating multiple .tgz files at\nthe same time.\n\nThe doubled ZLIB_VERNUM is unsightly and I'm not sure the BUG check is\nuseful -- I omitted error checking because there is no recurse for us if\ndeflateSetHeader() doesn't work, and on ancient zlib versions we\nsilently continue anyway.\n\nBut that's all just minor quibbling -- I'll include your changes in the\nnext version, they look fine overall.\n\nRené\n"},{"id":"457216","messageId":"0aa5c101-06bf-325c-efbc-6b4ef38616c5@web.de","threadId":"50928","inReplyTo":"nycvar.QRO.7.76.6.2206141043150.353@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 0/5] Avoid spawning gzip in git archive","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-14T20:05:07Z","receivedAt":"2022-06-14T20:05:39Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 14.06.22 um 13:28 schrieb Johannes Schindelin:\n>\n> By the way, the main reason why I did not work more is that in\n> http://madler.net/pipermail/zlib-devel_madler.net/2019-December/003308.html,\n> Mark Adler (the zlib maintainer) announced that...\n>\n>> [...] There are many well-tested performance improvements in zlib\n>> waiting in the wings that will be incorporated over the next several\n>> months. [...]\n>\n> This was in December 2019. And now it's June 2022 and I kind of wonder\n> whether those promised improvements will still come.\n>\n> In the meantime, however, a viable alternative seems to have cropped up:\n> https://github.com/zlib-ng/zlib-ng. Essentially, it looks as if it is what\n> zlib should have become after above-quoted announcement.\n>\n> In particular the CPU intrinsics support (think MMX, SSE2/3, etc) seem to\n> be very interesting and I would not be completely surprised if building\n> Git with your patches and linking against zlib-ng would paint a very\n> favorable picture not only in terms of CPU time but also in terms of\n> wallclock time. Sadly, I have not been able to set aside time to look into\n> that angle, but maybe I can peak your interest?\nI was unable to preload zlib-ng using DYLD_INSERT_LIBRARIES on macOS\n12.4 so far. The included demo proggy looks impressive, though:\n\n$ hyperfine -w3 -L gzip gzip,../zlib-ng/minigzip \"git -C ../linux archive --format=tar HEAD | {gzip} -c\"\nBenchmark #1: git -C ../linux archive --format=tar HEAD | gzip -c\n Time (mean ± σ): 20.424 s ± 0.006 s [User: 23.964 s, System: 0.432 s]\n Range (min … max): 20.414 s … 20.434 s 10 runs\n\nBenchmark #2: git -C ../linux archive --format=tar HEAD | ../zlib-ng/minigzip -c\n Time (mean ± σ): 12.158 s ± 0.006 s [User: 13.908 s, System: 0.376 s]\n Range (min … max): 12.145 s … 12.166 s 10 runs\n\nSummary\n 'git -C ../linux archive --format=tar HEAD | ../zlib-ng/minigzip -c' ran\n 1.68 ± 0.00 times faster than 'git -C ../linux archive --format=tar HEAD | gzip -c'\n"},{"id":"457288","messageId":"xmqqletx2826.fsf@gitster.g","threadId":"50928","inReplyTo":"4b4da5b3-7898-e97b-af74-a6874c8cb7e2@web.de","subject":"Re: [PATCH v3 5/5] archive-tar: use internal gzip by default","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-15T16:41:21Z","receivedAt":"2022-06-15T16:41:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> archive-tar.c (and archive-zip.c) use other static variables, so a\n> static gzhead won't break or block anything in this regard. There was\n> no interest in running it in parallel threads so far AFAIK, and it's\n> hard for me to imagine the usefulness of creating multiple .tgz files at\n> the same time.\n\n;-) FWIW I had exactly the same reaction. \n\nIf this were a lot isolated piece of helper function, perhaps, but\nno, reentrancy for this helper function specifically written for\nthis code path is not a very good argument.\n\nThe code structure the (not very good) argument tried to suport,\nhowever, is a good thing to have regardless, and if the code is\nwritten in such a way from the beginning, there is no reason to\nreject it. If it opens the door to a unified way to deal with all\nthe other static global variables (e.g. have a \"archiver_state\"\nstructure that collects all of them, with this one included, and\npass it around, or something), that would be great.\n\nShort of that, I do not care too much either way. As this topic is\nnot even in 'next', \"once written in one way, it is not worth the\ncode churn to rewrite it in the other way, when these two ways are\nboth reasonable\" does not yet apply here, so ...\n\n> But that's all just minor quibbling -- I'll include your changes in the\n> next version, they look fine overall.\n\n... that's fine, too.\n\nThanks, both.\n"},{"id":"457290","messageId":"9df761c3-355a-ede9-7971-b32687fe9abb@web.de","threadId":"50928","inReplyTo":"pull.145.git.gitgitgadget@gmail.com","subject":"[PATCH v4 0/6] Avoid spawning gzip in git archive","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-15T16:53:32Z","receivedAt":"2022-06-15T16:54:27Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Changes since v3:\n- Use deflateSetHeader() correctly, thanks to Dscho.\n- New patch to update the format-related documentation.\n\n archive: update format documentation\n archive: rename archiver data field to filter_command\n archive-tar: factor out write_block()\n archive-tar: add internal gzip implementation\n archive-tar: use OS_CODE 3 (Unix) for internal gzip\n archive-tar: use internal gzip by default\n\n Documentation/git-archive.txt | 21 +++++-----\n archive-tar.c | 77 ++++++++++++++++++++++++++++++-----\n archive.h | 2 +-\n t/t5000-tar-tree.sh | 28 ++++++++++---\n 4 files changed, 100 insertions(+), 28 deletions(-)\n\nRange-Diff vs. v3:\n-: ---------- > 1: 67369ed452 archive: update format documentation\n1: 73ccd190bd = 2: 6a7cce50ef archive: rename archiver data field to filter_command\n2: 352cff7163 = 3: c86e82bee8 archive-tar: factor out write_block()\n3: 4e7cf97631 ! 4: 6196b0e39d archive-tar: add internal gzip implementation\n @@ Commit message\n\n ## Documentation/git-archive.txt ##\n @@ Documentation/git-archive.txt: tar..command::\n - \tformat is given.\n + \tto the command (e.g., `-9`).\n +\n - The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n --`gzip -cn`. You may override them with custom commands.\n -+`gzip -cn`. You may override them with custom commands. An internal gzip\n -+implementation can be used by specifying the value `git archive gzip`.\n + The `tar.gz` and `tgz` formats are defined automatically and use the\n +-command `gzip -cn` by default.\n ++command `gzip -cn` by default. An internal gzip implementation can be\n ++used by specifying the value `git archive gzip`.\n\n tar..remote::\n - \tIf true, enable `` for use by remote clients via\n + \tIf true, enable the format for use by remote clients via\n\n ## archive-tar.c ##\n @@ archive-tar.c: static int write_tar_filter_archive(const struct archiver *ar,\n4: cb2bbe9f6d < -: ---------- archive-tar: use OS_CODE 3 (Unix) for internal gzip\n-: ---------- > 5: 19d286af6a archive-tar: use OS_CODE 3 (Unix) for internal gzip\n5: 5dd968ced1 ! 6: 74683137af archive-tar: use internal gzip by default\n @@ Commit message\n\n ## Documentation/git-archive.txt ##\n @@ Documentation/git-archive.txt: tar..command::\n - \tformat is given.\n + \tto the command (e.g., `-9`).\n +\n - The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n --`gzip -cn`. You may override them with custom commands. An internal gzip\n --implementation can be used by specifying the value `git archive gzip`.\n -+the magic value `git archive gzip`, which invokes an internal\n -+implementation of gzip. You may override them with custom commands.\n + The `tar.gz` and `tgz` formats are defined automatically and use the\n +-command `gzip -cn` by default. An internal gzip implementation can be\n +-used by specifying the value `git archive gzip`.\n ++magic command `git archive gzip` by default, which invokes an internal\n ++implementation of gzip.\n\n tar..remote::\n - \tIf true, enable `` for use by remote clients via\n + \tIf true, enable the format for use by remote clients via\n\n ## archive-tar.c ##\n @@ archive-tar.c: void init_tar_archiver(void)\n--\n2.36.1\n"},{"id":"457292","messageId":"56205881-ccb1-165b-889e-3aa641117981@web.de","threadId":"50928","inReplyTo":"9df761c3-355a-ede9-7971-b32687fe9abb@web.de","subject":"[PATCH v4 1/6] archive: update format documentation","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-15T16:58:38Z","receivedAt":"2022-06-15T16:59:21Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Mention all formats in the --format section, use backtick quoting for\nliteral values throughout, clarify the description of the configuration\noption.\n\nHelped-by: Junio C Hamano \nSigned-off-by: René Scharfe \n---\n Documentation/git-archive.txt | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\nindex 56989a2f34..ff3f7b0344 100644\n--- a/Documentation/git-archive.txt\n+++ b/Documentation/git-archive.txt\n@@ -34,10 +34,12 @@ OPTIONS\n -------\n\n --format=::\n-\tFormat of the resulting archive: 'tar' or 'zip'. If this option\n+\tFormat of the resulting archive. Possible values are `tar`,\n+\t`zip`, `tar.gz`, `tgz`, and any format defined using the\n+\tconfiguration option `tar..command`. If `--format`\n \tis not given, and the output file is specified, the format is\n-\tinferred from the filename if possible (e.g. writing to \"foo.zip\"\n-\tmakes the output to be in the zip format). Otherwise the output\n+\tinferred from the filename if possible (e.g. writing to `foo.zip`\n+\tmakes the output to be in the `zip` format). Otherwise the output\n \tformat is `tar`.\n\n -l::\n@@ -143,17 +145,15 @@ tar..command::\n \tis executed using the shell with the generated tar file on its\n \tstandard input, and should produce the final output on its\n \tstandard output. Any compression-level options will be passed\n-\tto the command (e.g., \"-9\"). An output file with the same\n-\textension as `` will be use this format if no other\n-\tformat is given.\n+\tto the command (e.g., `-9`).\n +\n-The \"tar.gz\" and \"tgz\" formats are defined automatically and default to\n-`gzip -cn`. You may override them with custom commands.\n+The `tar.gz` and `tgz` formats are defined automatically and use the\n+command `gzip -cn` by default.\n\n tar..remote::\n-\tIf true, enable `` for use by remote clients via\n+\tIf true, enable the format for use by remote clients via\n \tlinkgit:git-upload-archive[1]. Defaults to false for\n-\tuser-defined formats, but true for the \"tar.gz\" and \"tgz\"\n+\tuser-defined formats, but true for the `tar.gz` and `tgz`\n \tformats.\n\n [[ATTRIBUTES]]\n--\n2.36.1\n"},{"id":"457293","messageId":"00e5da9f-a3fb-8a5b-ad29-4016873b0e90@web.de","threadId":"50928","inReplyTo":"9df761c3-355a-ede9-7971-b32687fe9abb@web.de","subject":"[PATCH v4 2/6] archive: rename archiver data field to filter_command","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-15T16:59:57Z","receivedAt":"2022-06-15T17:00:56Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"The void pointer \"data\" in struct archiver is only used to store filter\ncommands to pass tar archives to, like gzip. Rename it accordingly and\nalso turn it into a char pointer to document the fact that it's a string\nreference.\n\nSigned-off-by: René Scharfe \n---\n archive-tar.c | 10 +++++-----\n archive.h | 2 +-\n 2 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 042feb66d2..2717e34a1d 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -383,8 +383,8 @@ static int tar_filter_config(const char *var, const char *value, void *data)\n \tif (!strcmp(type, \"command\")) {\n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n-\t\tfree(ar->data);\n-\t\tar->data = xstrdup(value);\n+\t\tfree(ar->filter_command);\n+\t\tar->filter_command = xstrdup(value);\n \t\treturn 0;\n \t}\n \tif (!strcmp(type, \"remote\")) {\n@@ -432,10 +432,10 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tstruct child_process filter = CHILD_PROCESS_INIT;\n \tint r;\n\n-\tif (!ar->data)\n+\tif (!ar->filter_command)\n \t\tBUG(\"tar-filter archiver called with no filter defined\");\n\n-\tstrbuf_addstr(&cmd, ar->data);\n+\tstrbuf_addstr(&cmd, ar->filter_command);\n \tif (args->compression_level >= 0)\n \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\n\n@@ -478,7 +478,7 @@ void init_tar_archiver(void)\n \tgit_config(git_tar_config, NULL);\n \tfor (i = 0; i < nr_tar_filters; i++) {\n \t\t/* omit any filters that never had a command configured */\n-\t\tif (tar_filters[i]->data)\n+\t\tif (tar_filters[i]->filter_command)\n \t\t\tregister_archiver(tar_filters[i]);\n \t}\n }\ndiff --git a/archive.h b/archive.h\nindex 49fab71aaf..08bed3ed3a 100644\n--- a/archive.h\n+++ b/archive.h\n@@ -43,7 +43,7 @@ struct archiver {\n \tconst char *name;\n \tint (*write_archive)(const struct archiver *, struct archiver_args *);\n \tunsigned flags;\n-\tvoid *data;\n+\tchar *filter_command;\n };\n void register_archiver(struct archiver *);\n\n--\n2.36.1\n"},{"id":"457294","messageId":"2415cea0-47c4-b63f-7ef8-1d8f3e08d523@web.de","threadId":"50928","inReplyTo":"9df761c3-355a-ede9-7971-b32687fe9abb@web.de","subject":"[PATCH v4 3/6] archive-tar: factor out write_block()","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-15T17:01:14Z","receivedAt":"2022-06-15T17:01:57Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"All tar archive writes have the same size and are done to the same file\ndescriptor. Move them to a common function, write_block(), to reduce\ncode duplication and make it easy to change the destination.\n\nOriginal-patch-by: Rohit Ashiwal \nSigned-off-by: René Scharfe \n---\n archive-tar.c | 13 +++++++++----\n 1 file changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 2717e34a1d..4e6a3deb80 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -38,11 +38,16 @@ static int write_tar_filter_archive(const struct archiver *ar,\n #define USTAR_MAX_MTIME 077777777777ULL\n #endif\n\n+static void write_block(const void *buf)\n+{\n+\twrite_or_die(1, buf, BLOCKSIZE);\n+}\n+\n /* writes out the whole block, but only if it is full */\n static void write_if_needed(void)\n {\n \tif (offset == BLOCKSIZE) {\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_block(block);\n \t\toffset = 0;\n \t}\n }\n@@ -66,7 +71,7 @@ static void do_write_blocked(const void *data, unsigned long size)\n \t\twrite_if_needed();\n \t}\n \twhile (size >= BLOCKSIZE) {\n-\t\twrite_or_die(1, buf, BLOCKSIZE);\n+\t\twrite_block(buf);\n \t\tsize -= BLOCKSIZE;\n \t\tbuf += BLOCKSIZE;\n \t}\n@@ -101,10 +106,10 @@ static void write_trailer(void)\n {\n \tint tail = BLOCKSIZE - offset;\n \tmemset(block + offset, 0, tail);\n-\twrite_or_die(1, block, BLOCKSIZE);\n+\twrite_block(block);\n \tif (tail < 2 * RECORDSIZE) {\n \t\tmemset(block, 0, offset);\n-\t\twrite_or_die(1, block, BLOCKSIZE);\n+\t\twrite_block(block);\n \t}\n }\n\n--\n2.36.1\n"},{"id":"457295","messageId":"1328fe72-1a27-b214-c226-d239099be673@web.de","threadId":"50928","inReplyTo":"9df761c3-355a-ede9-7971-b32687fe9abb@web.de","subject":"[PATCH v4 4/6] archive-tar: add internal gzip implementation","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-15T17:02:33Z","receivedAt":"2022-06-15T17:02:51Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Git uses zlib for its own object store, but calls gzip when creating tgz\narchives. Add an option to perform the gzip compression for the latter\nusing zlib, without depending on the external gzip binary.\n\nPlug it in by making write_block a function pointer and switching to a\ncompressing variant if the filter command has the magic value \"git\narchive gzip\". Does that indirection slow down tar creation? Not\nreally, at least not in this test:\n\n$ hyperfine -w3 -L rev HEAD,origin/main -p 'git checkout {rev} && make' \\\n'./git -C ../linux archive --format=tar HEAD # {rev}'\nBenchmark #1: ./git -C ../linux archive --format=tar HEAD # HEAD\n Time (mean ± σ): 4.044 s ± 0.007 s [User: 3.901 s, System: 0.137 s]\n Range (min … max): 4.038 s … 4.059 s 10 runs\n\nBenchmark #2: ./git -C ../linux archive --format=tar HEAD # origin/main\n Time (mean ± σ): 4.047 s ± 0.009 s [User: 3.903 s, System: 0.138 s]\n Range (min … max): 4.038 s … 4.066 s 10 runs\n\nHow does tgz creation perform?\n\n$ hyperfine -w3 -L command 'gzip -cn','git archive gzip' \\\n'./git -c tar.tgz.command=\"{command}\" -C ../linux archive --format=tgz HEAD'\nBenchmark #1: ./git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD\n Time (mean ± σ): 20.404 s ± 0.006 s [User: 23.943 s, System: 0.401 s]\n Range (min … max): 20.395 s … 20.414 s 10 runs\n\nBenchmark #2: ./git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD\n Time (mean ± σ): 23.807 s ± 0.023 s [User: 23.655 s, System: 0.145 s]\n Range (min … max): 23.782 s … 23.857 s 10 runs\n\nSummary\n './git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD' ran\n 1.17 ± 0.00 times faster than './git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD'\n\nSo the internal implementation takes 17% longer on the Linux repo, but\nuses 2% less CPU time. That's because the external gzip can run in\nparallel on its own processor, while the internal one works sequentially\nand avoids the inter-process communication overhead.\n\nWhat are the benefits? Only an internal sequential implementation can\noffer this eco mode, and it allows avoiding the gzip(1) requirement.\n\nThis implementation uses the helper functions from our zlib.c instead of\nthe convenient gz* functions from zlib, because the latter doesn't give\nthe control over the generated gzip header that the next patch requires.\n\nOriginal-patch-by: Rohit Ashiwal \nSigned-off-by: René Scharfe \n---\n Documentation/git-archive.txt | 3 ++-\n archive-tar.c | 45 ++++++++++++++++++++++++++++++++++-\n t/t5000-tar-tree.sh | 16 +++++++++++++\n 3 files changed, 62 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\nindex ff3f7b0344..b2d1b63d31 100644\n--- a/Documentation/git-archive.txt\n+++ b/Documentation/git-archive.txt\n@@ -148,7 +148,8 @@ tar..command::\n \tto the command (e.g., `-9`).\n +\n The `tar.gz` and `tgz` formats are defined automatically and use the\n-command `gzip -cn` by default.\n+command `gzip -cn` by default. An internal gzip implementation can be\n+used by specifying the value `git archive gzip`.\n\n tar..remote::\n \tIf true, enable the format for use by remote clients via\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 4e6a3deb80..53d0ef685c 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -38,11 +38,13 @@ static int write_tar_filter_archive(const struct archiver *ar,\n #define USTAR_MAX_MTIME 077777777777ULL\n #endif\n\n-static void write_block(const void *buf)\n+static void tar_write_block(const void *buf)\n {\n \twrite_or_die(1, buf, BLOCKSIZE);\n }\n\n+static void (*write_block)(const void *) = tar_write_block;\n+\n /* writes out the whole block, but only if it is full */\n static void write_if_needed(void)\n {\n@@ -430,6 +432,34 @@ static int write_tar_archive(const struct archiver *ar,\n \treturn err;\n }\n\n+static git_zstream gzstream;\n+static unsigned char outbuf[16384];\n+\n+static void tgz_deflate(int flush)\n+{\n+\twhile (gzstream.avail_in || flush == Z_FINISH) {\n+\t\tint status = git_deflate(&gzstream, flush);\n+\t\tif (!gzstream.avail_out || status == Z_STREAM_END) {\n+\t\t\twrite_or_die(1, outbuf, gzstream.next_out - outbuf);\n+\t\t\tgzstream.next_out = outbuf;\n+\t\t\tgzstream.avail_out = sizeof(outbuf);\n+\t\t\tif (status == Z_STREAM_END)\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR)\n+\t\t\tdie(_(\"deflate error (%d)\"), status);\n+\t}\n+}\n+\n+static void tgz_write_block(const void *data)\n+{\n+\tgzstream.next_in = (void *)data;\n+\tgzstream.avail_in = BLOCKSIZE;\n+\ttgz_deflate(Z_NO_FLUSH);\n+}\n+\n+static const char internal_gzip_command[] = \"git archive gzip\";\n+\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args)\n {\n@@ -440,6 +470,19 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!ar->filter_command)\n \t\tBUG(\"tar-filter archiver called with no filter defined\");\n\n+\tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n+\t\twrite_block = tgz_write_block;\n+\t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n+\t\tgzstream.next_out = outbuf;\n+\t\tgzstream.avail_out = sizeof(outbuf);\n+\n+\t\tr = write_tar_archive(ar, args);\n+\n+\t\ttgz_deflate(Z_FINISH);\n+\t\tgit_deflate_end(&gzstream);\n+\t\treturn r;\n+\t}\n+\n \tstrbuf_addstr(&cmd, ar->filter_command);\n \tif (args->compression_level >= 0)\n \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\ndiff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh\nindex 7f8d2ab0a7..9ac0ec67fe 100755\n--- a/t/t5000-tar-tree.sh\n+++ b/t/t5000-tar-tree.sh\n@@ -374,6 +374,22 @@ test_expect_success GZIP 'remote tar.gz can be disabled' '\n \t\t>remote.tar.gz\n '\n\n+test_expect_success 'git archive --format=tgz (internal gzip)' '\n+\ttest_config tar.tgz.command \"git archive gzip\" &&\n+\tgit archive --format=tgz HEAD >internal_gzip.tgz\n+'\n+\n+test_expect_success 'git archive --format=tar.gz (internal gzip)' '\n+\ttest_config tar.tar.gz.command \"git archive gzip\" &&\n+\tgit archive --format=tar.gz HEAD >internal_gzip.tar.gz &&\n+\ttest_cmp_bin internal_gzip.tgz internal_gzip.tar.gz\n+'\n+\n+test_expect_success GZIP 'extract tgz file (internal gzip)' '\n+\tgzip -d -c internal_gzip.tar &&\n+\ttest_cmp_bin b.tar internal_gzip.tar\n+'\n+\n test_expect_success 'archive and :(glob)' '\n \tgit archive -v HEAD -- \":(glob)**/sh\" >/dev/null 2>actual &&\n \tcat >expect <\nSigned-off-by: René Scharfe \n---\n archive-tar.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 53d0ef685c..efba78118b 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -463,6 +463,9 @@ static const char internal_gzip_command[] = \"git archive gzip\";\n static int write_tar_filter_archive(const struct archiver *ar,\n \t\t\t\t struct archiver_args *args)\n {\n+#if ZLIB_VERNUM >= 0x1221\n+\tstruct gz_header_s gzhead = { .os = 3 }; /* Unix, for reproducibility */\n+#endif\n \tstruct strbuf cmd = STRBUF_INIT;\n \tstruct child_process filter = CHILD_PROCESS_INIT;\n \tint r;\n@@ -473,6 +476,10 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n \t\twrite_block = tgz_write_block;\n \t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n+#if ZLIB_VERNUM >= 0x1221\n+\t\tif (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK)\n+\t\t\tBUG(\"deflateSetHeader() called too late\");\n+#endif\n \t\tgzstream.next_out = outbuf;\n \t\tgzstream.avail_out = sizeof(outbuf);\n\n--\n2.36.1\n"},{"id":"457297","messageId":"d0c2281e-1860-0808-8e80-9881d7d5cd63@web.de","threadId":"50928","inReplyTo":"9df761c3-355a-ede9-7971-b32687fe9abb@web.de","subject":"[PATCH v4 6/6] archive-tar: use internal gzip by default","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-15T17:05:03Z","receivedAt":"2022-06-15T17:05:23Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Drop the dependency on gzip(1) and use our internal implementation to\ncreate tar.gz and tgz files.\n\nSigned-off-by: René Scharfe \n---\n Documentation/git-archive.txt | 4 ++--\n archive-tar.c | 4 ++--\n t/t5000-tar-tree.sh | 32 ++++++++++++++++----------------\n 3 files changed, 20 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt\nindex b2d1b63d31..60c040988b 100644\n--- a/Documentation/git-archive.txt\n+++ b/Documentation/git-archive.txt\n@@ -148,8 +148,8 @@ tar..command::\n \tto the command (e.g., `-9`).\n +\n The `tar.gz` and `tgz` formats are defined automatically and use the\n-command `gzip -cn` by default. An internal gzip implementation can be\n-used by specifying the value `git archive gzip`.\n+magic command `git archive gzip` by default, which invokes an internal\n+implementation of gzip.\n\n tar..remote::\n \tIf true, enable the format for use by remote clients via\ndiff --git a/archive-tar.c b/archive-tar.c\nindex efba78118b..3d77e0f750 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -526,9 +526,9 @@ void init_tar_archiver(void)\n \tint i;\n \tregister_archiver(&tar_archiver);\n\n-\ttar_filter_config(\"tar.tgz.command\", \"gzip -cn\", NULL);\n+\ttar_filter_config(\"tar.tgz.command\", internal_gzip_command, NULL);\n \ttar_filter_config(\"tar.tgz.remote\", \"true\", NULL);\n-\ttar_filter_config(\"tar.tar.gz.command\", \"gzip -cn\", NULL);\n+\ttar_filter_config(\"tar.tar.gz.command\", internal_gzip_command, NULL);\n \ttar_filter_config(\"tar.tar.gz.remote\", \"true\", NULL);\n \tgit_config(git_tar_config, NULL);\n \tfor (i = 0; i < nr_tar_filters; i++) {\ndiff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh\nindex 9ac0ec67fe..1a68e89a55 100755\n--- a/t/t5000-tar-tree.sh\n+++ b/t/t5000-tar-tree.sh\n@@ -339,21 +339,21 @@ test_expect_success 'only enabled filters are available remotely' '\n \ttest_cmp_bin remote.bar config.bar\n '\n\n-test_expect_success GZIP 'git archive --format=tgz' '\n+test_expect_success 'git archive --format=tgz' '\n \tgit archive --format=tgz HEAD >j.tgz\n '\n\n-test_expect_success GZIP 'git archive --format=tar.gz' '\n+test_expect_success 'git archive --format=tar.gz' '\n \tgit archive --format=tar.gz HEAD >j1.tar.gz &&\n \ttest_cmp_bin j.tgz j1.tar.gz\n '\n\n-test_expect_success GZIP 'infer tgz from .tgz filename' '\n+test_expect_success 'infer tgz from .tgz filename' '\n \tgit archive --output=j2.tgz HEAD &&\n \ttest_cmp_bin j.tgz j2.tgz\n '\n\n-test_expect_success GZIP 'infer tgz from .tar.gz filename' '\n+test_expect_success 'infer tgz from .tar.gz filename' '\n \tgit archive --output=j3.tar.gz HEAD &&\n \ttest_cmp_bin j.tgz j3.tar.gz\n '\n@@ -363,31 +363,31 @@ test_expect_success GZIP 'extract tgz file' '\n \ttest_cmp_bin b.tar j.tar\n '\n\n-test_expect_success GZIP 'remote tar.gz is allowed by default' '\n+test_expect_success 'remote tar.gz is allowed by default' '\n \tgit archive --remote=. --format=tar.gz HEAD >remote.tar.gz &&\n \ttest_cmp_bin j.tgz remote.tar.gz\n '\n\n-test_expect_success GZIP 'remote tar.gz can be disabled' '\n+test_expect_success 'remote tar.gz can be disabled' '\n \tgit config tar.tar.gz.remote false &&\n \ttest_must_fail git archive --remote=. --format=tar.gz HEAD \\\n \t\t>remote.tar.gz\n '\n\n-test_expect_success 'git archive --format=tgz (internal gzip)' '\n-\ttest_config tar.tgz.command \"git archive gzip\" &&\n-\tgit archive --format=tgz HEAD >internal_gzip.tgz\n+test_expect_success GZIP 'git archive --format=tgz (external gzip)' '\n+\ttest_config tar.tgz.command \"gzip -cn\" &&\n+\tgit archive --format=tgz HEAD >external_gzip.tgz\n '\n\n-test_expect_success 'git archive --format=tar.gz (internal gzip)' '\n-\ttest_config tar.tar.gz.command \"git archive gzip\" &&\n-\tgit archive --format=tar.gz HEAD >internal_gzip.tar.gz &&\n-\ttest_cmp_bin internal_gzip.tgz internal_gzip.tar.gz\n+test_expect_success GZIP 'git archive --format=tar.gz (external gzip)' '\n+\ttest_config tar.tar.gz.command \"gzip -cn\" &&\n+\tgit archive --format=tar.gz HEAD >external_gzip.tar.gz &&\n+\ttest_cmp_bin external_gzip.tgz external_gzip.tar.gz\n '\n\n-test_expect_success GZIP 'extract tgz file (internal gzip)' '\n-\tgzip -d -c internal_gzip.tar &&\n-\ttest_cmp_bin b.tar internal_gzip.tar\n+test_expect_success GZIP 'extract tgz file (external gzip)' '\n+\tgzip -d -c external_gzip.tar &&\n+\ttest_cmp_bin b.tar external_gzip.tar\n '\n\n test_expect_success 'archive and :(glob)' '\n--\n2.36.1\n"},{"id":"457315","messageId":"220615.86wndhwt9a.gmgdl@evledraar.gmail.com","threadId":"50928","inReplyTo":"1328fe72-1a27-b214-c226-d239099be673@web.de","subject":"Re: [PATCH v4 4/6] archive-tar: add internal gzip implementation","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-15T20:32:04Z","receivedAt":"2022-06-15T20:45:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 15 2022, René Scharfe wrote:\n\n> Git uses zlib for its own object store, but calls gzip when creating tgz\n> archives. Add an option to perform the gzip compression for the latter\n> using zlib, without depending on the external gzip binary.\n>\n> Plug it in by making write_block a function pointer and switching to a\n> compressing variant if the filter command has the magic value \"git\n> archive gzip\". Does that indirection slow down tar creation? Not\n> really, at least not in this test:\n>\n> $ hyperfine -w3 -L rev HEAD,origin/main -p 'git checkout {rev} && make' \\\n> './git -C ../linux archive --format=tar HEAD # {rev}'\n\nShameless plug: https://lore.kernel.org/git/211201.86r1aw9gbd.gmgdl@evledraar.gmail.com/\n\nI.e. a \"hyperfine\" wrapper I wrote to make exactly this sort of thing\neasier.\n\nYou'll find that you need less or no --warmup with it, since the\ncheckout flip-flopping and re-making (and resulting FS and other cache\neviction) will go away, as we'll use different \"git worktree\"'s for the\ntwo \"rev\".\n\n(Also, putting those on a ramdisk really helps)\n\n> Benchmark #1: ./git -C ../linux archive --format=tar HEAD # HEAD\n> Time (mean ± σ): 4.044 s ± 0.007 s [User: 3.901 s, System: 0.137 s]\n> Range (min … max): 4.038 s … 4.059 s 10 runs\n>\n> Benchmark #2: ./git -C ../linux archive --format=tar HEAD # origin/main\n> Time (mean ± σ): 4.047 s ± 0.009 s [User: 3.903 s, System: 0.138 s]\n> Range (min … max): 4.038 s … 4.066 s 10 runs\n>\n> How does tgz creation perform?\n>\n> $ hyperfine -w3 -L command 'gzip -cn','git archive gzip' \\\n> './git -c tar.tgz.command=\"{command}\" -C ../linux archive --format=tgz HEAD'\n> Benchmark #1: ./git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD\n> Time (mean ± σ): 20.404 s ± 0.006 s [User: 23.943 s, System: 0.401 s]\n> Range (min … max): 20.395 s … 20.414 s 10 runs\n>\n> Benchmark #2: ./git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD\n> Time (mean ± σ): 23.807 s ± 0.023 s [User: 23.655 s, System: 0.145 s]\n> Range (min … max): 23.782 s … 23.857 s 10 runs\n>\n> Summary\n> './git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD' ran\n> 1.17 ± 0.00 times faster than './git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD'\n>\n> So the internal implementation takes 17% longer on the Linux repo, but\n> uses 2% less CPU time. That's because the external gzip can run in\n> parallel on its own processor, while the internal one works sequentially\n> and avoids the inter-process communication overhead.\n>\n> What are the benefits? Only an internal sequential implementation can\n> offer this eco mode, and it allows avoiding the gzip(1) requirement.\n\nI had been keeping one eye on this series, but didn't look at it in any\ndetail.\n\nI found this after reading 6/6, which I think in any case could really\nuse some \"why\" summary, which seems to mostly be covered here.\n\nI.e. it's unclear if the \"drop the dependency on gzip(1)\" in 6/6 is a\nreference to the GZIP test dependency, or that our users are unlikely to\nhave \"gzip(1)\" on their systems.\n\nIf it's the latter I'd much rather (as a user) take a 17% wallclock\nimprovement over a 2% cost of CPU. I mostly care about my own time, not\nthat of the CPU.\n\nCan't we have our 6/6 cake much easier and eat it too by learning a\n\"fallback\" mode, i.e. we try to invoke gzip, and if that doesn't work\nuse the \"internal\" one?\n\nRe the \"eco mode\": I also wonder how much of the overhead you're seeing\nfor both that 17% and 2% would go away if you pin both processes to the\nsame CPU, I can't recall the command offhand, but IIRC taskset or\nnumactl can do that. I.e. is this really measuring IPC overhead, or\nI-CPU overhead on your system?\n"},{"id":"457398","messageId":"3ed80afd-34b3-afd8-5ffb-0187a4475ee1@web.de","threadId":"50928","inReplyTo":"220615.86wndhwt9a.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 4/6] archive-tar: add internal gzip implementation","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-16T18:55:18Z","receivedAt":"2022-06-16T18:55:57Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 15.06.22 um 22:32 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Wed, Jun 15 2022, René Scharfe wrote:\n>\n>> Git uses zlib for its own object store, but calls gzip when creating tgz\n>> archives. Add an option to perform the gzip compression for the latter\n>> using zlib, without depending on the external gzip binary.\n>>\n>> Plug it in by making write_block a function pointer and switching to a\n>> compressing variant if the filter command has the magic value \"git\n>> archive gzip\". Does that indirection slow down tar creation? Not\n>> really, at least not in this test:\n>>\n>> $ hyperfine -w3 -L rev HEAD,origin/main -p 'git checkout {rev} && make' \\\n>> './git -C ../linux archive --format=tar HEAD # {rev}'\n>\n> Shameless plug: https://lore.kernel.org/git/211201.86r1aw9gbd.gmgdl@evledraar.gmail.com/\n>\n> I.e. a \"hyperfine\" wrapper I wrote to make exactly this sort of thing\n> easier.\n>\n> You'll find that you need less or no --warmup with it, since the\n> checkout flip-flopping and re-making (and resulting FS and other cache\n> eviction) will go away, as we'll use different \"git worktree\"'s for the\n> two \"rev\".\n\nOK, but requiring hyperfine alone is burden enough for reviewers.\n\nI had a try anyway and it took me a while to realize that git-hyperfine\nrequires setting the Git config option hyperfine.run-dir band that it\nignores it on my system. Had to hard-code it in the script.\n\n> (Also, putting those on a ramdisk really helps)\n>\n>> Benchmark #1: ./git -C ../linux archive --format=tar HEAD # HEAD\n>> Time (mean ± σ): 4.044 s ± 0.007 s [User: 3.901 s, System: 0.137 s]\n>> Range (min … max): 4.038 s … 4.059 s 10 runs\n>>\n>> Benchmark #2: ./git -C ../linux archive --format=tar HEAD # origin/main\n>> Time (mean ± σ): 4.047 s ± 0.009 s [User: 3.903 s, System: 0.138 s]\n>> Range (min … max): 4.038 s … 4.066 s 10 runs\n>>\n>> How does tgz creation perform?\n>>\n>> $ hyperfine -w3 -L command 'gzip -cn','git archive gzip' \\\n>> './git -c tar.tgz.command=\"{command}\" -C ../linux archive --format=tgz HEAD'\n>> Benchmark #1: ./git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD\n>> Time (mean ± σ): 20.404 s ± 0.006 s [User: 23.943 s, System: 0.401 s]\n>> Range (min … max): 20.395 s … 20.414 s 10 runs\n>>\n>> Benchmark #2: ./git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD\n>> Time (mean ± σ): 23.807 s ± 0.023 s [User: 23.655 s, System: 0.145 s]\n>> Range (min … max): 23.782 s … 23.857 s 10 runs\n>>\n>> Summary\n>> './git -c tar.tgz.command=\"gzip -cn\" -C ../linux archive --format=tgz HEAD' ran\n>> 1.17 ± 0.00 times faster than './git -c tar.tgz.command=\"git archive gzip\" -C ../linux archive --format=tgz HEAD'\n>>\n>> So the internal implementation takes 17% longer on the Linux repo, but\n>> uses 2% less CPU time. That's because the external gzip can run in\n>> parallel on its own processor, while the internal one works sequentially\n>> and avoids the inter-process communication overhead.\n>>\n>> What are the benefits? Only an internal sequential implementation can\n>> offer this eco mode, and it allows avoiding the gzip(1) requirement.\n>\n> I had been keeping one eye on this series, but didn't look at it in any\n> detail.\n>\n> I found this after reading 6/6, which I think in any case could really\n> use some \"why\" summary, which seems to mostly be covered here.\n>\n> I.e. it's unclear if the \"drop the dependency on gzip(1)\" in 6/6 is a\n> reference to the GZIP test dependency, or that our users are unlikely to\n> have \"gzip(1)\" on their systems.\n\nIt's to avoid a run dependency; the build/test dependency remains.\n\n> If it's the latter I'd much rather (as a user) take a 17% wallclock\n> improvement over a 2% cost of CPU. I mostly care about my own time, not\n> that of the CPU.\n\nUnderstandable, and you can set tar.tgz.command='gzip -cn' to get the\nold behavior. Saving energy is a better default, though.\n\nThe runtime in the real world probably includes lots more I/O time. The\ntests above are repeated and warmed up to get consistent measurements,\nbut big repos are probably not fully kept in memory like that.\n\n> Can't we have our 6/6 cake much easier and eat it too by learning a\n> \"fallback\" mode, i.e. we try to invoke gzip, and if that doesn't work\n> use the \"internal\" one?\n\nInteresting idea, but I think the existing config option suffices. E.g.\na distro could set it in the system-wide config file if/when gzip is\ninstalled.\n\n> Re the \"eco mode\": I also wonder how much of the overhead you're seeing\n> for both that 17% and 2% would go away if you pin both processes to the\n> same CPU, I can't recall the command offhand, but IIRC taskset or\n> numactl can do that. I.e. is this really measuring IPC overhead, or\n> I-CPU overhead on your system?\n\nI'd expect that running git archive and gzip at the same CPU core takes\nmore wall-clock time than using zlib because inflating the object files\nand deflating the archive are done sequentially in both scenarios.\nCan't test it on macOS because it doesn't offer a way to pin programs to\na certain core, but e.g. someone with access to a Linux system can check\nthat using taskset(1).\n\nRené\n"},{"id":"457840","messageId":"220624.8635fujn3o.gmgdl@evledraar.gmail.com","threadId":"50928","inReplyTo":"3ed80afd-34b3-afd8-5ffb-0187a4475ee1@web.de","subject":"Re: [PATCH v4 4/6] archive-tar: add internal gzip implementation","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-24T11:13:20Z","receivedAt":"2022-06-24T11:51:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 16 2022, René Scharfe wrote:\n\n> Am 15.06.22 um 22:32 schrieb Ævar Arnfjörð Bjarmason:\n>> [...]\n> Understandable, and you can set tar.tgz.command='gzip -cn' to get the\n> old behavior. Saving energy is a better default, though.\n\nI disagree with that in general, a big reason for why git won out over\nother VCS's is that it wasn't as slow. I think we should primarily be\ninterested in the time a user might end up staring at the screen.\n\nI understand the concern to have \"git archive\" just work, e.g. if you\nuninstall gzip(1) (although that seems rather obscure, but perhaps this\nis for more minimal setups).\n\nI don't think saving energy is a virtue, *maybe* it is, but maybe your\ncomputer is powered by hydro, solar or nuclear instead of coal, so even\nif we're taking global energy policy into account for changes to git\nit's highly context dependant.\n\nIn any case, this is also true for pretty much any other git command\nthat might spawn processes or threads, e.g. \"git grep\":\n\n\t$ hyperfine -w3 -L cpus 0,0-7 'taskset --cpu-list {cpus} ./git grep foo.*bar' -r 10\n\tBenchmark 1: taskset --cpu-list 0 ./git grep foo.*bar\n\t Time (mean ± σ): 39.3 ms ± 1.2 ms [User: 20.0 ms, System: 18.6 ms]\n\t Range (min … max): 38.2 ms … 41.8 ms 10 runs\n\t\n\tBenchmark 2: taskset --cpu-list 0-7 ./git grep foo.*bar\n\t Time (mean ± σ): 28.1 ms ± 1.3 ms [User: 43.5 ms, System: 51.0 ms]\n\t Range (min … max): 26.6 ms … 31.2 ms 10 runs\n\t\n\tSummary\n\t 'taskset --cpu-list 0-7 ./git grep foo.*bar' ran\n\t 1.40 ± 0.08 times faster than 'taskset --cpu-list 0 ./git grep foo.*bar'\n\nHere we use less than 1/2 the user/system time when I pin it to 1 cpu,\nbut we're 40% slower.\n\nSo this is a bit of a digression, but this particular thing seems much\nbetter left to the OS or your hardware's CPU throttling policy. To the\nextent that we care perhaps more fitting would be to have a global\ncore.wrapper-cmd option or something, so you could pass all git commands\nthrough \"taskset\" (or your local equivalent), or just use shell aliases.\n\n> The runtime in the real world probably includes lots more I/O time. The\n> tests above are repeated and warmed up to get consistent measurements,\n> but big repos are probably not fully kept in memory like that.\n>\n>> Can't we have our 6/6 cake much easier and eat it too by learning a\n>> \"fallback\" mode, i.e. we try to invoke gzip, and if that doesn't work\n>> use the \"internal\" one?\n>\n> Interesting idea, but I think the existing config option suffices. E.g.\n> a distro could set it in the system-wide config file if/when gzip is\n> installed.\n\nI think in practice distros are unlikely to have such triggers for\n\"package X is installed, let's set config Y\". I mean, e.g. Debian can do\nthat with its packaging system, but it's expecting a lot. Why not flip\nthe default depending on if start_command() fails?\n\n>> Re the \"eco mode\": I also wonder how much of the overhead you're seeing\n>> for both that 17% and 2% would go away if you pin both processes to the\n>> same CPU, I can't recall the command offhand, but IIRC taskset or\n>> numactl can do that. I.e. is this really measuring IPC overhead, or\n>> I-CPU overhead on your system?\n>\n> I'd expect that running git archive and gzip at the same CPU core takes\n> more wall-clock time than using zlib because inflating the object files\n> and deflating the archive are done sequentially in both scenarios.\n> Can't test it on macOS because it doesn't offer a way to pin programs to\n> a certain core, but e.g. someone with access to a Linux system can check\n> that using taskset(1).\n\nHere's a benchmark, this is your hyperfine command, just with taskset\nadded. It's an 8-core box, so 0-7 is \"all CPUs\" (I think...):\n\n\thyperfine -w3 \\\n\t\t-L cpus 0,0-7 \\\n\t\t-L command 'gzip -cn','git archive gzip' \\\n\t\t'taskset --cpu-list {cpus} ./git -c tar.tgz.command=\"{command}\" archive --format=tgz HEAD'\n\nWhich gives me:\n\n\tBenchmark 1: taskset --cpu-list 0 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD\n\t Time (mean ± σ): 1.561 s ± 0.029 s [User: 1.503 s, System: 0.058 s]\n\t Range (min … max): 1.522 s … 1.622 s 10 runs\n\t \n\tBenchmark 2: taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD\n\t Time (mean ± σ): 1.337 s ± 0.029 s [User: 1.535 s, System: 0.075 s]\n\t Range (min … max): 1.298 s … 1.388 s 10 runs\n\t \n\tBenchmark 3: taskset --cpu-list 0 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD\n\t Time (mean ± σ): 1.493 s ± 0.032 s [User: 1.453 s, System: 0.040 s]\n\t Range (min … max): 1.462 s … 1.572 s 10 runs\n\t \n\tBenchmark 4: taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD\n\t Time (mean ± σ): 1.503 s ± 0.026 s [User: 1.466 s, System: 0.036 s]\n\t Range (min … max): 1.469 s … 1.542 s 10 runs\n\t \n\tSummary\n\t 'taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD' ran\n\t 1.12 ± 0.03 times faster than 'taskset --cpu-list 0 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD'\n\t 1.12 ± 0.03 times faster than 'taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD'\n\t 1.17 ± 0.03 times faster than 'taskset --cpu-list 0 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD'\n\nWhic I think should control for the IPC overhead v.s. the advantage of\nmulticore. I.e. we're faster with \"gzip -cn\" on multicore, but the\ninternal implementation has an advantage when it comes to \n\nI tried out the fallback method, memory leaks aside (needs to do a\nproper cleanup) this seems to work. Most of the diff is moving the\nexisting code into a function:\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 3d77e0f7509..a1b08812ee3 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -458,14 +458,36 @@ static void tgz_write_block(const void *data)\n \ttgz_deflate(Z_NO_FLUSH);\n }\n \n+static const char default_gzip_command[] = \"gzip -cn\";\n static const char internal_gzip_command[] = \"git archive gzip\";\n \n-static int write_tar_filter_archive(const struct archiver *ar,\n-\t\t\t\t struct archiver_args *args)\n+static int do_internal_gzip(const struct archiver *ar,\n+\t\t\t struct archiver_args *args)\n {\n #if ZLIB_VERNUM >= 0x1221\n \tstruct gz_header_s gzhead = { .os = 3 }; /* Unix, for reproducibility */\n #endif\n+\tint r;\n+\n+\twrite_block = tgz_write_block;\n+\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n+#if ZLIB_VERNUM >= 0x1221\n+\tif (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK)\n+\t\tBUG(\"deflateSetHeader() called too late\");\n+#endif\n+\tgzstream.next_out = outbuf;\n+\tgzstream.avail_out = sizeof(outbuf);\n+\n+\tr = write_tar_archive(ar, args);\n+\n+\ttgz_deflate(Z_FINISH);\n+\tgit_deflate_end(&gzstream);\n+\treturn r;\n+}\n+\n+static int write_tar_filter_archive(const struct archiver *ar,\n+\t\t\t\t struct archiver_args *args)\n+{\n \tstruct strbuf cmd = STRBUF_INIT;\n \tstruct child_process filter = CHILD_PROCESS_INIT;\n \tint r;\n@@ -473,33 +495,24 @@ static int write_tar_filter_archive(const struct archiver *ar,\n \tif (!ar->filter_command)\n \t\tBUG(\"tar-filter archiver called with no filter defined\");\n \n-\tif (!strcmp(ar->filter_command, internal_gzip_command)) {\n-\t\twrite_block = tgz_write_block;\n-\t\tgit_deflate_init_gzip(&gzstream, args->compression_level);\n-#if ZLIB_VERNUM >= 0x1221\n-\t\tif (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK)\n-\t\t\tBUG(\"deflateSetHeader() called too late\");\n-#endif\n-\t\tgzstream.next_out = outbuf;\n-\t\tgzstream.avail_out = sizeof(outbuf);\n-\n-\t\tr = write_tar_archive(ar, args);\n-\n-\t\ttgz_deflate(Z_FINISH);\n-\t\tgit_deflate_end(&gzstream);\n-\t\treturn r;\n-\t}\n+\tif (!strcmp(ar->filter_command, internal_gzip_command))\n+\t\treturn do_internal_gzip(ar, args);\n \n \tstrbuf_addstr(&cmd, ar->filter_command);\n \tif (args->compression_level >= 0)\n \t\tstrbuf_addf(&cmd, \" -%d\", args->compression_level);\n \n-\tstrvec_push(&filter.args, cmd.buf);\n-\tfilter.use_shell = 1;\n+\tstrvec_split(&filter.args, cmd.buf);\n \tfilter.in = -1;\n \n-\tif (start_command(&filter) < 0)\n+\tif (start_command(&filter) < 0) {\n+\t\tif (!strcmp(ar->filter_command, default_gzip_command)) {\n+\t\t\twarning_errno(_(\"could not start '%s' filter, falling back to '%s'\"),\n+\t\t\t\t ar->filter_command, internal_gzip_command);\n+\t\t\treturn do_internal_gzip(ar, args);\n+\t\t}\n \t\tdie_errno(_(\"unable to start '%s' filter\"), cmd.buf);\n+\t}\n \tclose(1);\n \tif (dup2(filter.in, 1) < 0)\n \t\tdie_errno(_(\"unable to redirect descriptor\"));\n"},{"id":"457858","messageId":"9c3fc76e-a0ec-c401-07b1-748c006eddda@web.de","threadId":"50928","inReplyTo":"220624.8635fujn3o.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 4/6] archive-tar: add internal gzip implementation","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-06-24T20:24:48Z","receivedAt":"2022-06-24T20:25:27Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 24.06.22 um 13:13 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Thu, Jun 16 2022, René Scharfe wrote:\n>\n>> Am 15.06.22 um 22:32 schrieb Ævar Arnfjörð Bjarmason:\n>>> [...]\n>> Understandable, and you can set tar.tgz.command='gzip -cn' to get the\n>> old behavior. Saving energy is a better default, though.\n>\n> I disagree with that in general, a big reason for why git won out over\n> other VCS's is that it wasn't as slow. I think we should primarily be\n> interested in the time a user might end up staring at the screen.\n>\n> I understand the concern to have \"git archive\" just work, e.g. if you\n> uninstall gzip(1) (although that seems rather obscure, but perhaps this\n> is for more minimal setups).\n\nThe previous attempt came from/via Git on Windows.\n\n> I don't think saving energy is a virtue, *maybe* it is, but maybe your\n> computer is powered by hydro, solar or nuclear instead of coal, so even\n> if we're taking global energy policy into account for changes to git\n> it's highly context dependant.\n\nOr a device runs on battery power and saving energy keeps it running a\nbit longer. Or it's housed in a data center and saving energy helps\nreduce cooling requirements.\n\n> In any case, this is also true for pretty much any other git command\n> that might spawn processes or threads, e.g. \"git grep\":\n>\n> \t$ hyperfine -w3 -L cpus 0,0-7 'taskset --cpu-list {cpus} ./git grep foo.*bar' -r 10\n> \tBenchmark 1: taskset --cpu-list 0 ./git grep foo.*bar\n> \t Time (mean ± σ): 39.3 ms ± 1.2 ms [User: 20.0 ms, System: 18.6 ms]\n> \t Range (min … max): 38.2 ms … 41.8 ms 10 runs\n>\n> \tBenchmark 2: taskset --cpu-list 0-7 ./git grep foo.*bar\n> \t Time (mean ± σ): 28.1 ms ± 1.3 ms [User: 43.5 ms, System: 51.0 ms]\n> \t Range (min … max): 26.6 ms … 31.2 ms 10 runs\n>\n> \tSummary\n> \t 'taskset --cpu-list 0-7 ./git grep foo.*bar' ran\n> \t 1.40 ± 0.08 times faster than 'taskset --cpu-list 0 ./git grep foo.*bar'\n>\n> Here we use less than 1/2 the user/system time when I pin it to 1 cpu,\n> but we're 40% slower.\n>\n> So this is a bit of a digression, but this particular thing seems much\n> better left to the OS or your hardware's CPU throttling policy. To the\n> extent that we care perhaps more fitting would be to have a global\n> core.wrapper-cmd option or something, so you could pass all git commands\n> through \"taskset\" (or your local equivalent), or just use shell aliases.\n\nNot sure what conclusion to draw from these numbers. Perhaps that\ncomputation is not the bottleneck here (increasing the number of cores by\n700% increases speed only by 40%)? That coordination overhead makes up a\nbig percentage and there might be room for improvement/tuning?\n\nIn any case, I agree we should leave scheduling decisions at runtime to\nthe OS.\n\n>> The runtime in the real world probably includes lots more I/O time. The\n>> tests above are repeated and warmed up to get consistent measurements,\n>> but big repos are probably not fully kept in memory like that.\n\nOn top of that I guess only few people create tgz files at all. Most of\nthem I would expect to be created automatically (and cached) by sites\nlike kernel.org. So I imagine people rather create tar.xz, tar.zst or\nzip archives these days. Or use git at both ends (push/pull), as they\nshould. ;-) I have no data to support this guess, though.\n\nBut yeah, the tradeoff sounds a bit weird: Give 17% duration, get 2% CPU\ntime back -- sounds like a ripoff. In your example below it's 12%\nlonger duration for 5% saved CPU time, which sounds a bit better, but\nstill not terribly attractive.\n\nLook at it from a different angle: This basic sequential implementation\nis better for non-interactive tgz creation due to its slightly lower\nCPU usage, which we cannot achieve with any parallel process setup.\nIt's easier to deploy because it doesn't need gzip. Its runtime hit\nisn't *that* hard, and people interested primarily in speed should\nparallelize the expensive part, deflate, not run the cheap tar creation\nparallel to a single-threaded deflate. I.e. they should already run\npigz (https://zlib.net/pigz/).\n\n$ hyperfine -L gz gzip,pigz -w3 'git -C ../linux archive --format=tar HEAD | {gz} -cn'\nBenchmark 1: git -C ../linux archive --format=tar HEAD | gzip -cn\n Time (mean ± σ): 20.764 s ± 0.007 s [User: 24.119 s, System: 0.606 s]\n Range (min … max): 20.758 s … 20.781 s 10 runs\n\nBenchmark 2: git -C ../linux archive --format=tar HEAD | pigz -cn\n Time (mean ± σ): 6.077 s ± 0.023 s [User: 29.708 s, System: 1.599 s]\n Range (min … max): 6.037 s … 6.125 s 10 runs\n\nSummary\n 'git -C ../linux archive --format=tar HEAD | pigz -cn' ran\n 3.42 ± 0.01 times faster than 'git -C ../linux archive --format=tar HEAD | gzip -cn'\n\n>>> Can't we have our 6/6 cake much easier and eat it too by learning a\n>>> \"fallback\" mode, i.e. we try to invoke gzip, and if that doesn't work\n>>> use the \"internal\" one?\n>>\n>> Interesting idea, but I think the existing config option suffices. E.g.\n>> a distro could set it in the system-wide config file if/when gzip is\n>> installed.\n>\n> I think in practice distros are unlikely to have such triggers for\n> \"package X is installed, let's set config Y\". I mean, e.g. Debian can do\n> that with its packaging system, but it's expecting a lot.\n\nI don't *expect* any reaction either way, but packagers *can* go with a\ncustom config if they see the need.\n\n> Why not flip\n> the default depending on if start_command() fails?\n\nBecause it's harder to test and support due to its more complicated\nbehavior, and I don't see why it would be needed.\n\n>>> Re the \"eco mode\": I also wonder how much of the overhead you're seeing\n>>> for both that 17% and 2% would go away if you pin both processes to the\n>>> same CPU, I can't recall the command offhand, but IIRC taskset or\n>>> numactl can do that. I.e. is this really measuring IPC overhead, or\n>>> I-CPU overhead on your system?\n>>\n>> I'd expect that running git archive and gzip at the same CPU core takes\n>> more wall-clock time than using zlib because inflating the object files\n>> and deflating the archive are done sequentially in both scenarios.\n>> Can't test it on macOS because it doesn't offer a way to pin programs to\n>> a certain core, but e.g. someone with access to a Linux system can check\n>> that using taskset(1).\n>\n> Here's a benchmark, this is your hyperfine command, just with taskset\n> added. It's an 8-core box, so 0-7 is \"all CPUs\" (I think...):\n>\n> \thyperfine -w3 \\\n> \t\t-L cpus 0,0-7 \\\n> \t\t-L command 'gzip -cn','git archive gzip' \\\n> \t\t'taskset --cpu-list {cpus} ./git -c tar.tgz.command=\"{command}\" archive --format=tgz HEAD'\n>\n> Which gives me:\n>\n> \tBenchmark 1: taskset --cpu-list 0 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD\n> \t Time (mean ± σ): 1.561 s ± 0.029 s [User: 1.503 s, System: 0.058 s]\n> \t Range (min … max): 1.522 s … 1.622 s 10 runs\n>\n> \tBenchmark 2: taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD\n> \t Time (mean ± σ): 1.337 s ± 0.029 s [User: 1.535 s, System: 0.075 s]\n> \t Range (min … max): 1.298 s … 1.388 s 10 runs\n>\n> \tBenchmark 3: taskset --cpu-list 0 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD\n> \t Time (mean ± σ): 1.493 s ± 0.032 s [User: 1.453 s, System: 0.040 s]\n> \t Range (min … max): 1.462 s … 1.572 s 10 runs\n>\n> \tBenchmark 4: taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD\n> \t Time (mean ± σ): 1.503 s ± 0.026 s [User: 1.466 s, System: 0.036 s]\n> \t Range (min … max): 1.469 s … 1.542 s 10 runs\n>\n> \tSummary\n> \t 'taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD' ran\n> \t 1.12 ± 0.03 times faster than 'taskset --cpu-list 0 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD'\n> \t 1.12 ± 0.03 times faster than 'taskset --cpu-list 0-7 ./git -c tar.tgz.command=\"git archive gzip\" archive --format=tgz HEAD'\n> \t 1.17 ± 0.03 times faster than 'taskset --cpu-list 0 ./git -c tar.tgz.command=\"gzip -cn\" archive --format=tgz HEAD'\n>\n> Whic I think should control for the IPC overhead v.s. the advantage of\n> multicore. I.e. we're faster with \"gzip -cn\" on multicore, but the\n> internal implementation has an advantage when it comes to\n\nRight, #1, #3 and #4 all run sequentially, but #1 has the pipe overhead\nto deal with as well, which adds 5 percentage points to its runtime.\n\nRené\n"},{"id":"458241","messageId":"ps52p06s-01nr-4ss2-r802-6nsp5nqq5199@tzk.qr","threadId":"50928","inReplyTo":"0aa5c101-06bf-325c-efbc-6b4ef38616c5@web.de","subject":"Re: [PATCH v3 0/5] Avoid spawning gzip in git archive","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-30T18:55:38Z","receivedAt":"2022-06-30T18:56:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi René,\n\nOn Tue, 14 Jun 2022, René Scharfe wrote:\n\n> Am 14.06.22 um 13:28 schrieb Johannes Schindelin:\n> >\n> > By the way, the main reason why I did not work more is that in\n> > http://madler.net/pipermail/zlib-devel_madler.net/2019-December/003308.html,\n> > Mark Adler (the zlib maintainer) announced that...\n> >\n> >> [...] There are many well-tested performance improvements in zlib\n> >> waiting in the wings that will be incorporated over the next several\n> >> months. [...]\n> >\n> > This was in December 2019. And now it's June 2022 and I kind of wonder\n> > whether those promised improvements will still come.\n> >\n> > In the meantime, however, a viable alternative seems to have cropped up:\n> > https://github.com/zlib-ng/zlib-ng. Essentially, it looks as if it is what\n> > zlib should have become after above-quoted announcement.\n> >\n> > In particular the CPU intrinsics support (think MMX, SSE2/3, etc) seem to\n> > be very interesting and I would not be completely surprised if building\n> > Git with your patches and linking against zlib-ng would paint a very\n> > favorable picture not only in terms of CPU time but also in terms of\n> > wallclock time. Sadly, I have not been able to set aside time to look into\n> > that angle, but maybe I can peak your interest?\n> I was unable to preload zlib-ng using DYLD_INSERT_LIBRARIES on macOS\n> 12.4 so far. The included demo proggy looks impressive, though:\n>\n> $ hyperfine -w3 -L gzip gzip,../zlib-ng/minigzip \"git -C ../linux archive --format=tar HEAD | {gzip} -c\"\n> Benchmark #1: git -C ../linux archive --format=tar HEAD | gzip -c\n> Time (mean ± σ): 20.424 s ± 0.006 s [User: 23.964 s, System: 0.432 s]\n> Range (min … max): 20.414 s … 20.434 s 10 runs\n>\n> Benchmark #2: git -C ../linux archive --format=tar HEAD | ../zlib-ng/minigzip -c\n> Time (mean ± σ): 12.158 s ± 0.006 s [User: 13.908 s, System: 0.376 s]\n> Range (min … max): 12.145 s … 12.166 s 10 runs\n>\n> Summary\n> 'git -C ../linux archive --format=tar HEAD | ../zlib-ng/minigzip -c' ran\n> 1.68 ± 0.00 times faster than 'git -C ../linux archive --format=tar HEAD | gzip -c'\n\nIntriguing.\n\nI finally managed to play around with building and packaging zlib-ng [*1*]\n(since I want to use it as a drop-in replacement for zlib, I think it is\nbest to configure it with `--zlib-compat`, that way I do not have to\nfiddle with any equivalent of `LD_PRELOAD`). Here are my numbers:\n\n\tzlib-ng: 14.409 s ± 0.209 s\n\tzlib: 26.843 s ± 0.636 s\n\nThese are pretty good, which made me think that they might actually even\nhelp regular Git operations (because we zlib every loose object).\n\nSo I tried to `fast-import` some 2500 commits from linux.git into a fresh\nrepository, and the zlib-ng version takes ~51s and the zlib version takes\n~58s. At first I thought that it might be noise, but the trend seems to be\nsteady. It's not a huge improvement, of course, but I think that might be\nbecause most of the time is spent parsing.\n\nI then tried to test the performance focusing on writing loose object, by\nusing p0008 (increasing the number of files from 50 to 1500 and\nrestricting it to fsyncMethod=none).\n\nUnfortunately, the numbers are not really conclusive. I do see minor\nspeed-ups with zlib-ng, mostly, in the single digit percentages, though\noccasionally in the other direction. In other words, there is no clear-cut\nchange, just a vague tendency. My guess: Git writes too small files (their\ncontents are of the form \"$basedir$test_tick.$counter\") and zlib-ng's\nsuperior performance does not come to bear.\n\nStill, for larger workloads, zlib-ng seems to offer a quite nice and\nsubstantial performance improvement over zlib.\n\nCiao,\nDscho\n\nFootnote *1*: https://github.com/msys2/MINGW-packages/compare/master...dscho:zlib-ng\n"},{"id":"458389","messageId":"038r075o-5s5r-9sop-5o02-8s84428o0r54@tzk.qr","threadId":"50928","inReplyTo":"ps52p06s-01nr-4ss2-r802-6nsp5nqq5199@tzk.qr","subject":"Re: [PATCH v3 0/5] Avoid spawning gzip in git archive","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-07-01T16:05:59Z","receivedAt":"2022-07-01T16:06:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Me again,\n\nOn Thu, 30 Jun 2022, Johannes Schindelin wrote:\n\n> I finally managed to play around with building and packaging zlib-ng\n> [*1*] (since I want to use it as a drop-in replacement for zlib, I think\n> it is best to configure it with `--zlib-compat`, that way I do not have\n> to fiddle with any equivalent of `LD_PRELOAD`). Here are my numbers:\n>\n> \tzlib-ng: 14.409 s ± 0.209 s\n> \tzlib: 26.843 s ± 0.636 s\n>\n> These are pretty good, which made me think that they might actually even\n> help regular Git operations (because we zlib every loose object).\n>\n> So I tried to `fast-import` some 2500 commits from linux.git into a fresh\n> repository, and the zlib-ng version takes ~51s and the zlib version takes\n> ~58s. At first I thought that it might be noise, but the trend seems to be\n> steady. It's not a huge improvement, of course, but I think that might be\n> because most of the time is spent parsing.\n>\n> I then tried to test the performance focusing on writing loose object, by\n> using p0008 (increasing the number of files from 50 to 1500 and\n> restricting it to fsyncMethod=none).\n>\n> Unfortunately, the numbers are not really conclusive. I do see minor\n> speed-ups with zlib-ng, mostly, in the single digit percentages, though\n> occasionally in the other direction. In other words, there is no clear-cut\n> change, just a vague tendency. My guess: Git writes too small files (their\n> contents are of the form \"$basedir$test_tick.$counter\") and zlib-ng's\n> superior performance does not come to bear.\n>\n> Still, for larger workloads, zlib-ng seems to offer a quite nice and\n> substantial performance improvement over zlib.\n\nStolee pointed out to me that objects inside pack files are also\nzlib-compressed, and that measuring the speed of `git rev-list --objects\n--all --count` might therefore be a better test.\n\nAnd this is where things get a little messy: in the context of Git for\nWindows, my local measurements indicate that zlib is better, with ~41\nseconds using zlib vs ~52 seconds using zlib-ng (but the latter has a\nrather large variance).\n\nThese measurements were done with a relatively straight-forward build of\nzlib-ng v2.0.6, and on a hunch I then tried to build the tip of zlib-ng's\n`develop` branch (which was much less straight-forward) and now get\nvirtually the same speed with that `rev-list` command.\n\nBut then I repeated the `archive` measurement with the `develop` version\nof zlib-ng, and while it was still substantially faster than zlib, it was\nslightly slower than zlib-ng v2.0.6 (zlib: ~26 seconds, zlib-ng v2.0.6:\n~14 seconds, zlib-ng develop: ~16 seconds). Still, much, much faster than\nusing `-c tar.tgz.command=\"gzip -cn\"` at ~24 seconds.\n\nSo: the picture is messy. The latest official release of zlib-ng seems to\noffer performance wins using `archive` but slight losses using `rev-list.\nUpgrading to the latest revision of zlib-ng offers slightly smaller\nperformance wins using `archive` and equivalent performance using\n`rev-list`. Both blow `gzip -cn` out of the water, thanks to using MMX or\nwhatever my laptop's CPU offers.\n\nThe take-away as far as Git for Windows is concerned: It seems not _quite_\nthe time yet to switch from zlib to zlib-ng, I want to wait until there is\nan official zlib-ng release with favorable speed.\n\nCiao,\nDscho\n\nP.S.: I pushed a WIP update to this branch:\n\n> Footnote *1*: https://github.com/msys2/MINGW-packages/compare/master...dscho:zlib-ng\n"},{"id":"458391","messageId":"Yr8gZT6dbCpzaR9n@coredump.intra.peff.net","threadId":"50928","inReplyTo":"038r075o-5s5r-9sop-5o02-8s84428o0r54@tzk.qr","subject":"Re: [PATCH v3 0/5] Avoid spawning gzip in git archive","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-07-01T16:27:17Z","receivedAt":"2022-07-01T16:27:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 01, 2022 at 06:05:59PM +0200, Johannes Schindelin wrote:\n\n> Stolee pointed out to me that objects inside pack files are also\n> zlib-compressed, and that measuring the speed of `git rev-list --objects\n> --all --count` might therefore be a better test.\n\nThat will spend quite a lot of time doing hash-lookups for each tree\nentry. A better raw zlib test might be:\n\n git cat-file --batch --batch-all-objects --unordered >/dev/null\n\nwhich will just dump each object, and should mostly be zlib and delta\nreconstruction (the --unordered is important to hit the deltas in the\nright order).\n\n> And this is where things get a little messy: in the context of Git for\n> Windows, my local measurements indicate that zlib is better, with ~41\n> seconds using zlib vs ~52 seconds using zlib-ng (but the latter has a\n> rather large variance).\n\nThat is a surprising slow-down between the two. I'd expect the command\nabove to show even more pronounced results, though, as it's spending\nless time doing non-zlib things. But it's still just inflating (as\nopposed to git-archive, which is both inflating and deflating).\n\n-Peff\n"},{"id":"458395","messageId":"xmqqy1xc913j.fsf@gitster.g","threadId":"50928","inReplyTo":"Yr8gZT6dbCpzaR9n@coredump.intra.peff.net","subject":"Re: [PATCH v3 0/5] Avoid spawning gzip in git archive","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-01T17:47:28Z","receivedAt":"2022-07-01T17:47:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> That will spend quite a lot of time doing hash-lookups for each tree\n> entry. A better raw zlib test might be:\n>\n> git cat-file --batch --batch-all-objects --unordered >/dev/null\n>\n> which will just dump each object, and should mostly be zlib and delta\n> reconstruction (the --unordered is important to hit the deltas in the\n> right order).\n\n;-)\n\nI like --unordered has the meaning \"use the order Git likes\" (which\nis probably the packfile offset order, which we optimize for\nminimizing seek during delta reconstruction).\n\n"}]}