{"thread":{"id":"55434","subject":"[PATCH 2/5] test-lib: use exact match for test_subcommand","startedAt":"2021-04-05T13:04:24Z","lastAt":"2021-04-16T18:31:35Z","messageCount":72,"participants":["Derrick Stolee via GitGitGadget","Tom Saeger","Eric Sunshine","Junio C Hamano","Derrick Stolee","Ævar Arnfjörð Bjarmason","Josh Steadmon","Emily Shaffer","Jonathan Tan","Ramsay Jones"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"420980","messageId":"2b74889c2a323f03be477ffdf9ff388405779c3b.1617627856.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.git.1617627856.gitgitgadget@gmail.com","subject":"[PATCH 2/5] test-lib: use exact match for test_subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-05T13:04:12Z","receivedAt":"2021-04-05T13:04:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe use of 'grep' inside test_subcommand uses general patterns, leading\nto sometimes needing escape characters to avoid incorrect matches.\nFurther, some platforms interpret different glob characters differently.\n\nUse 'grep -F' to use an exact match. This requires removing escape\ncharacters from existing callers. Luckily, this is only one test that\nexpects refspecs as part of the subcommand.\n\nReported-by: Eric Sunshine \nSigned-off-by: Derrick Stolee \n---\n t/t7900-maintenance.sh | 4 ++--\n t/test-lib-functions.sh | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 2412d8c5c006..fc2315edec11 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -142,8 +142,8 @@ test_expect_success 'prefetch multiple remotes' '\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n-\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* \n\nThe previous logic filled a string list with the names of each remote,\nbut instead we could simply run the appropriate 'git fetch' data\ndirectly in the remote iterator. Do this for reduced code size, but also\nbecuase it sets up an upcoming change to use the remote's refspec. This\ndata is accessible from the 'struct remote' data that is now accessible\nin fetch_remote().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 33 ++++++++-------------------------\n 1 file changed, 8 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ef7226d7bca4..fa8128de9ae1 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n+static int fetch_remote(struct remote *remote, void *cbdata)\n {\n+\tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n-\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n+\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n \t\t \"--refmap=\", NULL);\n \n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n+\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n \n \treturn !!run_command(&child);\n }\n \n-static int append_remote(struct remote *remote, void *cbdata)\n-{\n-\tstruct string_list *remotes = (struct string_list *)cbdata;\n-\n-\tstring_list_append(remotes, remote->name);\n-\treturn 0;\n-}\n-\n static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n {\n-\tint result = 0;\n-\tstruct string_list_item *item;\n-\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n-\n \tgit_config_set_multivar_gently(\"log.excludedecoration\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n \t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n \n-\tif (for_each_remote(append_remote, &remotes)) {\n-\t\terror(_(\"failed to fill remotes\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n+\tif (for_each_remote(fetch_remote, opts)) {\n+\t\terror(_(\"failed to prefetch remotes\"));\n+\t\treturn 1;\n \t}\n \n-\tfor_each_string_list_item(item, &remotes)\n-\t\tresult |= fetch_remote(item->string, opts);\n-\n-cleanup:\n-\tstring_list_clear(&remotes, 0);\n-\treturn result;\n+\treturn 0;\n }\n \n static int maintenance_task_gc(struct maintenance_run_opts *opts)\n-- \ngitgitgadget\n\n"},{"id":"420983","messageId":"e10007e1cf8ff0005295f648b9489c11a9427122.1617627856.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.git.1617627856.gitgitgadget@gmail.com","subject":"[PATCH 3/5] refspec: output a refspec item","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-05T13:04:13Z","receivedAt":"2021-04-05T13:04:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a new method, refspec_item_format(), that takes a 'struct\nrefspec_item' pointer as input and returns a string for how that refspec\nitem should be written to Git's config or a subcommand, such as 'git\nfetch'.\n\nThere are several subtleties regarding special-case refspecs that can\noccur and are represented in t5511-refspec.sh. These cases will be\nexplored in new tests in the following change. It requires adding a new\ntest helper in order to test this format directly, so that is saved for\na separate change to keep this one focused on the logic of the format\nmethod.\n\nA future change will consume this method when translating refspecs in\nthe 'prefetch' task of the 'git maintenance' builtin.\n\nSigned-off-by: Derrick Stolee \n---\n refspec.c | 25 +++++++++++++++++++++++++\n refspec.h | 5 +++++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/refspec.c b/refspec.c\nindex e3d852c0bfec..ca65ba01bfe6 100644\n--- a/refspec.c\n+++ b/refspec.c\n@@ -180,6 +180,31 @@ void refspec_item_clear(struct refspec_item *item)\n \titem->exact_sha1 = 0;\n }\n \n+const char *refspec_item_format(const struct refspec_item *rsi)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_reset(&buf);\n+\n+\tif (rsi->matching)\n+\t\treturn \":\";\n+\n+\tif (rsi->negative)\n+\t\tstrbuf_addch(&buf, '^');\n+\telse if (rsi->force)\n+\t\tstrbuf_addch(&buf, '+');\n+\n+\tif (rsi->src)\n+\t\tstrbuf_addstr(&buf, rsi->src);\n+\n+\tif (rsi->dst) {\n+\t\tstrbuf_addch(&buf, ':');\n+\t\tstrbuf_addstr(&buf, rsi->dst);\n+\t}\n+\n+\treturn buf.buf;\n+}\n+\n void refspec_init(struct refspec *rs, int fetch)\n {\n \tmemset(rs, 0, sizeof(*rs));\ndiff --git a/refspec.h b/refspec.h\nindex 8b79891d3218..92a312f5b4e6 100644\n--- a/refspec.h\n+++ b/refspec.h\n@@ -56,6 +56,11 @@ int refspec_item_init(struct refspec_item *item, const char *refspec,\n void refspec_item_init_or_die(struct refspec_item *item, const char *refspec,\n \t\t\t int fetch);\n void refspec_item_clear(struct refspec_item *item);\n+/*\n+ * Output a given refspec item to a string.\n+ */\n+const char *refspec_item_format(const struct refspec_item *rsi);\n+\n void refspec_init(struct refspec *rs, int fetch);\n void refspec_append(struct refspec *rs, const char *refspec);\n __attribute__((format (printf,2,3)))\n-- \ngitgitgadget\n\n"},{"id":"420984","messageId":"c8d1de06f84499f2f56b3a06df665630806f94ce.1617627856.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.git.1617627856.gitgitgadget@gmail.com","subject":"[PATCH 4/5] test-tool: test refspec input/output","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-05T13:04:14Z","receivedAt":"2021-04-05T13:04:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a new test-helper, 'test-tool refspec', that currently reads stdin\nline-by-line and translates the refspecs using the parsing logic of\nrefspec_item_init() and writes them to output.\n\nCreate a test in t5511-refspec.sh that uses this helper to test several\nknown special cases. This includes all of the special properties of the\n'struct refspec_item', including:\n\n * force: The refspec starts with '+'.\n * pattern: Each side of the refspec has a glob character ('*')\n * matching: The refspec is simply the string \":\".\n * exact_sha1: The 'src' string is a 40-character hex string.\n * negative: The refspec starts with '^' and 'dst' is NULL.\n\nWhile the exact_sha1 property doesn't require special logic in\nrefspec_item_format, it is still tested here for completeness.\n\nThere is also the special-case refspec \"@\" which translates to \"HEAD\".\n\nNote that if a refspec does not start with \"refs/\", then that is not\nincorporated as part of the 'struct refspec_item'. This behavior is\nconfirmed by these tests. These refspecs still work in the wild because\nthe refs layer interprets them appropriately as branches, prepending\n\"refs/\" or \"refs/heads/\" as necessary. I spent some time attempting to\ninsert these prefixes explicitly in parse_refspec(), but these are\nseveral subtleties I was unable to overcome. If such a change were to be\nmade, then this new test in t5511-refspec.sh will need to be updated\nwith new output. For example, the input lines ending with \"translated\"\nare designed to demonstrate these subtleties.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n t/helper/test-refspec.c | 39 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5511-refspec.sh | 41 +++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 83 insertions(+)\n create mode 100644 t/helper/test-refspec.c\n\ndiff --git a/Makefile b/Makefile\nindex a6a73c574191..f858c9f25976 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -734,6 +734,7 @@ TEST_BUILTINS_OBJS += test-reach.o\n TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n+TEST_BUILTINS_OBJS += test-refspec.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\ndiff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\nnew file mode 100644\nindex 000000000000..08cf441a0a06\n--- /dev/null\n+++ b/t/helper/test-refspec.c\n@@ -0,0 +1,39 @@\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"refspec.h\"\n+#include \"strbuf.h\"\n+#include \"test-tool.h\"\n+\n+static const char * const refspec_usage[] = {\n+\tN_(\"test-tool refspec [--fetch]\"),\n+\tNULL\n+};\n+\n+int cmd__refspec(int argc, const char **argv)\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\tint fetch = 0;\n+\n+\tstruct option refspec_options [] = {\n+\t\tOPT_BOOL(0, \"fetch\", &fetch,\n+\t\t\t N_(\"enable the 'fetch' option for parsing refpecs\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, refspec_options,\n+\t\t\t refspec_usage, 0);\n+\n+\twhile (strbuf_getline(&line, stdin) != EOF) {\n+\t\tstruct refspec_item rsi;\n+\n+\t\tif (!refspec_item_init(&rsi, line.buf, fetch)) {\n+\t\t\tprintf(\"failed to parse %s\\n\", line.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tprintf(\"%s\\n\", refspec_item_format(&rsi));\n+\t\trefspec_item_clear(&rsi);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 287aa6002307..f534ad1731a9 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -55,6 +55,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n+\t{ \"refspec\", cmd__refspec },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9ea4b31011dd..46a0b8850f17 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -44,6 +44,7 @@ int cmd__reach(int argc, const char **argv);\n int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n+int cmd__refspec(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\ndiff --git a/t/t5511-refspec.sh b/t/t5511-refspec.sh\nindex be025b90f989..7614b6adf932 100755\n--- a/t/t5511-refspec.sh\n+++ b/t/t5511-refspec.sh\n@@ -93,4 +93,45 @@ test_refspec fetch \"refs/heads/${good}\"\n bad=$(printf '\\011tab')\n test_refspec fetch \"refs/heads/${bad}\"\t\t\t\tinvalid\n \n+test_expect_success 'test input/output round trip' '\n+\tcat >input <<-\\EOF &&\n+\t\t+refs/heads/*:refs/remotes/origin/*\n+\t\trefs/heads/*:refs/remotes/origin/*\n+\t\trefs/heads/main:refs/remotes/frotz/xyzzy\n+\t\t:refs/remotes/frotz/deleteme\n+\t\t^refs/heads/secrets\n+\t\trefs/heads/secret:refs/heads/translated\n+\t\trefs/heads/secret:heads/translated\n+\t\trefs/heads/secret:remotes/translated\n+\t\tsecret:translated\n+\t\trefs/heads/*:remotes/xxy/*\n+\t\trefs/heads*/for-linus:refs/remotes/mine/*\n+\t\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n+\t\tHEAD\n+\t\t@\n+\t\t:\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\t\t+refs/heads/*:refs/remotes/origin/*\n+\t\trefs/heads/*:refs/remotes/origin/*\n+\t\trefs/heads/main:refs/remotes/frotz/xyzzy\n+\t\t:refs/remotes/frotz/deleteme\n+\t\t^refs/heads/secrets\n+\t\trefs/heads/secret:refs/heads/translated\n+\t\trefs/heads/secret:heads/translated\n+\t\trefs/heads/secret:remotes/translated\n+\t\tsecret:translated\n+\t\trefs/heads/*:remotes/xxy/*\n+\t\trefs/heads*/for-linus:refs/remotes/mine/*\n+\t\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n+\t\tHEAD\n+\t\tHEAD\n+\t\t:\n+\tEOF\n+\ttest-tool refspec output &&\n+\ttest_cmp expect output &&\n+\ttest-tool refspec --fetch output &&\n+\ttest_cmp expect output\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"420985","messageId":"7f6c127dac48409ddc8d30ad236182bee21c1957.1617627856.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.git.1617627856.gitgitgadget@gmail.com","subject":"[PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-05T13:04:15Z","receivedAt":"2021-04-05T13:04:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe prefetch task previously used the default refspec source plus a\ncustom refspec destination to avoid colliding with remote refs:\n\n\t+refs/heads/*:refs/prefetch//*\n\nHowever, some users customize their refspec to reduce how much data they\ndownload from specific remotes. This can involve restrictive patterns\nfor fetching or negative patterns to avoid downloading some refs.\n\nModify fetch_remote() to iterate over the remote's refspec list and\ntranslate that into the appropriate prefetch scenario. Specifically,\nre-parse the raw form of the refspec into a new 'struct refspec' and\nmodify the 'dst' member to replace a leading \"refs/\" substring with\n\"refs/prefetch/\", or prepend \"refs/prefetch/\" to 'dst' otherwise.\nNegative refspecs do not have a 'dst' so they can be transferred to the\n'git fetch' command unmodified.\n\nThis prefix change provides the benefit of keeping whatever collisions\nmay exist in the custom refspecs, if that is a desirable outcome.\n\nThis changes the names of the refs that would be fetched by the default\nrefspec. Instead of \"refs/prefetch//\" they will now go\nto \"refs/prefetch/remotes//\". While this is a change, it\nis not a seriously breaking one: these refs are intended to be hidden\nand not used.\n\nUpdate the documentation to be more generic about the destination refs.\nDo not mention custom refpecs explicitly, as that does not need to be\nhighlighted in this documentation. The important part of placing refs in\nrefs/prefetch remains.\n\nReported-by: Tom Saeger \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 3 +--\n builtin/gc.c | 34 +++++++++++++++++++++++-\n t/t7900-maintenance.sh | 43 ++++++++++++++++++++++++++-----\n 3 files changed, 71 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 80ddd33ceba0..95a24264eb10 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -94,8 +94,7 @@ prefetch::\n \tobjects from all registered remotes. For each remote, a `git fetch`\n \tcommand is run. The refmap is custom to avoid updating local or remote\n \tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n-\tremote refs are stored in `refs/prefetch//`. Also, tags are\n-\tnot updated.\n+\trefs are stored in `refs/prefetch/`. Also, tags are not updated.\n +\n This is done to avoid disrupting the remote-tracking branches. The end users\n expect these refs to stay unmoved unless they initiate a fetch. With prefetch\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex fa8128de9ae1..92cb8b4e0bfa 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -32,6 +32,7 @@\n #include \"remote.h\"\n #include \"object-store.h\"\n #include \"exec-cmd.h\"\n+#include \"refspec.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -877,6 +878,7 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n {\n \tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n+\tint i;\n \n \tchild.git_cmd = 1;\n \tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n@@ -886,7 +888,37 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n+\tfor (i = 0; i < remote->fetch.nr; i++) {\n+\t\tstruct refspec_item replace;\n+\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n+\t\tstruct strbuf new_dst = STRBUF_INIT;\n+\t\tsize_t ignore_len = 0;\n+\n+\t\tif (rsi->negative) {\n+\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n+\n+\t\t/*\n+\t\t * If a refspec dst starts with \"refs/\" at the start,\n+\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n+\t\t * Otherwise, we will prepend the dst string with\n+\t\t * \"refs/prefetch/\".\n+\t\t */\n+\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n+\t\t\tignore_len = 5;\n+\n+\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n+\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n+\t\tfree(replace.dst);\n+\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n+\n+\t\tstrvec_push(&child.args, refspec_item_format(&replace));\n+\n+\t\trefspec_item_clear(&replace);\n+\t}\n \n \treturn !!run_command(&child);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex fc2315edec11..3366ea188782 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -142,20 +142,51 @@ test_expect_success 'prefetch multiple remotes' '\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n-\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* log &&\n \t! grep \"prefetch\" log\n '\n \n+test_expect_success 'prefetch custom refspecs' '\n+\tgit -C clone1 branch -f special/fetched HEAD &&\n+\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n+\n+\t# create multiple refspecs for remote1\n+\tgit config --add remote.remote1.fetch +refs/heads/special/fetched:refs/heads/fetched &&\n+\tgit config --add remote.remote1.fetch ^refs/heads/special/secret/not-fetched &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n+\n+\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n+\n+\t# skips second refspec because it is not a pattern type\n+\trs1=\"+refs/heads/*:refs/prefetch/remotes/remote1/*\" &&\n+\trs2=\"+refs/heads/special/fetched:refs/prefetch/heads/fetched\" &&\n+\trs3=\"^refs/heads/special/secret/not-fetched\" &&\n+\n+\ttest_subcommand git fetch remote1 $fetchargs $rs1 $rs2 $rs3 From: Derrick Stolee \n> \n> Add a new method, refspec_item_format(), that takes a 'struct\n> refspec_item' pointer as input and returns a string for how that refspec\n> item should be written to Git's config or a subcommand, such as 'git\n> fetch'.\n> \n> There are several subtleties regarding special-case refspecs that can\n> occur and are represented in t5511-refspec.sh. These cases will be\n> explored in new tests in the following change. It requires adding a new\n> test helper in order to test this format directly, so that is saved for\n> a separate change to keep this one focused on the logic of the format\n> method.\n> \n> A future change will consume this method when translating refspecs in\n> the 'prefetch' task of the 'git maintenance' builtin.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> refspec.c | 25 +++++++++++++++++++++++++\n> refspec.h | 5 +++++\n> 2 files changed, 30 insertions(+)\n> \n> diff --git a/refspec.c b/refspec.c\n> index e3d852c0bfec..ca65ba01bfe6 100644\n> --- a/refspec.c\n> +++ b/refspec.c\n> @@ -180,6 +180,31 @@ void refspec_item_clear(struct refspec_item *item)\n> \titem->exact_sha1 = 0;\n> }\n> \n> +const char *refspec_item_format(const struct refspec_item *rsi)\n> +{\n> +\tstatic struct strbuf buf = STRBUF_INIT;\n> +\n> +\tstrbuf_reset(&buf);\n\nis this even needed?\n\n> +\n> +\tif (rsi->matching)\n> +\t\treturn \":\";\n> +\n> +\tif (rsi->negative)\n> +\t\tstrbuf_addch(&buf, '^');\n> +\telse if (rsi->force)\n> +\t\tstrbuf_addch(&buf, '+');\n> +\n> +\tif (rsi->src)\n> +\t\tstrbuf_addstr(&buf, rsi->src);\n> +\n> +\tif (rsi->dst) {\n> +\t\tstrbuf_addch(&buf, ':');\n> +\t\tstrbuf_addstr(&buf, rsi->dst);\n> +\t}\n> +\n> +\treturn buf.buf;\n\nshould this be strbuf_detach?\n\n> +}\n> +\n> void refspec_init(struct refspec *rs, int fetch)\n> {\n> \tmemset(rs, 0, sizeof(*rs));\n> diff --git a/refspec.h b/refspec.h\n> index 8b79891d3218..92a312f5b4e6 100644\n> --- a/refspec.h\n> +++ b/refspec.h\n> @@ -56,6 +56,11 @@ int refspec_item_init(struct refspec_item *item, const char *refspec,\n> void refspec_item_init_or_die(struct refspec_item *item, const char *refspec,\n> \t\t\t int fetch);\n> void refspec_item_clear(struct refspec_item *item);\n> +/*\n> + * Output a given refspec item to a string.\n> + */\n> +const char *refspec_item_format(const struct refspec_item *rsi);\n> +\n> void refspec_init(struct refspec *rs, int fetch);\n> void refspec_append(struct refspec *rs, const char *refspec);\n> __attribute__((format (printf,2,3)))\n> -- \n> gitgitgadget\n> \n"},{"id":"420992","messageId":"20210405170105.fujb5hc5usthuw2y@brm-x62-17.us.oracle.com","threadId":"55434","inReplyTo":"3a94ff80657cd8e0142c1007be4f183c6a18587e.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/5] maintenance: simplify prefetch logic","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-05T17:01:05Z","receivedAt":"2021-04-05T17:01:21Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Mon, Apr 05, 2021 at 01:04:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The previous logic filled a string list with the names of each remote,\n> but instead we could simply run the appropriate 'git fetch' data\n> directly in the remote iterator. Do this for reduced code size, but also\n> becuase it sets up an upcoming change to use the remote's refspec. This\n\n*NIT* because\n\n> data is accessible from the 'struct remote' data that is now accessible\n> in fetch_remote().\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/gc.c | 33 ++++++++-------------------------\n> 1 file changed, 8 insertions(+), 25 deletions(-)\n> \n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index ef7226d7bca4..fa8128de9ae1 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n> \treturn 0;\n> }\n> \n> -static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n> +static int fetch_remote(struct remote *remote, void *cbdata)\n> {\n> +\tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n> \n> \tchild.git_cmd = 1;\n> -\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n> +\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n> \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n> \t\t \"--refmap=\", NULL);\n> \n> \tif (opts->quiet)\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \n> -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n> +\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n> \n> \treturn !!run_command(&child);\n> }\n> \n> -static int append_remote(struct remote *remote, void *cbdata)\n> -{\n> -\tstruct string_list *remotes = (struct string_list *)cbdata;\n> -\n> -\tstring_list_append(remotes, remote->name);\n> -\treturn 0;\n> -}\n> -\n> static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n> {\n> -\tint result = 0;\n> -\tstruct string_list_item *item;\n> -\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n> -\n> \tgit_config_set_multivar_gently(\"log.excludedecoration\",\n> \t\t\t\t\t\"refs/prefetch/\",\n> \t\t\t\t\t\"refs/prefetch/\",\n> \t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n> \t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n> \n> -\tif (for_each_remote(append_remote, &remotes)) {\n> -\t\terror(_(\"failed to fill remotes\"));\n> -\t\tresult = 1;\n> -\t\tgoto cleanup;\n> +\tif (for_each_remote(fetch_remote, opts)) {\n> +\t\terror(_(\"failed to prefetch remotes\"));\n> +\t\treturn 1;\n> \t}\n> \n> -\tfor_each_string_list_item(item, &remotes)\n> -\t\tresult |= fetch_remote(item->string, opts);\n> -\n> -cleanup:\n> -\tstring_list_clear(&remotes, 0);\n> -\treturn result;\n> +\treturn 0;\n> }\n> \n> static int maintenance_task_gc(struct maintenance_run_opts *opts)\n> -- \n> gitgitgadget\n> \n"},{"id":"420994","messageId":"20210405171652.xp63czmtsswqor2m@brm-x62-17.us.oracle.com","threadId":"55434","inReplyTo":"7f6c127dac48409ddc8d30ad236182bee21c1957.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-05T17:16:52Z","receivedAt":"2021-04-05T17:17:10Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Mon, Apr 05, 2021 at 01:04:15PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The prefetch task previously used the default refspec source plus a\n> custom refspec destination to avoid colliding with remote refs:\n> \n> \t+refs/heads/*:refs/prefetch//*\n> \n> However, some users customize their refspec to reduce how much data they\n> download from specific remotes. This can involve restrictive patterns\n> for fetching or negative patterns to avoid downloading some refs.\n> \n> Modify fetch_remote() to iterate over the remote's refspec list and\n> translate that into the appropriate prefetch scenario. Specifically,\n> re-parse the raw form of the refspec into a new 'struct refspec' and\n> modify the 'dst' member to replace a leading \"refs/\" substring with\n> \"refs/prefetch/\", or prepend \"refs/prefetch/\" to 'dst' otherwise.\n> Negative refspecs do not have a 'dst' so they can be transferred to the\n> 'git fetch' command unmodified.\n> \n> This prefix change provides the benefit of keeping whatever collisions\n> may exist in the custom refspecs, if that is a desirable outcome.\n> \n> This changes the names of the refs that would be fetched by the default\n> refspec. Instead of \"refs/prefetch//\" they will now go\n> to \"refs/prefetch/remotes//\". While this is a change, it\n> is not a seriously breaking one: these refs are intended to be hidden\n> and not used.\n> \n> Update the documentation to be more generic about the destination refs.\n> Do not mention custom refpecs explicitly, as that does not need to be\n> highlighted in this documentation. The important part of placing refs in\n> refs/prefetch remains.\n> \n> Reported-by: Tom Saeger \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-maintenance.txt | 3 +--\n> builtin/gc.c | 34 +++++++++++++++++++++++-\n> t/t7900-maintenance.sh | 43 ++++++++++++++++++++++++++-----\n> 3 files changed, 71 insertions(+), 9 deletions(-)\n> \n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 80ddd33ceba0..95a24264eb10 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -94,8 +94,7 @@ prefetch::\n> \tobjects from all registered remotes. For each remote, a `git fetch`\n> \tcommand is run. The refmap is custom to avoid updating local or remote\n> \tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n> -\tremote refs are stored in `refs/prefetch//`. Also, tags are\n> -\tnot updated.\n> +\trefs are stored in `refs/prefetch/`. Also, tags are not updated.\n> +\n> This is done to avoid disrupting the remote-tracking branches. The end users\n> expect these refs to stay unmoved unless they initiate a fetch. With prefetch\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index fa8128de9ae1..92cb8b4e0bfa 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -32,6 +32,7 @@\n> #include \"remote.h\"\n> #include \"object-store.h\"\n> #include \"exec-cmd.h\"\n> +#include \"refspec.h\"\n> \n> #define FAILED_RUN \"failed to run %s\"\n> \n> @@ -877,6 +878,7 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> {\n> \tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n> +\tint i;\n> \n> \tchild.git_cmd = 1;\n> \tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n> @@ -886,7 +888,37 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> \tif (opts->quiet)\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \n> -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n> +\tfor (i = 0; i < remote->fetch.nr; i++) {\n> +\t\tstruct refspec_item replace;\n> +\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n> +\t\tstruct strbuf new_dst = STRBUF_INIT;\n> +\t\tsize_t ignore_len = 0;\n> +\n> +\t\tif (rsi->negative) {\n> +\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n> +\n> +\t\t/*\n> +\t\t * If a refspec dst starts with \"refs/\" at the start,\n> +\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n> +\t\t * Otherwise, we will prepend the dst string with\n> +\t\t * \"refs/prefetch/\".\n> +\t\t */\n> +\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n> +\t\t\tignore_len = 5;\n> +\n> +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> +\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n> +\t\tfree(replace.dst);\n> +\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n> +\n> +\t\tstrvec_push(&child.args, refspec_item_format(&replace));\n\nsee comment on 3/5, think refspec_item_format is leaking here.\nthis code looks fine though.\n\n> +\n> +\t\trefspec_item_clear(&replace);\n> +\t}\n> \n> \treturn !!run_command(&child);\n> }\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index fc2315edec11..3366ea188782 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -142,20 +142,51 @@ test_expect_success 'prefetch multiple remotes' '\n> \ttest_commit -C clone2 two &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n> -\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* -\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remote2/* +\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote1/* +\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote2/* \ttest_path_is_missing .git/refs/remotes &&\n> -\tgit log prefetch/remote1/one &&\n> -\tgit log prefetch/remote2/two &&\n> +\tgit log prefetch/remotes/remote1/one &&\n> +\tgit log prefetch/remotes/remote2/two &&\n> \tgit fetch --all &&\n> -\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remote1/one &&\n> -\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remote2/two &&\n> +\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remotes/remote1/one &&\n> +\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remotes/remote2/two &&\n> \n> \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n> \tgit log --oneline --decorate --all >log &&\n> \t! grep \"prefetch\" log\n> '\n> \n> +test_expect_success 'prefetch custom refspecs' '\n> +\tgit -C clone1 branch -f special/fetched HEAD &&\n> +\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n> +\n> +\t# create multiple refspecs for remote1\n> +\tgit config --add remote.remote1.fetch +refs/heads/special/fetched:refs/heads/fetched &&\n> +\tgit config --add remote.remote1.fetch ^refs/heads/special/secret/not-fetched &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> +\n> +\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n> +\n> +\t# skips second refspec because it is not a pattern type\n> +\trs1=\"+refs/heads/*:refs/prefetch/remotes/remote1/*\" &&\n> +\trs2=\"+refs/heads/special/fetched:refs/prefetch/heads/fetched\" &&\n> +\trs3=\"^refs/heads/special/secret/not-fetched\" &&\n> +\n> +\ttest_subcommand git fetch remote1 $fetchargs $rs1 $rs2 $rs3 +\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote2/* +\n> +\t# first refspec is overridden by second\n> +\ttest_must_fail git rev-parse refs/prefetch/special/fetched &&\n> +\tgit rev-parse refs/prefetch/heads/fetched &&\n> +\n> +\t# possible incorrect places for the non-fetched ref\n> +\ttest_must_fail git rev-parse refs/prefetch/remotes/remote1/secret/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/prefetch/remotes/remote1/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/heads/secret/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/heads/not-fetched\n> +'\n> +\n> test_expect_success 'prefetch and existing log.excludeDecoration values' '\n> \tgit config --unset-all log.excludeDecoration &&\n> \tgit config log.excludeDecoration refs/remotes/remote1/ &&\n> -- \n> gitgitgadget\n"},{"id":"420995","messageId":"CAPig+cRaysm=-UcqY4zY5cV0KzAwd+RwzxXN-ZMV0nz=L4_+DA@mail.gmail.com","threadId":"55434","inReplyTo":"2b74889c2a323f03be477ffdf9ff388405779c3b.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/5] test-lib: use exact match for test_subcommand","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-04-05T17:31:42Z","receivedAt":"2021-04-05T17:31:55Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 5, 2021 at 9:04 AM Derrick Stolee via GitGitGadget\n wrote:\n> The use of 'grep' inside test_subcommand uses general patterns, leading\n> to sometimes needing escape characters to avoid incorrect matches.\n> Further, some platforms interpret different glob characters differently.\n\nThese are regular expression metacharacters, not glob characters. A\nmore general way to say this might be:\n\n Furthermore, it can be difficult to know which characters need\n escaping since the actual regular expression language implemented\n by various `grep`s differs between platforms; for instance, some\n may employ pure BRE, whereas others a mix of BRE & ERE.\n\n Sidestep this difficulty by using `grep -F`...\n\n> Use 'grep -F' to use an exact match. This requires removing escape\n> characters from existing callers. Luckily, this is only one test that\n> expects refspecs as part of the subcommand.\n>\n> Reported-by: Eric Sunshine \n> Signed-off-by: Derrick Stolee \n\nThe Reported-by: feels a bit unusual in this context. Perhaps\nHelped-by: would be more appropriate.\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> @@ -142,8 +142,8 @@ test_expect_success 'prefetch multiple remotes' '\n> - test_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* - test_subcommand git fetch remote2 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote2/\\\\* + test_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* + test_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remote2/* wrote:\n> On Mon, Apr 05, 2021 at 01:04:13PM +0000, Derrick Stolee via GitGitGadget wrote:\n> > +const char *refspec_item_format(const struct refspec_item *rsi)\n> > +{\n> > + static struct strbuf buf = STRBUF_INIT;\n> > +\n> > + strbuf_reset(&buf);\n>\n> is this even needed?\n\nThis is needed due to the `static` strbuf declaration (which is easy\nto overlook).\n\n> > + if (rsi->matching)\n> > + return \":\";\n> > +\n> > + if (rsi->negative)\n> > + strbuf_addch(&buf, '^');\n> > + else if (rsi->force)\n> > + strbuf_addch(&buf, '+');\n> > +\n> > + if (rsi->src)\n> > + strbuf_addstr(&buf, rsi->src);\n> > +\n> > + if (rsi->dst) {\n> > + strbuf_addch(&buf, ':');\n> > + strbuf_addstr(&buf, rsi->dst);\n> > + }\n> > +\n> > + return buf.buf;\n>\n> should this be strbuf_detach?\n\nIn normal circumstances, yes, however, with the `static` strbuf, this\nis correct.\n\nHowever, a more significant question, perhaps, is why this is using a\n`static` strbuf in the first place? Does this need to be optimized\nbecause it is on a hot path? If not, then the only obvious reason why\n`static` was chosen was that sometimes the function returns a string\nliteral and sometimes a constructed string. However, that's minor, and\nit would feel cleaner to avoid the `static` strbuf altogether by using\nstrbuf_detach() for the constructed case and xstrdup() for the string\nliteral case, and making it the caller's responsibility to free the\nresult. (The comment in the header file would need to be updated to\nsay as much.)\n"},{"id":"420997","messageId":"xmqq4kgk7hk9.fsf@gitster.g","threadId":"55434","inReplyTo":"CAPig+cRaysm=-UcqY4zY5cV0KzAwd+RwzxXN-ZMV0nz=L4_+DA@mail.gmail.com","subject":"Re: [PATCH 2/5] test-lib: use exact match for test_subcommand","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-05T17:43:50Z","receivedAt":"2021-04-05T17:43:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> On Mon, Apr 5, 2021 at 9:04 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> The use of 'grep' inside test_subcommand uses general patterns, leading\n>> to sometimes needing escape characters to avoid incorrect matches.\n>> Further, some platforms interpret different glob characters differently.\n>\n> These are regular expression metacharacters, not glob characters. A\n> more general way to say this might be:\n>\n> Furthermore, it can be difficult to know which characters need\n> escaping since the actual regular expression language implemented\n> by various `grep`s differs between platforms; for instance, some\n> may employ pure BRE, whereas others a mix of BRE & ERE.\n>\n> Sidestep this difficulty by using `grep -F`...\n>\n>> Use 'grep -F' to use an exact match. This requires removing escape\n>> characters from existing callers. Luckily, this is only one test that\n>> expects refspecs as part of the subcommand.\n>>\n>> Reported-by: Eric Sunshine \n>> Signed-off-by: Derrick Stolee \n>\n> The Reported-by: feels a bit unusual in this context. Perhaps\n> Helped-by: would be more appropriate.\n>\n>> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n>> @@ -142,8 +142,8 @@ test_expect_success 'prefetch multiple remotes' '\n>> - test_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* > - test_subcommand git fetch remote2 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote2/\\\\* > + test_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* > + test_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remote2/* \n> To be really robust and avoid accidental glob expansion (as unlikely\n> as it is), you should quote any arguments which contain glob\n> metacharacters such as \"*\" rather than supplying them bare like this.\n\nYup, just enclose the whole refspec inside dq-pair, like\n\n\ttest_subcommand git fetch remote2 $fetchargs \\\n\t\t\"+refs/heads/*:refs/prefetch/remote2/*\" writes:\n\n> On Mon, Apr 5, 2021 at 12:58 PM Tom Saeger wrote:\n>> On Mon, Apr 05, 2021 at 01:04:13PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> > +const char *refspec_item_format(const struct refspec_item *rsi)\n>> > +{\n>> > + static struct strbuf buf = STRBUF_INIT;\n>> > +\n>> > + strbuf_reset(&buf);\n>>\n>> is this even needed?\n>\n> This is needed due to the `static` strbuf declaration (which is easy\n> to overlook).\n>\n>> > + if (rsi->matching)\n>> > + return \":\";\n>> > +\n>> > + if (rsi->negative)\n>> > + strbuf_addch(&buf, '^');\n>> > + else if (rsi->force)\n>> > + strbuf_addch(&buf, '+');\n>> > +\n>> > + if (rsi->src)\n>> > + strbuf_addstr(&buf, rsi->src);\n>> > +\n>> > + if (rsi->dst) {\n>> > + strbuf_addch(&buf, ':');\n>> > + strbuf_addstr(&buf, rsi->dst);\n>> > + }\n>> > +\n>> > + return buf.buf;\n>>\n>> should this be strbuf_detach?\n>\n> In normal circumstances, yes, however, with the `static` strbuf, this\n> is correct.\n>\n> However, a more significant question, perhaps, is why this is using a\n> `static` strbuf in the first place? Does this need to be optimized\n> because it is on a hot path? If not, then the only obvious reason why\n> `static` was chosen was that sometimes the function returns a string\n> literal and sometimes a constructed string. However, that's minor, and\n> it would feel cleaner to avoid the `static` strbuf altogether by using\n> strbuf_detach() for the constructed case and xstrdup() for the string\n> literal case, and making it the caller's responsibility to free the\n> result. (The comment in the header file would need to be updated to\n> say as much.)\n\nVery good suggestion. That would also make this codepath\nthread-safe (I do not offhand know how important that is, though).\n\nThanks.\n"},{"id":"421000","messageId":"CAPig+cTCeED-Q_64+S5MDVFjCJdnCqYXWNS2ZDWKm4Ehjn=VXw@mail.gmail.com","threadId":"55434","inReplyTo":"c8d1de06f84499f2f56b3a06df665630806f94ce.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/5] test-tool: test refspec input/output","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-04-05T17:52:43Z","receivedAt":"2021-04-05T17:53:01Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 5, 2021 at 9:04 AM Derrick Stolee via GitGitGadget\n wrote:\n> Add a new test-helper, 'test-tool refspec', that currently reads stdin\n> line-by-line and translates the refspecs using the parsing logic of\n> refspec_item_init() and writes them to output.\n> [...]\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\n> @@ -0,0 +1,39 @@\n> +int cmd__refspec(int argc, const char **argv)\n> +{\n> + struct strbuf line = STRBUF_INIT;\n> + [...]\n> + return 0;\n> +}\n\nLeaking `strbuf line` here. Yes, I realize that the function is\nreturning and test-tool exiting immediately after this, so not a big\ndeal, but it's easy to do this correctly by releasing the strbuf, thus\nsetting good precedence for people who might use this as a template\nfor new test-tool functions they add in the future.\n\n> diff --git a/t/t5511-refspec.sh b/t/t5511-refspec.sh\n> @@ -93,4 +93,45 @@ test_refspec fetch \"refs/heads/${good}\"\n> +test_expect_success 'test input/output round trip' '\n> + cat >input <<-\\EOF &&\n> + +refs/heads/*:refs/remotes/origin/*\n> + refs/heads/*:refs/remotes/origin/*\n> + refs/heads/main:refs/remotes/frotz/xyzzy\n> + :refs/remotes/frotz/deleteme\n> + ^refs/heads/secrets\n> + refs/heads/secret:refs/heads/translated\n> + refs/heads/secret:heads/translated\n> + refs/heads/secret:remotes/translated\n> + secret:translated\n> + refs/heads/*:remotes/xxy/*\n> + refs/heads*/for-linus:refs/remotes/mine/*\n> + 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n> + HEAD\n> + @\n> + :\n> + EOF\n\nOver-indented heredoc body. It is customary[1] in this codebase for\nthe body and EOF to have the same indentation as the command which\nstarts the heredoc.\n\n> + cat >expect <<-\\EOF &&\n> + +refs/heads/*:refs/remotes/origin/*\n> + refs/heads/*:refs/remotes/origin/*\n> + refs/heads/main:refs/remotes/frotz/xyzzy\n> + :refs/remotes/frotz/deleteme\n> + ^refs/heads/secrets\n> + refs/heads/secret:refs/heads/translated\n> + refs/heads/secret:heads/translated\n> + refs/heads/secret:remotes/translated\n> + secret:translated\n> + refs/heads/*:remotes/xxy/*\n> + refs/heads*/for-linus:refs/remotes/mine/*\n> + 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n> + HEAD\n> + HEAD\n> + :\n> + EOF\n\nDitto.\n\n> + test-tool refspec output &&\n> + test_cmp expect output &&\n> + test-tool refspec --fetch output &&\n> + test_cmp expect output\n> +'\n\n[1]: https://lore.kernel.org/git/CAPig+cSBVG0AdyqXH2mZp6Ohrcb8_ec1Mm_vGbQM4zWT_7yYxQ@mail.gmail.com/\n"},{"id":"421054","messageId":"0c3989de-8fe9-b812-edb3-cab1e7227505@gmail.com","threadId":"55434","inReplyTo":"CAPig+cTCeED-Q_64+S5MDVFjCJdnCqYXWNS2ZDWKm4Ehjn=VXw@mail.gmail.com","subject":"Re: [PATCH 4/5] test-tool: test refspec input/output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-06T11:13:50Z","receivedAt":"2021-04-06T11:13:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/5/2021 1:52 PM, Eric Sunshine wrote:\n> On Mon, Apr 5, 2021 at 9:04 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> Add a new test-helper, 'test-tool refspec', that currently reads stdin\n>> line-by-line and translates the refspecs using the parsing logic of\n>> refspec_item_init() and writes them to output.\n>> [...]\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> diff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\n>> @@ -0,0 +1,39 @@\n>> +int cmd__refspec(int argc, const char **argv)\n>> +{\n>> + struct strbuf line = STRBUF_INIT;\n>> + [...]\n>> + return 0;\n>> +}\n> \n> Leaking `strbuf line` here. Yes, I realize that the function is\n> returning and test-tool exiting immediately after this, so not a big\n> deal, but it's easy to do this correctly by releasing the strbuf, thus\n> setting good precedence for people who might use this as a template\n> for new test-tool functions they add in the future.\n> \n>> diff --git a/t/t5511-refspec.sh b/t/t5511-refspec.sh\n>> @@ -93,4 +93,45 @@ test_refspec fetch \"refs/heads/${good}\"\n>> +test_expect_success 'test input/output round trip' '\n>> + cat >input <<-\\EOF &&\n>> + +refs/heads/*:refs/remotes/origin/*\n>> + refs/heads/*:refs/remotes/origin/*\n>> + refs/heads/main:refs/remotes/frotz/xyzzy\n>> + :refs/remotes/frotz/deleteme\n>> + ^refs/heads/secrets\n>> + refs/heads/secret:refs/heads/translated\n>> + refs/heads/secret:heads/translated\n>> + refs/heads/secret:remotes/translated\n>> + secret:translated\n>> + refs/heads/*:remotes/xxy/*\n>> + refs/heads*/for-linus:refs/remotes/mine/*\n>> + 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n>> + HEAD\n>> + @\n>> + :\n>> + EOF\n> \n> Over-indented heredoc body. It is customary[1] in this codebase for\n> the body and EOF to have the same indentation as the command which\n> starts the heredoc.\n\nGood catches. Thanks!\n-Stolee\n"},{"id":"421055","messageId":"41a4b531-d315-9743-34cc-d9f86c335f7d@gmail.com","threadId":"55434","inReplyTo":"20210405171652.xp63czmtsswqor2m@brm-x62-17.us.oracle.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-06T11:15:30Z","receivedAt":"2021-04-06T11:15:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/5/2021 1:16 PM, Tom Saeger wrote:\n> On Mon, Apr 05, 2021 at 01:04:15PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>> +\t\tstrvec_push(&child.args, refspec_item_format(&replace));\n> \n> see comment on 3/5, think refspec_item_format is leaking here.\n> this code looks fine though.\n\nI will respond to the comments on patch 3, but this is the reason\na static strbuf is used: we can print like this without needing\nto store the buffer in a variable and free() it here. Seemed like\nan easier-to-use API for a non-critical area of code. I'll\ncontinue the discussion over on that patch thread.\n\nThanks,\n-Stolee\n"},{"id":"421056","messageId":"37f0ff6c-b493-35b5-5ca0-92703f82e333@gmail.com","threadId":"55434","inReplyTo":"xmqqzgyc62yl.fsf@gitster.g","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-06T11:21:07Z","receivedAt":"2021-04-06T11:21:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/5/2021 1:44 PM, Junio C Hamano wrote:\n> Eric Sunshine writes:\n> \n>> On Mon, Apr 5, 2021 at 12:58 PM Tom Saeger wrote:\n>>> On Mon, Apr 05, 2021 at 01:04:13PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>>> +const char *refspec_item_format(const struct refspec_item *rsi)\n>>>> +{\n>>>> + static struct strbuf buf = STRBUF_INIT;\n>>>> +\n>>>> + strbuf_reset(&buf);\n>>>\n>>> is this even needed?\n>>\n>> This is needed due to the `static` strbuf declaration (which is easy\n>> to overlook).\n>>\n>>>> + if (rsi->matching)\n>>>> + return \":\";\n>>>> +\n>>>> + if (rsi->negative)\n>>>> + strbuf_addch(&buf, '^');\n>>>> + else if (rsi->force)\n>>>> + strbuf_addch(&buf, '+');\n>>>> +\n>>>> + if (rsi->src)\n>>>> + strbuf_addstr(&buf, rsi->src);\n>>>> +\n>>>> + if (rsi->dst) {\n>>>> + strbuf_addch(&buf, ':');\n>>>> + strbuf_addstr(&buf, rsi->dst);\n>>>> + }\n>>>> +\n>>>> + return buf.buf;\n>>>\n>>> should this be strbuf_detach?\n>>\n>> In normal circumstances, yes, however, with the `static` strbuf, this\n>> is correct.\n>>\n>> However, a more significant question, perhaps, is why this is using a\n>> `static` strbuf in the first place? Does this need to be optimized\n>> because it is on a hot path? If not, then the only obvious reason why\n>> `static` was chosen was that sometimes the function returns a string\n>> literal and sometimes a constructed string. However, that's minor, and\n>> it would feel cleaner to avoid the `static` strbuf altogether by using\n>> strbuf_detach() for the constructed case and xstrdup() for the string\n>> literal case, and making it the caller's responsibility to free the\n>> result. (The comment in the header file would need to be updated to\n>> say as much.)\n\nYes, we could get around the return of \":\" very easily.\n\n> Very good suggestion. That would also make this codepath\n> thread-safe (I do not offhand know how important that is, though).\n\nI was not intending to make this re-entrant/thread safe. The intention\nwas to make it easy to consume the formatted string into output such\nas a printf without needing to store a temporary 'char *' and free() it\nafterwards. This ensures that the only lost memory over the life of the\nprocess is at most one buffer. At minimum, these are things that could\nbe part of the message to justify this design.\n\nSo, I'm torn. This seems like a case where there is value in having\nthe return buffer be \"owned\" by this method, and the expected\nconsumers will use the buffer before calling it again. I'm not sure\nhow important it is to do this the other way.\n\nWould it be sufficient to justify this choice in the commit message\nand comment about it in the method declaration? Or is it worth adding\nthis templating around every caller:\n\n\tchar *buf = refspec_item_format(rsi);\n\t...\n\t\n\t...\n\tfree(buf);\n\nI don't need much convincing to do this, but I hadn't properly\ndescribed my opinion before. Just a small nudge would convince me to\ndo it this way.\n\nThanks,\n-Stolee\n"},{"id":"421069","messageId":"CAPig+cT8aqS1zaVqDSdycj6b9f3y7Mgn6TB9Sc1RY4_WomsY7Q@mail.gmail.com","threadId":"55434","inReplyTo":"37f0ff6c-b493-35b5-5ca0-92703f82e333@gmail.com","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-04-06T15:23:06Z","receivedAt":"2021-04-06T15:23:26Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Apr 6, 2021 at 7:21 AM Derrick Stolee wrote:\n> I was not intending to make this re-entrant/thread safe. The intention\n> was to make it easy to consume the formatted string into output such\n> as a printf without needing to store a temporary 'char *' and free() it\n> afterwards. This ensures that the only lost memory over the life of the\n> process is at most one buffer. At minimum, these are things that could\n> be part of the message to justify this design.\n\nThis has the failing that it won't work if someone calls it twice in\nthe same printf() or calls it again before even consuming the first\nreturned value, so this fails:\n\n printf(\"foo: %s\\nbar: %s\\n\",\n refspec_item_format(...),\n refspec_item_format(...));\n\nas does this:\n\n const char *a = refspec_item_format(...);\n const char *b = refspec_item_format(...);\n\nHistorically this project would \"work around\" that problem by using\nrotating static buffers in the function, but we've mostly been moving\naway from that for several reasons (can't predict how many buffers\nwill be needed, re-entrancy, etc.).\n\n> So, I'm torn. This seems like a case where there is value in having\n> the return buffer be \"owned\" by this method, and the expected\n> consumers will use the buffer before calling it again. I'm not sure\n> how important it is to do this the other way.\n\nIf history is any indication, we'd probably end up moving away from\nsuch an API eventually anyhow.\n\n> Would it be sufficient to justify this choice in the commit message\n> and comment about it in the method declaration? Or is it worth adding\n> this templating around every caller:\n>\n> char *buf = refspec_item_format(rsi);\n> ...\n> \n> ...\n> free(buf);\n\nAn alternative would be to have the caller pass in a strbuf to be\npopulated by the function. It doesn't reduce the boilerplate needed by\nthe caller (still need to create and release the strbuf), but may\navoid some memory allocations. But if this isn't a critical path and\nwon't likely ever be, then passing in strbuf may be overkill.\n\n> I don't need much convincing to do this, but I hadn't properly\n> described my opinion before. Just a small nudge would convince me to\n> do it this way.\n\nFor the reasons described above and earlier in the thread, avoiding\nthe static buffer seems the best course of action.\n"},{"id":"421071","messageId":"c5137224-abca-5ed1-8c8a-540d1a51ab96@gmail.com","threadId":"55434","inReplyTo":"CAPig+cT8aqS1zaVqDSdycj6b9f3y7Mgn6TB9Sc1RY4_WomsY7Q@mail.gmail.com","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-06T16:51:31Z","receivedAt":"2021-04-06T16:51:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/6/2021 11:23 AM, Eric Sunshine wrote:\n> On Tue, Apr 6, 2021 at 7:21 AM Derrick Stolee wrote:\n>> I was not intending to make this re-entrant/thread safe. The intention\n>> was to make it easy to consume the formatted string into output such\n>> as a printf without needing to store a temporary 'char *' and free() it\n>> afterwards. This ensures that the only lost memory over the life of the\n>> process is at most one buffer. At minimum, these are things that could\n>> be part of the message to justify this design.\n> \n> This has the failing that it won't work if someone calls it twice in\n> the same printf() or calls it again before even consuming the first\n> returned value, so this fails:\n> \n> printf(\"foo: %s\\nbar: %s\\n\",\n> refspec_item_format(...),\n> refspec_item_format(...));\n> \n> as does this:\n> \n> const char *a = refspec_item_format(...);\n> const char *b = refspec_item_format(...);\n> \n> Historically this project would \"work around\" that problem by using\n> rotating static buffers in the function, but we've mostly been moving\n> away from that for several reasons (can't predict how many buffers\n> will be needed, re-entrancy, etc.).\n> \n>> So, I'm torn. This seems like a case where there is value in having\n>> the return buffer be \"owned\" by this method, and the expected\n>> consumers will use the buffer before calling it again. I'm not sure\n>> how important it is to do this the other way.\n> \n> If history is any indication, we'd probably end up moving away from\n> such an API eventually anyhow.\n> \n>> Would it be sufficient to justify this choice in the commit message\n>> and comment about it in the method declaration? Or is it worth adding\n>> this templating around every caller:\n>>\n>> char *buf = refspec_item_format(rsi);\n>> ...\n>> \n>> ...\n>> free(buf);\n> \n> An alternative would be to have the caller pass in a strbuf to be\n> populated by the function. It doesn't reduce the boilerplate needed by\n> the caller (still need to create and release the strbuf), but may\n> avoid some memory allocations. But if this isn't a critical path and\n> won't likely ever be, then passing in strbuf may be overkill.\n> \n>> I don't need much convincing to do this, but I hadn't properly\n>> described my opinion before. Just a small nudge would convince me to\n>> do it this way.\n> \n> For the reasons described above and earlier in the thread, avoiding\n> the static buffer seems the best course of action.\n\nOK, convinced. I'll return a string that must be freed in my\nnext version. Thanks!\n\n-Stolee\n"},{"id":"421082","messageId":"5aa0cb06c3f2e31a51bb6747fa84de69c0c1212f.1617734870.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v2.git.1617734870.gitgitgadget@gmail.com","subject":"[PATCH v2 1/5] maintenance: simplify prefetch logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-06T18:47:46Z","receivedAt":"2021-04-06T18:47:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous logic filled a string list with the names of each remote,\nbut instead we could simply run the appropriate 'git fetch' data\ndirectly in the remote iterator. Do this for reduced code size, but also\nbecause it sets up an upcoming change to use the remote's refspec. This\ndata is accessible from the 'struct remote' data that is now accessible\nin fetch_remote().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 33 ++++++++-------------------------\n 1 file changed, 8 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ef7226d7bca4..fa8128de9ae1 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n+static int fetch_remote(struct remote *remote, void *cbdata)\n {\n+\tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n-\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n+\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n \t\t \"--refmap=\", NULL);\n \n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n+\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n \n \treturn !!run_command(&child);\n }\n \n-static int append_remote(struct remote *remote, void *cbdata)\n-{\n-\tstruct string_list *remotes = (struct string_list *)cbdata;\n-\n-\tstring_list_append(remotes, remote->name);\n-\treturn 0;\n-}\n-\n static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n {\n-\tint result = 0;\n-\tstruct string_list_item *item;\n-\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n-\n \tgit_config_set_multivar_gently(\"log.excludedecoration\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n \t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n \n-\tif (for_each_remote(append_remote, &remotes)) {\n-\t\terror(_(\"failed to fill remotes\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n+\tif (for_each_remote(fetch_remote, opts)) {\n+\t\terror(_(\"failed to prefetch remotes\"));\n+\t\treturn 1;\n \t}\n \n-\tfor_each_string_list_item(item, &remotes)\n-\t\tresult |= fetch_remote(item->string, opts);\n-\n-cleanup:\n-\tstring_list_clear(&remotes, 0);\n-\treturn result;\n+\treturn 0;\n }\n \n static int maintenance_task_gc(struct maintenance_run_opts *opts)\n-- \ngitgitgadget\n\n"},{"id":"421083","messageId":"d58a3e042ee84641290b15496e96f959ef86a122.1617734870.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v2.git.1617734870.gitgitgadget@gmail.com","subject":"[PATCH v2 2/5] test-lib: use exact match for test_subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-06T18:47:47Z","receivedAt":"2021-04-06T18:47:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe use of 'grep' inside test_subcommand uses general patterns, leading\nto sometimes needing escape characters to avoid incorrect matches.\nFurther, some platforms interpret regular expression metacharacters\ndifferently. Furthermore, it can be difficult to know which characters\nneed escaping since the actual regular expression language implemented\nby various `grep`s differs between platforms; for instance, some may\nemploy pure BRE, whereas others a mix of BRE & ERE.\n\nSidestep this difficulty by using `grep -F` to use an exact match. This\nrequires removing escape characters from existing callers. Luckily,\nthis is only one test that expects refspecs as part of the subcommand.\n\nHelped-by: Eric Sunshine \nSigned-off-by: Derrick Stolee \n---\n t/t7900-maintenance.sh | 4 ++--\n t/test-lib-functions.sh | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 2412d8c5c006..37eed6ed3aa3 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -142,8 +142,8 @@ test_expect_success 'prefetch multiple remotes' '\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n-\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* \n\nAdd a new test-helper, 'test-tool refspec', that currently reads stdin\nline-by-line and translates the refspecs using the parsing logic of\nrefspec_item_init() and writes them to output.\n\nCreate a test in t5511-refspec.sh that uses this helper to test several\nknown special cases. This includes all of the special properties of the\n'struct refspec_item', including:\n\n * force: The refspec starts with '+'.\n * pattern: Each side of the refspec has a glob character ('*')\n * matching: The refspec is simply the string \":\".\n * exact_sha1: The 'src' string is a 40-character hex string.\n * negative: The refspec starts with '^' and 'dst' is NULL.\n\nWhile the exact_sha1 property doesn't require special logic in\nrefspec_item_format, it is still tested here for completeness.\n\nThere is also the special-case refspec \"@\" which translates to \"HEAD\".\n\nNote that if a refspec does not start with \"refs/\", then that is not\nincorporated as part of the 'struct refspec_item'. This behavior is\nconfirmed by these tests. These refspecs still work in the wild because\nthe refs layer interprets them appropriately as branches, prepending\n\"refs/\" or \"refs/heads/\" as necessary. I spent some time attempting to\ninsert these prefixes explicitly in parse_refspec(), but these are\nseveral subtleties I was unable to overcome. If such a change were to be\nmade, then this new test in t5511-refspec.sh will need to be updated\nwith new output. For example, the input lines ending with \"translated\"\nare designed to demonstrate these subtleties.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n t/helper/test-refspec.c | 44 +++++++++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5511-refspec.sh | 41 ++++++++++++++++++++++++++++++++++++++\n 5 files changed, 88 insertions(+)\n create mode 100644 t/helper/test-refspec.c\n\ndiff --git a/Makefile b/Makefile\nindex a6a73c574191..f858c9f25976 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -734,6 +734,7 @@ TEST_BUILTINS_OBJS += test-reach.o\n TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n+TEST_BUILTINS_OBJS += test-refspec.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\ndiff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\nnew file mode 100644\nindex 000000000000..b06735ded208\n--- /dev/null\n+++ b/t/helper/test-refspec.c\n@@ -0,0 +1,44 @@\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"refspec.h\"\n+#include \"strbuf.h\"\n+#include \"test-tool.h\"\n+\n+static const char * const refspec_usage[] = {\n+\tN_(\"test-tool refspec [--fetch]\"),\n+\tNULL\n+};\n+\n+int cmd__refspec(int argc, const char **argv)\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\tint fetch = 0;\n+\n+\tstruct option refspec_options [] = {\n+\t\tOPT_BOOL(0, \"fetch\", &fetch,\n+\t\t\t N_(\"enable the 'fetch' option for parsing refpecs\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, refspec_options,\n+\t\t\t refspec_usage, 0);\n+\n+\twhile (strbuf_getline(&line, stdin) != EOF) {\n+\t\tstruct refspec_item rsi;\n+\t\tchar *buf;\n+\n+\t\tif (!refspec_item_init(&rsi, line.buf, fetch)) {\n+\t\t\tprintf(\"failed to parse %s\\n\", line.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbuf = refspec_item_format(&rsi);\n+\t\tprintf(\"%s\\n\", buf);\n+\t\tfree(buf);\n+\n+\t\trefspec_item_clear(&rsi);\n+\t}\n+\n+\tstrbuf_release(&line);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 287aa6002307..f534ad1731a9 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -55,6 +55,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n+\t{ \"refspec\", cmd__refspec },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9ea4b31011dd..46a0b8850f17 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -44,6 +44,7 @@ int cmd__reach(int argc, const char **argv);\n int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n+int cmd__refspec(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\ndiff --git a/t/t5511-refspec.sh b/t/t5511-refspec.sh\nindex be025b90f989..489bec08d570 100755\n--- a/t/t5511-refspec.sh\n+++ b/t/t5511-refspec.sh\n@@ -93,4 +93,45 @@ test_refspec fetch \"refs/heads/${good}\"\n bad=$(printf '\\011tab')\n test_refspec fetch \"refs/heads/${bad}\"\t\t\t\tinvalid\n \n+test_expect_success 'test input/output round trip' '\n+\tcat >input <<-\\EOF &&\n+\t+refs/heads/*:refs/remotes/origin/*\n+\trefs/heads/*:refs/remotes/origin/*\n+\trefs/heads/main:refs/remotes/frotz/xyzzy\n+\t:refs/remotes/frotz/deleteme\n+\t^refs/heads/secrets\n+\trefs/heads/secret:refs/heads/translated\n+\trefs/heads/secret:heads/translated\n+\trefs/heads/secret:remotes/translated\n+\tsecret:translated\n+\trefs/heads/*:remotes/xxy/*\n+\trefs/heads*/for-linus:refs/remotes/mine/*\n+\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n+\tHEAD\n+\t@\n+\t:\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\t+refs/heads/*:refs/remotes/origin/*\n+\trefs/heads/*:refs/remotes/origin/*\n+\trefs/heads/main:refs/remotes/frotz/xyzzy\n+\t:refs/remotes/frotz/deleteme\n+\t^refs/heads/secrets\n+\trefs/heads/secret:refs/heads/translated\n+\trefs/heads/secret:heads/translated\n+\trefs/heads/secret:remotes/translated\n+\tsecret:translated\n+\trefs/heads/*:remotes/xxy/*\n+\trefs/heads*/for-linus:refs/remotes/mine/*\n+\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n+\tHEAD\n+\tHEAD\n+\t:\n+\tEOF\n+\ttest-tool refspec output &&\n+\ttest_cmp expect output &&\n+\ttest-tool refspec --fetch output &&\n+\ttest_cmp expect output\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"421085","messageId":"pull.924.v2.git.1617734870.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.git.1617627856.gitgitgadget@gmail.com","subject":"[PATCH v2 0/5] Maintenance: adapt custom refspecs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-06T18:47:45Z","receivedAt":"2021-04-06T18:47:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Tom Saeger rightly pointed out [1] that the prefetch task ignores custom\nrefspecs. This can lead to downloading more data than requested, and it\ndoesn't even help the future foreground fetches that use that custom\nrefspec.\n\n[1]\nhttps://lore.kernel.org/git/20210401184914.qmr7jhjbhp2mt3h6@dhcp-10-154-148-175.vpn.oracle.com/\n\nThis series fixes this problem by carefully replacing the start of each\nrefspec's destination with \"refs/prefetch/\". If the destination already\nstarts with \"refs/\", then that is replaced. Otherwise \"refs/prefetch/\" is\njust prepended.\n\nIn order to accomplish this safely, a new refspec_item_format() method is\ncreated and tested.\n\nPatch 1 is just a preparation patch that makes the code simpler (and in\nhindsight it should have been written this way from the start).\n\nPatch 2 is a simplification of test_subcommand that removes the need for\nescaping glob characters. Thanks, Eric Sunshine, for the tip of why my tests\nwere failing on FreeBSD.\n\nPatches 3-4 add refspec_item_format().\n\nPatch 5 finally modifies the logic in the prefetch task to translate these\nrefspecs.\n\n\nUpdates in V2\n=============\n\nThanks for the close eye on this series. I appreciate the recommendations,\nwhich I believe I have responded to them all:\n\n * Fixed typos.\n * Made refspec_item_format() re-entrant. Consumers must free the buffer.\n * Cleaned up style (quoting and tabbing).\n\nThanks, -Stolee\n\nDerrick Stolee (5):\n maintenance: simplify prefetch logic\n test-lib: use exact match for test_subcommand\n refspec: output a refspec item\n test-tool: test refspec input/output\n maintenance: allow custom refspecs during prefetch\n\n Documentation/git-maintenance.txt | 3 +-\n Makefile | 1 +\n builtin/gc.c | 66 ++++++++++++++++++++-----------\n refspec.c | 23 +++++++++++\n refspec.h | 2 +\n t/helper/test-refspec.c | 44 +++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5511-refspec.sh | 41 +++++++++++++++++++\n t/t7900-maintenance.sh | 43 +++++++++++++++++---\n t/test-lib-functions.sh | 4 +-\n 11 files changed, 195 insertions(+), 34 deletions(-)\n create mode 100644 t/helper/test-refspec.c\n\n\nbase-commit: 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-924%2Fderrickstolee%2Fmaintenance%2Frefspec-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-924/derrickstolee/maintenance/refspec-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/924\n\nRange-diff vs v1:\n\n 1: 3a94ff80657c ! 1: 5aa0cb06c3f2 maintenance: simplify prefetch logic\n @@ Commit message\n The previous logic filled a string list with the names of each remote,\n but instead we could simply run the appropriate 'git fetch' data\n directly in the remote iterator. Do this for reduced code size, but also\n - becuase it sets up an upcoming change to use the remote's refspec. This\n + because it sets up an upcoming change to use the remote's refspec. This\n data is accessible from the 'struct remote' data that is now accessible\n in fetch_remote().\n \n 2: 2b74889c2a32 ! 2: d58a3e042ee8 test-lib: use exact match for test_subcommand\n @@ Commit message\n \n The use of 'grep' inside test_subcommand uses general patterns, leading\n to sometimes needing escape characters to avoid incorrect matches.\n - Further, some platforms interpret different glob characters differently.\n + Further, some platforms interpret regular expression metacharacters\n + differently. Furthermore, it can be difficult to know which characters\n + need escaping since the actual regular expression language implemented\n + by various `grep`s differs between platforms; for instance, some may\n + employ pure BRE, whereas others a mix of BRE & ERE.\n \n - Use 'grep -F' to use an exact match. This requires removing escape\n - characters from existing callers. Luckily, this is only one test that\n - expects refspecs as part of the subcommand.\n + Sidestep this difficulty by using `grep -F` to use an exact match. This\n + requires removing escape characters from existing callers. Luckily,\n + this is only one test that expects refspecs as part of the subcommand.\n \n - Reported-by: Eric Sunshine \n + Helped-by: Eric Sunshine \n Signed-off-by: Derrick Stolee \n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n -\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* exact_sha1 = 0;\n }\n \n -+const char *refspec_item_format(const struct refspec_item *rsi)\n ++char *refspec_item_format(const struct refspec_item *rsi)\n +{\n -+\tstatic struct strbuf buf = STRBUF_INIT;\n -+\n -+\tstrbuf_reset(&buf);\n ++\tstruct strbuf buf = STRBUF_INIT;\n +\n +\tif (rsi->matching)\n -+\t\treturn \":\";\n ++\t\treturn xstrdup(\":\");\n +\n +\tif (rsi->negative)\n +\t\tstrbuf_addch(&buf, '^');\n @@ refspec.c: void refspec_item_clear(struct refspec_item *item)\n +\t\tstrbuf_addstr(&buf, rsi->dst);\n +\t}\n +\n -+\treturn buf.buf;\n ++\treturn strbuf_detach(&buf, NULL);\n +}\n +\n void refspec_init(struct refspec *rs, int fetch)\n @@ refspec.h: int refspec_item_init(struct refspec_item *item, const char *refspec,\n void refspec_item_init_or_die(struct refspec_item *item, const char *refspec,\n \t\t\t int fetch);\n void refspec_item_clear(struct refspec_item *item);\n -+/*\n -+ * Output a given refspec item to a string.\n -+ */\n -+const char *refspec_item_format(const struct refspec_item *rsi);\n ++char *refspec_item_format(const struct refspec_item *rsi);\n +\n void refspec_init(struct refspec *rs, int fetch);\n void refspec_append(struct refspec *rs, const char *refspec);\n 4: c8d1de06f844 ! 4: bf296282323a test-tool: test refspec input/output\n @@ t/helper/test-refspec.c (new)\n +\n +\twhile (strbuf_getline(&line, stdin) != EOF) {\n +\t\tstruct refspec_item rsi;\n ++\t\tchar *buf;\n +\n +\t\tif (!refspec_item_init(&rsi, line.buf, fetch)) {\n +\t\t\tprintf(\"failed to parse %s\\n\", line.buf);\n +\t\t\tcontinue;\n +\t\t}\n +\n -+\t\tprintf(\"%s\\n\", refspec_item_format(&rsi));\n ++\t\tbuf = refspec_item_format(&rsi);\n ++\t\tprintf(\"%s\\n\", buf);\n ++\t\tfree(buf);\n ++\n +\t\trefspec_item_clear(&rsi);\n +\t}\n +\n ++\tstrbuf_release(&line);\n +\treturn 0;\n +}\n \n @@ t/t5511-refspec.sh: test_refspec fetch \"refs/heads/${good}\"\n \n +test_expect_success 'test input/output round trip' '\n +\tcat >input <<-\\EOF &&\n -+\t\t+refs/heads/*:refs/remotes/origin/*\n -+\t\trefs/heads/*:refs/remotes/origin/*\n -+\t\trefs/heads/main:refs/remotes/frotz/xyzzy\n -+\t\t:refs/remotes/frotz/deleteme\n -+\t\t^refs/heads/secrets\n -+\t\trefs/heads/secret:refs/heads/translated\n -+\t\trefs/heads/secret:heads/translated\n -+\t\trefs/heads/secret:remotes/translated\n -+\t\tsecret:translated\n -+\t\trefs/heads/*:remotes/xxy/*\n -+\t\trefs/heads*/for-linus:refs/remotes/mine/*\n -+\t\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n -+\t\tHEAD\n -+\t\t@\n -+\t\t:\n ++\t+refs/heads/*:refs/remotes/origin/*\n ++\trefs/heads/*:refs/remotes/origin/*\n ++\trefs/heads/main:refs/remotes/frotz/xyzzy\n ++\t:refs/remotes/frotz/deleteme\n ++\t^refs/heads/secrets\n ++\trefs/heads/secret:refs/heads/translated\n ++\trefs/heads/secret:heads/translated\n ++\trefs/heads/secret:remotes/translated\n ++\tsecret:translated\n ++\trefs/heads/*:remotes/xxy/*\n ++\trefs/heads*/for-linus:refs/remotes/mine/*\n ++\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n ++\tHEAD\n ++\t@\n ++\t:\n +\tEOF\n +\tcat >expect <<-\\EOF &&\n -+\t\t+refs/heads/*:refs/remotes/origin/*\n -+\t\trefs/heads/*:refs/remotes/origin/*\n -+\t\trefs/heads/main:refs/remotes/frotz/xyzzy\n -+\t\t:refs/remotes/frotz/deleteme\n -+\t\t^refs/heads/secrets\n -+\t\trefs/heads/secret:refs/heads/translated\n -+\t\trefs/heads/secret:heads/translated\n -+\t\trefs/heads/secret:remotes/translated\n -+\t\tsecret:translated\n -+\t\trefs/heads/*:remotes/xxy/*\n -+\t\trefs/heads*/for-linus:refs/remotes/mine/*\n -+\t\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n -+\t\tHEAD\n -+\t\tHEAD\n -+\t\t:\n ++\t+refs/heads/*:refs/remotes/origin/*\n ++\trefs/heads/*:refs/remotes/origin/*\n ++\trefs/heads/main:refs/remotes/frotz/xyzzy\n ++\t:refs/remotes/frotz/deleteme\n ++\t^refs/heads/secrets\n ++\trefs/heads/secret:refs/heads/translated\n ++\trefs/heads/secret:heads/translated\n ++\trefs/heads/secret:remotes/translated\n ++\tsecret:translated\n ++\trefs/heads/*:remotes/xxy/*\n ++\trefs/heads*/for-linus:refs/remotes/mine/*\n ++\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n ++\tHEAD\n ++\tHEAD\n ++\t:\n +\tEOF\n +\ttest-tool refspec output &&\n +\ttest_cmp expect output &&\n 5: 7f6c127dac48 ! 5: 9592224e3d42 maintenance: allow custom refspecs during prefetch\n @@ builtin/gc.c: static int fetch_remote(struct remote *remote, void *cbdata)\n +\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n +\t\tstruct strbuf new_dst = STRBUF_INIT;\n +\t\tsize_t ignore_len = 0;\n ++\t\tchar *replace_string;\n +\n +\t\tif (rsi->negative) {\n +\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n @@ builtin/gc.c: static int fetch_remote(struct remote *remote, void *cbdata)\n +\t\tfree(replace.dst);\n +\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n +\n -+\t\tstrvec_push(&child.args, refspec_item_format(&replace));\n ++\t\treplace_string = refspec_item_format(&replace);\n ++\t\tstrvec_push(&child.args, replace_string);\n ++\t\tfree(replace_string);\n +\n +\t\trefspec_item_clear(&replace);\n +\t}\n @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n --\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* /dev/null &&\n +\n @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n +\trs2=\"+refs/heads/special/fetched:refs/prefetch/heads/fetched\" &&\n +\trs3=\"^refs/heads/special/secret/not-fetched\" &&\n +\n -+\ttest_subcommand git fetch remote1 $fetchargs $rs1 $rs2 $rs3 \n\nAdd a new method, refspec_item_format(), that takes a 'struct\nrefspec_item' pointer as input and returns a string for how that refspec\nitem should be written to Git's config or a subcommand, such as 'git\nfetch'.\n\nThere are several subtleties regarding special-case refspecs that can\noccur and are represented in t5511-refspec.sh. These cases will be\nexplored in new tests in the following change. It requires adding a new\ntest helper in order to test this format directly, so that is saved for\na separate change to keep this one focused on the logic of the format\nmethod.\n\nA future change will consume this method when translating refspecs in\nthe 'prefetch' task of the 'git maintenance' builtin.\n\nSigned-off-by: Derrick Stolee \n---\n refspec.c | 23 +++++++++++++++++++++++\n refspec.h | 2 ++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/refspec.c b/refspec.c\nindex e3d852c0bfec..e79cde3c58be 100644\n--- a/refspec.c\n+++ b/refspec.c\n@@ -180,6 +180,29 @@ void refspec_item_clear(struct refspec_item *item)\n \titem->exact_sha1 = 0;\n }\n \n+char *refspec_item_format(const struct refspec_item *rsi)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tif (rsi->matching)\n+\t\treturn xstrdup(\":\");\n+\n+\tif (rsi->negative)\n+\t\tstrbuf_addch(&buf, '^');\n+\telse if (rsi->force)\n+\t\tstrbuf_addch(&buf, '+');\n+\n+\tif (rsi->src)\n+\t\tstrbuf_addstr(&buf, rsi->src);\n+\n+\tif (rsi->dst) {\n+\t\tstrbuf_addch(&buf, ':');\n+\t\tstrbuf_addstr(&buf, rsi->dst);\n+\t}\n+\n+\treturn strbuf_detach(&buf, NULL);\n+}\n+\n void refspec_init(struct refspec *rs, int fetch)\n {\n \tmemset(rs, 0, sizeof(*rs));\ndiff --git a/refspec.h b/refspec.h\nindex 8b79891d3218..9f2ddc7949a1 100644\n--- a/refspec.h\n+++ b/refspec.h\n@@ -56,6 +56,8 @@ int refspec_item_init(struct refspec_item *item, const char *refspec,\n void refspec_item_init_or_die(struct refspec_item *item, const char *refspec,\n \t\t\t int fetch);\n void refspec_item_clear(struct refspec_item *item);\n+char *refspec_item_format(const struct refspec_item *rsi);\n+\n void refspec_init(struct refspec *rs, int fetch);\n void refspec_append(struct refspec *rs, const char *refspec);\n __attribute__((format (printf,2,3)))\n-- \ngitgitgadget\n\n"},{"id":"421087","messageId":"9592224e3d428762c6f9b38a0bcc2cee5c3dff6c.1617734871.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v2.git.1617734870.gitgitgadget@gmail.com","subject":"[PATCH v2 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-06T18:47:50Z","receivedAt":"2021-04-06T18:48:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe prefetch task previously used the default refspec source plus a\ncustom refspec destination to avoid colliding with remote refs:\n\n\t+refs/heads/*:refs/prefetch//*\n\nHowever, some users customize their refspec to reduce how much data they\ndownload from specific remotes. This can involve restrictive patterns\nfor fetching or negative patterns to avoid downloading some refs.\n\nModify fetch_remote() to iterate over the remote's refspec list and\ntranslate that into the appropriate prefetch scenario. Specifically,\nre-parse the raw form of the refspec into a new 'struct refspec' and\nmodify the 'dst' member to replace a leading \"refs/\" substring with\n\"refs/prefetch/\", or prepend \"refs/prefetch/\" to 'dst' otherwise.\nNegative refspecs do not have a 'dst' so they can be transferred to the\n'git fetch' command unmodified.\n\nThis prefix change provides the benefit of keeping whatever collisions\nmay exist in the custom refspecs, if that is a desirable outcome.\n\nThis changes the names of the refs that would be fetched by the default\nrefspec. Instead of \"refs/prefetch//\" they will now go\nto \"refs/prefetch/remotes//\". While this is a change, it\nis not a seriously breaking one: these refs are intended to be hidden\nand not used.\n\nUpdate the documentation to be more generic about the destination refs.\nDo not mention custom refpecs explicitly, as that does not need to be\nhighlighted in this documentation. The important part of placing refs in\nrefs/prefetch remains.\n\nReported-by: Tom Saeger \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 3 +--\n builtin/gc.c | 37 +++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 43 ++++++++++++++++++++++++++-----\n 3 files changed, 74 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 80ddd33ceba0..95a24264eb10 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -94,8 +94,7 @@ prefetch::\n \tobjects from all registered remotes. For each remote, a `git fetch`\n \tcommand is run. The refmap is custom to avoid updating local or remote\n \tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n-\tremote refs are stored in `refs/prefetch//`. Also, tags are\n-\tnot updated.\n+\trefs are stored in `refs/prefetch/`. Also, tags are not updated.\n +\n This is done to avoid disrupting the remote-tracking branches. The end users\n expect these refs to stay unmoved unless they initiate a fetch. With prefetch\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex fa8128de9ae1..76f347dd6b11 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -32,6 +32,7 @@\n #include \"remote.h\"\n #include \"object-store.h\"\n #include \"exec-cmd.h\"\n+#include \"refspec.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -877,6 +878,7 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n {\n \tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n+\tint i;\n \n \tchild.git_cmd = 1;\n \tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n@@ -886,7 +888,40 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n+\tfor (i = 0; i < remote->fetch.nr; i++) {\n+\t\tstruct refspec_item replace;\n+\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n+\t\tstruct strbuf new_dst = STRBUF_INIT;\n+\t\tsize_t ignore_len = 0;\n+\t\tchar *replace_string;\n+\n+\t\tif (rsi->negative) {\n+\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n+\n+\t\t/*\n+\t\t * If a refspec dst starts with \"refs/\" at the start,\n+\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n+\t\t * Otherwise, we will prepend the dst string with\n+\t\t * \"refs/prefetch/\".\n+\t\t */\n+\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n+\t\t\tignore_len = 5;\n+\n+\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n+\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n+\t\tfree(replace.dst);\n+\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n+\n+\t\treplace_string = refspec_item_format(&replace);\n+\t\tstrvec_push(&child.args, replace_string);\n+\t\tfree(replace_string);\n+\n+\t\trefspec_item_clear(&replace);\n+\t}\n \n \treturn !!run_command(&child);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 37eed6ed3aa3..03487be3af38 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -142,20 +142,51 @@ test_expect_success 'prefetch multiple remotes' '\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n-\ttest_subcommand git fetch remote1 $fetchargs \"+refs/heads/*:refs/prefetch/remote1/*\" log &&\n \t! grep \"prefetch\" log\n '\n \n+test_expect_success 'prefetch custom refspecs' '\n+\tgit -C clone1 branch -f special/fetched HEAD &&\n+\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n+\n+\t# create multiple refspecs for remote1\n+\tgit config --add remote.remote1.fetch \"+refs/heads/special/fetched:refs/heads/fetched\" &&\n+\tgit config --add remote.remote1.fetch \"^refs/heads/special/secret/not-fetched\" &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n+\n+\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n+\n+\t# skips second refspec because it is not a pattern type\n+\trs1=\"+refs/heads/*:refs/prefetch/remotes/remote1/*\" &&\n+\trs2=\"+refs/heads/special/fetched:refs/prefetch/heads/fetched\" &&\n+\trs3=\"^refs/heads/special/secret/not-fetched\" &&\n+\n+\ttest_subcommand git fetch remote1 $fetchargs \"$rs1\" \"$rs2\" \"$rs3\" From: Derrick Stolee \n> \n> The prefetch task previously used the default refspec source plus a\n> custom refspec destination to avoid colliding with remote refs:\n> \n> \t+refs/heads/*:refs/prefetch//*\n> \n> However, some users customize their refspec to reduce how much data they\n> download from specific remotes. This can involve restrictive patterns\n> for fetching or negative patterns to avoid downloading some refs.\n> \n> Modify fetch_remote() to iterate over the remote's refspec list and\n> translate that into the appropriate prefetch scenario. Specifically,\n> re-parse the raw form of the refspec into a new 'struct refspec' and\n> modify the 'dst' member to replace a leading \"refs/\" substring with\n> \"refs/prefetch/\", or prepend \"refs/prefetch/\" to 'dst' otherwise.\n> Negative refspecs do not have a 'dst' so they can be transferred to the\n> 'git fetch' command unmodified.\n> \n> This prefix change provides the benefit of keeping whatever collisions\n> may exist in the custom refspecs, if that is a desirable outcome.\n> \n> This changes the names of the refs that would be fetched by the default\n> refspec. Instead of \"refs/prefetch//\" they will now go\n> to \"refs/prefetch/remotes//\". While this is a change, it\n> is not a seriously breaking one: these refs are intended to be hidden\n> and not used.\n> \n> Update the documentation to be more generic about the destination refs.\n> Do not mention custom refpecs explicitly, as that does not need to be\n> highlighted in this documentation. The important part of placing refs in\n> refs/prefetch remains.\n> \n> Reported-by: Tom Saeger \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-maintenance.txt | 3 +--\n> builtin/gc.c | 37 +++++++++++++++++++++++++-\n> t/t7900-maintenance.sh | 43 ++++++++++++++++++++++++++-----\n> 3 files changed, 74 insertions(+), 9 deletions(-)\n> \n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 80ddd33ceba0..95a24264eb10 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -94,8 +94,7 @@ prefetch::\n> \tobjects from all registered remotes. For each remote, a `git fetch`\n> \tcommand is run. The refmap is custom to avoid updating local or remote\n> \tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n> -\tremote refs are stored in `refs/prefetch//`. Also, tags are\n> -\tnot updated.\n> +\trefs are stored in `refs/prefetch/`. Also, tags are not updated.\n> +\n> This is done to avoid disrupting the remote-tracking branches. The end users\n> expect these refs to stay unmoved unless they initiate a fetch. With prefetch\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index fa8128de9ae1..76f347dd6b11 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -32,6 +32,7 @@\n> #include \"remote.h\"\n> #include \"object-store.h\"\n> #include \"exec-cmd.h\"\n> +#include \"refspec.h\"\n> \n> #define FAILED_RUN \"failed to run %s\"\n> \n> @@ -877,6 +878,7 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> {\n> \tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n> +\tint i;\n> \n> \tchild.git_cmd = 1;\n> \tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n> @@ -886,7 +888,40 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> \tif (opts->quiet)\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \n> -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n> +\tfor (i = 0; i < remote->fetch.nr; i++) {\n> +\t\tstruct refspec_item replace;\n> +\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n> +\t\tstruct strbuf new_dst = STRBUF_INIT;\n> +\t\tsize_t ignore_len = 0;\n> +\t\tchar *replace_string;\n> +\n> +\t\tif (rsi->negative) {\n> +\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n> +\n> +\t\t/*\n> +\t\t * If a refspec dst starts with \"refs/\" at the start,\n> +\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n> +\t\t * Otherwise, we will prepend the dst string with\n> +\t\t * \"refs/prefetch/\".\n> +\t\t */\n> +\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n> +\t\t\tignore_len = 5;\n> +\n> +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> +\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n> +\t\tfree(replace.dst);\n> +\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n> +\n> +\t\treplace_string = refspec_item_format(&replace);\n> +\t\tstrvec_push(&child.args, replace_string);\n> +\t\tfree(replace_string);\n> +\n> +\t\trefspec_item_clear(&replace);\n> +\t}\n> \n> \treturn !!run_command(&child);\n> }\n\nJunio brought up the point about configs which 'fetch' have no dst\nhttps://lore.kernel.org/git/c06a198a-2043-27a2-cab3-3471190754cc@gmail.com/\n\n [remote \"submaintainer1\"]\n url = ... repository of submaintainer #1 ...\n fetch = master\n tagopt = --no-tags\n\n\nThis patch fixes segfault for config like above.\nYou might have ideas on a cleaner way to do this.\nI did add `child_process_clear`.\n\n\n--Tom\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 76f347dd6b11..921266ee30a5 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -879,6 +879,7 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n struct maintenance_run_opts *opts = cbdata;\n struct child_process child = CHILD_PROCESS_INIT;\n int i;\n+ int nargs;\n\n child.git_cmd = 1;\n strvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n@@ -888,6 +889,8 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n if (opts->quiet)\n strvec_push(&child.args, \"--quiet\");\n\n+ nargs = child.args.nr;\n+\n for (i = 0; i < remote->fetch.nr; i++) {\n struct refspec_item replace;\n struct refspec_item *rsi = &remote->fetch.items[i];\n@@ -900,6 +903,10 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n continue;\n }\n\n+ if (!rsi->dst) {\n+ continue;\n+ }\n+\n refspec_item_init(&replace, remote->fetch.raw[i], 1);\n\n /*\n@@ -923,6 +930,12 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n refspec_item_clear(&replace);\n }\n\n+ /* skip remote if no refspecs to fetch */\n+ if (child.args.nr - nargs <= 0) {\n+ child_process_clear(&child);\n+ return 0;\n+ }\n"},{"id":"421091","messageId":"35fa99cf-ba24-f13e-0607-f9acce2905cf@gmail.com","threadId":"55434","inReplyTo":"20210406193637.x2xqkkyqbjabgzge@dhcp-10-154-113-61.vpn.oracle.com","subject":"Re: [PATCH v2 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-06T19:45:32Z","receivedAt":"2021-04-06T19:45:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/6/2021 3:36 PM, Tom Saeger wrote:\n>\n> Junio brought up the point about configs which 'fetch' have no dst\n> https://lore.kernel.org/git/c06a198a-2043-27a2-cab3-3471190754cc@gmail.com/\n\nThank you for reminding me about this. It was on the other thread,\nand I forgot to go back to it as I was preparing this version.\n\n> [remote \"submaintainer1\"]\n> url = ... repository of submaintainer #1 ...\n> fetch = master\n> tagopt = --no-tags\n> \n> \n> This patch fixes segfault for config like above.\n> You might have ideas on a cleaner way to do this.\n> I did add `child_process_clear`.\n\nI will also add a test to ensure this scenario does not regress.\n-Stolee\n"},{"id":"421111","messageId":"87r1jmjxdg.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"e10007e1cf8ff0005295f648b9489c11a9427122.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-07T08:46:03Z","receivedAt":"2021-04-07T08:47:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> Add a new method, refspec_item_format(), that takes a 'struct\n> refspec_item' pointer as input and returns a string for how that refspec\n> item should be written to Git's config or a subcommand, such as 'git\n> fetch'.\n>\n> There are several subtleties regarding special-case refspecs that can\n> occur and are represented in t5511-refspec.sh. These cases will be\n> explored in new tests in the following change. It requires adding a new\n> test helper in order to test this format directly, so that is saved for\n> a separate change to keep this one focused on the logic of the format\n> method.\n>\n> A future change will consume this method when translating refspecs in\n> the 'prefetch' task of the 'git maintenance' builtin.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> refspec.c | 25 +++++++++++++++++++++++++\n> refspec.h | 5 +++++\n> 2 files changed, 30 insertions(+)\n>\n> diff --git a/refspec.c b/refspec.c\n> index e3d852c0bfec..ca65ba01bfe6 100644\n> --- a/refspec.c\n> +++ b/refspec.c\n> @@ -180,6 +180,31 @@ void refspec_item_clear(struct refspec_item *item)\n> \titem->exact_sha1 = 0;\n> }\n> \n> +const char *refspec_item_format(const struct refspec_item *rsi)\n> +{\n> +\tstatic struct strbuf buf = STRBUF_INIT;\n> +\n> +\tstrbuf_reset(&buf);\n> +\n> +\tif (rsi->matching)\n> +\t\treturn \":\";\n> +\n> +\tif (rsi->negative)\n> +\t\tstrbuf_addch(&buf, '^');\n> +\telse if (rsi->force)\n> +\t\tstrbuf_addch(&buf, '+');\n> +\n> +\tif (rsi->src)\n> +\t\tstrbuf_addstr(&buf, rsi->src);\n> +\n> +\tif (rsi->dst) {\n> +\t\tstrbuf_addch(&buf, ':');\n> +\t\tstrbuf_addstr(&buf, rsi->dst);\n> +\t}\n> +\n> +\treturn buf.buf;\n\nThere's a downthread discussion about the strbuf usage here so that's\ncovered.\n\nBut I'm still confused about the need for this function and the\nfollowing two patches. If we apply this on top of your series:\n \n diff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\n index 08cf441a0a0..9e099e43ebf 100644\n --- a/t/helper/test-refspec.c\n +++ b/t/helper/test-refspec.c\n @@ -31,7 +31,7 @@ int cmd__refspec(int argc, const char **argv)\n continue;\n }\n \n - printf(\"%s\\n\", refspec_item_format(&rsi));\n + puts(line.buf);\n refspec_item_clear(&rsi);\n }\n\nThe only failing test is:\n \n + diff -u expect output\n --- expect 2021-04-07 08:12:05.577598038 +0000\n +++ output 2021-04-07 08:12:05.577598038 +0000\n @@ -11,5 +11,5 @@\n refs/heads*/for-linus:refs/remotes/mine/*\n 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n HEAD\n -HEAD\n +@\n :\n\nOther than that applying this on top makes everything pass:\n \n diff --git a/builtin/gc.c b/builtin/gc.c\n index 92cb8b4e0bf..ea5572d15c5 100644\n --- a/builtin/gc.c\n +++ b/builtin/gc.c\n @@ -889,35 +889,21 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n \tfor (i = 0; i < remote->fetch.nr; i++) {\n -\t\tstruct refspec_item replace;\n \t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n -\t\tstruct strbuf new_dst = STRBUF_INIT;\n -\t\tsize_t ignore_len = 0;\n +\t\tstruct strbuf new_spec = STRBUF_INIT;\n +\t\tchar *pos;\n \n \t\tif (rsi->negative) {\n \t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n \t\t\tcontinue;\n \t\t}\n \n -\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n -\n -\t\t/*\n -\t\t * If a refspec dst starts with \"refs/\" at the start,\n -\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n -\t\t * Otherwise, we will prepend the dst string with\n -\t\t * \"refs/prefetch/\".\n -\t\t */\n -\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n -\t\t\tignore_len = 5;\n -\n -\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n -\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n -\t\tfree(replace.dst);\n -\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n -\n -\t\tstrvec_push(&child.args, refspec_item_format(&replace));\n -\n -\t\trefspec_item_clear(&replace);\n +\t\tstrbuf_addstr(&new_spec, remote->fetch.raw[i]);\n +\t\tif ((pos = strrchr(new_spec.buf, ':')) != NULL)\n +\t\t\tstrbuf_splice(&new_spec, pos - new_spec.buf + 1, sizeof(\"refs/\") - 1,\n +\t\t\t\t \"refs/prefetch/\", sizeof(\"refs/prefetch/\") - 1);\n +\t\tstrvec_push(&child.args, new_spec.buf);\n +\t\tstrbuf_release(&new_spec);\n \t}\n \n \treturn !!run_command(&child);\n\nSo the purpose of this new API is that we don't want to make the\nassumption that strrchr(buf, ':') is a safe way to find the delimiter in\nthe refspec, or is there some case where we grok \"HEAD\" but not \"@\"\nthat's buggy, but not tested for in this series?\n"},{"id":"421112","messageId":"87o8eqjx0w.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"7f6c127dac48409ddc8d30ad236182bee21c1957.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-07T08:53:35Z","receivedAt":"2021-04-07T08:54:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n\n> [...]\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index fc2315edec11..3366ea188782 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -142,20 +142,51 @@ test_expect_success 'prefetch multiple remotes' '\n> \ttest_commit -C clone2 two &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n> -\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* -\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remote2/* +\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote1/* +\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote2/* \ttest_path_is_missing .git/refs/remotes &&\n> -\tgit log prefetch/remote1/one &&\n> -\tgit log prefetch/remote2/two &&\n> +\tgit log prefetch/remotes/remote1/one &&\n> +\tgit log prefetch/remotes/remote2/two &&\n> \tgit fetch --all &&\n> -\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remote1/one &&\n> -\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remote2/two &&\n> +\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remotes/remote1/one &&\n> +\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remotes/remote2/two &&\n> \n> \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n> \tgit log --oneline --decorate --all >log &&\n> \t! grep \"prefetch\" log\n> '\n> \n> +test_expect_success 'prefetch custom refspecs' '\n> +\tgit -C clone1 branch -f special/fetched HEAD &&\n> +\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n> +\n> +\t# create multiple refspecs for remote1\n> +\tgit config --add remote.remote1.fetch +refs/heads/special/fetched:refs/heads/fetched &&\n> +\tgit config --add remote.remote1.fetch ^refs/heads/special/secret/not-fetched &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n\nI see this is following some established convention in the file, but is\nthere really not a way to make this pass without directing stderr to\n/dev/null? It makes ad-hoc debugging when reviewing harder.\n\n\nI tried just removing it, but then (in an earlier test case) the\n\"test_subcommand\" fails because it can't find the line we're looking\nfor, so us piping stderr to /dev/null impacts our trace2 output?\n\n"},{"id":"421113","messageId":"87lf9ujwym.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"c8d1de06f84499f2f56b3a06df665630806f94ce.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/5] test-tool: test refspec input/output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-07T08:54:57Z","receivedAt":"2021-04-07T08:55:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n\n> +static const char * const refspec_usage[] = {\n> +\tN_(\"test-tool refspec [--fetch]\"),\n> +\tNULL\n> +};\n> +\n> +int cmd__refspec(int argc, const char **argv)\n> +{\n> +\tstruct strbuf line = STRBUF_INIT;\n> +\tint fetch = 0;\n> +\n> +\tstruct option refspec_options [] = {\n> +\t\tOPT_BOOL(0, \"fetch\", &fetch,\n> +\t\t\t N_(\"enable the 'fetch' option for parsing refpecs\")),\n> +\t\tOPT_END()\n> +\t};\n\nI don't think we should waste translator time by marking these (or\nanything else in t/helper) with N_()).\n\nI see you probably copied this from elsewhere & we should probably fix\nthe existing ones, but no reason to add new ones...\n"},{"id":"421114","messageId":"87im4yjspp.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"87o8eqjx0w.fsf@evledraar.gmail.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-07T10:26:42Z","receivedAt":"2021-04-07T10:26:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 07 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n>\n>> [...]\n>> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n>> index fc2315edec11..3366ea188782 100755\n>> --- a/t/t7900-maintenance.sh\n>> +++ b/t/t7900-maintenance.sh\n>> @@ -142,20 +142,51 @@ test_expect_success 'prefetch multiple remotes' '\n>> \ttest_commit -C clone2 two &&\n>> \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n>> \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n>> -\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* > -\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remote2/* > +\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote1/* > +\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote2/* > \ttest_path_is_missing .git/refs/remotes &&\n>> -\tgit log prefetch/remote1/one &&\n>> -\tgit log prefetch/remote2/two &&\n>> +\tgit log prefetch/remotes/remote1/one &&\n>> +\tgit log prefetch/remotes/remote2/two &&\n>> \tgit fetch --all &&\n>> -\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remote1/one &&\n>> -\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remote2/two &&\n>> +\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remotes/remote1/one &&\n>> +\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remotes/remote2/two &&\n>> \n>> \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n>> \tgit log --oneline --decorate --all >log &&\n>> \t! grep \"prefetch\" log\n>> '\n>> \n>> +test_expect_success 'prefetch custom refspecs' '\n>> +\tgit -C clone1 branch -f special/fetched HEAD &&\n>> +\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n>> +\n>> +\t# create multiple refspecs for remote1\n>> +\tgit config --add remote.remote1.fetch +refs/heads/special/fetched:refs/heads/fetched &&\n>> +\tgit config --add remote.remote1.fetch ^refs/heads/special/secret/not-fetched &&\n>> +\n>> +\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n>\n> I see this is following some established convention in the file, but is\n> there really not a way to make this pass without directing stderr to\n> /dev/null? It makes ad-hoc debugging when reviewing harder.\n\nAs I later found out this is copy/pasted to get around the fact that\n--quiet is dependent on isatty(), so without this the result would be\ndifferent under --verbose and non-verbose testing.\n\nSo that dates back to 3ddaad0e060 (maintenance: add --quiet option,\n2020-09-17), but I see other quiet=isatty(2) in related code. I wish we\ncould isolate that particular behavior so removing the 2>/dev/null when\ndebugging the tests doesn't cause you to run into this, maybe an\nexplicit --quiet or --no-quiet option for all but one test that's\nchecking that isatty() behavior?\n\n> I tried just removing it, but then (in an earlier test case) the\n> \"test_subcommand\" fails because it can't find the line we're looking\n> for, so us piping stderr to /dev/null impacts our trace2 output?\n\nI hadn't seen seen test_subcommand before, sorry to be blunt, but \"ew!\".\n\nSo we're ad-hoc grepping trace2 JSON output just to find out whether we\ninvoked some subcommand. But unlike test_expect_code etc. this one\ndoesn't run git for you, but instead we have temp *.txt files and the\ncommand disconnected from the run.\n\nAnd because you're using \"grep\" and \"! grep\" to test, you're hiding the\ndifference between \"did not find this line\" v.s. \"did not find anything\nat all\".\n\nBecause of that the second test using test_subcommand is either buggy or\npainfully non-obvious. We check that \"run --auto\" doesn't contain a\n\"auto --quiet\", but in reality it doesn't contain any subcommands at\nall. We didn't run any because it exited with \"nothing to pack\".\n\nI think converting the whole thing to something like the WIP/RFC patch\nbelow is much better and more readable.\n\nThe pattern is basically stolen from test_commit and\ncheck_sub_test_lib_test_err, respectively.\n\nAs an aside: The new test_expect_process_tree function would be much\nless painful if we had a helper that took the JSON output and emitted\nsome sensible subset of the information therein. For now I just ad-hoc\ngrepped the \"perf\" output. AFAICT the only way to get \"depth\" from\ntrace2's JSON so to count slashes in SIDs.\n\nIn the case of your tests you're mostly/(only?) interested in a slice of\nthe \"child_start\" events. If we had a helper to spew out a pretty-print\nversion of that (or a subset of it) we could test_cmp against that\ndirectly.\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 3366ea18878..d03fb361562 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -30,28 +30,41 @@ test_expect_success 'help text' '\n '\n \n test_expect_success 'run [--auto|--quiet]' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" \\\n-\t\tgit maintenance run 2>/dev/null &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" \\\n-\t\tgit maintenance run --auto 2>/dev/null &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-quiet.txt\" \\\n-\t\tgit maintenance run --no-quiet 2>/dev/null &&\n-\ttest_subcommand git gc --quiet .enabled' '\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex a5915dec22d..cd1187b473c 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1625,6 +1625,38 @@ test_path_is_hidden () {\n \treturn 1\n }\n \n+test_expect_process_tree () {\n+\tdepth= &&\n+\t>actual &&\n+\tcat >expect &&\n+\tcat <&3 >expect.err\n+\twhile test $# != 0\n+\tdo\n+\t\tcase \"$1\" in\n+\t\t--depth)\n+\t\t\tdepth=\"$2\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t*)\n+\t\t\tbreak\n+\t\t\t;;\n+\t\tesac\n+\t\tshift\n+\tdone &&\n+\tlog=\"$(pwd)/proc-tree.txt\" &&\n+\t>\"$log\" &&\n+\tGIT_TRACE2_PERF=\"$log\" \"$@\" 2>actual.err &&\n+\tgrep \"child_start\" proc-tree.txt >proc-tree-start.txt || : &&\n+\tif test -n \"$depth\"\n+\tthen\n+\t\tgrep \" d$depth \" proc-tree-start.txt >tmp.txt || : &&\n+\t\tmv tmp.txt proc-tree-start.txt\n+\tfi &&\n+\tsed -e 's/^.*argv:\\[//' -e 's/\\]$//' actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect.err actual.err\n+} 7>&2 2>&4\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n"},{"id":"421119","messageId":"87ft02jjfq.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"7f6c127dac48409ddc8d30ad236182bee21c1957.1617627856.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-07T13:47:05Z","receivedAt":"2021-04-07T13:47:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n\n> This changes the names of the refs that would be fetched by the default\n> refspec. Instead of \"refs/prefetch//\" they will now go\n> to \"refs/prefetch/remotes//\". While this is a change, it\n> is not a seriously breaking one: these refs are intended to be hidden\n> and not used.\n\nNot \"a seriously breaking one\" just because we'll assume nobody had a\nremote they'd named \"remotes\" and they'll need to manually clean that\nmess up (if needed), or ...?\n\n> [...]\n> \tobjects from all registered remotes. For each remote, a `git fetch`\n> \tcommand is run. The refmap is custom to avoid updating local or remote\n> \tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n> -\tremote refs are stored in `refs/prefetch//`. Also, tags are\n> -\tnot updated.\n> +\trefs are stored in `refs/prefetch/`. Also, tags are not updated.\n\nSo, \"tags are not updated\", but:\n\n> \n> -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n> +\tfor (i = 0; i < remote->fetch.nr; i++) {\n> +\t\tstruct refspec_item replace;\n> +\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n> +\t\tstruct strbuf new_dst = STRBUF_INIT;\n> +\t\tsize_t ignore_len = 0;\n> +\n> +\t\tif (rsi->negative) {\n> +\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n> +\n> +\t\t/*\n> +\t\t * If a refspec dst starts with \"refs/\" at the start,\n> +\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n> +\t\t * Otherwise, we will prepend the dst string with\n> +\t\t * \"refs/prefetch/\".\n> +\t\t */\n> +\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n> +\t\t\tignore_len = 5;\n> +\n> +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> +\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n> +\t\tfree(replace.dst);\n> +\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n> +\n> +\t\tstrvec_push(&child.args, refspec_item_format(&replace));\n> +\n> +\t\trefspec_item_clear(&replace);\n> +\t}\n\nIsn't a blanket replacement of refs/heads/* with refs/* going to change\nthat? I haven't tested this so maybe it still doesn't work, but:\n\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n> -\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* -\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remote2/* +\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote1/* +\ttest_subcommand git fetch remote2 $fetchargs +refs/heads/*:refs/prefetch/remotes/remote2/* [...]\n>\n> +\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n\nIt seems we should at least have a test for the case of having a refspec\nthat pulls down tags.\n\nI suspect that we could document this as an absolute before, as\nrefs/heads/* is the only namespace that'll refuse tags, but now that we\nfetch refs/ that'll no longer be the case.\n\nI'd think that this new behavior (if I'm right) is a feature, but that\nwe need to update the docs/tests appropriately.\n"},{"id":"421152","messageId":"b3e00d3e-c782-9f2a-14e0-f576e50a7e55@gmail.com","threadId":"55434","inReplyTo":"87r1jmjxdg.fsf@evledraar.gmail.com","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-07T20:53:10Z","receivedAt":"2021-04-07T20:53:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/7/2021 4:46 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n>> +\treturn buf.buf;\n> \n> There's a downthread discussion about the strbuf usage here so that's\n> covered.\n\nAnd it's fixed in v2.\n\n> But I'm still confused about the need for this function and the\n> following two patches. If we apply this on top of your series:\n> \n> diff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\n> index 08cf441a0a0..9e099e43ebf 100644\n> --- a/t/helper/test-refspec.c\n> +++ b/t/helper/test-refspec.c\n> @@ -31,7 +31,7 @@ int cmd__refspec(int argc, const char **argv)\n> continue;\n> }\n> \n> - printf(\"%s\\n\", refspec_item_format(&rsi));\n> + puts(line.buf);\n> refspec_item_clear(&rsi);\n> }\n> \n> The only failing test is:\n> \n> + diff -u expect output\n> --- expect 2021-04-07 08:12:05.577598038 +0000\n> +++ output 2021-04-07 08:12:05.577598038 +0000\n> @@ -11,5 +11,5 @@\n> refs/heads*/for-linus:refs/remotes/mine/*\n> 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n> HEAD\n> -HEAD\n> +@\n> :\n\nIt should be obvious that taking refspecs as input, parsing them,\nthen reformatting them for output should be almost equivalent to\nprinting the input line.\n\nThe point is to exercise the logic that actually formats the\nrefspec for output. The test-tool clearly does this.\n\nThe logic for converting a 'struct refspec_item' to a string is\nnon-trivial and worth testing. I don't understand why you are\nconcerned that the black-box of the test-tool could be done\nmore easily to \"trick\" the test script.\n\n> So the purpose of this new API is that we don't want to make the\n> assumption that strrchr(buf, ':') is a safe way to find the delimiter in\n> the refspec, or is there some case where we grok \"HEAD\" but not \"@\"\n> that's buggy, but not tested for in this series?\n\nThe purpose is to allow us to modify a 'struct refspec_item' andproduce a refspec string instead of munging a refspec string\ndirectly.\n\nThanks,\n-Stolee\n \n"},{"id":"421160","messageId":"87czv5kaxw.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"b3e00d3e-c782-9f2a-14e0-f576e50a7e55@gmail.com","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-07T22:05:15Z","receivedAt":"2021-04-07T22:05:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 07 2021, Derrick Stolee wrote:\n\n> On 4/7/2021 4:46 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n>>> +\treturn buf.buf;\n>> \n>> There's a downthread discussion about the strbuf usage here so that's\n>> covered.\n>\n> And it's fixed in v2.\n>\n>> But I'm still confused about the need for this function and the\n>> following two patches. If we apply this on top of your series:\n>> \n>> diff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\n>> index 08cf441a0a0..9e099e43ebf 100644\n>> --- a/t/helper/test-refspec.c\n>> +++ b/t/helper/test-refspec.c\n>> @@ -31,7 +31,7 @@ int cmd__refspec(int argc, const char **argv)\n>> continue;\n>> }\n>> \n>> - printf(\"%s\\n\", refspec_item_format(&rsi));\n>> + puts(line.buf);\n>> refspec_item_clear(&rsi);\n>> }\n>> \n>> The only failing test is:\n>> \n>> + diff -u expect output\n>> --- expect 2021-04-07 08:12:05.577598038 +0000\n>> +++ output 2021-04-07 08:12:05.577598038 +0000\n>> @@ -11,5 +11,5 @@\n>> refs/heads*/for-linus:refs/remotes/mine/*\n>> 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n>> HEAD\n>> -HEAD\n>> +@\n>> :\n>\n> It should be obvious that taking refspecs as input, parsing them,\n> then reformatting them for output should be almost equivalent to\n> printing the input line.\n>\n> The point is to exercise the logic that actually formats the\n> refspec for output. The test-tool clearly does this.\n>\n> The logic for converting a 'struct refspec_item' to a string is\n> non-trivial and worth testing. I don't understand why you are\n> concerned that the black-box of the test-tool could be done\n> more easily to \"trick\" the test script.\n\nYes, but why do we need to convert it to a struct refspec_item in the\nfirst place?\n\nMaybe I'm just overly comfortable with string munging but I think the\nsmaller patch-on-top to use strbuf_splice() is simpler than adding a new\nAPI just for this use-case.\n\nBut I'm still wondering if that @ v.s. HEAD case is something this\nseries actually needs in its end goal (but then has a missing test?), or\nif it was just a \"let's test the guts of the refspec.c while we're at\nit\".\n\n>> So the purpose of this new API is that we don't want to make the\n>> assumption that strrchr(buf, ':') is a safe way to find the delimiter in\n>> the refspec, or is there some case where we grok \"HEAD\" but not \"@\"\n>> that's buggy, but not tested for in this series?\n>\n> The purpose is to allow us to modify a 'struct refspec_item' andproduce a refspec string instead of munging a refspec string\n> directly.\n\nBut aren't we doing that all over the place, e.g. the grep results for\n\"refspec_appendf\". Even for things purely constructed on the C API level\nwe pass a const char* now.\n\nI'm not saying it wouldn't be nice to have the refspec.c API changed to\nhave a clear delimitation between its const char* handling, and C-level\nuses which could construct and pass a \"struct refspec_item\" instead.\n\nBut is it *needed* here in a way that I've missed, or is this just a\npartial testing/refactoring of that API while we're at it?\n\n[Guessing ahead here because of our TZ difference]:\n\nIt seems to me that if this is such a partial refactoring it's a strange\nway to go about it.\n\nWe're left with freeing/munging the \"struct refspec\" src/dst in-place\nand re-constructing a string that has \"+\" etc., but we already had that\ndata in parse_refspec() just before we'd call\nrefspec_item_format(). That function could then just spew out a\npre-formatted string we'd squirreled away in \"struct refspec_item\".\n\nIf the lengthy paragraph you have at the end of 4/5 holds true, then\nsuch an internal representation doesn't need to have the \"refs/\" prefix\nstores as a const char* (in cases where it's not just \"*\" or whatever),\nno?. We'd then be able to more easily init/copy/munge the refspec for\nformatting.\n"},{"id":"421168","messageId":"xmqqy2dtww0h.fsf@gitster.g","threadId":"55434","inReplyTo":"87czv5kaxw.fsf@evledraar.gmail.com","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-07T22:49:18Z","receivedAt":"2021-04-07T22:49:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Wed, Apr 07 2021, Derrick Stolee wrote:\n>\n>> The purpose is to allow us to modify a 'struct refspec_item'\n>> andproduce a refspec string instead of munging a refspec string\n>> directly.\n\nOuch. I thought the goal was to take\n\n [remote \"origin\"]\n\tfetch = $src:$dst\n\nlet the code that is used in the actual fetching to parse it into\nthe in-core \"refspec_item\", and then transform the refspec_item by\n\n - discarding it if the item does not result in storing in the real\n fetch\n\n - tweaking $dst side so that it won't touch anywhere outside\n refs/prefetch/ to avoid disturbing end-user's notion of what the\n latest state of the remote ref is.\n\nso that the \"parsed\" refspec_item is passed to the fetch machinery\nwithout ever having to be converted back to textual form.\n\nWhy do we even need to \"andproduce a refspec string\"?\n"},{"id":"421179","messageId":"871rblk8ct.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"xmqqy2dtww0h.fsf@gitster.g","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-07T23:01:06Z","receivedAt":"2021-04-07T23:01:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Apr 08 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> On Wed, Apr 07 2021, Derrick Stolee wrote:\n>>\n>>> The purpose is to allow us to modify a 'struct refspec_item'\n>>> andproduce a refspec string instead of munging a refspec string\n>>> directly.\n>\n> Ouch. I thought the goal was to take\n>\n> [remote \"origin\"]\n> \tfetch = $src:$dst\n>\n> let the code that is used in the actual fetching to parse it into\n> the in-core \"refspec_item\", and then transform the refspec_item by\n>\n> - discarding it if the item does not result in storing in the real\n> fetch\n>\n> - tweaking $dst side so that it won't touch anywhere outside\n> refs/prefetch/ to avoid disturbing end-user's notion of what the\n> latest state of the remote ref is.\n>\n> so that the \"parsed\" refspec_item is passed to the fetch machinery\n> without ever having to be converted back to textual form.\n>\n> Why do we even need to \"andproduce a refspec string\"?\n\nWe're shelling out to \"git fetch\", but we first munge the refspec on in\n\"git gc\".\n\nBut yes, it seems even more straightforward to do away with passing the\nrefspec at all to \"git fetch\", and instead pass some (maybe internal\nonly, and documented as such) \"--refspec-dst-prefix=refs/prefetch/\"\noption to \"git fetch\".\n\nI.e. get_ref_map() over there is already doing a version of this loop\nover \"remote->fetch.nr\".\n\nSo instead of \"git gc\" doing the loop, then passing all the refspecs on\nthe command-line, it could tell \"git fetch\" to do the same munging when\nit does the same iteration.\n\nDoing the munging in builtin/gc.c's fetch_remote() just seems like a\nrelic from when we didn't care what decision builtin/fetch.c made about\nrefspecs, we always wanted our custom one.\n\n"},{"id":"421183","messageId":"YG47Yu8VPiRuUBsM@google.com","threadId":"55434","inReplyTo":"bf296282323aa74f7c5b1a49c74d5aaa4b6f37ae.1617734870.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/5] test-tool: test refspec input/output","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-04-07T23:08:18Z","receivedAt":"2021-04-07T23:08:26Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.04.06 18:47, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Add a new test-helper, 'test-tool refspec', that currently reads stdin\n> line-by-line and translates the refspecs using the parsing logic of\n> refspec_item_init() and writes them to output.\n> \n> Create a test in t5511-refspec.sh that uses this helper to test several\n> known special cases. This includes all of the special properties of the\n> 'struct refspec_item', including:\n> \n> * force: The refspec starts with '+'.\n> * pattern: Each side of the refspec has a glob character ('*')\n> * matching: The refspec is simply the string \":\".\n> * exact_sha1: The 'src' string is a 40-character hex string.\n> * negative: The refspec starts with '^' and 'dst' is NULL.\n> \n> While the exact_sha1 property doesn't require special logic in\n> refspec_item_format, it is still tested here for completeness.\n> \n> There is also the special-case refspec \"@\" which translates to \"HEAD\".\n> \n> Note that if a refspec does not start with \"refs/\", then that is not\n> incorporated as part of the 'struct refspec_item'. This behavior is\n> confirmed by these tests. These refspecs still work in the wild because\n> the refs layer interprets them appropriately as branches, prepending\n> \"refs/\" or \"refs/heads/\" as necessary. I spent some time attempting to\n> insert these prefixes explicitly in parse_refspec(), but these are\n> several subtleties I was unable to overcome. If such a change were to be\n> made, then this new test in t5511-refspec.sh will need to be updated\n> with new output. For example, the input lines ending with \"translated\"\n> are designed to demonstrate these subtleties.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Makefile | 1 +\n> t/helper/test-refspec.c | 44 +++++++++++++++++++++++++++++++++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/t5511-refspec.sh | 41 ++++++++++++++++++++++++++++++++++++++\n> 5 files changed, 88 insertions(+)\n> create mode 100644 t/helper/test-refspec.c\n> \n> diff --git a/Makefile b/Makefile\n> index a6a73c574191..f858c9f25976 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -734,6 +734,7 @@ TEST_BUILTINS_OBJS += test-reach.o\n> TEST_BUILTINS_OBJS += test-read-cache.o\n> TEST_BUILTINS_OBJS += test-read-graph.o\n> TEST_BUILTINS_OBJS += test-read-midx.o\n> +TEST_BUILTINS_OBJS += test-refspec.o\n> TEST_BUILTINS_OBJS += test-ref-store.o\n> TEST_BUILTINS_OBJS += test-regex.o\n> TEST_BUILTINS_OBJS += test-repository.o\n> diff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c\n> new file mode 100644\n> index 000000000000..b06735ded208\n> --- /dev/null\n> +++ b/t/helper/test-refspec.c\n> @@ -0,0 +1,44 @@\n> +#include \"cache.h\"\n> +#include \"parse-options.h\"\n> +#include \"refspec.h\"\n> +#include \"strbuf.h\"\n> +#include \"test-tool.h\"\n> +\n> +static const char * const refspec_usage[] = {\n> +\tN_(\"test-tool refspec [--fetch]\"),\n> +\tNULL\n> +};\n> +\n> +int cmd__refspec(int argc, const char **argv)\n> +{\n> +\tstruct strbuf line = STRBUF_INIT;\n> +\tint fetch = 0;\n> +\n> +\tstruct option refspec_options [] = {\n> +\t\tOPT_BOOL(0, \"fetch\", &fetch,\n> +\t\t\t N_(\"enable the 'fetch' option for parsing refpecs\")),\n\nTypo here: s/refpecs/refspecs/\n\n\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, NULL, refspec_options,\n> +\t\t\t refspec_usage, 0);\n> +\n> +\twhile (strbuf_getline(&line, stdin) != EOF) {\n> +\t\tstruct refspec_item rsi;\n> +\t\tchar *buf;\n> +\n> +\t\tif (!refspec_item_init(&rsi, line.buf, fetch)) {\n> +\t\t\tprintf(\"failed to parse %s\\n\", line.buf);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tbuf = refspec_item_format(&rsi);\n> +\t\tprintf(\"%s\\n\", buf);\n> +\t\tfree(buf);\n> +\n> +\t\trefspec_item_clear(&rsi);\n> +\t}\n> +\n> +\tstrbuf_release(&line);\n> +\treturn 0;\n> +}\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index 287aa6002307..f534ad1731a9 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -55,6 +55,7 @@ static struct test_cmd cmds[] = {\n> \t{ \"read-cache\", cmd__read_cache },\n> \t{ \"read-graph\", cmd__read_graph },\n> \t{ \"read-midx\", cmd__read_midx },\n> +\t{ \"refspec\", cmd__refspec },\n> \t{ \"ref-store\", cmd__ref_store },\n> \t{ \"regex\", cmd__regex },\n> \t{ \"repository\", cmd__repository },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index 9ea4b31011dd..46a0b8850f17 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -44,6 +44,7 @@ int cmd__reach(int argc, const char **argv);\n> int cmd__read_cache(int argc, const char **argv);\n> int cmd__read_graph(int argc, const char **argv);\n> int cmd__read_midx(int argc, const char **argv);\n> +int cmd__refspec(int argc, const char **argv);\n> int cmd__ref_store(int argc, const char **argv);\n> int cmd__regex(int argc, const char **argv);\n> int cmd__repository(int argc, const char **argv);\n> diff --git a/t/t5511-refspec.sh b/t/t5511-refspec.sh\n> index be025b90f989..489bec08d570 100755\n> --- a/t/t5511-refspec.sh\n> +++ b/t/t5511-refspec.sh\n> @@ -93,4 +93,45 @@ test_refspec fetch \"refs/heads/${good}\"\n> bad=$(printf '\\011tab')\n> test_refspec fetch \"refs/heads/${bad}\"\t\t\t\tinvalid\n> \n> +test_expect_success 'test input/output round trip' '\n> +\tcat >input <<-\\EOF &&\n> +\t+refs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/main:refs/remotes/frotz/xyzzy\n> +\t:refs/remotes/frotz/deleteme\n> +\t^refs/heads/secrets\n> +\trefs/heads/secret:refs/heads/translated\n> +\trefs/heads/secret:heads/translated\n> +\trefs/heads/secret:remotes/translated\n> +\tsecret:translated\n> +\trefs/heads/*:remotes/xxy/*\n> +\trefs/heads*/for-linus:refs/remotes/mine/*\n> +\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n> +\tHEAD\n> +\t@\n> +\t:\n> +\tEOF\n> +\tcat >expect <<-\\EOF &&\n> +\t+refs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/main:refs/remotes/frotz/xyzzy\n> +\t:refs/remotes/frotz/deleteme\n> +\t^refs/heads/secrets\n> +\trefs/heads/secret:refs/heads/translated\n> +\trefs/heads/secret:heads/translated\n> +\trefs/heads/secret:remotes/translated\n> +\tsecret:translated\n> +\trefs/heads/*:remotes/xxy/*\n> +\trefs/heads*/for-linus:refs/remotes/mine/*\n> +\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n> +\tHEAD\n> +\tHEAD\n> +\t:\n> +\tEOF\n> +\ttest-tool refspec output &&\n> +\ttest_cmp expect output &&\n> +\ttest-tool refspec --fetch output &&\n> +\ttest_cmp expect output\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"421184","messageId":"YG47jV2+qcRVwY5N@google.com","threadId":"55434","inReplyTo":"9592224e3d428762c6f9b38a0bcc2cee5c3dff6c.1617734871.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-04-07T23:09:01Z","receivedAt":"2021-04-07T23:09:10Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.04.06 18:47, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The prefetch task previously used the default refspec source plus a\n> custom refspec destination to avoid colliding with remote refs:\n> \n> \t+refs/heads/*:refs/prefetch//*\n> \n> However, some users customize their refspec to reduce how much data they\n> download from specific remotes. This can involve restrictive patterns\n> for fetching or negative patterns to avoid downloading some refs.\n> \n> Modify fetch_remote() to iterate over the remote's refspec list and\n> translate that into the appropriate prefetch scenario. Specifically,\n> re-parse the raw form of the refspec into a new 'struct refspec' and\n> modify the 'dst' member to replace a leading \"refs/\" substring with\n> \"refs/prefetch/\", or prepend \"refs/prefetch/\" to 'dst' otherwise.\n> Negative refspecs do not have a 'dst' so they can be transferred to the\n> 'git fetch' command unmodified.\n> \n> This prefix change provides the benefit of keeping whatever collisions\n> may exist in the custom refspecs, if that is a desirable outcome.\n> \n> This changes the names of the refs that would be fetched by the default\n> refspec. Instead of \"refs/prefetch//\" they will now go\n> to \"refs/prefetch/remotes//\". While this is a change, it\n> is not a seriously breaking one: these refs are intended to be hidden\n> and not used.\n> \n> Update the documentation to be more generic about the destination refs.\n> Do not mention custom refpecs explicitly, as that does not need to be\n\nTypo here: s/refpecs/refspecs/\n\n\n> highlighted in this documentation. The important part of placing refs in\n> refs/prefetch remains.\n> \n> Reported-by: Tom Saeger \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-maintenance.txt | 3 +--\n> builtin/gc.c | 37 +++++++++++++++++++++++++-\n> t/t7900-maintenance.sh | 43 ++++++++++++++++++++++++++-----\n> 3 files changed, 74 insertions(+), 9 deletions(-)\n> \n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 80ddd33ceba0..95a24264eb10 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -94,8 +94,7 @@ prefetch::\n> \tobjects from all registered remotes. For each remote, a `git fetch`\n> \tcommand is run. The refmap is custom to avoid updating local or remote\n> \tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n> -\tremote refs are stored in `refs/prefetch//`. Also, tags are\n> -\tnot updated.\n> +\trefs are stored in `refs/prefetch/`. Also, tags are not updated.\n> +\n> This is done to avoid disrupting the remote-tracking branches. The end users\n> expect these refs to stay unmoved unless they initiate a fetch. With prefetch\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index fa8128de9ae1..76f347dd6b11 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -32,6 +32,7 @@\n> #include \"remote.h\"\n> #include \"object-store.h\"\n> #include \"exec-cmd.h\"\n> +#include \"refspec.h\"\n> \n> #define FAILED_RUN \"failed to run %s\"\n> \n> @@ -877,6 +878,7 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> {\n> \tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n> +\tint i;\n> \n> \tchild.git_cmd = 1;\n> \tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n> @@ -886,7 +888,40 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> \tif (opts->quiet)\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \n> -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n> +\tfor (i = 0; i < remote->fetch.nr; i++) {\n> +\t\tstruct refspec_item replace;\n> +\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n> +\t\tstruct strbuf new_dst = STRBUF_INIT;\n> +\t\tsize_t ignore_len = 0;\n> +\t\tchar *replace_string;\n> +\n> +\t\tif (rsi->negative) {\n> +\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n> +\n> +\t\t/*\n> +\t\t * If a refspec dst starts with \"refs/\" at the start,\n> +\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n> +\t\t * Otherwise, we will prepend the dst string with\n> +\t\t * \"refs/prefetch/\".\n> +\t\t */\n> +\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n> +\t\t\tignore_len = 5;\n> +\n> +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> +\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n> +\t\tfree(replace.dst);\n> +\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n> +\n> +\t\treplace_string = refspec_item_format(&replace);\n> +\t\tstrvec_push(&child.args, replace_string);\n> +\t\tfree(replace_string);\n> +\n> +\t\trefspec_item_clear(&replace);\n> +\t}\n> \n> \treturn !!run_command(&child);\n> }\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 37eed6ed3aa3..03487be3af38 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -142,20 +142,51 @@ test_expect_success 'prefetch multiple remotes' '\n> \ttest_commit -C clone2 two &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n> -\ttest_subcommand git fetch remote1 $fetchargs \"+refs/heads/*:refs/prefetch/remote1/*\" -\ttest_subcommand git fetch remote2 $fetchargs \"+refs/heads/*:refs/prefetch/remote2/*\" +\ttest_subcommand git fetch remote1 $fetchargs \"+refs/heads/*:refs/prefetch/remotes/remote1/*\" +\ttest_subcommand git fetch remote2 $fetchargs \"+refs/heads/*:refs/prefetch/remotes/remote2/*\" \ttest_path_is_missing .git/refs/remotes &&\n> -\tgit log prefetch/remote1/one &&\n> -\tgit log prefetch/remote2/two &&\n> +\tgit log prefetch/remotes/remote1/one &&\n> +\tgit log prefetch/remotes/remote2/two &&\n> \tgit fetch --all &&\n> -\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remote1/one &&\n> -\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remote2/two &&\n> +\ttest_cmp_rev refs/remotes/remote1/one refs/prefetch/remotes/remote1/one &&\n> +\ttest_cmp_rev refs/remotes/remote2/two refs/prefetch/remotes/remote2/two &&\n> \n> \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n> \tgit log --oneline --decorate --all >log &&\n> \t! grep \"prefetch\" log\n> '\n> \n> +test_expect_success 'prefetch custom refspecs' '\n> +\tgit -C clone1 branch -f special/fetched HEAD &&\n> +\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n> +\n> +\t# create multiple refspecs for remote1\n> +\tgit config --add remote.remote1.fetch \"+refs/heads/special/fetched:refs/heads/fetched\" &&\n> +\tgit config --add remote.remote1.fetch \"^refs/heads/special/secret/not-fetched\" &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> +\n> +\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n> +\n> +\t# skips second refspec because it is not a pattern type\n> +\trs1=\"+refs/heads/*:refs/prefetch/remotes/remote1/*\" &&\n> +\trs2=\"+refs/heads/special/fetched:refs/prefetch/heads/fetched\" &&\n> +\trs3=\"^refs/heads/special/secret/not-fetched\" &&\n> +\n> +\ttest_subcommand git fetch remote1 $fetchargs \"$rs1\" \"$rs2\" \"$rs3\" +\ttest_subcommand git fetch remote2 $fetchargs \"+refs/heads/*:refs/prefetch/remotes/remote2/*\" +\n> +\t# first refspec is overridden by second\n> +\ttest_must_fail git rev-parse refs/prefetch/special/fetched &&\n> +\tgit rev-parse refs/prefetch/heads/fetched &&\n> +\n> +\t# possible incorrect places for the non-fetched ref\n> +\ttest_must_fail git rev-parse refs/prefetch/remotes/remote1/secret/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/prefetch/remotes/remote1/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/heads/secret/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/heads/not-fetched\n> +'\n> +\n> test_expect_success 'prefetch and existing log.excludeDecoration values' '\n> \tgit config --unset-all log.excludeDecoration &&\n> \tgit config log.excludeDecoration refs/remotes/remote1/ &&\n> -- \n> gitgitgadget\n"},{"id":"421185","messageId":"YG4+67oe39IRnUzb@google.com","threadId":"55434","inReplyTo":"5aa0cb06c3f2e31a51bb6747fa84de69c0c1212f.1617734870.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/5] maintenance: simplify prefetch logic","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-04-07T23:23:23Z","receivedAt":"2021-04-07T23:23:33Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Overall, the series looks pretty solid to me - which is why I've got a\nhandful of small nits to relay. :)\n\nOn Tue, Apr 06, 2021 at 06:47:46PM +0000, Derrick Stolee via GitGitGadget wrote:\n> -static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n> +static int fetch_remote(struct remote *remote, void *cbdata)\n> {\n> +\tstruct maintenance_run_opts *opts = cbdata;\n[snip]\n> \tif (opts->quiet)\nI worry about the lack of null-checking here.\n\n - Emily\n"},{"id":"421186","messageId":"YG4/qltdbWKSv5zT@google.com","threadId":"55434","inReplyTo":"bf296282323aa74f7c5b1a49c74d5aaa4b6f37ae.1617734870.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/5] test-tool: test refspec input/output","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-04-07T23:26:34Z","receivedAt":"2021-04-07T23:26:44Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Apr 06, 2021 at 06:47:49PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/t/t5511-refspec.sh b/t/t5511-refspec.sh\n> index be025b90f989..489bec08d570 100755\n> --- a/t/t5511-refspec.sh\n> +++ b/t/t5511-refspec.sh\n> @@ -93,4 +93,45 @@ test_refspec fetch \"refs/heads/${good}\"\n> bad=$(printf '\\011tab')\n> test_refspec fetch \"refs/heads/${bad}\"\t\t\t\tinvalid\n> \n> +test_expect_success 'test input/output round trip' '\n> +\tcat >input <<-\\EOF &&\n> +\t+refs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/main:refs/remotes/frotz/xyzzy\n> +\t:refs/remotes/frotz/deleteme\n> +\t^refs/heads/secrets\n> +\trefs/heads/secret:refs/heads/translated\n> +\trefs/heads/secret:heads/translated\n> +\trefs/heads/secret:remotes/translated\n> +\tsecret:translated\n> +\trefs/heads/*:remotes/xxy/*\n> +\trefs/heads*/for-linus:refs/remotes/mine/*\n> +\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n> +\tHEAD\n> +\t@\n> +\t:\n> +\tEOF\n> +\tcat >expect <<-\\EOF &&\n> +\t+refs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/*:refs/remotes/origin/*\n> +\trefs/heads/main:refs/remotes/frotz/xyzzy\n> +\t:refs/remotes/frotz/deleteme\n> +\t^refs/heads/secrets\n> +\trefs/heads/secret:refs/heads/translated\n> +\trefs/heads/secret:heads/translated\n> +\trefs/heads/secret:remotes/translated\n> +\tsecret:translated\n> +\trefs/heads/*:remotes/xxy/*\n> +\trefs/heads*/for-linus:refs/remotes/mine/*\n> +\t2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed\n> +\tHEAD\n> +\tHEAD\n> +\t:\n> +\tEOF\n\nI don't like these expect/actual here. They are almost entirely\nidentical, which means that the reader either A) spends a toilsome few\nminutes checking every single line to be sure they are not identical, or\nB) reads the first three lines, decides they're the same, and misses the\n@->HEAD special case.\n\nWhy not instead run the test once for all the lines which should be the\nsame before and after the parse, and again for all the lines which\nshould differ, to reduce burden on the reader?\n\n - Emily\n"},{"id":"421188","messageId":"YG5CTFMIfSYcqZkJ@google.com","threadId":"55434","inReplyTo":"9592224e3d428762c6f9b38a0bcc2cee5c3dff6c.1617734871.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-04-07T23:37:48Z","receivedAt":"2021-04-07T23:37:59Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Apr 06, 2021 at 06:47:50PM +0000, Derrick Stolee via GitGitGadget wrote:\n> @@ -877,6 +878,7 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n[snip]\n> +\t\t/*\n> +\t\t * If a refspec dst starts with \"refs/\" at the start,\n> +\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n> +\t\t * Otherwise, we will prepend the dst string with\n> +\t\t * \"refs/prefetch/\".\n> +\t\t */\n> +\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n> +\t\t\tignore_len = 5;\nUsing a literal string plus the literal value of the string length,\ntwice, doesn't sit great with me...\n\n> +\n> +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> +\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n...plus with some ugly array pointer math. :) Why not use\ngit-compat-util.h:skip_prefix() instead of doing your own math? (In\nfact, the doc comment on skip_prefix() talks about using it exactly for\nstripping \"refs/\" off the beginning of a string :) )\n\n - Emily\n"},{"id":"421190","messageId":"20210408002353.2467818-1-jonathantanmy@google.com","threadId":"55434","inReplyTo":"9592224e3d428762c6f9b38a0bcc2cee5c3dff6c.1617734871.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-04-08T00:23:53Z","receivedAt":"2021-04-08T00:23:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +test_expect_success 'prefetch custom refspecs' '\n> +\tgit -C clone1 branch -f special/fetched HEAD &&\n> +\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n> +\n> +\t# create multiple refspecs for remote1\n> +\tgit config --add remote.remote1.fetch \"+refs/heads/special/fetched:refs/heads/fetched\" &&\n> +\tgit config --add remote.remote1.fetch \"^refs/heads/special/secret/not-fetched\" &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> +\n> +\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n> +\n> +\t# skips second refspec because it is not a pattern type\n\nWhat second refspec is being skipped?\n\n> +\trs1=\"+refs/heads/*:refs/prefetch/remotes/remote1/*\" &&\n> +\trs2=\"+refs/heads/special/fetched:refs/prefetch/heads/fetched\" &&\n> +\trs3=\"^refs/heads/special/secret/not-fetched\" &&\n> +\n> +\ttest_subcommand git fetch remote1 $fetchargs \"$rs1\" \"$rs2\" \"$rs3\" +\ttest_subcommand git fetch remote2 $fetchargs \"+refs/heads/*:refs/prefetch/remotes/remote2/*\" +\n> +\t# first refspec is overridden by second\n> +\ttest_must_fail git rev-parse refs/prefetch/special/fetched &&\n> +\tgit rev-parse refs/prefetch/heads/fetched &&\n> +\n> +\t# possible incorrect places for the non-fetched ref\n> +\ttest_must_fail git rev-parse refs/prefetch/remotes/remote1/secret/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/prefetch/remotes/remote1/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/heads/secret/not-fetched &&\n> +\ttest_must_fail git rev-parse refs/heads/not-fetched\n> +'\n\nOther than this and the other comments that others have brought up, this\nseries looks good.\n"},{"id":"421197","messageId":"xmqqsg41ut72.fsf@gitster.g","threadId":"55434","inReplyTo":"871rblk8ct.fsf@evledraar.gmail.com","subject":"Re: [PATCH 3/5] refspec: output a refspec item","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-08T07:33:05Z","receivedAt":"2021-04-08T07:33:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\nÆvar Arnfjörð Bjarmason writes:\n\n> But yes, it seems even more straightforward to do away with passing the\n> refspec at all to \"git fetch\", and instead pass some (maybe internal\n> only, and documented as such) \"--refspec-dst-prefix=refs/prefetch/\"\n> option to \"git fetch\".\n>\n> I.e. get_ref_map() over there is already doing a version of this loop\n> over \"remote->fetch.nr\".\n>\n> So instead of \"git gc\" doing the loop, then passing all the refspecs on\n> the command-line, it could tell \"git fetch\" to do the same munging when\n> it does the same iteration.\n\nThat direction makes quite a lot of sense to me.\n\n> Doing the munging in builtin/gc.c's fetch_remote() just seems like a\n> relic from when we didn't care what decision builtin/fetch.c made about\n> refspecs, we always wanted our custom one.\n"},{"id":"421384","messageId":"243a0aa1-7c0a-d2ab-49ff-8ea8cacc1b81@gmail.com","threadId":"55434","inReplyTo":"87im4yjspp.fsf@evledraar.gmail.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-09T11:48:31Z","receivedAt":"2021-04-09T11:48:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/7/2021 6:26 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Apr 07 2021, Ævar Arnfjörð Bjarmason wrote:\n> \n>> On Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n>>> +\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n>>\n>> I see this is following some established convention in the file, but is\n>> there really not a way to make this pass without directing stderr to\n>> /dev/null? It makes ad-hoc debugging when reviewing harder.\n> \n> As I later found out this is copy/pasted to get around the fact that\n> --quiet is dependent on isatty(), so without this the result would be\n> different under --verbose and non-verbose testing.\n\nYes, adding --quiet directly is a better pattern.\n\n> So that dates back to 3ddaad0e060 (maintenance: add --quiet option,\n> 2020-09-17), but I see other quiet=isatty(2) in related code. I wish we\n> could isolate that particular behavior so removing the 2>/dev/null when\n> debugging the tests doesn't cause you to run into this, maybe an\n> explicit --quiet or --no-quiet option for all but one test that's\n> checking that isatty() behavior?\n> \n>> I tried just removing it, but then (in an earlier test case) the\n>> \"test_subcommand\" fails because it can't find the line we're looking\n>> for, so us piping stderr to /dev/null impacts our trace2 output?\n> \n> I hadn't seen seen test_subcommand before, sorry to be blunt, but \"ew!\".\n\nSaying \"I'm about to be rude\" before being rude doesn't excuse it.\nI had to step away and get over this comment before I could examine\nthe actually constructive feedback below.\n\nA better way to communicate the same information could be \"This test\nhelper seems more complicated than necessary, and has some gaps that\ncould be filled.\" I completely agree with this statement.\n\n> So we're ad-hoc grepping trace2 JSON output just to find out whether we\n> invoked some subcommand. But unlike test_expect_code etc. this one\n> doesn't run git for you, but instead we have temp *.txt files and the\n> command disconnected from the run.\n> \n> And because you're using \"grep\" and \"! grep\" to test, you're hiding the\n> difference between \"did not find this line\" v.s. \"did not find anything\n> at all\".\n\nYou're right that there is value in comparing the ordered list of\nsubcommands run by a given Git command. That will catch buggy tests\nthat are checking that a subcommand doesn't run.\n\n> Because of that the second test using test_subcommand is either buggy or\n> painfully non-obvious. We check that \"run --auto\" doesn't contain a\n> \"auto --quiet\", but in reality it doesn't contain any subcommands at\n> all. We didn't run any because it exited with \"nothing to pack\".\n\nThat exit is from the third command, which does not pass the --auto\ncommand. The \"nothing to pack\" output is from the 'git gc --no-quiet'\nsubcommand that is being checked in this third test.\n\n> I think converting the whole thing to something like the WIP/RFC patch\n> below is much better and more readable.\n\nThis is an interesting approach. I don't see you using the ERR that you\nare inputting anywhere, so that seems like an unnecessary bloat to the\nconsumers. But maybe I haven't discovered all of the places where this\nwould be useful, but it seems better to pipe stderr to a file for later\ncomparison when needed.\n> +test_expect_process_tree () {\n> +\tdepth= &&\n> +\t>actual &&\n> +\tcat >expect &&\n> +\tcat <&3 >expect.err\n> +\twhile test $# != 0\n> +\tdo\n> +\t\tcase \"$1\" in\n> +\t\t--depth)\n> +\t\t\tdepth=\"$2\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t\t*)\n> +\t\t\tbreak\n> +\t\t\t;;\n> +\t\tesac\n> +\t\tshift\n> +\tdone &&\nDo you have an example where this is being checked? Or can depth\nbe left as 1 for now?\n\n> +\tlog=\"$(pwd)/proc-tree.txt\" &&\n> +\t>\"$log\" &&\n> +\tGIT_TRACE2_PERF=\"$log\" \"$@\" 2>actual.err &&\n> +\tgrep \"child_start\" proc-tree.txt >proc-tree-start.txt || : &&\n> +\tif test -n \"$depth\"\n> +\tthen\n> +\t\tgrep \" d$depth \" proc-tree-start.txt >tmp.txt || : &&\n> +\t\tmv tmp.txt proc-tree-start.txt\n> +\tfi &&\n> +\tsed -e 's/^.*argv:\\[//' -e 's/\\]$//' actual &&\n> +\ttest_cmp expect actual &&\n> +\ttest_cmp expect.err actual.err\n> +} 7>&2 2>&4\n\nI think similar ideas could apply to test_region. Giving it a try\nnow.\n\n-Stolee\n"},{"id":"421466","messageId":"91f3d4f5-429c-8a9e-eea3-a02ced426b7b@gmail.com","threadId":"55434","inReplyTo":"YG4+67oe39IRnUzb@google.com","subject":"Re: [PATCH v2 1/5] maintenance: simplify prefetch logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-09T19:00:21Z","receivedAt":"2021-04-09T19:00:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/7/2021 7:23 PM, Emily Shaffer wrote:\n> Overall, the series looks pretty solid to me - which is why I've got a\n> handful of small nits to relay. :)\n> \n> On Tue, Apr 06, 2021 at 06:47:46PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> -static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n>> +static int fetch_remote(struct remote *remote, void *cbdata)\n>> {\n>> +\tstruct maintenance_run_opts *opts = cbdata;\n> [snip]\n>> \tif (opts->quiet)\n> I worry about the lack of null-checking here.\n\nIf this was a general-purpose method, I'd agree with you.\nBut since this is a static method being called exactly once\n(and always passing a non-NULL value) then I don't believe\nthat NULL check is valuable here.\n\nThanks,\n-Stolee\n"},{"id":"421468","messageId":"87im4vgsvx.fsf@evledraar.gmail.com","threadId":"55434","inReplyTo":"243a0aa1-7c0a-d2ab-49ff-8ea8cacc1b81@gmail.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-09T19:28:02Z","receivedAt":"2021-04-09T19:28:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 09 2021, Derrick Stolee wrote:\n\n> On 4/7/2021 6:26 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Wed, Apr 07 2021, Ævar Arnfjörð Bjarmason wrote:\n>> \n>>> On Mon, Apr 05 2021, Derrick Stolee via GitGitGadget wrote:\n>>>> +\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n>>>\n>>> I see this is following some established convention in the file, but is\n>>> there really not a way to make this pass without directing stderr to\n>>> /dev/null? It makes ad-hoc debugging when reviewing harder.\n>> \n>> As I later found out this is copy/pasted to get around the fact that\n>> --quiet is dependent on isatty(), so without this the result would be\n>> different under --verbose and non-verbose testing.\n>\n> Yes, adding --quiet directly is a better pattern.\n>\n>> So that dates back to 3ddaad0e060 (maintenance: add --quiet option,\n>> 2020-09-17), but I see other quiet=isatty(2) in related code. I wish we\n>> could isolate that particular behavior so removing the 2>/dev/null when\n>> debugging the tests doesn't cause you to run into this, maybe an\n>> explicit --quiet or --no-quiet option for all but one test that's\n>> checking that isatty() behavior?\n>> \n>>> I tried just removing it, but then (in an earlier test case) the\n>>> \"test_subcommand\" fails because it can't find the line we're looking\n>>> for, so us piping stderr to /dev/null impacts our trace2 output?\n>> \n>> I hadn't seen seen test_subcommand before, sorry to be blunt, but \"ew!\".\n>\n> Saying \"I'm about to be rude\" before being rude doesn't excuse it.\n> I had to step away and get over this comment before I could examine\n> the actually constructive feedback below.\n>\n> A better way to communicate the same information could be \"This test\n> helper seems more complicated than necessary, and has some gaps that\n> could be filled.\" I completely agree with this statement.\n\nI didn't mean to be rude, my understanding of \"blunt\" is more like\n\"forthright\" or \"without mincing words\" or something to that effect, but\nI'm not a native speaker of English, so I'll take your understanding\nover mine and apologize, sorry.\n\nWhat I meant to say is: When we look at some prior art in the test suite\nthat does similar things this pattern is needlessly convoluted, and\nseparates the command invocation from the test assert itself. Something\nsimilar-ish to check_sub_test_lib_test would allow you to run the\ncommand, but do any asserting of the proc tree in the helper.\n\n>> So we're ad-hoc grepping trace2 JSON output just to find out whether we\n>> invoked some subcommand. But unlike test_expect_code etc. this one\n>> doesn't run git for you, but instead we have temp *.txt files and the\n>> command disconnected from the run.\n>> \n>> And because you're using \"grep\" and \"! grep\" to test, you're hiding the\n>> difference between \"did not find this line\" v.s. \"did not find anything\n>> at all\".\n>\n> You're right that there is value in comparing the ordered list of\n> subcommands run by a given Git command. That will catch buggy tests\n> that are checking that a subcommand doesn't run.\n>\n>> Because of that the second test using test_subcommand is either buggy or\n>> painfully non-obvious. We check that \"run --auto\" doesn't contain a\n>> \"auto --quiet\", but in reality it doesn't contain any subcommands at\n>> all. We didn't run any because it exited with \"nothing to pack\".\n>\n> That exit is from the third command, which does not pass the --auto\n> command. The \"nothing to pack\" output is from the 'git gc --no-quiet'\n> subcommand that is being checked in this third test.\n>\n>> I think converting the whole thing to something like the WIP/RFC patch\n>> below is much better and more readable.\n>\n> This is an interesting approach. I don't see you using the ERR that you\n> are inputting anywhere, so that seems like an unnecessary bloat to the\n> consumers. But maybe I haven't discovered all of the places where this\n> would be useful, but it seems better to pipe stderr to a file for later\n> comparison when needed.\n\nYes, it's probably not a good default here. For the test-lib.sh tests\nthere's check_sub_test_lib_test and check_sub_test_lib_test_err, most of\nthe tests only test stdout.\n\n>> +test_expect_process_tree () {\n>> +\tdepth= &&\n>> +\t>actual &&\n>> +\tcat >expect &&\n>> +\tcat <&3 >expect.err\n>> +\twhile test $# != 0\n>> +\tdo\n>> +\t\tcase \"$1\" in\n>> +\t\t--depth)\n>> +\t\t\tdepth=\"$2\"\n>> +\t\t\tshift\n>> +\t\t\t;;\n>> +\t\t*)\n>> +\t\t\tbreak\n>> +\t\t\t;;\n>> +\t\tesac\n>> +\t\tshift\n>> +\tdone &&\n> Do you have an example where this is being checked? Or can depth\n> be left as 1 for now?\n\nIt can probably be hardcoded, but I was hoping someone more familiar\nwith trace2 would chime in, but I'm fairly sure there's not a way to do\nit without parsing the existing output with either some clever\ngrep/awk-ing of the PERF output, or stateful parsing of the JSON.\n\nI thought that for git maintenance tests perhaps something wanted to\nassert that we didn't have maintenance invoking maintenance, or that\nsomething expected to prune refs really invoked the relevant prune\ncommand via \"gc\".\n\n>> +\tlog=\"$(pwd)/proc-tree.txt\" &&\n>> +\t>\"$log\" &&\n>> +\tGIT_TRACE2_PERF=\"$log\" \"$@\" 2>actual.err &&\n>> +\tgrep \"child_start\" proc-tree.txt >proc-tree-start.txt || : &&\n>> +\tif test -n \"$depth\"\n>> +\tthen\n>> +\t\tgrep \" d$depth \" proc-tree-start.txt >tmp.txt || : &&\n>> +\t\tmv tmp.txt proc-tree-start.txt\n>> +\tfi &&\n>> +\tsed -e 's/^.*argv:\\[//' -e 's/\\]$//' actual &&\n>> +\ttest_cmp expect actual &&\n>> +\ttest_cmp expect.err actual.err\n>> +} 7>&2 2>&4\n>\n> I think similar ideas could apply to test_region. Giving it a try\n> now.\n\nProbably, I didn't even notice that one...\n"},{"id":"421498","messageId":"36aa6837-722c-9ef0-84cc-77e982db9f6e@gmail.com","threadId":"55434","inReplyTo":"87im4vgsvx.fsf@evledraar.gmail.com","subject":"Re: [PATCH 5/5] maintenance: allow custom refspecs during prefetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-10T00:56:31Z","receivedAt":"2021-04-10T00:56:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/9/2021 3:28 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Apr 09 2021, Derrick Stolee wrote:\n> \n>> On 4/7/2021 6:26 AM, Ævar Arnfjörð Bjarmason wrote:\n>>> I think converting the whole thing to something like the WIP/RFC patch\n>>> below is much better and more readable.\n>>\n>> This is an interesting approach. I don't see you using the ERR that you\n>> are inputting anywhere, so that seems like an unnecessary bloat to the\n>> consumers. But maybe I haven't discovered all of the places where this\n>> would be useful, but it seems better to pipe stderr to a file for later\n>> comparison when needed.\n> \n> Yes, it's probably not a good default here. For the test-lib.sh tests\n> there's check_sub_test_lib_test and check_sub_test_lib_test_err, most of\n> the tests only test stdout.\n> \n>>> +test_expect_process_tree () {\n>>> +\tdepth= &&\n>>> +\t>actual &&\n>>> +\tcat >expect &&\n>>> +\tcat <&3 >expect.err\n>>> +\twhile test $# != 0\n>>> +\tdo\n>>> +\t\tcase \"$1\" in\n>>> +\t\t--depth)\n>>> +\t\t\tdepth=\"$2\"\n>>> +\t\t\tshift\n>>> +\t\t\t;;\n>>> +\t\t*)\n>>> +\t\t\tbreak\n>>> +\t\t\t;;\n>>> +\t\tesac\n>>> +\t\tshift\n>>> +\tdone &&\n>> Do you have an example where this is being checked? Or can depth\n>> be left as 1 for now?\n> \n> It can probably be hardcoded, but I was hoping someone more familiar\n> with trace2 would chime in, but I'm fairly sure there's not a way to do\n> it without parsing the existing output with either some clever\n> grep/awk-ing of the PERF output, or stateful parsing of the JSON.\n> \n> I thought that for git maintenance tests perhaps something wanted to\n> assert that we didn't have maintenance invoking maintenance, or that\n> something expected to prune refs really invoked the relevant prune\n> command via \"gc\".\n> \n>>> +\tlog=\"$(pwd)/proc-tree.txt\" &&\n>>> +\t>\"$log\" &&\n>>> +\tGIT_TRACE2_PERF=\"$log\" \"$@\" 2>actual.err &&\n>>> +\tgrep \"child_start\" proc-tree.txt >proc-tree-start.txt || : &&\n>>> +\tif test -n \"$depth\"\n>>> +\tthen\n>>> +\t\tgrep \" d$depth \" proc-tree-start.txt >tmp.txt || : &&\n>>> +\t\tmv tmp.txt proc-tree-start.txt\n>>> +\tfi &&\n>>> +\tsed -e 's/^.*argv:\\[//' -e 's/\\]$//' actual &&\n>>> +\ttest_cmp expect actual &&\n>>> +\ttest_cmp expect.err actual.err\n>>> +} 7>&2 2>&4\n>>\n>> I think similar ideas could apply to test_region. Giving it a try\n>> now.\n> \n> Probably, I didn't even notice that one...\n\nI gave this a few hours today, and I'm giving up. I'm the first to\nadmit that I don't have the correct scripting skills to do some of\nthese things.\n\nI've got what I tried below. It certainly looks like it would work.\nIt solves the problem of \"what if the test is flaky?\" by ensuring that\nall subcommands (at depth 0) match the inputs exactly.\n\nHowever, the problem comes when trying to make that work for all of\nthe maintenance tests, specifically the 'incremental-repack' task.\nThat task dynamically computes a --batch-size=X parameter, and that\nis not stable across runs of the script.\n\nThis was avoided in the past by only checking for the first of three\nsubcommands when verifying that the 'incremental-repack' task worked.\nThat is, except for the EXPENSIVE test that checks that the --batch-size\nmaxes out at 2g.\n\nThe thing that might make these changing parameters work is to allow\nthe specified lines be a _prefix_ of the actual parameters. Or, let\neach line be a pattern that is checked against that line. Issues come\nup with how to handle this line-by-line check that I was unable to\novercome.\n\nThe good news is that the idea of adding a '--prefetch' option to\n'git fetch' makes the change to t7900-maintenance.sh much easier,\nmaking this change to test_subcommand less of a priority.\n\nI include my attempt here as a patch. Feel free to take whatever\nyou want of it, or none of it and start over. I do think that it\nmakes the test script look much nicer.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nFrom 449d098f2a13860f44b2e6fb96fb2dd5872b511b Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Fri, 9 Apr 2021 08:42:26 -0400\nSubject: [PATCH 1/2] test-lib: add test_subcommands\nMIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\n\nThe test_subcommand helper in test-lib-functions.sh satisfied a need to\ncheck for certain subcommands in Git processes. This is especially\nneeded in t7900-maintenance.sh to ensure that the maintenance builtin\nproperly calls certain subcommands based on command-line options, config\nvalues, and the state of the repository.\n\nHowever, test_subcommand has some complexities in its use that can be\nimproved. First, it requires that the caller knows to create a log file\nwith GIT_TRACE2_EVENT then supply that to test_subcommand. Further, it\nonly checks that some exact subcommands exist or do not exist. It does\nnot guarantee that the list of subcommands exactly matches a given list.\nBecause of this drawback, the tests that check a subcommand does _not_\nrun are particularly flaky to slight changes in behavior.\n\nIntroduce a new helper, test_subcommands, that resolves these drawbacks:\n\n1. It runs the supplied command and handles the trace log itself.\n\n2. It takes a list of commands overs stdin and compares this to the\n complete list of subcommands run from the top-level command.\n\nThe helper does not test that the full subcommand tree matches, because\nthat would cause tests to be too fragile to changes unrelated to the\ncomponent being tested. This could easily be extended to allow the full\ntree with an option, if desired.\n\nTo ensure we only check the first level, use GIT_TRACE2_PERF output and\nscan for \" d0 \" in the rows that include the \"child_start\" event. The\nlast column includes a way to scrape the subcommand itself from the\ntrace. Sometimes arguments are quoted, such as when passing a refspec\nwith '*' to the subcommand. This makes it difficult to create a matching\nstring within the single-quoted test definitions, so strip these single\nquotes from the arguments before matching the input.\n\nOnly modify one test in t7900-maintenance.sh. The rest of the callers to\ntest_subcommand will be converted to test_subcommands in a later change.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n t/t7900-maintenance.sh | 16 +++++++---------\n t/test-lib-functions.sh | 24 ++++++++++++++++++++++++\n 2 files changed, 31 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 2412d8c5c0..e170ab7862 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -30,15 +30,13 @@ test_expect_success 'help text' '\n '\n \n test_expect_success 'run [--auto|--quiet]' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" \\\n-\t\tgit maintenance run 2>/dev/null &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" \\\n-\t\tgit maintenance run --auto 2>/dev/null &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-quiet.txt\" \\\n-\t\tgit maintenance run --no-quiet 2>/dev/null &&\n-\ttest_subcommand git gc --quiet expect &&\n+\n+\tlog=\"$(pwd)\"/subcommand-trace.txt &&\n+\n+\tGIT_TRACE2_PERF=\"$log\" \"$@\" 2>err &&\n+\tgrep \"child_start\" \"$log\" | grep \" d0 \" >processes || : &&\n+\n+\tsed -e 's/^.*argv:\\[//' -e 's/\\]$//' -e \"s/'//g\" actual &&\n+\ttest_cmp expect actual &&\n+\n+\trm -f \"$log\" expect actual processes\n+}\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n-- \n2.31.1.vfs.0.0\n\n--- >8 ---\n\nAnd here is the follow-up that attempts to change the rest of\nthe tests in t7900-maintenance.sh. However, this leads to flaky\ntests because of the --batch-size changes.\n\n--- >8 ---\n\nFrom 61f142ba7fcde8ae2a84f18005f384a4544b7741 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Fri, 9 Apr 2021 11:13:42 -0400\nSubject: [PATCH 2/2] t7900: convert to test_subcommands\n\nReplace the remaining uses of test_subcommand with test_subcommands,\nallowing us to delete the old helper.\n\nMost of these replacements are straightforward. However, some are a bit\nmore subtle, specifically because we now are checking the full ordered\nset of subcommands. Some places we were only testing one of multiple\nsubcommands that would be run by a task. These are expanded to include\nthe full set.\n\nWhen working with the prefetch task, the refspecs that are passed to the\nsubcommand are normally quoted with single quotes in the GIT_TRACE2_PERF\noutput, but those characters are removed in test_subcommands, allowing\nour tests in t7900-maintenance.sh to avoid escaping such quotes.\n\nSigned-off-by: Derrick Stolee \n---\n t/t7900-maintenance.sh | 285 +++++++++++++++++++---------------------\n t/test-lib-functions.sh | 33 -----\n 2 files changed, 133 insertions(+), 185 deletions(-)\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e170ab7862..8861435dcf 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -40,78 +40,65 @@ test_expect_success 'run [--auto|--quiet]' '\n '\n \n test_expect_success 'maintenance.auto config option' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/default\" git commit --quiet --allow-empty -m 1 &&\n-\ttest_subcommand git maintenance run --auto --quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-config.txt\" git maintenance run 2>err &&\n-\ttest_subcommand ! git gc --quiet ' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" \\\n-\t\tgit maintenance run --task=commit-graph 2>/dev/null &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-gc.txt\" \\\n-\t\tgit maintenance run --task=gc 2>/dev/null &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" \\\n-\t\tgit maintenance run --task=commit-graph 2>/dev/null &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" \\\n-\t\tgit maintenance run --task=commit-graph --task=gc 2>/dev/null &&\n-\ttest_subcommand ! git gc --quiet /dev/null &&\n-\ttest_subcommand ! git commit-graph write --split --reachable --no-progress \\\n-\t\tdid-run.txt <<-EOF &&\n+\tgit commit-graph write --split --reachable --no-progress\n+\tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/cg-no.txt\" \\\n-\t\tgit -c maintenance.commit-graph.auto=1 $COMMAND &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/cg-negative-means-yes.txt\" \\\n-\t\tgit -c maintenance.commit-graph.auto=\"-1\" $COMMAND &&\n+\ttest_subcommands git -c maintenance.commit-graph.auto=1 $COMMAND /dev/null &&\n \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n-\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* /dev/null &&\n-\ttest_subcommand ! git prune-packed --quiet /dev/null &&\n-\ttest_subcommand ! git prune-packed --quiet /dev/null &&\n-\ttest_subcommand git prune-packed --quiet /dev/null &&\n-\ttest_subcommand git prune-packed --quiet /dev/null &&\n-\ttest_subcommand git multi-pack-index repack \\\n-\t\t --no-progress --batch-size=2147483647 /dev/null &&\n-\ttest_subcommand ! git multi-pack-index write --no-progress /dev/null &&\n-\ttest_subcommand ! git multi-pack-index write --no-progress /dev/null &&\n-\ttest_subcommand git multi-pack-index write --no-progress after &&\n-\ttest_must_be_empty after &&\n-\ttest_subcommand git pack-refs --all --prune daily -> hourly' '\n \tgit config maintenance.incremental-repack.enabled true &&\n \tgit config maintenance.incremental-repack.schedule weekly &&\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n-\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n-\ttest_subcommand git prune-packed --quiet hourly <<-EOF &&\n+\tgit prune-packed --quiet\n+\tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/daily.txt\" \\\n-\t\tgit maintenance run --schedule=daily 2>/dev/null &&\n-\ttest_subcommand git prune-packed --quiet daily <<-EOF &&\n+\tgit prune-packed --quiet\n+\tgit commit-graph write --split --reachable --no-progress\n+\tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/weekly.txt\" \\\n-\t\tgit maintenance run --schedule=weekly 2>/dev/null &&\n-\ttest_subcommand git prune-packed --quiet weekly <<-EOF &&\n+\tgit prune-packed --quiet\n+\tgit multi-pack-index write --no-progress\n+\tgit multi-pack-index expire --no-progress\n+\tgit multi-pack-index repack --no-progress --batch-size=655\n+\tgit commit-graph write --split --reachable --no-progress\n+\tEOF\n+\n+\ttest_subcommands git maintenance run --schedule=hourly ... < \n-#\n-# For example, to look for an invocation of \"git upload-pack\n-# /path/to/repo\"\n-#\n-#\tGIT_TRACE2_EVENT=event.log git fetch ... &&\n-#\ttest_subcommand git upload-pack \"$PATH\" \n\nThe --prefetch option will be used by the 'prefetch' maintenance task\ninstead of sending refspecs explicitly across the command-line. The\nintention is to modify the refspec to place all results in\nrefs/prefetch/ instead of anywhere else.\n\nCreate helper method filter_prefetch_refspec() to modify a given refspec\nto fit the rules expected of the prefetch task:\n\n * Negative refspecs are preserved.\n * Refspecs without a destination are removed.\n * Refspecs whose source starts with \"refs/tags/\" are removed.\n * Other refspecs are placed within \"refs/prefetch/\".\n\nFinally, we add the 'force' option to ensure that prefetch refs are\nreplaced as necessary.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/fetch-options.txt | 5 +++\n builtin/fetch.c | 56 +++++++++++++++++++++++++++++++\n t/t5582-fetch-negative-refspec.sh | 30 +++++++++++++++++\n 3 files changed, 91 insertions(+)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 07783deee309..9e7b4e189ce0 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -110,6 +110,11 @@ ifndef::git-pull[]\n \tsetting `fetch.writeCommitGraph`.\n endif::git-pull[]\n \n+--prefetch::\n+\tModify the configured refspec to place all refs into the\n+\t`refs/prefetch/` namespace. See the `prefetch` task in\n+\tlinkgit:git-maintenance[1].\n+\n -p::\n --prune::\n \tBefore fetching, remove any remote-tracking references that no\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 0b90de87c7a2..30856b442b79 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -48,6 +48,7 @@ enum {\n static int fetch_prune_config = -1; /* unspecified */\n static int fetch_show_forced_updates = 1;\n static uint64_t forced_updates_ms = 0;\n+static int prefetch = 0;\n static int prune = -1; /* unspecified */\n #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */\n \n@@ -158,6 +159,8 @@ static struct option builtin_fetch_options[] = {\n \t\t N_(\"do not fetch all tags (--no-tags)\"), TAGS_UNSET),\n \tOPT_INTEGER('j', \"jobs\", &max_jobs,\n \t\t N_(\"number of submodules fetched in parallel\")),\n+\tOPT_BOOL(0, \"prefetch\", &prefetch,\n+\t\t N_(\"modify the refspec to place all refs within refs/prefetch/\")),\n \tOPT_BOOL('p', \"prune\", &prune,\n \t\t N_(\"prune remote-tracking branches no longer on remote\")),\n \tOPT_BOOL('P', \"prune-tags\", &prune_tags,\n@@ -436,6 +439,55 @@ static void find_non_local_tags(const struct ref *refs,\n \toidset_clear(&fetch_oids);\n }\n \n+static void filter_prefetch_refspec(struct refspec *rs)\n+{\n+\tint i;\n+\n+\tif (!prefetch)\n+\t\treturn;\n+\n+\tfor (i = 0; i < rs->nr; i++) {\n+\t\tstruct strbuf new_dst = STRBUF_INIT;\n+\t\tchar *old_dst;\n+\t\tconst char *sub = NULL;\n+\n+\t\tif (rs->items[i].negative)\n+\t\t\tcontinue;\n+\t\tif (!rs->items[i].dst ||\n+\t\t (rs->items[i].src &&\n+\t\t !strncmp(rs->items[i].src, \"refs/tags/\", 10))) {\n+\t\t\tint j;\n+\n+\t\t\tfree(rs->items[i].src);\n+\t\t\tfree(rs->items[i].dst);\n+\n+\t\t\tfor (j = i + 1; j < rs->nr; j++) {\n+\t\t\t\trs->items[j - 1] = rs->items[j];\n+\t\t\t\trs->raw[j - 1] = rs->raw[j];\n+\t\t\t}\n+\t\t\trs->nr--;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\told_dst = rs->items[i].dst;\n+\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n+\n+\t\t/*\n+\t\t * If old_dst starts with \"refs/\", then place\n+\t\t * sub after that prefix. Otherwise, start at\n+\t\t * the beginning of the string.\n+\t\t */\n+\t\tif (!skip_prefix(old_dst, \"refs/\", &sub))\n+\t\t\tsub = old_dst;\n+\t\tstrbuf_addstr(&new_dst, sub);\n+\n+\t\trs->items[i].dst = strbuf_detach(&new_dst, NULL);\n+\t\trs->items[i].force = 1;\n+\n+\t\tfree(old_dst);\n+\t}\n+}\n+\n static struct ref *get_ref_map(struct remote *remote,\n \t\t\t const struct ref *remote_refs,\n \t\t\t struct refspec *rs,\n@@ -452,6 +504,10 @@ static struct ref *get_ref_map(struct remote *remote,\n \tstruct hashmap existing_refs;\n \tint existing_refs_populated = 0;\n \n+\tfilter_prefetch_refspec(rs);\n+\tif (remote)\n+\t\tfilter_prefetch_refspec(&remote->fetch);\n+\n \tif (rs->nr) {\n \t\tstruct refspec *fetch_refspec;\n \ndiff --git a/t/t5582-fetch-negative-refspec.sh b/t/t5582-fetch-negative-refspec.sh\nindex f34509727702..030e6f978c4e 100755\n--- a/t/t5582-fetch-negative-refspec.sh\n+++ b/t/t5582-fetch-negative-refspec.sh\n@@ -240,4 +240,34 @@ test_expect_success \"push with matching +: and negative refspec\" '\n \tgit -C two push -v one\n '\n \n+test_expect_success '--prefetch correctly modifies refspecs' '\n+\tgit -C one config --unset-all remote.origin.fetch &&\n+\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n+\tgit -C one config --add remote.origin.fetch ^refs/heads/bogus/ignore &&\n+\tgit -C one config --add remote.origin.fetch \"refs/heads/bogus/*:bogus/*\" &&\n+\n+\tgit tag -a -m never never-fetch-tag HEAD &&\n+\n+\tgit branch bogus/fetched HEAD~1 &&\n+\tgit branch bogus/ignore HEAD &&\n+\n+\tgit -C one fetch --prefetch --no-tags &&\n+\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n+\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n+\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore &&\n+\n+\t# correctly handle when refspec set becomes empty\n+\t# after removing the refs/tags/* refspec.\n+\tgit -C one config --unset-all remote.origin.fetch &&\n+\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n+\n+\tgit -C one fetch --prefetch --no-tags &&\n+\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n+\n+\t# The refspec for refs that are not fully qualified\n+\t# are filtered multiple times.\n+\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n+\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"421512","messageId":"ed055d772452f2cd445d4b8d42ea6b92f363466e.1618020226.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v3.git.1618020225.gitgitgadget@gmail.com","subject":"[PATCH v3 3/3] maintenance: use 'git fetch --prefetch'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-10T02:03:45Z","receivedAt":"2021-04-10T02:03:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'prefetch' maintenance task previously forced the following refspec\nfor each remote:\n\n\t+refs/heads/*:refs/prefetch//*\n\nIf a user has specified a more strict refspec for the remote, then this\nprefetch task downloads more objects than necessary.\n\nThe previous change introduced the '--prefetch' option to 'git fetch'\nwhich manipulates the remote's refspec to place all resulting refs into\nrefs/prefetch/, with further partitioning based on the destinations of\nthose refspecs.\n\nUpdate the documentation to be more generic about the destination refs.\nDo not mention custom refspecs explicitly, as that does not need to be\nhighlighted in this documentation. The important part of placing refs in\nrefs/prefetch/ remains.\n\nReported-by: Tom Saeger \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 6 ++----\n builtin/gc.c | 7 +++----\n t/t7900-maintenance.sh | 14 +++++++-------\n 3 files changed, 12 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 80ddd33ceba0..1e738ad39832 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -92,10 +92,8 @@ commit-graph::\n prefetch::\n \tThe `prefetch` task updates the object directory with the latest\n \tobjects from all registered remotes. For each remote, a `git fetch`\n-\tcommand is run. The refmap is custom to avoid updating local or remote\n-\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n-\tremote refs are stored in `refs/prefetch//`. Also, tags are\n-\tnot updated.\n+\tcommand is run. The configured refspec is modified to place all\n+\trequested refs within `refs/prefetch/`. Also, tags are not updated.\n +\n This is done to avoid disrupting the remote-tracking branches. The end users\n expect these refs to stay unmoved unless they initiate a fetch. With prefetch\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex fa8128de9ae1..9d35f7da50d8 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -879,15 +879,14 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n-\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n+\tstrvec_pushl(&child.args, \"fetch\", remote->name,\n+\t\t \"--prefetch\", \"--prune\", \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n-\t\t \"--refmap=\", NULL);\n+\t\t NULL);\n \n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n-\n \treturn !!run_command(&child);\n }\n \ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 2412d8c5c006..eadb800c08cc 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -141,15 +141,15 @@ test_expect_success 'prefetch multiple remotes' '\n \ttest_commit -C clone1 one &&\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n-\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n-\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* log &&\n-- \ngitgitgadget\n"},{"id":"421513","messageId":"4c0e983ba56f030851370ef71887117db45e8479.1618020225.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v3.git.1618020225.gitgitgadget@gmail.com","subject":"[PATCH v3 1/3] maintenance: simplify prefetch logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-10T02:03:43Z","receivedAt":"2021-04-10T02:04:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous logic filled a string list with the names of each remote,\nbut instead we could simply run the appropriate 'git fetch' data\ndirectly in the remote iterator. Do this for reduced code size, but also\nbecause it sets up an upcoming change to use the remote's refspec. This\ndata is accessible from the 'struct remote' data that is now accessible\nin fetch_remote().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 33 ++++++++-------------------------\n 1 file changed, 8 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ef7226d7bca4..fa8128de9ae1 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n+static int fetch_remote(struct remote *remote, void *cbdata)\n {\n+\tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n-\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n+\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n \t\t \"--refmap=\", NULL);\n \n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n+\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n \n \treturn !!run_command(&child);\n }\n \n-static int append_remote(struct remote *remote, void *cbdata)\n-{\n-\tstruct string_list *remotes = (struct string_list *)cbdata;\n-\n-\tstring_list_append(remotes, remote->name);\n-\treturn 0;\n-}\n-\n static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n {\n-\tint result = 0;\n-\tstruct string_list_item *item;\n-\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n-\n \tgit_config_set_multivar_gently(\"log.excludedecoration\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n \t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n \n-\tif (for_each_remote(append_remote, &remotes)) {\n-\t\terror(_(\"failed to fill remotes\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n+\tif (for_each_remote(fetch_remote, opts)) {\n+\t\terror(_(\"failed to prefetch remotes\"));\n+\t\treturn 1;\n \t}\n \n-\tfor_each_string_list_item(item, &remotes)\n-\t\tresult |= fetch_remote(item->string, opts);\n-\n-cleanup:\n-\tstring_list_clear(&remotes, 0);\n-\treturn result;\n+\treturn 0;\n }\n \n static int maintenance_task_gc(struct maintenance_run_opts *opts)\n-- \ngitgitgadget\n\n"},{"id":"421514","messageId":"pull.924.v3.git.1618020225.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v2.git.1617734870.gitgitgadget@gmail.com","subject":"[PATCH v3 0/3] Maintenance: adapt custom refspecs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-10T02:03:42Z","receivedAt":"2021-04-10T02:04:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Tom Saeger rightly pointed out [1] that the prefetch task ignores custom\nrefspecs. This can lead to downloading more data than requested, and it\ndoesn't even help the future foreground fetches that use that custom\nrefspec.\n\n[1]\nhttps://lore.kernel.org/git/20210401184914.qmr7jhjbhp2mt3h6@dhcp-10-154-148-175.vpn.oracle.com/\n\nThis series fixes this problem by carefully replacing the start of each\nrefspec's destination with \"refs/prefetch/\". If the destination already\nstarts with \"refs/\", then that is replaced. Otherwise \"refs/prefetch/\" is\njust prepended.\n\nThis happens inside of git fetch when a --prefetch option is given. This\nallows us to maniuplate a struct refspec_item instead of a full refspec\nstring. It also simplifies our logic in testing the prefetch task.\n\n\nUpdate in V3\n============\n\n * The fix is almost completely rewritten as an update to 'git fetch'. See\n the new PATCH 2 for this update.\n\n * There was some discussion of rewriting test_subcommand, but that can be\n delayed until a proper solution is found to complications around softer\n matches.\n\n\nUpdates in V2\n=============\n\nThanks for the close eye on this series. I appreciate the recommendations,\nwhich I believe I have responded to them all:\n\n * Fixed typos.\n * Made refspec_item_format() re-entrant. Consumers must free the buffer.\n * Cleaned up style (quoting and tabbing).\n\nThanks, -Stolee\n\nDerrick Stolee (3):\n maintenance: simplify prefetch logic\n fetch: add --prefetch option\n maintenance: use 'git fetch --prefetch'\n\n Documentation/fetch-options.txt | 5 +++\n Documentation/git-maintenance.txt | 6 ++--\n builtin/fetch.c | 56 +++++++++++++++++++++++++++++++\n builtin/gc.c | 36 +++++---------------\n t/t5582-fetch-negative-refspec.sh | 30 +++++++++++++++++\n t/t7900-maintenance.sh | 14 ++++----\n 6 files changed, 109 insertions(+), 38 deletions(-)\n\n\nbase-commit: 89b43f80a514aee58b662ad606e6352e03eaeee4\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-924%2Fderrickstolee%2Fmaintenance%2Frefspec-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-924/derrickstolee/maintenance/refspec-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/924\n\nRange-diff vs v2:\n\n 1: 5aa0cb06c3f2 = 1: 4c0e983ba56f maintenance: simplify prefetch logic\n 2: d58a3e042ee8 < -: ------------ test-lib: use exact match for test_subcommand\n 3: 96388d949b98 < -: ------------ refspec: output a refspec item\n 4: bf296282323a < -: ------------ test-tool: test refspec input/output\n -: ------------ > 2: 7f488eea6dbd fetch: add --prefetch option\n 5: 9592224e3d42 ! 3: ed055d772452 maintenance: allow custom refspecs during prefetch\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: allow custom refspecs during prefetch\n + maintenance: use 'git fetch --prefetch'\n \n - The prefetch task previously used the default refspec source plus a\n - custom refspec destination to avoid colliding with remote refs:\n + The 'prefetch' maintenance task previously forced the following refspec\n + for each remote:\n \n +refs/heads/*:refs/prefetch//*\n \n - However, some users customize their refspec to reduce how much data they\n - download from specific remotes. This can involve restrictive patterns\n - for fetching or negative patterns to avoid downloading some refs.\n + If a user has specified a more strict refspec for the remote, then this\n + prefetch task downloads more objects than necessary.\n \n - Modify fetch_remote() to iterate over the remote's refspec list and\n - translate that into the appropriate prefetch scenario. Specifically,\n - re-parse the raw form of the refspec into a new 'struct refspec' and\n - modify the 'dst' member to replace a leading \"refs/\" substring with\n - \"refs/prefetch/\", or prepend \"refs/prefetch/\" to 'dst' otherwise.\n - Negative refspecs do not have a 'dst' so they can be transferred to the\n - 'git fetch' command unmodified.\n -\n - This prefix change provides the benefit of keeping whatever collisions\n - may exist in the custom refspecs, if that is a desirable outcome.\n -\n - This changes the names of the refs that would be fetched by the default\n - refspec. Instead of \"refs/prefetch//\" they will now go\n - to \"refs/prefetch/remotes//\". While this is a change, it\n - is not a seriously breaking one: these refs are intended to be hidden\n - and not used.\n + The previous change introduced the '--prefetch' option to 'git fetch'\n + which manipulates the remote's refspec to place all resulting refs into\n + refs/prefetch/, with further partitioning based on the destinations of\n + those refspecs.\n \n Update the documentation to be more generic about the destination refs.\n - Do not mention custom refpecs explicitly, as that does not need to be\n + Do not mention custom refspecs explicitly, as that does not need to be\n highlighted in this documentation. The important part of placing refs in\n - refs/prefetch remains.\n + refs/prefetch/ remains.\n \n Reported-by: Tom Saeger \n Signed-off-by: Derrick Stolee \n \n ## Documentation/git-maintenance.txt ##\n -@@ Documentation/git-maintenance.txt: prefetch::\n +@@ Documentation/git-maintenance.txt: commit-graph::\n + prefetch::\n + \tThe `prefetch` task updates the object directory with the latest\n \tobjects from all registered remotes. For each remote, a `git fetch`\n - \tcommand is run. The refmap is custom to avoid updating local or remote\n - \tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n +-\tcommand is run. The refmap is custom to avoid updating local or remote\n +-\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n -\tremote refs are stored in `refs/prefetch//`. Also, tags are\n -\tnot updated.\n -+\trefs are stored in `refs/prefetch/`. Also, tags are not updated.\n ++\tcommand is run. The configured refspec is modified to place all\n ++\trequested refs within `refs/prefetch/`. Also, tags are not updated.\n +\n This is done to avoid disrupting the remote-tracking branches. The end users\n expect these refs to stay unmoved unless they initiate a fetch. With prefetch\n \n ## builtin/gc.c ##\n -@@\n - #include \"remote.h\"\n - #include \"object-store.h\"\n - #include \"exec-cmd.h\"\n -+#include \"refspec.h\"\n - \n - #define FAILED_RUN \"failed to run %s\"\n - \n @@ builtin/gc.c: static int fetch_remote(struct remote *remote, void *cbdata)\n - {\n - \tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n -+\tint i;\n \n \tchild.git_cmd = 1;\n - \tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n -@@ builtin/gc.c: static int fetch_remote(struct remote *remote, void *cbdata)\n +-\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n ++\tstrvec_pushl(&child.args, \"fetch\", remote->name,\n ++\t\t \"--prefetch\", \"--prune\", \"--no-tags\",\n + \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n +-\t\t \"--refmap=\", NULL);\n ++\t\t NULL);\n + \n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n -+\tfor (i = 0; i < remote->fetch.nr; i++) {\n -+\t\tstruct refspec_item replace;\n -+\t\tstruct refspec_item *rsi = &remote->fetch.items[i];\n -+\t\tstruct strbuf new_dst = STRBUF_INIT;\n -+\t\tsize_t ignore_len = 0;\n -+\t\tchar *replace_string;\n -+\n -+\t\tif (rsi->negative) {\n -+\t\t\tstrvec_push(&child.args, remote->fetch.raw[i]);\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n -+\t\trefspec_item_init(&replace, remote->fetch.raw[i], 1);\n -+\n -+\t\t/*\n -+\t\t * If a refspec dst starts with \"refs/\" at the start,\n -+\t\t * then we will replace \"refs/\" with \"refs/prefetch/\".\n -+\t\t * Otherwise, we will prepend the dst string with\n -+\t\t * \"refs/prefetch/\".\n -+\t\t */\n -+\t\tif (!strncmp(replace.dst, \"refs/\", 5))\n -+\t\t\tignore_len = 5;\n -+\n -+\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n -+\t\tstrbuf_addstr(&new_dst, replace.dst + ignore_len);\n -+\t\tfree(replace.dst);\n -+\t\treplace.dst = strbuf_detach(&new_dst, NULL);\n -+\n -+\t\treplace_string = refspec_item_format(&replace);\n -+\t\tstrvec_push(&child.args, replace_string);\n -+\t\tfree(replace_string);\n -+\n -+\t\trefspec_item_clear(&replace);\n -+\t}\n - \n +-\n \treturn !!run_command(&child);\n }\n + \n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n + \ttest_commit -C clone1 one &&\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n - \tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n --\ttest_subcommand git fetch remote1 $fetchargs \"+refs/heads/*:refs/prefetch/remote1/*\" log &&\n - \t! grep \"prefetch\" log\n - '\n - \n -+test_expect_success 'prefetch custom refspecs' '\n -+\tgit -C clone1 branch -f special/fetched HEAD &&\n -+\tgit -C clone1 branch -f special/secret/not-fetched HEAD &&\n -+\n -+\t# create multiple refspecs for remote1\n -+\tgit config --add remote.remote1.fetch \"+refs/heads/special/fetched:refs/heads/fetched\" &&\n -+\tgit config --add remote.remote1.fetch \"^refs/heads/special/secret/not-fetched\" &&\n -+\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/prefetch-refspec.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n -+\n -+\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n -+\n -+\t# skips second refspec because it is not a pattern type\n -+\trs1=\"+refs/heads/*:refs/prefetch/remotes/remote1/*\" &&\n -+\trs2=\"+refs/heads/special/fetched:refs/prefetch/heads/fetched\" &&\n -+\trs3=\"^refs/heads/special/secret/not-fetched\" &&\n -+\n -+\ttest_subcommand git fetch remote1 $fetchargs \"$rs1\" \"$rs2\" \"$rs3\" On 4/9/2021 3:28 PM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Fri, Apr 09 2021, Derrick Stolee wrote:\n>> \n>>> On 4/7/2021 6:26 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>> I think converting the whole thing to something like the WIP/RFC patch\n>>>> below is much better and more readable.\n>>>\n>>> This is an interesting approach. I don't see you using the ERR that you\n>>> are inputting anywhere, so that seems like an unnecessary bloat to the\n>>> consumers. But maybe I haven't discovered all of the places where this\n>>> would be useful, but it seems better to pipe stderr to a file for later\n>>> comparison when needed.\n>> \n>> Yes, it's probably not a good default here. For the test-lib.sh tests\n>> there's check_sub_test_lib_test and check_sub_test_lib_test_err, most of\n>> the tests only test stdout.\n>> \n>>>> +test_expect_process_tree () {\n>>>> +\tdepth= &&\n>>>> +\t>actual &&\n>>>> +\tcat >expect &&\n>>>> +\tcat <&3 >expect.err\n>>>> +\twhile test $# != 0\n>>>> +\tdo\n>>>> +\t\tcase \"$1\" in\n>>>> +\t\t--depth)\n>>>> +\t\t\tdepth=\"$2\"\n>>>> +\t\t\tshift\n>>>> +\t\t\t;;\n>>>> +\t\t*)\n>>>> +\t\t\tbreak\n>>>> +\t\t\t;;\n>>>> +\t\tesac\n>>>> +\t\tshift\n>>>> +\tdone &&\n>>> Do you have an example where this is being checked? Or can depth\n>>> be left as 1 for now?\n>> \n>> It can probably be hardcoded, but I was hoping someone more familiar\n>> with trace2 would chime in, but I'm fairly sure there's not a way to do\n>> it without parsing the existing output with either some clever\n>> grep/awk-ing of the PERF output, or stateful parsing of the JSON.\n>> \n>> I thought that for git maintenance tests perhaps something wanted to\n>> assert that we didn't have maintenance invoking maintenance, or that\n>> something expected to prune refs really invoked the relevant prune\n>> command via \"gc\".\n>> \n>>>> +\tlog=\"$(pwd)/proc-tree.txt\" &&\n>>>> +\t>\"$log\" &&\n>>>> +\tGIT_TRACE2_PERF=\"$log\" \"$@\" 2>actual.err &&\n>>>> +\tgrep \"child_start\" proc-tree.txt >proc-tree-start.txt || : &&\n>>>> +\tif test -n \"$depth\"\n>>>> +\tthen\n>>>> +\t\tgrep \" d$depth \" proc-tree-start.txt >tmp.txt || : &&\n>>>> +\t\tmv tmp.txt proc-tree-start.txt\n>>>> +\tfi &&\n>>>> +\tsed -e 's/^.*argv:\\[//' -e 's/\\]$//' actual &&\n>>>> +\ttest_cmp expect actual &&\n>>>> +\ttest_cmp expect.err actual.err\n>>>> +} 7>&2 2>&4\n>>>\n>>> I think similar ideas could apply to test_region. Giving it a try\n>>> now.\n>> \n>> Probably, I didn't even notice that one...\n>\n> I gave this a few hours today, and I'm giving up. I'm the first to\n> admit that I don't have the correct scripting skills to do some of\n> these things.\n>\n> I've got what I tried below. It certainly looks like it would work.\n> It solves the problem of \"what if the test is flaky?\" by ensuring that\n> all subcommands (at depth 0) match the inputs exactly.\n\nLooks good!\n\n> However, the problem comes when trying to make that work for all of\n> the maintenance tests, specifically the 'incremental-repack' task.\n> That task dynamically computes a --batch-size=X parameter, and that\n> is not stable across runs of the script.\n>\n> This was avoided in the past by only checking for the first of three\n> subcommands when verifying that the 'incremental-repack' task worked.\n> That is, except for the EXPENSIVE test that checks that the --batch-size\n> maxes out at 2g.\n>\n> The thing that might make these changing parameters work is to allow\n> the specified lines be a _prefix_ of the actual parameters. Or, let\n> each line be a pattern that is checked against that line. Issues come\n> up with how to handle this line-by-line check that I was unable to\n> overcome.\n>\n> The good news is that the idea of adding a '--prefetch' option to\n> 'git fetch' makes the change to t7900-maintenance.sh much easier,\n> making this change to test_subcommand less of a priority.\n>\n> I include my attempt here as a patch. Feel free to take whatever\n> you want of it, or none of it and start over. I do think that it\n> makes the test script look much nicer.\n\nI think a good way to deal with that is to have a lower-level helper\nfunction that doesn't do the test_cmp, and instead just runs the\ncommand, and leaves the stdout/stderr files in-place for another \"check\"\nhelper.\n\nOn a WIP topic I split up the t0000-basic.sh \"test test-lib.sh itself\"\ncode to do pretty much that:\nhttps://github.com/avar/git/blob/avar/support-test-verbose-under-prove-2/t/lib-subtest.sh\n\nIt's basically back to the existing model of \"run a command and grep it\nlater\", except that we can pass the JSON through some basic parser, and\nextract common cases like test_cmp, prefix munging before test_cmp etc.\n"},{"id":"421562","messageId":"xmqqtuodiowj.fsf@gitster.g","threadId":"55434","inReplyTo":"pull.924.v3.git.1618020225.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/3] Maintenance: adapt custom refspecs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-11T01:35:40Z","receivedAt":"2021-04-11T01:35:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> * The fix is almost completely rewritten as an update to 'git fetch'. See\n> the new PATCH 2 for this update.\n\nI do agree that it gives us the most flexibility there with nice\nencapsulation. Nobody other than \"git fetch\" needs to know how it\ncomputes which remote refs are fetched given the real pathspec, and\nthe only thing the caller with \"--prefetch\" is interested in is that\nthe prefetch operation would not contaminate the remote-tracking\nrefs.\n\nGreat idea. I wish I were the one who thought of it first ;-)\n"},{"id":"421625","messageId":"9e009c7e-a2bf-1740-ba43-c64a97f1dc28@ramsayjones.plus.com","threadId":"55434","inReplyTo":"7f488eea6dbdd93ff1a97a2572de7ddb0d65b7e0.1618020225.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/3] fetch: add --prefetch option","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2021-04-11T21:09:01Z","receivedAt":"2021-04-11T21:09:17Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 10/04/2021 03:03, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The --prefetch option will be used by the 'prefetch' maintenance task\n> instead of sending refspecs explicitly across the command-line. The\n> intention is to modify the refspec to place all results in\n> refs/prefetch/ instead of anywhere else.\n> \n> Create helper method filter_prefetch_refspec() to modify a given refspec\n> to fit the rules expected of the prefetch task:\n> \n> * Negative refspecs are preserved.\n> * Refspecs without a destination are removed.\n> * Refspecs whose source starts with \"refs/tags/\" are removed.\n> * Other refspecs are placed within \"refs/prefetch/\".\n> \n> Finally, we add the 'force' option to ensure that prefetch refs are\n> replaced as necessary.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/fetch-options.txt | 5 +++\n> builtin/fetch.c | 56 +++++++++++++++++++++++++++++++\n> t/t5582-fetch-negative-refspec.sh | 30 +++++++++++++++++\n> 3 files changed, 91 insertions(+)\n> \n> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> index 07783deee309..9e7b4e189ce0 100644\n> --- a/Documentation/fetch-options.txt\n> +++ b/Documentation/fetch-options.txt\n> @@ -110,6 +110,11 @@ ifndef::git-pull[]\n> \tsetting `fetch.writeCommitGraph`.\n> endif::git-pull[]\n> \n> +--prefetch::\n> +\tModify the configured refspec to place all refs into the\n> +\t`refs/prefetch/` namespace. See the `prefetch` task in\n> +\tlinkgit:git-maintenance[1].\n> +\n> -p::\n> --prune::\n> \tBefore fetching, remove any remote-tracking references that no\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 0b90de87c7a2..30856b442b79 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -48,6 +48,7 @@ enum {\n> static int fetch_prune_config = -1; /* unspecified */\n> static int fetch_show_forced_updates = 1;\n> static uint64_t forced_updates_ms = 0;\n> +static int prefetch = 0;\n> static int prune = -1; /* unspecified */\n> #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */\n> \n> @@ -158,6 +159,8 @@ static struct option builtin_fetch_options[] = {\n> \t\t N_(\"do not fetch all tags (--no-tags)\"), TAGS_UNSET),\n> \tOPT_INTEGER('j', \"jobs\", &max_jobs,\n> \t\t N_(\"number of submodules fetched in parallel\")),\n> +\tOPT_BOOL(0, \"prefetch\", &prefetch,\n> +\t\t N_(\"modify the refspec to place all refs within refs/prefetch/\")),\n> \tOPT_BOOL('p', \"prune\", &prune,\n> \t\t N_(\"prune remote-tracking branches no longer on remote\")),\n> \tOPT_BOOL('P', \"prune-tags\", &prune_tags,\n> @@ -436,6 +439,55 @@ static void find_non_local_tags(const struct ref *refs,\n> \toidset_clear(&fetch_oids);\n> }\n> \n> +static void filter_prefetch_refspec(struct refspec *rs)\n> +{\n> +\tint i;\n> +\n> +\tif (!prefetch)\n> +\t\treturn;\n> +\n> +\tfor (i = 0; i < rs->nr; i++) {\n> +\t\tstruct strbuf new_dst = STRBUF_INIT;\n> +\t\tchar *old_dst;\n> +\t\tconst char *sub = NULL;\n> +\n> +\t\tif (rs->items[i].negative)\n> +\t\t\tcontinue;\n> +\t\tif (!rs->items[i].dst ||\n> +\t\t (rs->items[i].src &&\n> +\t\t !strncmp(rs->items[i].src, \"refs/tags/\", 10))) {\n> +\t\t\tint j;\n> +\n> +\t\t\tfree(rs->items[i].src);\n> +\t\t\tfree(rs->items[i].dst);\n> +\n> +\t\t\tfor (j = i + 1; j < rs->nr; j++) {\n> +\t\t\t\trs->items[j - 1] = rs->items[j];\n> +\t\t\t\trs->raw[j - 1] = rs->raw[j];\n> +\t\t\t}\n> +\t\t\trs->nr--;\n\nHmm, don't you need to do 'i--;' here?\n\n(Sorry in advance if this is nonsense, I am just skimming the\npatches without reading the whole series carefully).\n\nMaybe try a test which has an entry, which requires the 'prefetch'\nmodification, that immediately follows a 'tag' or 'empty dst' entry.\n(I can't quite tell, just reading the email, whether that is covered\nby the tests below - so please just ignore me if it already works ;)\n\nATB,\nRamsay Jones\n\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\told_dst = rs->items[i].dst;\n> +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> +\n> +\t\t/*\n> +\t\t * If old_dst starts with \"refs/\", then place\n> +\t\t * sub after that prefix. Otherwise, start at\n> +\t\t * the beginning of the string.\n> +\t\t */\n> +\t\tif (!skip_prefix(old_dst, \"refs/\", &sub))\n> +\t\t\tsub = old_dst;\n> +\t\tstrbuf_addstr(&new_dst, sub);\n> +\n> +\t\trs->items[i].dst = strbuf_detach(&new_dst, NULL);\n> +\t\trs->items[i].force = 1;\n> +\n> +\t\tfree(old_dst);\n> +\t}\n> +}\n> +\n> static struct ref *get_ref_map(struct remote *remote,\n> \t\t\t const struct ref *remote_refs,\n> \t\t\t struct refspec *rs,\n> @@ -452,6 +504,10 @@ static struct ref *get_ref_map(struct remote *remote,\n> \tstruct hashmap existing_refs;\n> \tint existing_refs_populated = 0;\n> \n> +\tfilter_prefetch_refspec(rs);\n> +\tif (remote)\n> +\t\tfilter_prefetch_refspec(&remote->fetch);\n> +\n> \tif (rs->nr) {\n> \t\tstruct refspec *fetch_refspec;\n> \n> diff --git a/t/t5582-fetch-negative-refspec.sh b/t/t5582-fetch-negative-refspec.sh\n> index f34509727702..030e6f978c4e 100755\n> --- a/t/t5582-fetch-negative-refspec.sh\n> +++ b/t/t5582-fetch-negative-refspec.sh\n> @@ -240,4 +240,34 @@ test_expect_success \"push with matching +: and negative refspec\" '\n> \tgit -C two push -v one\n> '\n> \n> +test_expect_success '--prefetch correctly modifies refspecs' '\n> +\tgit -C one config --unset-all remote.origin.fetch &&\n> +\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n> +\tgit -C one config --add remote.origin.fetch ^refs/heads/bogus/ignore &&\n> +\tgit -C one config --add remote.origin.fetch \"refs/heads/bogus/*:bogus/*\" &&\n> +\n> +\tgit tag -a -m never never-fetch-tag HEAD &&\n> +\n> +\tgit branch bogus/fetched HEAD~1 &&\n> +\tgit branch bogus/ignore HEAD &&\n> +\n> +\tgit -C one fetch --prefetch --no-tags &&\n> +\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n> +\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n> +\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore &&\n> +\n> +\t# correctly handle when refspec set becomes empty\n> +\t# after removing the refs/tags/* refspec.\n> +\tgit -C one config --unset-all remote.origin.fetch &&\n> +\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n> +\n> +\tgit -C one fetch --prefetch --no-tags &&\n> +\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n> +\n> +\t# The refspec for refs that are not fully qualified\n> +\t# are filtered multiple times.\n> +\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n> +\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore\n> +'\n> +\n> test_done\n> \n"},{"id":"421708","messageId":"20210412164805.pxq7acln43gbggkl@dhcp-10-154-186-72.vpn.oracle.com","threadId":"55434","inReplyTo":"xmqqtuodiowj.fsf@gitster.g","subject":"Re: [PATCH v3 0/3] Maintenance: adapt custom refspecs","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-12T16:48:05Z","receivedAt":"2021-04-12T16:52:55Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Sat, Apr 10, 2021 at 06:35:40PM -0700, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n> > * The fix is almost completely rewritten as an update to 'git fetch'. See\n> > the new PATCH 2 for this update.\n> \n> I do agree that it gives us the most flexibility there with nice\n> encapsulation. Nobody other than \"git fetch\" needs to know how it\n> computes which remote refs are fetched given the real pathspec, and\n> the only thing the caller with \"--prefetch\" is interested in is that\n> the prefetch operation would not contaminate the remote-tracking\n> refs.\n> \n> Great idea. I wish I were the one who thought of it first ;-)\n\nYes - this simplifies things greatly!\n\nI do have one case that fails prefetch though.\nIt's a case where all the remote's fetch configs are filtered out.\n\nExample:\n\n\t[remote \"pr-924\"]\n\t url = https://github.com/gitgitgadget/git\n\t fetch = +refs/tags/pr-924/derrickstolee/maintenance/refspec-v3\n\t skipfetchall = true\n\t tagopt = --no-tags\n\n\nIn this case, running `git fetch pr-924` will fetch and update\nFETCH_HEAD, but running with maintenance prefetch task results in:\n\nfatal: Couldn't find remote ref HEAD\nerror: failed to prefetch remotes\nerror: task 'prefetch' failed\n\nI tracked this down a bit, but don't have a suggestion how to fix it.\n\nbuiltin/fetch.c `get_ref_map` makes two calls to `filter_prefetch_refspec`,\none for 'rs' and another for 'remote->fetch'.\n\n`filter_prefetch_refspec` works and filters out the above fetch config.\nThis correctly yields condition\n`rs->nr == 0` and `remote->fetch.nr == 0`\n\nLater a call is made to `get_remote_ref(remote_refs, \"HEAD\")` which\nfails, leading to `fatal: Couldn't find remote ref HEAD`\n\nShould this be expected, or should this now be special-cased for 'prefetch'\nsomehow?\n\nRegards,\n\n--Tom\n"},{"id":"421738","messageId":"20210412172423.rjwv7gozmnnzcaep@dhcp-10-154-186-72.vpn.oracle.com","threadId":"55434","inReplyTo":"20210412164805.pxq7acln43gbggkl@dhcp-10-154-186-72.vpn.oracle.com","subject":"Re: [PATCH v3 0/3] Maintenance: adapt custom refspecs","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-12T17:24:23Z","receivedAt":"2021-04-12T17:24:39Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Mon, Apr 12, 2021 at 11:48:09AM -0500, Tom Saeger wrote:\n> On Sat, Apr 10, 2021 at 06:35:40PM -0700, Junio C Hamano wrote:\n> > \"Derrick Stolee via GitGitGadget\" writes:\n> > \n> > > * The fix is almost completely rewritten as an update to 'git fetch'. See\n> > > the new PATCH 2 for this update.\n> > \n> > I do agree that it gives us the most flexibility there with nice\n> > encapsulation. Nobody other than \"git fetch\" needs to know how it\n> > computes which remote refs are fetched given the real pathspec, and\n> > the only thing the caller with \"--prefetch\" is interested in is that\n> > the prefetch operation would not contaminate the remote-tracking\n> > refs.\n> > \n> > Great idea. I wish I were the one who thought of it first ;-)\n> \n> Yes - this simplifies things greatly!\n> \n> I do have one case that fails prefetch though.\n> It's a case where all the remote's fetch configs are filtered out.\n> \n> Example:\n> \n> \t[remote \"pr-924\"]\n> \t url = https://github.com/gitgitgadget/git\n> \t fetch = +refs/tags/pr-924/derrickstolee/maintenance/refspec-v3\n> \t skipfetchall = true\n> \t tagopt = --no-tags\n> \n> \n> In this case, running `git fetch pr-924` will fetch and update\n> FETCH_HEAD, but running with maintenance prefetch task results in:\n> \n> fatal: Couldn't find remote ref HEAD\n> error: failed to prefetch remotes\n> error: task 'prefetch' failed\n> \n> I tracked this down a bit, but don't have a suggestion how to fix it.\n\nThis ugly hack fixes this failure. I'll keep staring at it.\n\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 30856b442b79..6489ce7d8d3b 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -508,6 +508,9 @@ static struct ref *get_ref_map(struct remote *remote,\n if (remote)\n filter_prefetch_refspec(&remote->fetch);\n\n+ if (prefetch && !rs->nr && remote && !remote->fetch.nr)\n+ return NULL;\n+\n if (rs->nr) {\n struct refspec *fetch_refspec;\n\n--\n\n\n\n> \n> builtin/fetch.c `get_ref_map` makes two calls to `filter_prefetch_refspec`,\n> one for 'rs' and another for 'remote->fetch'.\n> \n> `filter_prefetch_refspec` works and filters out the above fetch config.\n> This correctly yields condition\n> `rs->nr == 0` and `remote->fetch.nr == 0`\n> \n> Later a call is made to `get_remote_ref(remote_refs, \"HEAD\")` which\n> fails, leading to `fatal: Couldn't find remote ref HEAD`\n> \n> Should this be expected, or should this now be special-cased for 'prefetch'\n> somehow?\n> \n> Regards,\n> \n> --Tom\n"},{"id":"421740","messageId":"20210412174113.g5iv4muedcklg7mv@dhcp-10-154-186-72.vpn.oracle.com","threadId":"55434","inReplyTo":"20210412172423.rjwv7gozmnnzcaep@dhcp-10-154-186-72.vpn.oracle.com","subject":"Re: [PATCH v3 0/3] Maintenance: adapt custom refspecs","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-12T17:41:13Z","receivedAt":"2021-04-12T17:41:38Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Mon, Apr 12, 2021 at 12:24:27PM -0500, Tom Saeger wrote:\n> On Mon, Apr 12, 2021 at 11:48:09AM -0500, Tom Saeger wrote:\n> > On Sat, Apr 10, 2021 at 06:35:40PM -0700, Junio C Hamano wrote:\n> > > \"Derrick Stolee via GitGitGadget\" writes:\n> > > \n> > > > * The fix is almost completely rewritten as an update to 'git fetch'. See\n> > > > the new PATCH 2 for this update.\n> > > \n> > > I do agree that it gives us the most flexibility there with nice\n> > > encapsulation. Nobody other than \"git fetch\" needs to know how it\n> > > computes which remote refs are fetched given the real pathspec, and\n> > > the only thing the caller with \"--prefetch\" is interested in is that\n> > > the prefetch operation would not contaminate the remote-tracking\n> > > refs.\n> > > \n> > > Great idea. I wish I were the one who thought of it first ;-)\n> > \n> > Yes - this simplifies things greatly!\n> > \n> > I do have one case that fails prefetch though.\n> > It's a case where all the remote's fetch configs are filtered out.\n> > \n> > Example:\n> > \n> > \t[remote \"pr-924\"]\n> > \t url = https://github.com/gitgitgadget/git\n> > \t fetch = +refs/tags/pr-924/derrickstolee/maintenance/refspec-v3\n> > \t skipfetchall = true\n> > \t tagopt = --no-tags\n> > \n> > \n> > In this case, running `git fetch pr-924` will fetch and update\n> > FETCH_HEAD, but running with maintenance prefetch task results in:\n> > \n> > fatal: Couldn't find remote ref HEAD\n> > error: failed to prefetch remotes\n> > error: task 'prefetch' failed\n> > \n> > I tracked this down a bit, but don't have a suggestion how to fix it.\n> \n> This ugly hack fixes this failure. I'll keep staring at it.\n> \n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 30856b442b79..6489ce7d8d3b 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -508,6 +508,9 @@ static struct ref *get_ref_map(struct remote *remote,\n> if (remote)\n> filter_prefetch_refspec(&remote->fetch);\n> \n> + if (prefetch && !rs->nr && remote && !remote->fetch.nr)\n> + return NULL;\n> +\n> if (rs->nr) {\n> struct refspec *fetch_refspec;\n> \n> --\n> \n\nLess ugly fix...\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 30856b442b79..5fbffbd17d7d 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -576,6 +576,8 @@ static struct ref *get_ref_map(struct remote *remote,\n if (has_merge &&\n !strcmp(branch->remote_name, remote->name))\n add_merge_config(&ref_map, remote_refs, branch, &tail);\n+ } else if (prefetch) {\n+ ;\n } else {\n ref_map = get_remote_ref(remote_refs, \"HEAD\");\n if (!ref_map)\n--\n\nOther ideas?\n\n\n> \n> \n> > \n> > builtin/fetch.c `get_ref_map` makes two calls to `filter_prefetch_refspec`,\n> > one for 'rs' and another for 'remote->fetch'.\n> > \n> > `filter_prefetch_refspec` works and filters out the above fetch config.\n> > This correctly yields condition\n> > `rs->nr == 0` and `remote->fetch.nr == 0`\n> > \n> > Later a call is made to `get_remote_ref(remote_refs, \"HEAD\")` which\n> > fails, leading to `fatal: Couldn't find remote ref HEAD`\n> > \n> > Should this be expected, or should this now be special-cased for 'prefetch'\n> > somehow?\n> > \n> > Regards,\n> > \n> > --Tom\n"},{"id":"421779","messageId":"20210412201308.teugitvky5cctcio@dhcp-10-154-186-72.vpn.oracle.com","threadId":"55434","inReplyTo":"4c0e983ba56f030851370ef71887117db45e8479.1618020225.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/3] maintenance: simplify prefetch logic","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-12T20:13:08Z","receivedAt":"2021-04-12T20:13:32Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Sat, Apr 10, 2021 at 02:03:43AM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The previous logic filled a string list with the names of each remote,\n> but instead we could simply run the appropriate 'git fetch' data\n> directly in the remote iterator. Do this for reduced code size, but also\n> because it sets up an upcoming change to use the remote's refspec. This\n> data is accessible from the 'struct remote' data that is now accessible\n> in fetch_remote().\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/gc.c | 33 ++++++++-------------------------\n> 1 file changed, 8 insertions(+), 25 deletions(-)\n> \n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index ef7226d7bca4..fa8128de9ae1 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n> \treturn 0;\n> }\n> \n> -static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n> +static int fetch_remote(struct remote *remote, void *cbdata)\n> {\n> +\tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n\n\nI think this might be appropriate to add:\n\n\n if (remote->skip_default_update)\n return 0;\n\n\nmaintenance prefetch is acting like `git fetch --all`\nSo it should also skip remotes with configs `skipfetchall = true`\nAgree?\n\n> \n> \tchild.git_cmd = 1;\n> -\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n> +\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n> \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n> \t\t \"--refmap=\", NULL);\n> \n> \tif (opts->quiet)\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \n> -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n> +\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n> \n> \treturn !!run_command(&child);\n> }\n> \n> -static int append_remote(struct remote *remote, void *cbdata)\n> -{\n> -\tstruct string_list *remotes = (struct string_list *)cbdata;\n> -\n> -\tstring_list_append(remotes, remote->name);\n> -\treturn 0;\n> -}\n> -\n> static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n> {\n> -\tint result = 0;\n> -\tstruct string_list_item *item;\n> -\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n> -\n> \tgit_config_set_multivar_gently(\"log.excludedecoration\",\n> \t\t\t\t\t\"refs/prefetch/\",\n> \t\t\t\t\t\"refs/prefetch/\",\n> \t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n> \t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n> \n> -\tif (for_each_remote(append_remote, &remotes)) {\n> -\t\terror(_(\"failed to fill remotes\"));\n> -\t\tresult = 1;\n> -\t\tgoto cleanup;\n> +\tif (for_each_remote(fetch_remote, opts)) {\n> +\t\terror(_(\"failed to prefetch remotes\"));\n> +\t\treturn 1;\n> \t}\n> \n> -\tfor_each_string_list_item(item, &remotes)\n> -\t\tresult |= fetch_remote(item->string, opts);\n> -\n> -cleanup:\n> -\tstring_list_clear(&remotes, 0);\n> -\treturn result;\n> +\treturn 0;\n> }\n> \n> static int maintenance_task_gc(struct maintenance_run_opts *opts)\n> -- \n> gitgitgadget\n> \n"},{"id":"421780","messageId":"28857e57-d9c6-ecb9-8a7d-d85a29f20f33@gmail.com","threadId":"55434","inReplyTo":"9e009c7e-a2bf-1740-ba43-c64a97f1dc28@ramsayjones.plus.com","subject":"Re: [PATCH v3 2/3] fetch: add --prefetch option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-12T20:23:51Z","receivedAt":"2021-04-12T20:24:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/11/21 5:09 PM, Ramsay Jones wrote:> On 10/04/2021 03:03, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>> +\tfor (i = 0; i < rs->nr; i++) {\n>> +\t\tstruct strbuf new_dst = STRBUF_INIT;\n>> +\t\tchar *old_dst;\n>> +\t\tconst char *sub = NULL;\n>> +\n>> +\t\tif (rs->items[i].negative)\n>> +\t\t\tcontinue;\n>> +\t\tif (!rs->items[i].dst ||\n>> +\t\t (rs->items[i].src &&\n>> +\t\t !strncmp(rs->items[i].src, \"refs/tags/\", 10))) {\n>> +\t\t\tint j;\n>> +\n>> +\t\t\tfree(rs->items[i].src);\n>> +\t\t\tfree(rs->items[i].dst);\n>> +\n>> +\t\t\tfor (j = i + 1; j < rs->nr; j++) {\n>> +\t\t\t\trs->items[j - 1] = rs->items[j];\n>> +\t\t\t\trs->raw[j - 1] = rs->raw[j];\n>> +\t\t\t}\n>> +\t\t\trs->nr--;\n> \n> Hmm, don't you need to do 'i--;' here?\n> \n> (Sorry in advance if this is nonsense, I am just skimming the\n> patches without reading the whole series carefully).\n> \n> Maybe try a test which has an entry, which requires the 'prefetch'\n> modification, that immediately follows a 'tag' or 'empty dst' entry.\n> (I can't quite tell, just reading the email, whether that is covered\n> by the tests below - so please just ignore me if it already works ;)\n\nYou are absolutely right! Thanks.\n\n-Stolee\n"},{"id":"421781","messageId":"e7f09182-3b75-f2a3-c9c7-0055cc377a32@gmail.com","threadId":"55434","inReplyTo":"20210412174113.g5iv4muedcklg7mv@dhcp-10-154-186-72.vpn.oracle.com","subject":"Re: [PATCH v3 0/3] Maintenance: adapt custom refspecs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-12T20:25:59Z","receivedAt":"2021-04-12T20:26:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/12/21 1:41 PM, Tom Saeger wrote:> \n> Less ugly fix...\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 30856b442b79..5fbffbd17d7d 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -576,6 +576,8 @@ static struct ref *get_ref_map(struct remote *remote,\n> if (has_merge &&\n> !strcmp(branch->remote_name, remote->name))\n> add_merge_config(&ref_map, remote_refs, branch, &tail);\n> + } else if (prefetch) {\n> + ;\n> } else {\n\nI'll give this a try, but with \"else if (!prefetch)\" for the\nlast block,instead.\n\nThanks for your diligent testing! It's helping a lot.\n\nThanks,\n-Stolee\n"},{"id":"421782","messageId":"2f4fa2b5-0d8b-b368-ab4d-411740595a4f@gmail.com","threadId":"55434","inReplyTo":"20210412201308.teugitvky5cctcio@dhcp-10-154-186-72.vpn.oracle.com","subject":"Re: [PATCH v3 1/3] maintenance: simplify prefetch logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-12T20:27:26Z","receivedAt":"2021-04-12T20:27:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/12/21 4:13 PM, Tom Saeger wrote:\n> On Sat, Apr 10, 2021 at 02:03:43AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> -static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n>> +static int fetch_remote(struct remote *remote, void *cbdata)\n>> {\n>> +\tstruct maintenance_run_opts *opts = cbdata;\n>> \tstruct child_process child = CHILD_PROCESS_INIT;\n> \n> \n> I think this might be appropriate to add:\n> \n> \n> if (remote->skip_default_update)\n> return 0;\n> \n> \n> maintenance prefetch is acting like `git fetch --all`\n> So it should also skip remotes with configs `skipfetchall = true`\n> Agree?\n\nTIL about skipfetchall. I think that's a good idea to introduce.\nIt'll be a new patch, not added in this one.\n\nThanks,\n-Stolee\n"},{"id":"422089","messageId":"4c0e983ba56f030851370ef71887117db45e8479.1618577399.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v4.git.1618577399.gitgitgadget@gmail.com","subject":"[PATCH v4 1/4] maintenance: simplify prefetch logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-16T12:49:56Z","receivedAt":"2021-04-16T12:50:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous logic filled a string list with the names of each remote,\nbut instead we could simply run the appropriate 'git fetch' data\ndirectly in the remote iterator. Do this for reduced code size, but also\nbecause it sets up an upcoming change to use the remote's refspec. This\ndata is accessible from the 'struct remote' data that is now accessible\nin fetch_remote().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 33 ++++++++-------------------------\n 1 file changed, 8 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ef7226d7bca4..fa8128de9ae1 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n+static int fetch_remote(struct remote *remote, void *cbdata)\n {\n+\tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n-\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n+\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n \t\t \"--refmap=\", NULL);\n \n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n+\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n \n \treturn !!run_command(&child);\n }\n \n-static int append_remote(struct remote *remote, void *cbdata)\n-{\n-\tstruct string_list *remotes = (struct string_list *)cbdata;\n-\n-\tstring_list_append(remotes, remote->name);\n-\treturn 0;\n-}\n-\n static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n {\n-\tint result = 0;\n-\tstruct string_list_item *item;\n-\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n-\n \tgit_config_set_multivar_gently(\"log.excludedecoration\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\t\"refs/prefetch/\",\n \t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n \t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n \n-\tif (for_each_remote(append_remote, &remotes)) {\n-\t\terror(_(\"failed to fill remotes\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n+\tif (for_each_remote(fetch_remote, opts)) {\n+\t\terror(_(\"failed to prefetch remotes\"));\n+\t\treturn 1;\n \t}\n \n-\tfor_each_string_list_item(item, &remotes)\n-\t\tresult |= fetch_remote(item->string, opts);\n-\n-cleanup:\n-\tstring_list_clear(&remotes, 0);\n-\treturn result;\n+\treturn 0;\n }\n \n static int maintenance_task_gc(struct maintenance_run_opts *opts)\n-- \ngitgitgadget\n\n"},{"id":"422090","messageId":"pull.924.v4.git.1618577399.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v3.git.1618020225.gitgitgadget@gmail.com","subject":"[PATCH v4 0/4] Maintenance: adapt custom refspecs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-16T12:49:55Z","receivedAt":"2021-04-16T12:50:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Tom Saeger rightly pointed out [1] that the prefetch task ignores custom\nrefspecs. This can lead to downloading more data than requested, and it\ndoesn't even help the future foreground fetches that use that custom\nrefspec.\n\n[1]\nhttps://lore.kernel.org/git/20210401184914.qmr7jhjbhp2mt3h6@dhcp-10-154-148-175.vpn.oracle.com/\n\nThis series fixes this problem by carefully replacing the start of each\nrefspec's destination with \"refs/prefetch/\". If the destination already\nstarts with \"refs/\", then that is replaced. Otherwise \"refs/prefetch/\" is\njust prepended.\n\nThis happens inside of git fetch when a --prefetch option is given. This\nallows us to maniuplate a struct refspec_item instead of a full refspec\nstring. It also simplifies our logic in testing the prefetch task.\n\n\nUpdates in V4\n=============\n\n * Two bugs were fixed. Thanks, Ramsay and Tom, for pointing out the issues.\n Tests are added that prevent regressions.\n * A new patch is added to respect remote..skipFetchAll. This is added\n at the end to take advantage of the simpler test design after --prefetch\n is added.\n\n\nUpdate in V3\n============\n\n * The fix is almost completely rewritten as an update to 'git fetch'. See\n the new PATCH 2 for this update.\n\n * There was some discussion of rewriting test_subcommand, but that can be\n delayed until a proper solution is found to complications around softer\n matches.\n\n\nUpdates in V2\n=============\n\nThanks for the close eye on this series. I appreciate the recommendations,\nwhich I believe I have responded to them all:\n\n * Fixed typos.\n * Made refspec_item_format() re-entrant. Consumers must free the buffer.\n * Cleaned up style (quoting and tabbing).\n\nThanks, -Stolee\n\nDerrick Stolee (4):\n maintenance: simplify prefetch logic\n fetch: add --prefetch option\n maintenance: use 'git fetch --prefetch'\n maintenance: respect remote.*.skipFetchAll\n\n Documentation/fetch-options.txt | 5 +++\n Documentation/git-maintenance.txt | 6 ++--\n builtin/fetch.c | 59 ++++++++++++++++++++++++++++++-\n builtin/gc.c | 39 +++++++-------------\n t/t5582-fetch-negative-refspec.sh | 43 ++++++++++++++++++++++\n t/t7900-maintenance.sh | 22 +++++++-----\n 6 files changed, 134 insertions(+), 40 deletions(-)\n\n\nbase-commit: 89b43f80a514aee58b662ad606e6352e03eaeee4\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-924%2Fderrickstolee%2Fmaintenance%2Frefspec-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-924/derrickstolee/maintenance/refspec-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/924\n\nRange-diff vs v3:\n\n 1: 4c0e983ba56f = 1: 4c0e983ba56f maintenance: simplify prefetch logic\n 2: 7f488eea6dbd ! 2: 73b4e8496746 fetch: add --prefetch option\n @@ Commit message\n Finally, we add the 'force' option to ensure that prefetch refs are\n replaced as necessary.\n \n + There are some interesting cases that are worth testing.\n +\n + An earlier version of this change dropped the \"i--\" from the loop that\n + deletes a refspec item and shifts the remaining entries down. This\n + allowed some refspecs to not be modified. The subtle part about the\n + first --prefetch test is that the \"refs/tags/*\" refspec appears directly\n + before the \"refs/heads/bogus/*\" refspec. Without that \"i--\", this\n + ordering would remove the \"refs/tags/*\" refspec and leave the last one\n + unmodified, placing the result in \"refs/heads/*\".\n +\n + It is possible to have an empty refspec. This is typically the case for\n + remotes other than the origin, where users want to fetch a specific tag\n + or branch. To correctly test this case, we need to further remove the\n + upstream remote for the local branch. Thus, we are testing a refspec\n + that will be deleted, leaving nothing to fetch.\n +\n + Helped-by: Tom Saeger \n + Helped-by: Ramsay Jones \n Signed-off-by: Derrick Stolee \n \n ## Documentation/fetch-options.txt ##\n @@ builtin/fetch.c: static void find_non_local_tags(const struct ref *refs,\n +\t\t\t\trs->raw[j - 1] = rs->raw[j];\n +\t\t\t}\n +\t\t\trs->nr--;\n ++\t\t\ti--;\n +\t\t\tcontinue;\n +\t\t}\n +\n @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,\n \tif (rs->nr) {\n \t\tstruct refspec *fetch_refspec;\n \n +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,\n + \t\t\tif (has_merge &&\n + \t\t\t !strcmp(branch->remote_name, remote->name))\n + \t\t\t\tadd_merge_config(&ref_map, remote_refs, branch, &tail);\n +-\t\t} else {\n ++\t\t} else if (!prefetch) {\n + \t\t\tref_map = get_remote_ref(remote_refs, \"HEAD\");\n + \t\t\tif (!ref_map)\n + \t\t\t\tdie(_(\"Couldn't find remote ref HEAD\"));\n \n ## t/t5582-fetch-negative-refspec.sh ##\n @@ t/t5582-fetch-negative-refspec.sh: test_expect_success \"push with matching +: and negative refspec\" '\n @@ t/t5582-fetch-negative-refspec.sh: test_expect_success \"push with matching +: an\n \n +test_expect_success '--prefetch correctly modifies refspecs' '\n +\tgit -C one config --unset-all remote.origin.fetch &&\n -+\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n +\tgit -C one config --add remote.origin.fetch ^refs/heads/bogus/ignore &&\n ++\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n +\tgit -C one config --add remote.origin.fetch \"refs/heads/bogus/*:bogus/*\" &&\n +\n +\tgit tag -a -m never never-fetch-tag HEAD &&\n @@ t/t5582-fetch-negative-refspec.sh: test_expect_success \"push with matching +: an\n +\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n +\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore\n +'\n ++\n ++test_expect_success '--prefetch succeeds when refspec becomes empty' '\n ++\tgit checkout bogus/fetched &&\n ++\ttest_commit extra &&\n ++\n ++\tgit -C one config --unset-all remote.origin.fetch &&\n ++\tgit -C one config --unset branch.main.remote &&\n ++\tgit -C one config remote.origin.fetch \"+refs/tags/extra\" &&\n ++\tgit -C one config remote.origin.skipfetchall true &&\n ++\tgit -C one config remote.origin.tagopt \"--no-tags\" &&\n ++\n ++\tgit -C one fetch --prefetch\n ++'\n +\n test_done\n 3: ed055d772452 = 3: 565ed8a18929 maintenance: use 'git fetch --prefetch'\n -: ------------ > 4: 92652fd9e6e1 maintenance: respect remote.*.skipFetchAll\n\n-- \ngitgitgadget\n"},{"id":"422091","messageId":"73b4e8496746a3d1387e568a5a898c02ec81df76.1618577399.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v4.git.1618577399.gitgitgadget@gmail.com","subject":"[PATCH v4 2/4] fetch: add --prefetch option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-16T12:49:57Z","receivedAt":"2021-04-16T12:50:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --prefetch option will be used by the 'prefetch' maintenance task\ninstead of sending refspecs explicitly across the command-line. The\nintention is to modify the refspec to place all results in\nrefs/prefetch/ instead of anywhere else.\n\nCreate helper method filter_prefetch_refspec() to modify a given refspec\nto fit the rules expected of the prefetch task:\n\n * Negative refspecs are preserved.\n * Refspecs without a destination are removed.\n * Refspecs whose source starts with \"refs/tags/\" are removed.\n * Other refspecs are placed within \"refs/prefetch/\".\n\nFinally, we add the 'force' option to ensure that prefetch refs are\nreplaced as necessary.\n\nThere are some interesting cases that are worth testing.\n\nAn earlier version of this change dropped the \"i--\" from the loop that\ndeletes a refspec item and shifts the remaining entries down. This\nallowed some refspecs to not be modified. The subtle part about the\nfirst --prefetch test is that the \"refs/tags/*\" refspec appears directly\nbefore the \"refs/heads/bogus/*\" refspec. Without that \"i--\", this\nordering would remove the \"refs/tags/*\" refspec and leave the last one\nunmodified, placing the result in \"refs/heads/*\".\n\nIt is possible to have an empty refspec. This is typically the case for\nremotes other than the origin, where users want to fetch a specific tag\nor branch. To correctly test this case, we need to further remove the\nupstream remote for the local branch. Thus, we are testing a refspec\nthat will be deleted, leaving nothing to fetch.\n\nHelped-by: Tom Saeger \nHelped-by: Ramsay Jones \nSigned-off-by: Derrick Stolee \n---\n Documentation/fetch-options.txt | 5 +++\n builtin/fetch.c | 59 ++++++++++++++++++++++++++++++-\n t/t5582-fetch-negative-refspec.sh | 43 ++++++++++++++++++++++\n 3 files changed, 106 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 07783deee309..9e7b4e189ce0 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -110,6 +110,11 @@ ifndef::git-pull[]\n \tsetting `fetch.writeCommitGraph`.\n endif::git-pull[]\n \n+--prefetch::\n+\tModify the configured refspec to place all refs into the\n+\t`refs/prefetch/` namespace. See the `prefetch` task in\n+\tlinkgit:git-maintenance[1].\n+\n -p::\n --prune::\n \tBefore fetching, remove any remote-tracking references that no\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 0b90de87c7a2..97c4fe6e6d66 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -48,6 +48,7 @@ enum {\n static int fetch_prune_config = -1; /* unspecified */\n static int fetch_show_forced_updates = 1;\n static uint64_t forced_updates_ms = 0;\n+static int prefetch = 0;\n static int prune = -1; /* unspecified */\n #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */\n \n@@ -158,6 +159,8 @@ static struct option builtin_fetch_options[] = {\n \t\t N_(\"do not fetch all tags (--no-tags)\"), TAGS_UNSET),\n \tOPT_INTEGER('j', \"jobs\", &max_jobs,\n \t\t N_(\"number of submodules fetched in parallel\")),\n+\tOPT_BOOL(0, \"prefetch\", &prefetch,\n+\t\t N_(\"modify the refspec to place all refs within refs/prefetch/\")),\n \tOPT_BOOL('p', \"prune\", &prune,\n \t\t N_(\"prune remote-tracking branches no longer on remote\")),\n \tOPT_BOOL('P', \"prune-tags\", &prune_tags,\n@@ -436,6 +439,56 @@ static void find_non_local_tags(const struct ref *refs,\n \toidset_clear(&fetch_oids);\n }\n \n+static void filter_prefetch_refspec(struct refspec *rs)\n+{\n+\tint i;\n+\n+\tif (!prefetch)\n+\t\treturn;\n+\n+\tfor (i = 0; i < rs->nr; i++) {\n+\t\tstruct strbuf new_dst = STRBUF_INIT;\n+\t\tchar *old_dst;\n+\t\tconst char *sub = NULL;\n+\n+\t\tif (rs->items[i].negative)\n+\t\t\tcontinue;\n+\t\tif (!rs->items[i].dst ||\n+\t\t (rs->items[i].src &&\n+\t\t !strncmp(rs->items[i].src, \"refs/tags/\", 10))) {\n+\t\t\tint j;\n+\n+\t\t\tfree(rs->items[i].src);\n+\t\t\tfree(rs->items[i].dst);\n+\n+\t\t\tfor (j = i + 1; j < rs->nr; j++) {\n+\t\t\t\trs->items[j - 1] = rs->items[j];\n+\t\t\t\trs->raw[j - 1] = rs->raw[j];\n+\t\t\t}\n+\t\t\trs->nr--;\n+\t\t\ti--;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\told_dst = rs->items[i].dst;\n+\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n+\n+\t\t/*\n+\t\t * If old_dst starts with \"refs/\", then place\n+\t\t * sub after that prefix. Otherwise, start at\n+\t\t * the beginning of the string.\n+\t\t */\n+\t\tif (!skip_prefix(old_dst, \"refs/\", &sub))\n+\t\t\tsub = old_dst;\n+\t\tstrbuf_addstr(&new_dst, sub);\n+\n+\t\trs->items[i].dst = strbuf_detach(&new_dst, NULL);\n+\t\trs->items[i].force = 1;\n+\n+\t\tfree(old_dst);\n+\t}\n+}\n+\n static struct ref *get_ref_map(struct remote *remote,\n \t\t\t const struct ref *remote_refs,\n \t\t\t struct refspec *rs,\n@@ -452,6 +505,10 @@ static struct ref *get_ref_map(struct remote *remote,\n \tstruct hashmap existing_refs;\n \tint existing_refs_populated = 0;\n \n+\tfilter_prefetch_refspec(rs);\n+\tif (remote)\n+\t\tfilter_prefetch_refspec(&remote->fetch);\n+\n \tif (rs->nr) {\n \t\tstruct refspec *fetch_refspec;\n \n@@ -520,7 +577,7 @@ static struct ref *get_ref_map(struct remote *remote,\n \t\t\tif (has_merge &&\n \t\t\t !strcmp(branch->remote_name, remote->name))\n \t\t\t\tadd_merge_config(&ref_map, remote_refs, branch, &tail);\n-\t\t} else {\n+\t\t} else if (!prefetch) {\n \t\t\tref_map = get_remote_ref(remote_refs, \"HEAD\");\n \t\t\tif (!ref_map)\n \t\t\t\tdie(_(\"Couldn't find remote ref HEAD\"));\ndiff --git a/t/t5582-fetch-negative-refspec.sh b/t/t5582-fetch-negative-refspec.sh\nindex f34509727702..e5d2e79ad382 100755\n--- a/t/t5582-fetch-negative-refspec.sh\n+++ b/t/t5582-fetch-negative-refspec.sh\n@@ -240,4 +240,47 @@ test_expect_success \"push with matching +: and negative refspec\" '\n \tgit -C two push -v one\n '\n \n+test_expect_success '--prefetch correctly modifies refspecs' '\n+\tgit -C one config --unset-all remote.origin.fetch &&\n+\tgit -C one config --add remote.origin.fetch ^refs/heads/bogus/ignore &&\n+\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n+\tgit -C one config --add remote.origin.fetch \"refs/heads/bogus/*:bogus/*\" &&\n+\n+\tgit tag -a -m never never-fetch-tag HEAD &&\n+\n+\tgit branch bogus/fetched HEAD~1 &&\n+\tgit branch bogus/ignore HEAD &&\n+\n+\tgit -C one fetch --prefetch --no-tags &&\n+\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n+\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n+\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore &&\n+\n+\t# correctly handle when refspec set becomes empty\n+\t# after removing the refs/tags/* refspec.\n+\tgit -C one config --unset-all remote.origin.fetch &&\n+\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n+\n+\tgit -C one fetch --prefetch --no-tags &&\n+\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n+\n+\t# The refspec for refs that are not fully qualified\n+\t# are filtered multiple times.\n+\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n+\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore\n+'\n+\n+test_expect_success '--prefetch succeeds when refspec becomes empty' '\n+\tgit checkout bogus/fetched &&\n+\ttest_commit extra &&\n+\n+\tgit -C one config --unset-all remote.origin.fetch &&\n+\tgit -C one config --unset branch.main.remote &&\n+\tgit -C one config remote.origin.fetch \"+refs/tags/extra\" &&\n+\tgit -C one config remote.origin.skipfetchall true &&\n+\tgit -C one config remote.origin.tagopt \"--no-tags\" &&\n+\n+\tgit -C one fetch --prefetch\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"422092","messageId":"565ed8a1892946a75c1116b0f351725e64fa90ba.1618577399.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v4.git.1618577399.gitgitgadget@gmail.com","subject":"[PATCH v4 3/4] maintenance: use 'git fetch --prefetch'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-16T12:49:58Z","receivedAt":"2021-04-16T12:50:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'prefetch' maintenance task previously forced the following refspec\nfor each remote:\n\n\t+refs/heads/*:refs/prefetch//*\n\nIf a user has specified a more strict refspec for the remote, then this\nprefetch task downloads more objects than necessary.\n\nThe previous change introduced the '--prefetch' option to 'git fetch'\nwhich manipulates the remote's refspec to place all resulting refs into\nrefs/prefetch/, with further partitioning based on the destinations of\nthose refspecs.\n\nUpdate the documentation to be more generic about the destination refs.\nDo not mention custom refspecs explicitly, as that does not need to be\nhighlighted in this documentation. The important part of placing refs in\nrefs/prefetch/ remains.\n\nReported-by: Tom Saeger \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 6 ++----\n builtin/gc.c | 7 +++----\n t/t7900-maintenance.sh | 14 +++++++-------\n 3 files changed, 12 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 80ddd33ceba0..1e738ad39832 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -92,10 +92,8 @@ commit-graph::\n prefetch::\n \tThe `prefetch` task updates the object directory with the latest\n \tobjects from all registered remotes. For each remote, a `git fetch`\n-\tcommand is run. The refmap is custom to avoid updating local or remote\n-\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n-\tremote refs are stored in `refs/prefetch//`. Also, tags are\n-\tnot updated.\n+\tcommand is run. The configured refspec is modified to place all\n+\trequested refs within `refs/prefetch/`. Also, tags are not updated.\n +\n This is done to avoid disrupting the remote-tracking branches. The end users\n expect these refs to stay unmoved unless they initiate a fetch. With prefetch\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex fa8128de9ae1..9d35f7da50d8 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -879,15 +879,14 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n-\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n+\tstrvec_pushl(&child.args, \"fetch\", remote->name,\n+\t\t \"--prefetch\", \"--prune\", \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n-\t\t \"--refmap=\", NULL);\n+\t\t NULL);\n \n \tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n-\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n-\n \treturn !!run_command(&child);\n }\n \ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 2412d8c5c006..eadb800c08cc 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -141,15 +141,15 @@ test_expect_success 'prefetch multiple remotes' '\n \ttest_commit -C clone1 one &&\n \ttest_commit -C clone2 two &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n-\tfetchargs=\"--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet\" &&\n-\ttest_subcommand git fetch remote1 $fetchargs +refs/heads/\\\\*:refs/prefetch/remote1/\\\\* log &&\n-- \ngitgitgadget\n\n"},{"id":"422093","messageId":"92652fd9e6e17654abdb30625c85937b6b56c38e.1618577399.git.gitgitgadget@gmail.com","threadId":"55434","inReplyTo":"pull.924.v4.git.1618577399.gitgitgadget@gmail.com","subject":"[PATCH v4 4/4] maintenance: respect remote.*.skipFetchAll","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-16T12:49:59Z","receivedAt":"2021-04-16T12:50:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a remote has the skipFetchAll setting enabled, then that remote is\nnot intended for frequent fetching. It makes sense to not fetch that\ndata during the 'prefetch' maintenance task. Skip that remote in the\niteration without error. The skip_default_update member is initialized\nin remote.c:handle_config() as part of initializing the 'struct remote'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 3 +++\n t/t7900-maintenance.sh | 8 +++++++-\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9d35f7da50d8..98a803196b88 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -878,6 +878,9 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \tstruct maintenance_run_opts *opts = cbdata;\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n+\tif (remote->skip_default_update)\n+\t\treturn 0;\n+\n \tchild.git_cmd = 1;\n \tstrvec_pushl(&child.args, \"fetch\", remote->name,\n \t\t \"--prefetch\", \"--prune\", \"--no-tags\",\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex eadb800c08cc..b93ae014ee58 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -153,7 +153,13 @@ test_expect_success 'prefetch multiple remotes' '\n \n \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n \tgit log --oneline --decorate --all >log &&\n-\t! grep \"prefetch\" log\n+\t! grep \"prefetch\" log &&\n+\n+\ttest_when_finished git config --unset remote.remote1.skipFetchAll &&\n+\tgit config remote.remote1.skipFetchAll true &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/skip-remote1.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n+\ttest_subcommand ! git fetch remote1 $fetchargs From: Derrick Stolee \n>\n> If a remote has the skipFetchAll setting enabled, then that remote is\n> not intended for frequent fetching. It makes sense to not fetch that\n> data during the 'prefetch' maintenance task. Skip that remote in the\n> iteration without error. The skip_default_update member is initialized\n> in remote.c:handle_config() as part of initializing the 'struct remote'.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/gc.c | 3 +++\n> t/t7900-maintenance.sh | 8 +++++++-\n> 2 files changed, 10 insertions(+), 1 deletion(-)\n>\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 9d35f7da50d8..98a803196b88 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -878,6 +878,9 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> \tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n> \n> +\tif (remote->skip_default_update)\n> +\t\treturn 0;\n> +\n> \tchild.git_cmd = 1;\n> \tstrvec_pushl(&child.args, \"fetch\", remote->name,\n> \t\t \"--prefetch\", \"--prune\", \"--no-tags\",\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index eadb800c08cc..b93ae014ee58 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -153,7 +153,13 @@ test_expect_success 'prefetch multiple remotes' '\n> \n> \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n> \tgit log --oneline --decorate --all >log &&\n> -\t! grep \"prefetch\" log\n> +\t! grep \"prefetch\" log &&\n> +\n> +\ttest_when_finished git config --unset remote.remote1.skipFetchAll &&\n> +\tgit config remote.remote1.skipFetchAll true &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/skip-remote1.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> +\ttest_subcommand ! git fetch remote1 $fetchargs +\ttest_subcommand git fetch remote2 $fetchargs '\n> \n> test_expect_success 'prefetch and existing log.excludeDecoration values' '\n\nWithout having read the code I'd have very much expected a\n\"remote.*.skipFetchAll\" to impact:\n\n git fetch --all\n\nOr:\n\n git remote update --all # --all does not exist yet\n\nAs e.g. remote..skipDefaultUpdate would do (i.e. impact \"git\nremote update\" ...).\n\nI suspect naming it like this started as a hack around the lack of\n4-level .ini config keys, i.e. so we could do:\n\n maintenance.remote..skipFetchAll = true\n\nBut I wonder if we couldn't give this a less confusing name still, even\nthe pseudo-command form of:\n\n maintenanceRemote..skipFetchAll = true\n\nOr something...\n\n"},{"id":"422098","messageId":"20210416143317.hoswhvk7ivdg6z7v@dhcp-10-154-163-120.vpn.oracle.com","threadId":"55434","inReplyTo":"87zgxytjwa.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 4/4] maintenance: respect remote.*.skipFetchAll","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-16T14:33:17Z","receivedAt":"2021-04-16T14:33:53Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Fri, Apr 16, 2021 at 03:54:13PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Apr 16 2021, Derrick Stolee via GitGitGadget wrote:\n> \n> > From: Derrick Stolee \n> >\n> > If a remote has the skipFetchAll setting enabled, then that remote is\n> > not intended for frequent fetching. It makes sense to not fetch that\n> > data during the 'prefetch' maintenance task. Skip that remote in the\n> > iteration without error. The skip_default_update member is initialized\n> > in remote.c:handle_config() as part of initializing the 'struct remote'.\n> >\n> > Signed-off-by: Derrick Stolee \n> > ---\n> > builtin/gc.c | 3 +++\n> > t/t7900-maintenance.sh | 8 +++++++-\n> > 2 files changed, 10 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/builtin/gc.c b/builtin/gc.c\n> > index 9d35f7da50d8..98a803196b88 100644\n> > --- a/builtin/gc.c\n> > +++ b/builtin/gc.c\n> > @@ -878,6 +878,9 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> > \tstruct maintenance_run_opts *opts = cbdata;\n> > \tstruct child_process child = CHILD_PROCESS_INIT;\n> > \n> > +\tif (remote->skip_default_update)\n> > +\t\treturn 0;\n> > +\n> > \tchild.git_cmd = 1;\n> > \tstrvec_pushl(&child.args, \"fetch\", remote->name,\n> > \t\t \"--prefetch\", \"--prune\", \"--no-tags\",\n> > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> > index eadb800c08cc..b93ae014ee58 100755\n> > --- a/t/t7900-maintenance.sh\n> > +++ b/t/t7900-maintenance.sh\n> > @@ -153,7 +153,13 @@ test_expect_success 'prefetch multiple remotes' '\n> > \n> > \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n> > \tgit log --oneline --decorate --all >log &&\n> > -\t! grep \"prefetch\" log\n> > +\t! grep \"prefetch\" log &&\n> > +\n> > +\ttest_when_finished git config --unset remote.remote1.skipFetchAll &&\n> > +\tgit config remote.remote1.skipFetchAll true &&\n> > +\tGIT_TRACE2_EVENT=\"$(pwd)/skip-remote1.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> > +\ttest_subcommand ! git fetch remote1 $fetchargs > +\ttest_subcommand git fetch remote2 $fetchargs > '\n> > \n> > test_expect_success 'prefetch and existing log.excludeDecoration values' '\n> \n> Without having read the code I'd have very much expected a\n> \"remote.*.skipFetchAll\" to impact:\n> \n> git fetch --all\n\n'skipFetchAll' indeed impacts 'git fetch --all'\n\nBut this patch doesn't add \"skipFetchAll\", instead it just honors that\nconfig if set during 'git maintenance --task=prefetch'\n\nSee v3 discussion: https://lore.kernel.org/git/2f4fa2b5-0d8b-b368-ab4d-411740595a4f@gmail.com/\n\n> \n> Or:\n> \n> git remote update --all # --all does not exist yet\n> \n> As e.g. remote..skipDefaultUpdate would do (i.e. impact \"git\n> remote update\" ...).\n> \n> I suspect naming it like this started as a hack around the lack of\n> 4-level .ini config keys, i.e. so we could do:\n> \n> maintenance.remote..skipFetchAll = true\n> \n> But I wonder if we couldn't give this a less confusing name still, even\n> the pseudo-command form of:\n> \n> maintenanceRemote..skipFetchAll = true\n> \n> Or something...\n> \n\nWhether or not additional maintenance specific configs are desired, that might be\nsomething to consider. I've thought about this for a few of my repos\nwhich have remotes which require a VPN connection. Perhaps\nI want to skip those during 'prefetch'? Or maybe instead define a\n'remotes.prefetch' group and only prefetch remotes listed?\n\nThat all said - I wouldn't hold-up this patch series for it.\n\n--Tom\n"},{"id":"422105","messageId":"20210416175214.lysy3amsb7urj3ix@tsaeger.oracle.com","threadId":"55434","inReplyTo":"73b4e8496746a3d1387e568a5a898c02ec81df76.1618577399.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/4] fetch: add --prefetch option","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-16T17:52:14Z","receivedAt":"2021-04-16T17:52:51Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Fri, Apr 16, 2021 at 12:49:57PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The --prefetch option will be used by the 'prefetch' maintenance task\n> instead of sending refspecs explicitly across the command-line. The\n> intention is to modify the refspec to place all results in\n> refs/prefetch/ instead of anywhere else.\n> \n> Create helper method filter_prefetch_refspec() to modify a given refspec\n> to fit the rules expected of the prefetch task:\n> \n> * Negative refspecs are preserved.\n> * Refspecs without a destination are removed.\n> * Refspecs whose source starts with \"refs/tags/\" are removed.\n> * Other refspecs are placed within \"refs/prefetch/\".\n> \n> Finally, we add the 'force' option to ensure that prefetch refs are\n> replaced as necessary.\n> \n> There are some interesting cases that are worth testing.\n> \n> An earlier version of this change dropped the \"i--\" from the loop that\n> deletes a refspec item and shifts the remaining entries down. This\n> allowed some refspecs to not be modified. The subtle part about the\n> first --prefetch test is that the \"refs/tags/*\" refspec appears directly\n> before the \"refs/heads/bogus/*\" refspec. Without that \"i--\", this\n> ordering would remove the \"refs/tags/*\" refspec and leave the last one\n> unmodified, placing the result in \"refs/heads/*\".\n> \n> It is possible to have an empty refspec. This is typically the case for\n> remotes other than the origin, where users want to fetch a specific tag\n> or branch. To correctly test this case, we need to further remove the\n> upstream remote for the local branch. Thus, we are testing a refspec\n> that will be deleted, leaving nothing to fetch.\n> \n> Helped-by: Tom Saeger \n> Helped-by: Ramsay Jones \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/fetch-options.txt | 5 +++\n> builtin/fetch.c | 59 ++++++++++++++++++++++++++++++-\n> t/t5582-fetch-negative-refspec.sh | 43 ++++++++++++++++++++++\n> 3 files changed, 106 insertions(+), 1 deletion(-)\n> \n> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> index 07783deee309..9e7b4e189ce0 100644\n> --- a/Documentation/fetch-options.txt\n> +++ b/Documentation/fetch-options.txt\n> @@ -110,6 +110,11 @@ ifndef::git-pull[]\n> \tsetting `fetch.writeCommitGraph`.\n> endif::git-pull[]\n> \n> +--prefetch::\n> +\tModify the configured refspec to place all refs into the\n> +\t`refs/prefetch/` namespace. See the `prefetch` task in\n> +\tlinkgit:git-maintenance[1].\n> +\n> -p::\n> --prune::\n> \tBefore fetching, remove any remote-tracking references that no\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 0b90de87c7a2..97c4fe6e6d66 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -48,6 +48,7 @@ enum {\n> static int fetch_prune_config = -1; /* unspecified */\n> static int fetch_show_forced_updates = 1;\n> static uint64_t forced_updates_ms = 0;\n> +static int prefetch = 0;\n> static int prune = -1; /* unspecified */\n> #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */\n> \n> @@ -158,6 +159,8 @@ static struct option builtin_fetch_options[] = {\n> \t\t N_(\"do not fetch all tags (--no-tags)\"), TAGS_UNSET),\n> \tOPT_INTEGER('j', \"jobs\", &max_jobs,\n> \t\t N_(\"number of submodules fetched in parallel\")),\n> +\tOPT_BOOL(0, \"prefetch\", &prefetch,\n> +\t\t N_(\"modify the refspec to place all refs within refs/prefetch/\")),\n> \tOPT_BOOL('p', \"prune\", &prune,\n> \t\t N_(\"prune remote-tracking branches no longer on remote\")),\n> \tOPT_BOOL('P', \"prune-tags\", &prune_tags,\n> @@ -436,6 +439,56 @@ static void find_non_local_tags(const struct ref *refs,\n> \toidset_clear(&fetch_oids);\n> }\n> \n> +static void filter_prefetch_refspec(struct refspec *rs)\n> +{\n> +\tint i;\n> +\n> +\tif (!prefetch)\n> +\t\treturn;\n> +\n> +\tfor (i = 0; i < rs->nr; i++) {\n> +\t\tstruct strbuf new_dst = STRBUF_INIT;\n> +\t\tchar *old_dst;\n> +\t\tconst char *sub = NULL;\n> +\n> +\t\tif (rs->items[i].negative)\n> +\t\t\tcontinue;\n> +\t\tif (!rs->items[i].dst ||\n> +\t\t (rs->items[i].src &&\n> +\t\t !strncmp(rs->items[i].src, \"refs/tags/\", 10))) {\n> +\t\t\tint j;\n> +\n> +\t\t\tfree(rs->items[i].src);\n> +\t\t\tfree(rs->items[i].dst);\n> +\n> +\t\t\tfor (j = i + 1; j < rs->nr; j++) {\n> +\t\t\t\trs->items[j - 1] = rs->items[j];\n> +\t\t\t\trs->raw[j - 1] = rs->raw[j];\n> +\t\t\t}\n> +\t\t\trs->nr--;\n> +\t\t\ti--;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\told_dst = rs->items[i].dst;\n> +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> +\n> +\t\t/*\n> +\t\t * If old_dst starts with \"refs/\", then place\n> +\t\t * sub after that prefix. Otherwise, start at\n> +\t\t * the beginning of the string.\n> +\t\t */\n> +\t\tif (!skip_prefix(old_dst, \"refs/\", &sub))\n> +\t\t\tsub = old_dst;\n> +\t\tstrbuf_addstr(&new_dst, sub);\n> +\n> +\t\trs->items[i].dst = strbuf_detach(&new_dst, NULL);\n> +\t\trs->items[i].force = 1;\n> +\n> +\t\tfree(old_dst);\n> +\t}\n> +}\n> +\n> static struct ref *get_ref_map(struct remote *remote,\n> \t\t\t const struct ref *remote_refs,\n> \t\t\t struct refspec *rs,\n> @@ -452,6 +505,10 @@ static struct ref *get_ref_map(struct remote *remote,\n> \tstruct hashmap existing_refs;\n> \tint existing_refs_populated = 0;\n> \n> +\tfilter_prefetch_refspec(rs);\n> +\tif (remote)\n> +\t\tfilter_prefetch_refspec(&remote->fetch);\n> +\n> \tif (rs->nr) {\n> \t\tstruct refspec *fetch_refspec;\n> \n> @@ -520,7 +577,7 @@ static struct ref *get_ref_map(struct remote *remote,\n> \t\t\tif (has_merge &&\n> \t\t\t !strcmp(branch->remote_name, remote->name))\n> \t\t\t\tadd_merge_config(&ref_map, remote_refs, branch, &tail);\n> -\t\t} else {\n> +\t\t} else if (!prefetch) {\n\nThat works for me.\n\n> \t\t\tref_map = get_remote_ref(remote_refs, \"HEAD\");\n> \t\t\tif (!ref_map)\n> \t\t\t\tdie(_(\"Couldn't find remote ref HEAD\"));\n> diff --git a/t/t5582-fetch-negative-refspec.sh b/t/t5582-fetch-negative-refspec.sh\n> index f34509727702..e5d2e79ad382 100755\n> --- a/t/t5582-fetch-negative-refspec.sh\n> +++ b/t/t5582-fetch-negative-refspec.sh\n> @@ -240,4 +240,47 @@ test_expect_success \"push with matching +: and negative refspec\" '\n> \tgit -C two push -v one\n> '\n> \n> +test_expect_success '--prefetch correctly modifies refspecs' '\n> +\tgit -C one config --unset-all remote.origin.fetch &&\n> +\tgit -C one config --add remote.origin.fetch ^refs/heads/bogus/ignore &&\n> +\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n> +\tgit -C one config --add remote.origin.fetch \"refs/heads/bogus/*:bogus/*\" &&\n> +\n> +\tgit tag -a -m never never-fetch-tag HEAD &&\n> +\n> +\tgit branch bogus/fetched HEAD~1 &&\n> +\tgit branch bogus/ignore HEAD &&\n> +\n> +\tgit -C one fetch --prefetch --no-tags &&\n> +\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n> +\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n> +\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore &&\n> +\n> +\t# correctly handle when refspec set becomes empty\n> +\t# after removing the refs/tags/* refspec.\n> +\tgit -C one config --unset-all remote.origin.fetch &&\n> +\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n> +\n> +\tgit -C one fetch --prefetch --no-tags &&\n> +\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n> +\n> +\t# The refspec for refs that are not fully qualified\n> +\t# are filtered multiple times.\n> +\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n> +\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore\n> +'\n> +\n> +test_expect_success '--prefetch succeeds when refspec becomes empty' '\n\ntechnically this will get skipped based only on \"skipfetchall\" right?\n\nThe remote could have an empty-set of refspecs or multiple\nvalid refspecs post filter_prefetch_refspec, but the remote gets skipped altogether.\n\nperhaps '--prefetch succeeds when remote.skipfetchall is true' '\n\nanyway this is looking pretty solid\n\nReviewed-by: Tom Saeger \n\n> +\tgit checkout bogus/fetched &&\n> +\ttest_commit extra &&\n> +\n> +\tgit -C one config --unset-all remote.origin.fetch &&\n> +\tgit -C one config --unset branch.main.remote &&\n> +\tgit -C one config remote.origin.fetch \"+refs/tags/extra\" &&\n> +\tgit -C one config remote.origin.skipfetchall true &&\n> +\tgit -C one config remote.origin.tagopt \"--no-tags\" &&\n> +\n> +\tgit -C one fetch --prefetch\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"422106","messageId":"20210416180241.sxcbs2fg52r7hmjz@tsaeger.oracle.com","threadId":"55434","inReplyTo":"4c0e983ba56f030851370ef71887117db45e8479.1618577399.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/4] maintenance: simplify prefetch logic","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-16T18:02:41Z","receivedAt":"2021-04-16T18:03:08Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Fri, Apr 16, 2021 at 12:49:56PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The previous logic filled a string list with the names of each remote,\n> but instead we could simply run the appropriate 'git fetch' data\n> directly in the remote iterator. Do this for reduced code size, but also\n> because it sets up an upcoming change to use the remote's refspec. This\n> data is accessible from the 'struct remote' data that is now accessible\n> in fetch_remote().\n> \n> Signed-off-by: Derrick Stolee \n\nReviewed-by: Tom Saeger \n\n> ---\n> builtin/gc.c | 33 ++++++++-------------------------\n> 1 file changed, 8 insertions(+), 25 deletions(-)\n> \n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index ef7226d7bca4..fa8128de9ae1 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n> \treturn 0;\n> }\n> \n> -static int fetch_remote(const char *remote, struct maintenance_run_opts *opts)\n> +static int fetch_remote(struct remote *remote, void *cbdata)\n> {\n> +\tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n> \n> \tchild.git_cmd = 1;\n> -\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n> +\tstrvec_pushl(&child.args, \"fetch\", remote->name, \"--prune\", \"--no-tags\",\n> \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n> \t\t \"--refmap=\", NULL);\n> \n> \tif (opts->quiet)\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \n> -\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n> +\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote->name);\n> \n> \treturn !!run_command(&child);\n> }\n> \n> -static int append_remote(struct remote *remote, void *cbdata)\n> -{\n> -\tstruct string_list *remotes = (struct string_list *)cbdata;\n> -\n> -\tstring_list_append(remotes, remote->name);\n> -\treturn 0;\n> -}\n> -\n> static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n> {\n> -\tint result = 0;\n> -\tstruct string_list_item *item;\n> -\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n> -\n> \tgit_config_set_multivar_gently(\"log.excludedecoration\",\n> \t\t\t\t\t\"refs/prefetch/\",\n> \t\t\t\t\t\"refs/prefetch/\",\n> \t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n> \t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n> \n> -\tif (for_each_remote(append_remote, &remotes)) {\n> -\t\terror(_(\"failed to fill remotes\"));\n> -\t\tresult = 1;\n> -\t\tgoto cleanup;\n> +\tif (for_each_remote(fetch_remote, opts)) {\n> +\t\terror(_(\"failed to prefetch remotes\"));\n> +\t\treturn 1;\n> \t}\n> \n> -\tfor_each_string_list_item(item, &remotes)\n> -\t\tresult |= fetch_remote(item->string, opts);\n> -\n> -cleanup:\n> -\tstring_list_clear(&remotes, 0);\n> -\treturn result;\n> +\treturn 0;\n> }\n> \n> static int maintenance_task_gc(struct maintenance_run_opts *opts)\n> -- \n> gitgitgadget\n> \n"},{"id":"422107","messageId":"20210416182635.wldgozqybdp7y5hw@tsaeger.oracle.com","threadId":"55434","inReplyTo":"20210416175214.lysy3amsb7urj3ix@tsaeger.oracle.com","subject":"Re: [PATCH v4 2/4] fetch: add --prefetch option","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-16T18:26:35Z","receivedAt":"2021-04-16T18:27:03Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Fri, Apr 16, 2021 at 12:52:21PM -0500, Tom Saeger wrote:\n> On Fri, Apr 16, 2021 at 12:49:57PM +0000, Derrick Stolee via GitGitGadget wrote:\n> > From: Derrick Stolee \n> > \n> > The --prefetch option will be used by the 'prefetch' maintenance task\n> > instead of sending refspecs explicitly across the command-line. The\n> > intention is to modify the refspec to place all results in\n> > refs/prefetch/ instead of anywhere else.\n> > \n> > Create helper method filter_prefetch_refspec() to modify a given refspec\n> > to fit the rules expected of the prefetch task:\n> > \n> > * Negative refspecs are preserved.\n> > * Refspecs without a destination are removed.\n> > * Refspecs whose source starts with \"refs/tags/\" are removed.\n> > * Other refspecs are placed within \"refs/prefetch/\".\n> > \n> > Finally, we add the 'force' option to ensure that prefetch refs are\n> > replaced as necessary.\n> > \n> > There are some interesting cases that are worth testing.\n> > \n> > An earlier version of this change dropped the \"i--\" from the loop that\n> > deletes a refspec item and shifts the remaining entries down. This\n> > allowed some refspecs to not be modified. The subtle part about the\n> > first --prefetch test is that the \"refs/tags/*\" refspec appears directly\n> > before the \"refs/heads/bogus/*\" refspec. Without that \"i--\", this\n> > ordering would remove the \"refs/tags/*\" refspec and leave the last one\n> > unmodified, placing the result in \"refs/heads/*\".\n> > \n> > It is possible to have an empty refspec. This is typically the case for\n> > remotes other than the origin, where users want to fetch a specific tag\n> > or branch. To correctly test this case, we need to further remove the\n> > upstream remote for the local branch. Thus, we are testing a refspec\n> > that will be deleted, leaving nothing to fetch.\n> > \n> > Helped-by: Tom Saeger \n> > Helped-by: Ramsay Jones \n> > Signed-off-by: Derrick Stolee \n> > ---\n> > Documentation/fetch-options.txt | 5 +++\n> > builtin/fetch.c | 59 ++++++++++++++++++++++++++++++-\n> > t/t5582-fetch-negative-refspec.sh | 43 ++++++++++++++++++++++\n> > 3 files changed, 106 insertions(+), 1 deletion(-)\n> > \n> > diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> > index 07783deee309..9e7b4e189ce0 100644\n> > --- a/Documentation/fetch-options.txt\n> > +++ b/Documentation/fetch-options.txt\n> > @@ -110,6 +110,11 @@ ifndef::git-pull[]\n> > \tsetting `fetch.writeCommitGraph`.\n> > endif::git-pull[]\n> > \n> > +--prefetch::\n> > +\tModify the configured refspec to place all refs into the\n> > +\t`refs/prefetch/` namespace. See the `prefetch` task in\n> > +\tlinkgit:git-maintenance[1].\n> > +\n> > -p::\n> > --prune::\n> > \tBefore fetching, remove any remote-tracking references that no\n> > diff --git a/builtin/fetch.c b/builtin/fetch.c\n> > index 0b90de87c7a2..97c4fe6e6d66 100644\n> > --- a/builtin/fetch.c\n> > +++ b/builtin/fetch.c\n> > @@ -48,6 +48,7 @@ enum {\n> > static int fetch_prune_config = -1; /* unspecified */\n> > static int fetch_show_forced_updates = 1;\n> > static uint64_t forced_updates_ms = 0;\n> > +static int prefetch = 0;\n> > static int prune = -1; /* unspecified */\n> > #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */\n> > \n> > @@ -158,6 +159,8 @@ static struct option builtin_fetch_options[] = {\n> > \t\t N_(\"do not fetch all tags (--no-tags)\"), TAGS_UNSET),\n> > \tOPT_INTEGER('j', \"jobs\", &max_jobs,\n> > \t\t N_(\"number of submodules fetched in parallel\")),\n> > +\tOPT_BOOL(0, \"prefetch\", &prefetch,\n> > +\t\t N_(\"modify the refspec to place all refs within refs/prefetch/\")),\n> > \tOPT_BOOL('p', \"prune\", &prune,\n> > \t\t N_(\"prune remote-tracking branches no longer on remote\")),\n> > \tOPT_BOOL('P', \"prune-tags\", &prune_tags,\n> > @@ -436,6 +439,56 @@ static void find_non_local_tags(const struct ref *refs,\n> > \toidset_clear(&fetch_oids);\n> > }\n> > \n> > +static void filter_prefetch_refspec(struct refspec *rs)\n> > +{\n> > +\tint i;\n> > +\n> > +\tif (!prefetch)\n> > +\t\treturn;\n> > +\n> > +\tfor (i = 0; i < rs->nr; i++) {\n> > +\t\tstruct strbuf new_dst = STRBUF_INIT;\n> > +\t\tchar *old_dst;\n> > +\t\tconst char *sub = NULL;\n> > +\n> > +\t\tif (rs->items[i].negative)\n> > +\t\t\tcontinue;\n> > +\t\tif (!rs->items[i].dst ||\n> > +\t\t (rs->items[i].src &&\n> > +\t\t !strncmp(rs->items[i].src, \"refs/tags/\", 10))) {\n> > +\t\t\tint j;\n> > +\n> > +\t\t\tfree(rs->items[i].src);\n> > +\t\t\tfree(rs->items[i].dst);\n> > +\n> > +\t\t\tfor (j = i + 1; j < rs->nr; j++) {\n> > +\t\t\t\trs->items[j - 1] = rs->items[j];\n> > +\t\t\t\trs->raw[j - 1] = rs->raw[j];\n> > +\t\t\t}\n> > +\t\t\trs->nr--;\n> > +\t\t\ti--;\n> > +\t\t\tcontinue;\n> > +\t\t}\n> > +\n> > +\t\told_dst = rs->items[i].dst;\n> > +\t\tstrbuf_addstr(&new_dst, \"refs/prefetch/\");\n> > +\n> > +\t\t/*\n> > +\t\t * If old_dst starts with \"refs/\", then place\n> > +\t\t * sub after that prefix. Otherwise, start at\n> > +\t\t * the beginning of the string.\n> > +\t\t */\n> > +\t\tif (!skip_prefix(old_dst, \"refs/\", &sub))\n> > +\t\t\tsub = old_dst;\n> > +\t\tstrbuf_addstr(&new_dst, sub);\n> > +\n> > +\t\trs->items[i].dst = strbuf_detach(&new_dst, NULL);\n> > +\t\trs->items[i].force = 1;\n> > +\n> > +\t\tfree(old_dst);\n> > +\t}\n> > +}\n> > +\n> > static struct ref *get_ref_map(struct remote *remote,\n> > \t\t\t const struct ref *remote_refs,\n> > \t\t\t struct refspec *rs,\n> > @@ -452,6 +505,10 @@ static struct ref *get_ref_map(struct remote *remote,\n> > \tstruct hashmap existing_refs;\n> > \tint existing_refs_populated = 0;\n> > \n> > +\tfilter_prefetch_refspec(rs);\n> > +\tif (remote)\n> > +\t\tfilter_prefetch_refspec(&remote->fetch);\n> > +\n> > \tif (rs->nr) {\n> > \t\tstruct refspec *fetch_refspec;\n> > \n> > @@ -520,7 +577,7 @@ static struct ref *get_ref_map(struct remote *remote,\n> > \t\t\tif (has_merge &&\n> > \t\t\t !strcmp(branch->remote_name, remote->name))\n> > \t\t\t\tadd_merge_config(&ref_map, remote_refs, branch, &tail);\n> > -\t\t} else {\n> > +\t\t} else if (!prefetch) {\n> \n> That works for me.\n> \n> > \t\t\tref_map = get_remote_ref(remote_refs, \"HEAD\");\n> > \t\t\tif (!ref_map)\n> > \t\t\t\tdie(_(\"Couldn't find remote ref HEAD\"));\n> > diff --git a/t/t5582-fetch-negative-refspec.sh b/t/t5582-fetch-negative-refspec.sh\n> > index f34509727702..e5d2e79ad382 100755\n> > --- a/t/t5582-fetch-negative-refspec.sh\n> > +++ b/t/t5582-fetch-negative-refspec.sh\n> > @@ -240,4 +240,47 @@ test_expect_success \"push with matching +: and negative refspec\" '\n> > \tgit -C two push -v one\n> > '\n> > \n> > +test_expect_success '--prefetch correctly modifies refspecs' '\n> > +\tgit -C one config --unset-all remote.origin.fetch &&\n> > +\tgit -C one config --add remote.origin.fetch ^refs/heads/bogus/ignore &&\n> > +\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n> > +\tgit -C one config --add remote.origin.fetch \"refs/heads/bogus/*:bogus/*\" &&\n> > +\n> > +\tgit tag -a -m never never-fetch-tag HEAD &&\n> > +\n> > +\tgit branch bogus/fetched HEAD~1 &&\n> > +\tgit branch bogus/ignore HEAD &&\n> > +\n> > +\tgit -C one fetch --prefetch --no-tags &&\n> > +\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n> > +\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n> > +\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore &&\n> > +\n> > +\t# correctly handle when refspec set becomes empty\n> > +\t# after removing the refs/tags/* refspec.\n> > +\tgit -C one config --unset-all remote.origin.fetch &&\n> > +\tgit -C one config --add remote.origin.fetch \"refs/tags/*:refs/tags/*\" &&\n> > +\n> > +\tgit -C one fetch --prefetch --no-tags &&\n> > +\ttest_must_fail git -C one rev-parse never-fetch-tag &&\n> > +\n> > +\t# The refspec for refs that are not fully qualified\n> > +\t# are filtered multiple times.\n> > +\tgit -C one rev-parse refs/prefetch/bogus/fetched &&\n> > +\ttest_must_fail git -C one rev-parse refs/prefetch/bogus/ignore\n> > +'\n> > +\n> > +test_expect_success '--prefetch succeeds when refspec becomes empty' '\n> \n> technically this will get skipped based only on \"skipfetchall\" right?\n> \n> The remote could have an empty-set of refspecs or multiple\n> valid refspecs post filter_prefetch_refspec, but the remote gets skipped altogether.\n> \n> perhaps '--prefetch succeeds when remote.skipfetchall is true' '\n\nForget what I said here. Now seeing this is using 'git fetch --prefetch'\ndirectly and not using `mainenance --task=prefetch`\n\n> \n> anyway this is looking pretty solid\n> \n> Reviewed-by: Tom Saeger \n> \n> > +\tgit checkout bogus/fetched &&\n> > +\ttest_commit extra &&\n> > +\n> > +\tgit -C one config --unset-all remote.origin.fetch &&\n> > +\tgit -C one config --unset branch.main.remote &&\n> > +\tgit -C one config remote.origin.fetch \"+refs/tags/extra\" &&\n> > +\tgit -C one config remote.origin.skipfetchall true &&\n> > +\tgit -C one config remote.origin.tagopt \"--no-tags\" &&\n> > +\n> > +\tgit -C one fetch --prefetch\n> > +'\n> > +\n> > test_done\n> > -- \n> > gitgitgadget\n> > \n"},{"id":"422108","messageId":"20210416183115.vzx2zj3q23bqtj73@tsaeger.oracle.com","threadId":"55434","inReplyTo":"92652fd9e6e17654abdb30625c85937b6b56c38e.1618577399.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 4/4] maintenance: respect remote.*.skipFetchAll","fromName":"Tom Saeger","fromEmail":"tom.saeger@oracle.com","sentAt":"2021-04-16T18:31:15Z","receivedAt":"2021-04-16T18:31:35Z","isPatch":true,"sender":{"key":"tom.saeger@oracle.com","avatar":null},"body":"On Fri, Apr 16, 2021 at 12:49:59PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> If a remote has the skipFetchAll setting enabled, then that remote is\n> not intended for frequent fetching. It makes sense to not fetch that\n> data during the 'prefetch' maintenance task. Skip that remote in the\n> iteration without error. The skip_default_update member is initialized\n> in remote.c:handle_config() as part of initializing the 'struct remote'.\n> \n> Signed-off-by: Derrick Stolee \n\nReviewed-by: Tom Saeger \n> ---\n> builtin/gc.c | 3 +++\n> t/t7900-maintenance.sh | 8 +++++++-\n> 2 files changed, 10 insertions(+), 1 deletion(-)\n> \n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 9d35f7da50d8..98a803196b88 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -878,6 +878,9 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n> \tstruct maintenance_run_opts *opts = cbdata;\n> \tstruct child_process child = CHILD_PROCESS_INIT;\n> \n> +\tif (remote->skip_default_update)\n> +\t\treturn 0;\n> +\n\nWell that is way easier than doing it in 'fetch'.\n\n\n> \tchild.git_cmd = 1;\n> \tstrvec_pushl(&child.args, \"fetch\", remote->name,\n> \t\t \"--prefetch\", \"--prune\", \"--no-tags\",\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index eadb800c08cc..b93ae014ee58 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -153,7 +153,13 @@ test_expect_success 'prefetch multiple remotes' '\n> \n> \ttest_cmp_config refs/prefetch/ log.excludedecoration &&\n> \tgit log --oneline --decorate --all >log &&\n> -\t! grep \"prefetch\" log\n> +\t! grep \"prefetch\" log &&\n> +\n> +\ttest_when_finished git config --unset remote.remote1.skipFetchAll &&\n> +\tgit config remote.remote1.skipFetchAll true &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/skip-remote1.txt\" git maintenance run --task=prefetch 2>/dev/null &&\n> +\ttest_subcommand ! git fetch remote1 $fetchargs +\ttest_subcommand git fetch remote2 $fetchargs '\n> \n> test_expect_success 'prefetch and existing log.excludeDecoration values' '\n> -- \n> gitgitgadget\n"}]}