{"thread":{"id":"59867","subject":"[PATCH 1/9] pack-objects: allow `--filter` without `--stdout`","startedAt":"2023-06-14T19:26:08Z","lastAt":"2023-10-02T20:14:26Z","messageCount":161,"participants":["Christian Couder","Junio C Hamano","Taylor Blau","Robert Coup","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"478386","messageId":"20230614192541.1599256-1-christian.couder@gmail.com","threadId":"59867","inReplyTo":null,"subject":"[PATCH 0/9] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:32Z","receivedAt":"2023-06-14T19:26:08Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"# Intro\n\nLast year, John Cai sent 2 versions of a patch series to implement\n`git repack --filter=` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\nThis could be useful for the following purposes:\n\n - As a way for servers to save storage costs by for example moving\n large blobs, or blobs in inactive repos, to separate storage\n (while still making them accessible using for example the\n alternates mechanism).\n\n - As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n - As a way for clients to reclaim some space when they cloned with a\n filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a separate\n directory or storage, then check that everything works well, and\n then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\n# Commit overview\n\n* 1/9 pack-objects: allow `--filter` without `--stdout`\n\n This patch is the same as the first patch in the previous series. To\n be able to later repack with a filter we need `git pack-objects` to\n write packfiles when it's filtering instead of just writing the pack\n without the filtered out objects to stdout.\n\n* 2/9 pack-objects: add `--print-filtered` to print omitted objects\n\n We need a way to know the objects that are filtered out of the\n packfile generated by `git pack-objects --filter=`. The\n simplest way is to teach pack-objects to print their oids to stdout.\n\n* 3/9 t/helper: add 'find-pack' test-tool\n\n For testing `git repack --filter=...` that we are going to\n implement, it's useful to have a test helper that can tell which\n packfiles contain a specific object.\n\n* - 4/9 repack: refactor piping an oid to a command\n - 5/9 repack: refactor finishing pack-objects command\n\n These are small refactorings so that `git repack --filter=...` will\n be able to reuse useful existing functions.\n\n* 6/9 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. It uses one\n `git pack-objects` process with both the `--filter` and the\n `--print-filtered` options. From this process it reads the oids of\n the filtered out objects and pass them to a separate `git\n pack-objects` process which will pack these objects into a separate\n packfile.\n\n* 7/9 gc: add `gc.repackFilter` config option\n\n This is a gc config option so that `git gc` can also repack using a\n filter and put the filtered out objects into a separate packfile.\n\n* 8/9 repack: implement `--filter-to` for storing filtered out objects\n\n For some use cases, it's interesting to create the packfile that\n contains the filtered out objects into a separate location. This is\n similar to the --expire-to option for cruft packfiles.\n\n* 9/9 gc: add `gc.repackFilterTo` config option\n\n This allows specifying the location of the packfile that contains\n the filtered out objects when using `gc.repackFilter`.\n\n\nChristian Couder (9):\n pack-objects: allow `--filter` without `--stdout`\n pack-objects: add `--print-filtered` to print omitted objects\n t/helper: add 'find-pack' test-tool\n repack: refactor piping an oid to a command\n repack: refactor finishing pack-objects command\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 11 ++\n Documentation/git-pack-objects.txt | 14 ++-\n Documentation/git-repack.txt | 11 ++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 55 ++++++--\n builtin/repack.c | 166 ++++++++++++++++++-------\n t/helper/test-find-pack.c | 35 ++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5317-pack-objects-filter-objects.sh | 27 ++++\n t/t6500-gc.sh | 23 ++++\n t/t7700-repack.sh | 43 +++++++\n 13 files changed, 345 insertions(+), 53 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n\n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478385","messageId":"20230614192541.1599256-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 1/9] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:33Z","receivedAt":"2023-06-14T19:26:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex a9995a932c..583270a85f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -298,8 +298,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9cfc8801f9..af007868c1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4388,12 +4388,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478387","messageId":"20230614192541.1599256-3-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:34Z","receivedAt":"2023-06-14T19:26:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When using the `--filter=` option, `git pack-objects` will\nomit some objects from the resulting packfile(s) it produces. It could\nbe useful to know about these omitted objects though.\n\nFor example, we might want to write these objects into a separate\npackfile by piping them into another `git pack-object` process.\nOr we might want to check if these objects are available from a\npromisor remote.\n\nAnyway, this patch implements a simple way to let us know about these\nobjects by simply printing their oid, one per line, on stdout when the\nnew `--print-filtered` flag is passed.\n\nAs `--print-filtered` doesn't make sense without `--filter`, it is\ndisallowed to use the former without the latter.\n\nUsing `--stdout` is likely to make the `--print-filtered` output\ndifficult to find or parse, so we also disallow using these two options\ntogether.\n\nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 10 ++++++\n builtin/pack-objects.c | 47 ++++++++++++++++++++++++--\n t/t5317-pack-objects-filter-objects.sh | 27 +++++++++++++++\n 3 files changed, 81 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 583270a85f..6469080029 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -305,6 +305,16 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n --no-filter::\n \tTurns off any previous `--filter=` argument.\n \n+--print-filtered::\n+\tRequires `--filter=`. Prints on stdout, one per line, the\n+\tobject IDs of the objects that are filtered out from the\n+\tresulting packfile by the filter. This is incompatible with\n+\t`--stdout`. As hashes are already written to stdout\n+\tbased on the resulting pack contents (see the `base-name`\n+\targument), a line containing only six `-` characters is\n+\twritten after those hashes, before the filtered object\n+\tIDs.\n+\n --missing=::\n \tA debug option to help with future \"partial clone\" development.\n \tThis option specifies how missing objects are handled.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex af007868c1..c8e2b6b859 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,6 +266,12 @@ static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n \n+/*\n+ * Objects omitted by filter\n+ */\n+static int print_filtered_out;\n+static struct oidset *omitted_by_filter;\n+\n /*\n * stats\n */\n@@ -4065,11 +4071,18 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \t\tdie(_(\"revision walk setup failed\"));\n \tmark_edges_uninteresting(revs, show_edge, sparse);\n \n+\tif (print_filtered_out) {\n+\t\tomitted_by_filter = xmalloc(sizeof(*omitted_by_filter));\n+\t\toidset_init(omitted_by_filter, 0);\n+\t}\n+\n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list(revs,\n-\t\t\t show_commit, fn_show_object,\n-\t\t\t NULL);\n+\ttraverse_commit_list_filtered(revs,\n+\t\t\t\t show_commit,\n+\t\t\t\t fn_show_object,\n+\t\t\t\t NULL,\n+\t\t\t\t omitted_by_filter);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs->ignore_missing_links = 1;\n@@ -4165,6 +4178,23 @@ static int option_parse_cruft_expiration(const struct option *opt,\n \treturn 0;\n }\n \n+static void print_omitted_by_filter(void)\n+{\n+\tstruct oidset_iter iter;\n+\tconst struct object_id *oid;\n+\n+\tfprintf_ln(stdout, \"%s\", \"------\");\n+\tfprintf_ln(stderr, \"%s\", _(\"Printing objects omitted by filter\"));\n+\n+\toidset_iter_init(omitted_by_filter, &iter);\n+\n+\twhile ((oid = oidset_iter_next(&iter)))\n+\t\tfprintf_ln(stdout, \"%s\", oid_to_hex(oid));\n+\n+\toidset_clear(omitted_by_filter);\n+\tFREE_AND_NULL(omitted_by_filter);\n+}\n+\n int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n {\n \tint use_internal_rev_list = 0;\n@@ -4278,6 +4308,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\tOPT_BOOL(0, \"print-filtered\", &print_filtered_out,\n+\t\t\t N_(\"print filtered out objects to stdout\")),\n \t\tOPT_END(),\n \t};\n \n@@ -4394,6 +4426,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n \n+\tif (print_filtered_out && !filter_options.choice)\n+\t\tdie(_(\"cannot use --print-filtered without --filter\"));\n+\n+\tif (print_filtered_out && pack_to_stdout)\n+\t\tdie(_(\"cannot use --print-filtered with --stdout\"));\n+\n \tif (cruft) {\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n@@ -4509,6 +4547,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t written, written_delta, reused, reused_delta,\n \t\t\t reuse_packfile_objects);\n \n+\tif (omitted_by_filter)\n+\t\tprint_omitted_by_filter();\n+\n cleanup:\n \tlist_objects_filter_release(&filter_options);\n \tstrvec_clear(&rp);\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..ec3a03d90a 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -438,6 +438,33 @@ test_expect_success 'verify sparse:oid=oid-ish' '\n \ttest_cmp expected observed\n '\n \n+# Test pack-objects with --print-filtered option\n+\n+test_expect_success 'pack-objects fails w/ both --print-filtered and --stdout' '\n+\ttest_must_fail git -C r1 pack-objects --revs --stdout \\\n+\t\t--filter=blob:none --print-filtered >filter.out <<-EOF\n+\tHEAD\n+\tEOF\n+'\n+\n+test_expect_success 'pack-objects w/ --print-filtered and a pack name' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none \\\n+\t\t--print-filtered filtered-pack >filter.out <<-EOF &&\n+\tHEAD\n+\tEOF\n+\n+\t# Check that the second line contains \"------\"\n+\thead -n 2 filter.out | tail -n 1 >actual &&\n+\techo \"------\" >expected &&\n+\ttest_cmp expected actual &&\n+\n+\t# Remove the first two lines and check there are all the blobs\n+\ttail -n +3 filter.out | sort >actual &&\n+\tgit -C r1 cat-file --batch-check --batch-all-objects | grep blob |\n+\t\tsed -e \"s/ blob.*//\" | sort >expected &&\n+\ttest_cmp expected actual\n+'\n+\n # Delete some loose objects and use pack-objects, but WITHOUT any filtering.\n # This models previously omitted objects that we did not receive.\n \n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478388","messageId":"20230614192541.1599256-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 3/9] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:35Z","receivedAt":"2023-06-14T19:26:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In a following commit, we will make it possible to separate objects in\ndifferent packfiles depending on a filter.\n\nTo make sure that the right objects are in the right packs, let's add a\nnew test-tool that can display which packfile(s) a given object is in.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 35 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 38 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..c1cd735b31 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..1928fe7329\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ */\n+\n+static const char *find_pack_usage = \"\\n\"\n+\" test-tool find-pack \";\n+\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\n+\tsetup_git_directory();\n+\n+\tif (argc != 2)\n+\t\tusage(find_pack_usage);\n+\n+\tif (repo_get_oid(the_repository, argv[1], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[1]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tif (find_pack_entry_one(oid.hash, p))\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex abe8a785eb..41da40c296 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ea2672436c..411dbf2db4 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478389","messageId":"20230614192541.1599256-6-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 5/9] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:37Z","receivedAt":"2023-06-14T19:26:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -712,12 +748,9 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct string_list *existing_kept_packs)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n-\tconst char *scratch;\n-\tint local = skip_prefix(destination, packdir, &scratch);\n \n \tprepare_pack_objects(&cmd, args, destination);\n \n@@ -758,27 +791,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, destination);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -789,10 +802,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1023,18 +1034,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, NULL);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478390","messageId":"20230614192541.1599256-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 4/9] repack: refactor piping an oid to a command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:36Z","receivedAt":"2023-06-14T19:26:30Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new write_oid_hex_cmd() function to send an oid to the standard\ninput of a running command. This new function will be used in a\nfollowing commit.\n\nSigned-off-by: Christian Couder \n---\n builtin/repack.c | 20 +++++++++++++-------\n 1 file changed, 13 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 0541c3ce15..e591c295cf 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -182,6 +182,17 @@ static void prepare_pack_objects(struct child_process *cmd,\n \tcmd->out = -1;\n }\n \n+static void write_oid_hex_cmd(const char *oid_hex,\n+\t\t\t struct child_process *cmd,\n+\t\t\t const char *err_msg)\n+{\n+\tif (cmd->in == -1 && start_command(cmd))\n+\t\tdie(\"%s\", err_msg);\n+\n+\txwrite(cmd->in, oid_hex, the_hash_algo->hexsz);\n+\txwrite(cmd->in, \"\\n\", 1);\n+}\n+\n /*\n * Write oid to the given struct child_process's stdin, starting it first if\n * necessary.\n@@ -192,13 +203,8 @@ static int write_oid(const struct object_id *oid,\n {\n \tstruct child_process *cmd = data;\n \n-\tif (cmd->in == -1) {\n-\t\tif (start_command(cmd))\n-\t\t\tdie(_(\"could not start pack-objects to repack promisor objects\"));\n-\t}\n-\n-\txwrite(cmd->in, oid_to_hex(oid), the_hash_algo->hexsz);\n-\txwrite(cmd->in, \"\\n\", 1);\n+\twrite_oid_hex_cmd(oid_to_hex(oid), cmd,\n+\t\t\t _(\"could not start pack-objects to repack promisor objects\"));\n \treturn 0;\n }\n \n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478391","messageId":"20230614192541.1599256-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:38Z","receivedAt":"2023-06-14T19:26:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"After cloning with --filter=, for example to avoid\ngetting unneeded large files on a user machine, it's possible\nthat some of these large files still get fetched for some reasons\n(like checking out old branches) over time.\n\nIn this case the repo size could grow too much for no good reason and a\nway to filter out some objects would be useful to remove the unneeded\nlarge files.\n\nDeleting objects right away could corrupt a repo though, so it might be\nbetter to put those objects into a separate packfile instead of\ndeleting them. The separate pack could then be removed after checking\nthat all the objects in it are still available on a promisor remote it\ncan access.\n\nAlso splitting a packfile into 2 packs depending on a filter could be\nuseful in other usecases. For example some large blobs might take a lot\nof precious space on fast storage while they are rarely accessed, and\nit could make sense to move them in a separate cheaper, though slower,\nstorage.\n\nThis commit implements a new `--filter=` option in\n`git repack` that moves filtered out objects into a separate pack.\n\nThis is done by reading filtered out objects from `git pack-objects`'s\noutput and piping them into a separate `git pack-objects` process that\nwill put them into a separate packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 5 +++\n builtin/repack.c | 75 ++++++++++++++++++++++++++++++++++--\n t/t7700-repack.sh | 16 ++++++++\n 3 files changed, 93 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..aa29c7e648 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,11 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. See\n+\tlinkgit:git-rev-list[1] for valid `` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f1adacf1d0..b13d7196de 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -53,6 +53,7 @@ struct pack_objects_args {\n \tconst char *depth;\n \tconst char *threads;\n \tconst char *max_pack_size;\n+\tconst char *filter;\n \tint no_reuse_delta;\n \tint no_reuse_object;\n \tint quiet;\n@@ -167,6 +168,10 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--threads=%s\", args->threads);\n \tif (args->max_pack_size)\n \t\tstrvec_pushf(&cmd->args, \"--max-pack-size=%s\", args->max_pack_size);\n+\tif (args->filter) {\n+\t\tstrvec_pushf(&cmd->args, \"--filter=%s\", args->filter);\n+\t\tstrvec_pushf(&cmd->args, \"--print-filtered\");\n+\t}\n \tif (args->no_reuse_delta)\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n \tif (args->no_reuse_object)\n@@ -703,13 +708,21 @@ static void remove_redundant_bitmaps(struct string_list *include,\n \tstrbuf_release(&path);\n }\n \n+static void pack_filtered(const char *oid_hex, struct child_process *cmd)\n+{\n+\twrite_oid_hex_cmd(oid_hex, cmd,\n+\t\t\t _(\"could not start pack-objects to pack filtered objects\"));\n+}\n+\n static int finish_pack_objects_cmd(struct child_process *cmd,\n \t\t\t\t struct string_list *names,\n-\t\t\t\t const char *destination)\n+\t\t\t\t const char *destination,\n+\t\t\t\t struct child_process *pack_filtered_cmd)\n {\n \tint local = 1;\n \tFILE *out;\n \tstruct strbuf line = STRBUF_INIT;\n+\tint filtered_start = 0;\n \n \tif (destination) {\n \t\tconst char *scratch;\n@@ -720,9 +733,20 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \twhile (strbuf_getline_lf(&line, out) != EOF) {\n \t\tstruct string_list_item *item;\n \n+\t\tif (!filtered_start && pack_filtered_cmd && !strcmp(line.buf, \"------\")) {\n+\t\t\tfiltered_start = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (line.len != the_hash_algo->hexsz)\n \t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n \t\t\t \"from pack-objects.\"));\n+\n+\t\tif (pack_filtered_cmd && filtered_start) {\n+\t\t\tpack_filtered(line.buf, pack_filtered_cmd);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\t/*\n \t\t * Avoid putting packs written outside of the repository in the\n \t\t * list of names.\n@@ -791,9 +815,44 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\treturn finish_pack_objects_cmd(&cmd, names, destination);\n+\treturn finish_pack_objects_cmd(&cmd, names, destination, NULL);\n }\n \n+/*\n+ * Prepare the command that will pack objects that have been filtered\n+ * out from the original pack, so that they will end up in a separate\n+ * pack.\n+ */\n+static void prepare_pack_filtered_cmd(struct child_process *cmd,\n+\t\t\t\t const struct pack_objects_args *args,\n+\t\t\t\t const char *destination)\n+{\n+\t/* We need to copy args to modify it */\n+\tstruct pack_objects_args new_args = *args;\n+\n+\t/* No need to filter again */\n+\tnew_args.filter = NULL;\n+\n+\tprepare_pack_objects(cmd, &new_args, destination);\n+\tcmd->in = -1;\n+}\n+\n+static void finish_pack_filtered_cmd(struct child_process *cmd,\n+\t\t\t\t struct string_list *names)\n+{\n+\tif (cmd->in == -1) {\n+\t\t/* No packed objects; cmd was never started */\n+\t\tchild_process_clear(cmd);\n+\t\treturn;\n+\t}\n+\n+\tclose(cmd->in);\n+\n+\tif (finish_pack_objects_cmd(cmd, names, NULL, NULL))\n+\t\tdie(_(\"could not finish pack-objects to pack filtered objects\"));\n+}\n+\n+\n int cmd_repack(int argc, const char **argv, const char *prefix)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -817,6 +876,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tstruct child_process pack_filtered_cmd = CHILD_PROCESS_INIT;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -858,6 +918,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_STRING(0, \"filter\", &po_args.filter, N_(\"args\"),\n+\t\t\t\tN_(\"object filtering\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -1011,6 +1073,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (po_args.filter)\n+\t\tprepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n+\n \tif (geometry)\n \t\tcmd.in = -1;\n \telse\n@@ -1034,7 +1099,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tret = finish_pack_objects_cmd(&cmd, &names, NULL);\n+\tret = finish_pack_objects_cmd(&cmd, &names, NULL,\n+\t\t\t\t po_args.filter ? &pack_filtered_cmd : NULL);\n \tif (ret)\n \t\tgoto cleanup;\n \n@@ -1102,6 +1168,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter)\n+\t\tfinish_pack_filtered_cmd(&pack_filtered_cmd, &names);\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex faa739eeb9..9e7654090f 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -270,6 +270,22 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n+\ttest -n \"$commit_pack\" &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -n \"$blob_pack\" &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478392","messageId":"20230614192541.1599256-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:40Z","receivedAt":"2023-06-14T19:26:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage. Even in a different directory this pack can be\naccessible if, for example, the Git alternates mechanism is used to\npoint to it.\n\nIf users want to remove a pack that contains filtered out objects after\nchecking that they are all already on a promisor remote, creating the\npack in a different directory makes it easier to do so.\n\nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 6 ++++++\n builtin/repack.c | 17 ++++++++++++-----\n t/t7700-repack.sh | 27 +++++++++++++++++++++++++++\n 3 files changed, 45 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex aa29c7e648..070dd22610 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -148,6 +148,12 @@ depth is 4095.\n \tresulting packfile and put them into a separate packfile. See\n \tlinkgit:git-rev-list[1] for valid `` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. This can be used for putting the pack on a\n+\tseparate object directory that is accessed through the Git\n+\talternates mechanism. Only useful with `--filter`.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex b13d7196de..8c71e8fd51 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -838,7 +838,8 @@ static void prepare_pack_filtered_cmd(struct child_process *cmd,\n }\n \n static void finish_pack_filtered_cmd(struct child_process *cmd,\n-\t\t\t\t struct string_list *names)\n+\t\t\t\t struct string_list *names,\n+\t\t\t\t const char *destination)\n {\n \tif (cmd->in == -1) {\n \t\t/* No packed objects; cmd was never started */\n@@ -848,7 +849,7 @@ static void finish_pack_filtered_cmd(struct child_process *cmd,\n \n \tclose(cmd->in);\n \n-\tif (finish_pack_objects_cmd(cmd, names, NULL, NULL))\n+\tif (finish_pack_objects_cmd(cmd, names, destination, NULL))\n \t\tdie(_(\"could not finish pack-objects to pack filtered objects\"));\n }\n \n@@ -877,6 +878,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n \tstruct child_process pack_filtered_cmd = CHILD_PROCESS_INIT;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -930,6 +932,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1073,8 +1077,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n-\tif (po_args.filter)\n-\t\tprepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n+\tif (po_args.filter) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\t\tprepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);\n+\t}\n \n \tif (geometry)\n \t\tcmd.in = -1;\n@@ -1169,7 +1176,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter)\n-\t\tfinish_pack_filtered_cmd(&pack_filtered_cmd, &names);\n+\t\tfinish_pack_filtered_cmd(&pack_filtered_cmd, &names, filter_to);\n \n \tstring_list_sort(&names);\n \ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 9e7654090f..898f8a01b4 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -286,6 +286,33 @@ test_expect_success 'repacking with a filter works' '\n \ttest \"$blob_pack2\" = \"$blob_pack\"\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n+\ttest -n \"$commit_pack\" &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -z \"$blob_pack\" &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack $blob_hash) &&\n+\ttest -n \"$blob_pack\" &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -n \"$blob_pack\" &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478393","messageId":"20230614192541.1599256-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 7/9] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:39Z","receivedAt":"2023-06-14T19:26:35Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 12 ++++++++++++\n 3 files changed, 23 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 7f95c866e1..055c4e0db6 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -130,6 +130,11 @@ or rebase occurring. Since these changes are not part of the current\n project most users will want to expire them sooner, which is why the\n default is more aggressive than `gc.reflogExpire`.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f3942188a6..1c57913214 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -61,6 +61,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -170,6 +171,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -355,6 +358,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..5b89faf505 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,18 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none -c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478394","messageId":"20230614192541.1599256-10-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH 9/9] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-14T19:25:41Z","receivedAt":"2023-06-14T19:26:36Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 6 ++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 055c4e0db6..699ad887b3 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -135,6 +135,12 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. See the\n+\t`--filter-to=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 1c57913214..87f5fc6946 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -172,6 +173,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -361,6 +363,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 5b89faf505..37056a824b 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -214,6 +213,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.41.0.37.gae45d9845e\n\n"},{"id":"478399","messageId":"xmqqsfatwwtv.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"Re: [PATCH 0/9] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-14T21:36:12Z","receivedAt":"2023-06-14T21:36:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> In some discussions, it was mentioned that such a feature, or a\n> similar feature in `git gc`, or in a new standalone command (perhaps\n> called `git prune-filtered`), should put the filtered out objects into\n> a new packfile instead of deleting them.\n>\n> Recently there were internal discussions at GitLab about either moving\n> blobs from inactive repos onto cheaper storage, or moving large blobs\n> onto cheaper storage. This lead us to rethink at repacking using a\n> filter, but moving the filtered out objects into a separate packfile\n> instead of deleting them.\n>\n> So here is a new patch series doing that while implementing the\n> `--filter=` option in `git repack`.\n\nVery interesting idea, indeed, and would be very useful.\nThanks.\n\n"},{"id":"478455","messageId":"xmqq7cs4uyqe.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-3-christian.couder@gmail.com","subject":"Re: [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-15T22:50:17Z","receivedAt":"2023-06-15T22:50:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> When using the `--filter=` option, `git pack-objects` will\n> omit some objects from the resulting packfile(s) it produces. It could\n> be useful to know about these omitted objects though.\n>\n> For example, we might want to write these objects into a separate\n> packfile by piping them into another `git pack-object` process.\n> Or we might want to check if these objects are available from a\n> promisor remote.\n>\n> Anyway, this patch implements a simple way to let us know about these\n> objects by simply printing their oid, one per line, on stdout when the\n> new `--print-filtered` flag is passed.\n\nMakes sense. It is a bit sad that we have to accumulate everything\nuntil the end at which time we have to dump the accumulated in bulk,\nbut that is a current limitation of list-objects-filter API and not\nwithin the scope of this change. We may in the longer term want to\nsee if we can make the collection of filtered-out objects streamable\nby replacing the .omits object array with a callback function, or do\nsomething along that line.\n"},{"id":"478456","messageId":"xmqq1qicuwrw.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-4-christian.couder@gmail.com","subject":"Re: [PATCH 3/9] t/helper: add 'find-pack' test-tool","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-15T23:32:35Z","receivedAt":"2023-06-15T23:32:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> In a following commit, we will make it possible to separate objects in\n> different packfiles depending on a filter.\n>\n> To make sure that the right objects are in the right packs, let's add a\n> new test-tool that can display which packfile(s) a given object is in.\n\nThis tool would be serviceable if we only are interested in checking\njust a few objects, but if we were to check many objects, I have to\nwonder if it would be more efficient to use show-index to dump the\nlist of objects per pack, which should be sorted by object name, so\nit should be trivial to run \"comm\" with the list of objects you want\nto check.\n\nOr if you only are checking about a dozen or so, taking one or more\narguments from the command line and looping over them may also be\nOK. The output format of course may have to be changed, if we were\nto go that route, though.\n\nIt really depends on the granularity at which this test helper wants\nto work at, I think.\n\nAnyway...\n\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index abe8a785eb..41da40c296 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n> \t{ \"env-helper\", cmd__env_helper },\n> \t{ \"example-decorate\", cmd__example_decorate },\n> \t{ \"fast-rebase\", cmd__fast_rebase },\n> +\t{ \"find-pack\", cmd__find_pack },\n> \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n> \t{ \"genrandom\", cmd__genrandom },\n> \t{ \"genzeros\", cmd__genzeros },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index ea2672436c..411dbf2db4 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n> int cmd__env_helper(int argc, const char **argv);\n> int cmd__example_decorate(int argc, const char **argv);\n> int cmd__fast_rebase(int argc, const char **argv);\n> +int cmd__find_pack(int argc, const char **argv);\n> int cmd__fsmonitor_client(int argc, const char **argv);\n> int cmd__genrandom(int argc, const char **argv);\n> int cmd__genzeros(int argc, const char **argv);\n"},{"id":"478457","messageId":"xmqqttv8thjw.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-5-christian.couder@gmail.com","subject":"Re: [PATCH 4/9] repack: refactor piping an oid to a command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-15T23:46:43Z","receivedAt":"2023-06-15T23:46:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Create a new write_oid_hex_cmd() function to send an oid to the standard\n> input of a running command. This new function will be used in a\n> following commit.\n>\n> Signed-off-by: Christian Couder \n> ---\n> builtin/repack.c | 20 +++++++++++++-------\n> 1 file changed, 13 insertions(+), 7 deletions(-)\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 0541c3ce15..e591c295cf 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -182,6 +182,17 @@ static void prepare_pack_objects(struct child_process *cmd,\n> \tcmd->out = -1;\n> }\n> \n> +static void write_oid_hex_cmd(const char *oid_hex,\n> +\t\t\t struct child_process *cmd,\n> +\t\t\t const char *err_msg)\n> +{\n> +\tif (cmd->in == -1 && start_command(cmd))\n> +\t\tdie(\"%s\", err_msg);\n\nI am not sure why we would want to conflate the \"if we haven't\nstarted the command, auto-start it upon our first attempt to write\"\nin these low-level \"I am designed to do one thing, which is to feed\nthe object name to the process, and do it well\" function.\n\nThe caller in the original shares the same issue, so we could say\nthat this patch is not creating a new problem, but this somehow\nfeels it is mak ng the existing problem even worse.\n\nAnd I think the error handling here shows why the API feels wrong.\nWhen auto-start fails, we have a message, but when write fails,\nthere is no custom message---it makes as if write_oid_hex_cmd() is\nprimarily about starting, which is so important relative to its\nother functionalities and deserves a custom error message, but that\nis not the message you want to be conveying.\n\n> +\txwrite(cmd->in, oid_hex, the_hash_algo->hexsz);\n> +\txwrite(cmd->in, \"\\n\", 1);\n\nI would have expected that the \"refactor\" at least would reduce the\nnumber of system calls by combining these two writes into one using\nan on-stack local variable char buf[GIT_MAX_HEZSZ+1] or something.\n"},{"id":"478458","messageId":"xmqqilbotgbp.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-6-christian.couder@gmail.com","subject":"Re: [PATCH 5/9] repack: refactor finishing pack-objects command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-16T00:13:14Z","receivedAt":"2023-06-16T00:13:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> +static int finish_pack_objects_cmd(struct child_process *cmd,\n> +\t\t\t\t struct string_list *names,\n> +\t\t\t\t const char *destination)\n> +{\n> +\tint local = 1;\n> +\tFILE *out;\n> +\tstruct strbuf line = STRBUF_INIT;\n> +\n> +\tif (destination) {\n> +\t\tconst char *scratch;\n> +\t\tlocal = skip_prefix(destination, packdir, &scratch);\n> +\t}\n> +\n> +\tout = xfdopen(cmd->out, \"r\");\n> +\twhile (strbuf_getline_lf(&line, out) != EOF) {\n> +\t\tstruct string_list_item *item;\n> +\n> +\t\tif (line.len != the_hash_algo->hexsz)\n> +\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n> +\t\t\t \"from pack-objects.\"));\n> +\t\t/*\n> +\t\t * Avoid putting packs written outside of the repository in the\n> +\t\t * list of names.\n> +\t\t */\n> +\t\tif (local) {\n> +\t\t\titem = string_list_append(names, line.buf);\n> +\t\t\titem->util = populate_pack_exts(line.buf);\n> +\t\t}\n> +\t}\n> +\tfclose(out);\n> +\n> +\tstrbuf_release(&line);\n> +\n> +\treturn finish_command(cmd);\n> +}\n\nComputing \"is it local?\" based on the value of \"destination\" feels\nit belongs to the caller (one of the callers that do need the\ncomputation), not to this function, especially given that the full\nvalue of \"destination\" is not even used in any other way in this\nfunction. And the \"is_local?\" bit can instead be passesd into this\nhelper function as a parameter.\n\nI wondered what \"beautify\" was about---the original looks OK to me\nalready, and while I do not mind to see a full sentence spelled in a\nmore gramatically correct way like in the postimage, I do not think\nthe change was worth wasting reviewer's time wondering if there are\nother improvements by pointing it out in the proposed log message.\n\nThanks.\n"},{"id":"478459","messageId":"xmqqmt10s0cw.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-7-christian.couder@gmail.com","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-16T00:43:27Z","receivedAt":"2023-06-16T00:43:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> After cloning with --filter=, for example to avoid\n> getting unneeded large files on a user machine, it's possible\n> that some of these large files still get fetched for some reasons\n> (like checking out old branches) over time.\n>\n> In this case the repo size could grow too much for no good reason and a\n> way to filter out some objects would be useful to remove the unneeded\n> large files.\n\nMakes sense.\n\nIf we repack without these objects, when the repository has a\npromisor remote, we should be able to rely on that remote to supply\nthem on demand, once we need them again, no?\n\n> Deleting objects right away could corrupt a repo though,...\n\nHmph, could you elaborate why it is the case? Isn't it the whole\npoint to have promisor remote and use a lazy clone with the --filter\noption, so that objects that _ought_ to exist from connectivity's\npoint of view _are_ allowed to be missing because the promisor\npromises to make them available on-demand?\n\n\tSide note: I think I know the answer. While trying to remove\n\tUNNEEDED large files, doing so may discard NEEDED large\n\tfiles when done carelessly (e.g. the file may have been\n\tcreated locally and haven't been pushed back). But (1) if\n\tthat is the problem, perhaps we should be more careful in\n\tthe first place? (2) if it inherently is impossible to tell\n\twhich ones are unneeded reliably, the reason why it is\n\timpossible, and the reason why \"try sifting into two bins,\n\tone that we _think_ are unneeded and another for the rest,\n\tand verify what we _thought_ are unneeded are all available\n\tfrom the promisor remote\" is the best we can do, must be\n\tdescribed, I think.\n\n> ... so it might be\n> better to put those objects into a separate packfile instead of\n> deleting them. The separate pack could then be removed after checking\n> that all the objects in it are still available on a promisor remote it\n> can access.\n\nSurely, sifting the objects into two bins (i.e. those that we\nwouldn't have received if we cloned from the promisor remote just\nnow, which are prunable, and those that we cannot lose because the\npromisor remote would not have them, e.g. we created them and have\nnot pushed them to the remote yet) without removing anything would\nbe safe, but if the result of such sifting must be verified, doesn't\nit indicate that the sifting step was buggy or misdesigned? It does\nnot sound like a very good justification to save them in a separate\npackfile. It does smell somewhat similar to the cruft packs but not\nreally (the choice over there is between exploding to loose and\nkeeping in a pack, and never involves loss of objects).\n\n> Also splitting a packfile into 2 packs depending on a filter could be\n> useful in other usecases. For example some large blobs might take a lot\n> of precious space on fast storage while they are rarely accessed, and\n> it could make sense to move them in a separate cheaper, though slower,\n> storage.\n\nThis one, outside the context of partial clone client, does make\ntons of sense.\n\nI guess what I suspect is that this option, while it would be very\nuseful for the \"in other usecases\" scenario above, may not become\nall that useful in the \"our lazy clone got bloated and we want to\ntrim objects we know we can retrieve from the promisor remote again\nif necessary\" scenario, until the repack machinery learns to use an\nextra piece of information (namely \"these are objects that we can\nfetch from the promisor remote\") at the same time.\n\n> This commit implements a new `--filter=` option in\n> `git repack` that moves filtered out objects into a separate pack.\n>\n> This is done by reading filtered out objects from `git pack-objects`'s\n> output and piping them into a separate `git pack-objects` process that\n> will put them into a separate packfile.\n\nSo, for example, you may say \"no blobs\" in the filter, and while\npacking the local repository with the filter, resulting in a pack\nthat exclude all blobs, we will learn what blob objects we did not\npack into that packfile. We can pack them into a separate one, and\nmost of the blobs are what we could retrieve again from the promisor\nremote, but some of the blobs are what we locally created ourselves\nand haven't pushed back to the promisor remote yet. Now what? My\nearlier suspicion that this mechanism may not be all that useful for\nthe \"slim bloated lazy clone\" comes from that I cannot think of a\ngood answer to this \"Now what?\" question---my naive solution would\ninvolve enumerating the objects in that \"separate packfile\" that is\na mixture of precious ones and expendable ones, and then learning\nwhich ones are precious, and creating a new pack that is a subset of\nthat \"separate packfile\" with only the precious ones. But if I do\nso, I do not think we need this new mechanism that seems to go only\nthe half-way.\n"},{"id":"478460","messageId":"xmqqfs6srvsr.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-9-christian.couder@gmail.com","subject":"Re: [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-16T02:21:56Z","receivedAt":"2023-06-16T02:22:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> A previous commit has implemented `git repack --filter=` to\n> allow users to filter out some objects from the main pack and move them\n> into a new different pack.\n>\n> It would be nice if this new different pack could be created in a\n> different directory than the regular pack. This would make it possible\n> to move large blobs into a pack on a different kind of storage, for\n> example cheaper storage. Even in a different directory this pack can be\n> accessible if, for example, the Git alternates mechanism is used to\n> point to it.\n\nMakes sense, I guess, for \"in other usecases\" scenario. I am not\nsure how this would be useful for the originally stated goal of\nunbloating a bloated repository with promisor remote(s), though. \n\n> If users want to remove a pack that contains filtered out objects after\n> checking that they are all already on a promisor remote, creating the\n> pack in a different directory makes it easier to do so.\n\nCare to elaborate? I do not see how a separate directory would make\nit easier. After separating the potential cruft into a packfile,\nyou'd walk its .idx and see if there are any objects that are not\navailable (yet) at the promisor remotes to check if it is safe to\nremove. That can be done regardless of the location of the packfile\nthat is suspected to be now removable.\n\n> Signed-off-by: Christian Couder \n> ---\n> Documentation/git-repack.txt | 6 ++++++\n> builtin/repack.c | 17 ++++++++++++-----\n> t/t7700-repack.sh | 27 +++++++++++++++++++++++++++\n> 3 files changed, 45 insertions(+), 5 deletions(-)\n>\n> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> index aa29c7e648..070dd22610 100644\n> --- a/Documentation/git-repack.txt\n> +++ b/Documentation/git-repack.txt\n> @@ -148,6 +148,12 @@ depth is 4095.\n> \tresulting packfile and put them into a separate packfile. See\n> \tlinkgit:git-rev-list[1] for valid `` forms.\n> \n> +--filter-to=::\n> +\tWrite the pack containing filtered out objects to the\n> +\tdirectory ``. This can be used for putting the pack on a\n> +\tseparate object directory that is accessed through the Git\n> +\talternates mechanism. Only useful with `--filter`.\n> +\n> -b::\n> --write-bitmap-index::\n> \tWrite a reachability bitmap index as part of the repack. This\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index b13d7196de..8c71e8fd51 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -838,7 +838,8 @@ static void prepare_pack_filtered_cmd(struct child_process *cmd,\n> }\n> \n> static void finish_pack_filtered_cmd(struct child_process *cmd,\n> -\t\t\t\t struct string_list *names)\n> +\t\t\t\t struct string_list *names,\n> +\t\t\t\t const char *destination)\n> {\n> \tif (cmd->in == -1) {\n> \t\t/* No packed objects; cmd was never started */\n> @@ -848,7 +849,7 @@ static void finish_pack_filtered_cmd(struct child_process *cmd,\n> \n> \tclose(cmd->in);\n> \n> -\tif (finish_pack_objects_cmd(cmd, names, NULL, NULL))\n> +\tif (finish_pack_objects_cmd(cmd, names, destination, NULL))\n> \t\tdie(_(\"could not finish pack-objects to pack filtered objects\"));\n> }\n> \n> @@ -877,6 +878,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \tconst char *cruft_expiration = NULL;\n> \tconst char *expire_to = NULL;\n> \tstruct child_process pack_filtered_cmd = CHILD_PROCESS_INIT;\n> +\tconst char *filter_to = NULL;\n> \n> \tstruct option builtin_repack_options[] = {\n> \t\tOPT_BIT('a', NULL, &pack_everything,\n> @@ -930,6 +932,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n> \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n> \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n> +\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n> +\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n> \t\tOPT_END()\n> \t};\n> \n> @@ -1073,8 +1077,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\tstrvec_push(&cmd.args, \"--incremental\");\n> \t}\n> \n> -\tif (po_args.filter)\n> -\t\tprepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n> +\tif (po_args.filter) {\n> +\t\tif (!filter_to)\n> +\t\t\tfilter_to = packtmp;\n> +\t\tprepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);\n> +\t}\n> \n> \tif (geometry)\n> \t\tcmd.in = -1;\n> @@ -1169,7 +1176,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t}\n> \n> \tif (po_args.filter)\n> -\t\tfinish_pack_filtered_cmd(&pack_filtered_cmd, &names);\n> +\t\tfinish_pack_filtered_cmd(&pack_filtered_cmd, &names, filter_to);\n> \n> \tstring_list_sort(&names);\n> \n> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n> index 9e7654090f..898f8a01b4 100755\n> --- a/t/t7700-repack.sh\n> +++ b/t/t7700-repack.sh\n> @@ -286,6 +286,33 @@ test_expect_success 'repacking with a filter works' '\n> \ttest \"$blob_pack2\" = \"$blob_pack\"\n> '\n> \n> +test_expect_success '--filter-to stores filtered out objects' '\n> +\tgit -C bare.git repack -a -d &&\n> +\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n> +\n> +\tgit init --bare filtered.git &&\n> +\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n> +\t\t--filter=blob:none \\\n> +\t\t--filter-to=../filtered.git/objects/pack/pack &&\n> +\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n> +\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n> +\n> +\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n> +\ttest -n \"$commit_pack\" &&\n> +\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n> +\ttest -z \"$blob_pack\" &&\n> +\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n> +\ttest -n \"$blob_hash\" &&\n> +\tblob_pack=$(test-tool -C filtered.git find-pack $blob_hash) &&\n> +\ttest -n \"$blob_pack\" &&\n> +\n> +\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n> +\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n> +\ttest -n \"$blob_pack\" &&\n> +\tblob_content=$(git -C bare.git show $blob_hash) &&\n> +\ttest \"$blob_content\" = \"content1\"\n> +'\n> +\n> objdir=.git/objects\n> midx=$objdir/pack/multi-pack-index\n"},{"id":"478461","messageId":"xmqqa5x0ruag.fsf@gitster.g","threadId":"59867","inReplyTo":"20230614192541.1599256-10-christian.couder@gmail.com","subject":"Re: [PATCH 9/9] gc: add `gc.repackFilterTo` config option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-16T02:54:31Z","receivedAt":"2023-06-16T02:55:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> A previous commit implemented the `gc.repackFilter` config option\n> to specify a filter that should be used by `git gc` when\n> performing repacks.\n>\n> Another previous commit has implemented\n> `git repack --filter-to=` to specify the location of the\n> packfile containing filtered out objects when using a filter.\n>\n> Let's implement the `gc.repackFilterTo` config option to specify\n> that location in the config when `gc.repackFilter` is used.\n>\n> Now when `git gc` will perform a repack with a configured\n> through this option and not empty, the repack process will be\n> passed a corresponding `--filter-to=` argument.\n>\n> Signed-off-by: Christian Couder \n> ---\n\nThat's an obvious follow-up on the previous step.\n\nThanks.\n"},{"id":"478462","messageId":"xmqq5y7ortmr.fsf@gitster.g","threadId":"59867","inReplyTo":"xmqqsfatwwtv.fsf@gitster.g","subject":"Re: [PATCH 0/9] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-16T03:08:44Z","receivedAt":"2023-06-16T03:08:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Christian Couder writes:\n>\n>> In some discussions, it was mentioned that such a feature, or a\n>> similar feature in `git gc`, or in a new standalone command (perhaps\n>> called `git prune-filtered`), should put the filtered out objects into\n>> a new packfile instead of deleting them.\n>>\n>> Recently there were internal discussions at GitLab about either moving\n>> blobs from inactive repos onto cheaper storage, or moving large blobs\n>> onto cheaper storage. This lead us to rethink at repacking using a\n>> filter, but moving the filtered out objects into a separate packfile\n>> instead of deleting them.\n>>\n>> So here is a new patch series doing that while implementing the\n>> `--filter=` option in `git repack`.\n>\n> Very interesting idea, indeed, and would be very useful.\n> Thanks.\n\nOverall, I have a split feeling on the series.\n\nOne side of my brain thinks that the series does a very good job to\naddress the needs of those who want to partition their objects into\ntwo classes, and the problem I saw in the series was mostly the way\nit was sold (in other words, if it did not mention unbloating lazily\ncloned repositories at all, I would have said \"Yes! It is an\nexcellent series.\", and if it said \"this mechanism is not meant to\nbe used to unbloat a lazily cloned repository, because the mechanism\ndoes not distinguish objects that are only locally available and\nobjects that are retrievable from the promisor remotes, among those\nthat match the filter\", it would have been even better)\n\nTo the other side of my brain, it smells as if the series wanted to\naddress the unbloating issue, but ended up with an unsatisfactory\nsolution, and used \"partitioning objects in a full repository on the\nserver side \" as an excuse for the resulting mechanism to still\nexist, even though it is not usable for the original purpose.\n\nIdeally, it would be great to have a mechanism that can be used for\nboth. The \"partitioning\" can be treated as a degenerate case where\nthe repository does not have its upstream promisor (hence, any\nobject that match the filtering criteria can be excluded from the\nprimary pack because there are no \"not available (yet) in our\npromisor\" objects), while the \"unbloat\" case can know who its\npromisors are and ask the promisors what objects, among those that\nmatch the filtering criteria, are still available from them to\nexclude only those objects from the primary pack.\n\nIn the second ideal world, we may not be ready to tackle the\nunbloating issue, but \"partitioning\" alone may still be a useful\nfeature. In that case, perhaps the series can be salvaged by\nupdating how the feature is sold, with some comments indicating the\nfuture direction to extend the mechanism later.\n\nThanks.\n\n\n\n\n"},{"id":"478621","messageId":"CAP8UFD0GSTHJcagxhkKXZ5Lx7QMzpn3ve2XvvHDt6N=caM1DrQ@mail.gmail.com","threadId":"59867","inReplyTo":"xmqq1qicuwrw.fsf@gitster.g","subject":"Re: [PATCH 3/9] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-21T10:40:38Z","receivedAt":"2023-06-21T10:43:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jun 16, 2023 at 1:32 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > In a following commit, we will make it possible to separate objects in\n> > different packfiles depending on a filter.\n> >\n> > To make sure that the right objects are in the right packs, let's add a\n> > new test-tool that can display which packfile(s) a given object is in.\n>\n> This tool would be serviceable if we only are interested in checking\n> just a few objects, but if we were to check many objects, I have to\n> wonder if it would be more efficient to use show-index to dump the\n> list of objects per pack, which should be sorted by object name, so\n> it should be trivial to run \"comm\" with the list of objects you want\n> to check.\n\nI agree that this new tool is for checking just a few objects.\n\n> Or if you only are checking about a dozen or so, taking one or more\n> arguments from the command line and looping over them may also be\n> OK. The output format of course may have to be changed, if we were\n> to go that route, though.\n>\n> It really depends on the granularity at which this test helper wants\n> to work at, I think.\n\nYeah, in the previous commit implementing --print-filtered, we check\nthat all the objects that should be printed to stdout are indeed\nprinted. So later when git repack --filter=... is implemented by using\ngit pack-objects --print-filtered and piping the printed objects to a\nregular git pack-objects command, I don't think it's necessary to test\na lot of objects, as hopefully both pack-objects command are supposed\nto work well at that point.\n"},{"id":"478622","messageId":"ZJLVzaQKBrgCARAG@nand.local","threadId":"59867","inReplyTo":"20230614192541.1599256-2-christian.couder@gmail.com","subject":"Re: [PATCH 1/9] pack-objects: allow `--filter` without `--stdout`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T10:49:49Z","receivedAt":"2023-06-21T10:51:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 14, 2023 at 09:25:33PM +0200, Christian Couder wrote:\n> 9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\n> taught `git pack-objects` to use `--filter`, but required the use of\n> `--stdout` since a partial clone mechanism was not yet in place to\n> handle missing objects. Since then, changes like 9e27beaa23\n> (promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\n> and others added support to dynamically fetch objects that were missing.\n>\n> Even without a promisor remote, filtering out objects can also be useful\n> if we can put the filtered out objects in a separate pack, and in this\n> case it also makes sense for pack-objects to write the packfile directly\n> to an actual file rather than on stdout.\n>\n> Remove the `--stdout` requirement when using `--filter`, so that in a\n> follow-up commit, repack can pass `--filter` to pack-objects to omit\n> certain objects from the resulting packfile.\n\nMakes sense.\n\nIs there any situation in which using --stdout with --filter would be a\npotential foot-gun? I am not as familiar with the partial clone\nmechanism as others CC'd, so I have no idea one way or the other.\n\nIf it is unsafe in certain situations (or, at the very least, could\nproduce surprising behavior), it may be worthwhile to only allow\n`--filter= --stdout` with some kind of\n`--filter-to-stdout-is-ok` flag to indicate that the caller knows what\nthey are doing.\n\nPresumably 'git repack --filter' would pass such a flag later on in the\nseries.\n\n> Signed-off-by: John Cai \n> Signed-off-by: Christian Couder \n> ---\n> Documentation/git-pack-objects.txt | 4 ++--\n> builtin/pack-objects.c | 8 ++------\n> 2 files changed, 4 insertions(+), 8 deletions(-)\n\nShould there be a trivial test here? I'm thinking something on the order\nof writing a filtered pack to stdout and redirecting it to a file,\nmoving it into place, and then indexing the pack to make sure that we\ngot the expected set of objects.\n\nThanks,\nTaylor\n"},{"id":"478623","messageId":"ZJLWgfCNoqE6xNqq@nand.local","threadId":"59867","inReplyTo":"xmqq7cs4uyqe.fsf@gitster.g","subject":"Re: [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T10:52:49Z","receivedAt":"2023-06-21T10:54:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jun 15, 2023 at 03:50:17PM -0700, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n> > When using the `--filter=` option, `git pack-objects` will\n> > omit some objects from the resulting packfile(s) it produces. It could\n> > be useful to know about these omitted objects though.\n> >\n> > For example, we might want to write these objects into a separate\n> > packfile by piping them into another `git pack-object` process.\n> > Or we might want to check if these objects are available from a\n> > promisor remote.\n> >\n> > Anyway, this patch implements a simple way to let us know about these\n> > objects by simply printing their oid, one per line, on stdout when the\n> > new `--print-filtered` flag is passed.\n>\n> Makes sense. It is a bit sad that we have to accumulate everything\n> until the end at which time we have to dump the accumulated in bulk,\n> but that is a current limitation of list-objects-filter API and not\n> within the scope of this change. We may in the longer term want to\n> see if we can make the collection of filtered-out objects streamable\n> by replacing the .omits object array with a callback function, or do\n> something along that line.\n\nHmm. I think it is possible to use something like `git pack-objects`'s\n`--stdin-packs` mode to accomplish this without needing to keep track of\nthe set of discarded objects (i.e. those which don't match the filter).\n\nIIUC, the set of objects which don't match the filter is the same as the\nset of all objects in packs beforehand, differenced with the set of\nobjects that shows up in the pack containing objects which *do* match\nthe filter.\n\nIf you mark all of the \"before\" packs with `-` in the input to\n`--stdin-packs`, and then pass along the pack containing the filtered\nset without `-` (to indicate that the resulting pack should not contain\nany objects which appear in that pack), I think you would end up with\nthe set of non-matching objects.\n\nThanks,\nTaylor\n"},{"id":"478624","messageId":"ZJLW4TPDpxjkHS4h@nand.local","threadId":"59867","inReplyTo":"xmqq1qicuwrw.fsf@gitster.g","subject":"Re: [PATCH 3/9] t/helper: add 'find-pack' test-tool","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T10:54:25Z","receivedAt":"2023-06-21T10:55:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jun 15, 2023 at 04:32:35PM -0700, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n> > In a following commit, we will make it possible to separate objects in\n> > different packfiles depending on a filter.\n> >\n> > To make sure that the right objects are in the right packs, let's add a\n> > new test-tool that can display which packfile(s) a given object is in.\n>\n> This tool would be serviceable if we only are interested in checking\n> just a few objects, but if we were to check many objects, I have to\n> wonder if it would be more efficient to use show-index to dump the\n> list of objects per pack, which should be sorted by object name, so\n> it should be trivial to run \"comm\" with the list of objects you want\n> to check.\n\nI was going to say the exact same thing. Even if we were checking many\nobjects, can't we dump the output of show-index to a file, and then grep\nit repeatedly? Presumably these tests are working on repositories with\ntens of objects, so I doubt it matters much either way.\n\nIf we do end up taking this approach to use the test-helper instead, the\nimplementation seems reasonable.\n\nThanks,\nTaylor\n"},{"id":"478625","messageId":"ZJLXLgE07bapFs9W@nand.local","threadId":"59867","inReplyTo":"xmqqttv8thjw.fsf@gitster.g","subject":"Re: [PATCH 4/9] repack: refactor piping an oid to a command","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T10:55:42Z","receivedAt":"2023-06-21T10:56:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jun 15, 2023 at 04:46:43PM -0700, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n> > Create a new write_oid_hex_cmd() function to send an oid to the standard\n> > input of a running command. This new function will be used in a\n> > following commit.\n> >\n> > Signed-off-by: Christian Couder \n> > ---\n> > builtin/repack.c | 20 +++++++++++++-------\n> > 1 file changed, 13 insertions(+), 7 deletions(-)\n> >\n> > diff --git a/builtin/repack.c b/builtin/repack.c\n> > index 0541c3ce15..e591c295cf 100644\n> > --- a/builtin/repack.c\n> > +++ b/builtin/repack.c\n> > @@ -182,6 +182,17 @@ static void prepare_pack_objects(struct child_process *cmd,\n> > \tcmd->out = -1;\n> > }\n> >\n> > +static void write_oid_hex_cmd(const char *oid_hex,\n> > +\t\t\t struct child_process *cmd,\n> > +\t\t\t const char *err_msg)\n> > +{\n> > +\tif (cmd->in == -1 && start_command(cmd))\n> > +\t\tdie(\"%s\", err_msg);\n>\n> I am not sure why we would want to conflate the \"if we haven't\n> started the command, auto-start it upon our first attempt to write\"\n> in these low-level \"I am designed to do one thing, which is to feed\n> the object name to the process, and do it well\" function.\n\nI agree, the implementation of `write_oid_hex_cmd()` seems too magical\nto me.\n\nPerhaps there was some awkwardness with using the pre-image w.r.t some\nlater change? Let's see...\n\nThanks,\nTaylor\n"},{"id":"478626","messageId":"CAP8UFD1MGX8yZsNuKmax2J1ZmwfxmzfbgJ_pNrbar6X2Dar8+g@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqttv8thjw.fsf@gitster.g","subject":"Re: [PATCH 4/9] repack: refactor piping an oid to a command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-21T10:56:25Z","receivedAt":"2023-06-21T10:56:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jun 16, 2023 at 1:46 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > Create a new write_oid_hex_cmd() function to send an oid to the standard\n> > input of a running command. This new function will be used in a\n> > following commit.\n> >\n> > Signed-off-by: Christian Couder \n> > ---\n> > builtin/repack.c | 20 +++++++++++++-------\n> > 1 file changed, 13 insertions(+), 7 deletions(-)\n> >\n> > diff --git a/builtin/repack.c b/builtin/repack.c\n> > index 0541c3ce15..e591c295cf 100644\n> > --- a/builtin/repack.c\n> > +++ b/builtin/repack.c\n> > @@ -182,6 +182,17 @@ static void prepare_pack_objects(struct child_process *cmd,\n> > cmd->out = -1;\n> > }\n> >\n> > +static void write_oid_hex_cmd(const char *oid_hex,\n> > + struct child_process *cmd,\n> > + const char *err_msg)\n> > +{\n> > + if (cmd->in == -1 && start_command(cmd))\n> > + die(\"%s\", err_msg);\n>\n> I am not sure why we would want to conflate the \"if we haven't\n> started the command, auto-start it upon our first attempt to write\"\n> in these low-level \"I am designed to do one thing, which is to feed\n> the object name to the process, and do it well\" function.\n>\n> The caller in the original shares the same issue, so we could say\n> that this patch is not creating a new problem, but this somehow\n> feels it is mak ng the existing problem even worse.\n\nOk. I think I will rework this patch from version 2 of this series to\nremove that code. It will perhaps look like there is a bit of\nduplicated code, but I don't think it will be too bad.\n\n> And I think the error handling here shows why the API feels wrong.\n> When auto-start fails, we have a message, but when write fails,\n> there is no custom message---it makes as if write_oid_hex_cmd() is\n> primarily about starting, which is so important relative to its\n> other functionalities and deserves a custom error message, but that\n> is not the message you want to be conveying.\n\nRight.\n\n> > + xwrite(cmd->in, oid_hex, the_hash_algo->hexsz);\n> > + xwrite(cmd->in, \"\\n\", 1);\n>\n> I would have expected that the \"refactor\" at least would reduce the\n> number of system calls by combining these two writes into one using\n> an on-stack local variable char buf[GIT_MAX_HEZSZ+1] or something.\n\nOk, I will change it to reduce the number of system calls.\n"},{"id":"478627","messageId":"ZJLZYNCQ2ApwePAz@nand.local","threadId":"59867","inReplyTo":"20230614192541.1599256-6-christian.couder@gmail.com","subject":"Re: [PATCH 5/9] repack: refactor finishing pack-objects command","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T11:05:04Z","receivedAt":"2023-06-21T11:05:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"> diff --git a/builtin/repack.c b/builtin/repack.c\n> index e591c295cf..f1adacf1d0 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -703,6 +703,42 @@ static void remove_redundant_bitmaps(struct string_list *include,\n> \tstrbuf_release(&path);\n> }\n>\n> +static int finish_pack_objects_cmd(struct child_process *cmd,\n> +\t\t\t\t struct string_list *names,\n> +\t\t\t\t const char *destination)\n> +{\n> +\tint local = 1;\n> +\tFILE *out;\n> +\tstruct strbuf line = STRBUF_INIT;\n> +\n> +\tif (destination) {\n> +\t\tconst char *scratch;\n\nMaybe stick the declaration above (and consider making it static), so\nthat this can become\n\n if (destination)\n local = skip_prefix(destination, packdir, &scratch);\n\nwithout the braces. Although it might be nice to either put this behind\na \"has_prefix()\" convenience function (which itself owns the scratch\nbuffer and hides that detail from the caller), or to make skip_prefix\nskip trying to assign into the buffer if given NULL.\n\n> +\t\tlocal = skip_prefix(destination, packdir, &scratch);\n> +\t}\n> +\n> +\tout = xfdopen(cmd->out, \"r\");\n> +\twhile (strbuf_getline_lf(&line, out) != EOF) {\n> +\t\tstruct string_list_item *item;\n> +\n> +\t\tif (line.len != the_hash_algo->hexsz)\n> +\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n> +\t\t\t \"from pack-objects.\"));\n> +\t\t/*\n> +\t\t * Avoid putting packs written outside of the repository in the\n> +\t\t * list of names.\n> +\t\t */\n> +\t\tif (local) {\n\nConsider moving the declaration of item into this block, since it's not\nused elsewhere throughout the body of the loop.\n\nAlternatively, if you want to leave it up there, it might be easier to\nadjust the if condition to be more in line with the comment, like:\n\n if (!local)\n continue;\n\n> +\tret = finish_pack_objects_cmd(&cmd, &names, NULL);\n\nOK, we don't even bother calling skip_prefix if given a NULL\ndestination. I wonder if it might make sense to force the caller to\ncompute \"is this local?\" ahead of time. This one would always pass \"1\"\ntrivially, and the cruft case would depend on whether or not we are\nhandling the `--expire-to` option.\n\nThanks,\nTaylor\n"},{"id":"478628","messageId":"ZJLZnS1olivaQ+GH@nand.local","threadId":"59867","inReplyTo":"xmqqilbotgbp.fsf@gitster.g","subject":"Re: [PATCH 5/9] repack: refactor finishing pack-objects command","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T11:06:05Z","receivedAt":"2023-06-21T11:07:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jun 15, 2023 at 05:13:14PM -0700, Junio C Hamano wrote:\n> Computing \"is it local?\" based on the value of \"destination\" feels\n> it belongs to the caller (one of the callers that do need the\n> computation), not to this function, especially given that the full\n> value of \"destination\" is not even used in any other way in this\n> function. And the \"is_local?\" bit can instead be passesd into this\n> helper function as a parameter.\n\nHah. I had the same suggestion down-thread, but hadn't read your reply\nyet. There are either a couple of changes we could make to\nskip_prefix(), or foist the responsibility onto the caller (I tend to\nprefer the latter).\n\nThanks,\nTaylor\n"},{"id":"478629","messageId":"CAP8UFD1cAMg-54YTmW6N9sbi3KKs2OmPRz7zshWDGkStgiZnUQ@mail.gmail.com","threadId":"59867","inReplyTo":"ZJLWgfCNoqE6xNqq@nand.local","subject":"Re: [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-21T11:11:38Z","receivedAt":"2023-06-21T11:11:55Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 12:52 PM Taylor Blau wrote:\n>\n> On Thu, Jun 15, 2023 at 03:50:17PM -0700, Junio C Hamano wrote:\n\n> > Makes sense. It is a bit sad that we have to accumulate everything\n> > until the end at which time we have to dump the accumulated in bulk,\n> > but that is a current limitation of list-objects-filter API and not\n> > within the scope of this change. We may in the longer term want to\n> > see if we can make the collection of filtered-out objects streamable\n> > by replacing the .omits object array with a callback function, or do\n> > something along that line.\n>\n> Hmm. I think it is possible to use something like `git pack-objects`'s\n> `--stdin-packs` mode to accomplish this without needing to keep track of\n> the set of discarded objects (i.e. those which don't match the filter).\n>\n> IIUC, the set of objects which don't match the filter is the same as the\n> set of all objects in packs beforehand, differenced with the set of\n> objects that shows up in the pack containing objects which *do* match\n> the filter.\n>\n> If you mark all of the \"before\" packs with `-` in the input to\n> `--stdin-packs`, and then pass along the pack containing the filtered\n> set without `-` (to indicate that the resulting pack should not contain\n> any objects which appear in that pack), I think you would end up with\n> the set of non-matching objects.\n\nI agree that it can be done like this, but I am not sure it's very\nefficient to do it like this. When we create the pack with filtered\nout objects, we know the set of objects we filtered out, so it doesn't\nseem efficient to make `git pack-objects --stdin-packs` read more\npackfiles or their indexes than necessary and compute that set of\nobjects again.\n\nNow I haven't checked if there is a real performance difference for\nlarge packfiles, and perhaps `git pack-objects --stdin-packs` is very\nefficient. But I hope that going the way I implemented it and perhaps\nusing some optimization ideas that Junio suggested above, will make it\neasier to improve performance in the future.\n"},{"id":"478630","messageId":"ZJLcS78DDOjn6MzP@nand.local","threadId":"59867","inReplyTo":"20230614192541.1599256-7-christian.couder@gmail.com","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T11:17:31Z","receivedAt":"2023-06-21T11:17:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 14, 2023 at 09:25:38PM +0200, Christian Couder wrote:\n> ---\n> Documentation/git-repack.txt | 5 +++\n> builtin/repack.c | 75 ++++++++++++++++++++++++++++++++++--\n> t/t7700-repack.sh | 16 ++++++++\n> 3 files changed, 93 insertions(+), 3 deletions(-)\n\nHaving read through the implementation in the repack builtin, I am\nalmost certain that my suggestion earlier in the thread to implement\nthis in terms of 'git pack-objects --filter' and 'git pack-objects\n--stdin-packs' would work.\n\n> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> index 4017157949..aa29c7e648 100644\n> --- a/Documentation/git-repack.txt\n> +++ b/Documentation/git-repack.txt\n> @@ -143,6 +143,11 @@ depth is 4095.\n> \ta larger and slower repository; see the discussion in\n> \t`pack.packSizeLimit`.\n>\n> +--filter=::\n> +\tRemove objects matching the filter specification from the\n> +\tresulting packfile and put them into a separate packfile. See\n> +\tlinkgit:git-rev-list[1] for valid `` forms.\n> +\n\nThis documentation leaves me with a handful of questions about how it\ninteracts with other options. Here are some:\n\n - What happens when you pass it with \"-d\"? Does it delete objects that\n didn't match the filter? Leave them alone? If the latter, should\n this combination be declared invalid instead of silently ignoring\n the user's request to delete redundant packs?\n\n - What happens with --max-pack-size? Does the filtered pack get split\n into multiple packs (as I think we would expect from such a\n combination)?\n\n - What about with `--cruft`? Does it split the cruft pack into two\n based on whether or not the unreachable object(s) matched or didn't\n match the filter?\n\n - What happens when passed with \"--geometric\"? I don't think there is\n a sensible interpretation (at least, I can't think of what it would\n mean to do \"--filter= --geometric=\" off the top of my\n head).\n\n - What about with \"--write-bitmap-index\"? Do we write one bitmap\n index? Two? If the latter, do we combine the packs into a MIDX\n before writing the bitmap? Should we?\n\nI think it may be worth spelling out answers to some of these questions\nin the documentation, and codifying those answers in the form of tests.\n\nThis makes me wonder whether or not this option should belong in repack\nat all, or whether there should be some new special-purpose builtin that\nis designed to split existing pack(s) based on whether or not they meet\nsome filter criteria.\n\nThanks,\nTaylor\n"},{"id":"478631","messageId":"CAP8UFD0F-n51+4V7Bykb27Px-fUNqk5rpKr5rOj9Ex2tZZXPnA@mail.gmail.com","threadId":"59867","inReplyTo":"ZJLZnS1olivaQ+GH@nand.local","subject":"Re: [PATCH 5/9] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-21T11:19:04Z","receivedAt":"2023-06-21T11:20:00Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 1:06 PM Taylor Blau wrote:\n>\n> On Thu, Jun 15, 2023 at 05:13:14PM -0700, Junio C Hamano wrote:\n> > Computing \"is it local?\" based on the value of \"destination\" feels\n> > it belongs to the caller (one of the callers that do need the\n> > computation), not to this function, especially given that the full\n> > value of \"destination\" is not even used in any other way in this\n> > function. And the \"is_local?\" bit can instead be passesd into this\n> > helper function as a parameter.\n>\n> Hah. I had the same suggestion down-thread, but hadn't read your reply\n> yet. There are either a couple of changes we could make to\n> skip_prefix(), or foist the responsibility onto the caller (I tend to\n> prefer the latter).\n\nOk, I will make callers compute the \"is_local?\" bit and pass it to the\nhelper function as a parameter.\n"},{"id":"478632","messageId":"ZJLdBHNFPfxHp3C0@nand.local","threadId":"59867","inReplyTo":"xmqqmt10s0cw.fsf@gitster.g","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T11:20:36Z","receivedAt":"2023-06-21T11:21:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jun 15, 2023 at 05:43:27PM -0700, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n> > After cloning with --filter=, for example to avoid\n> > getting unneeded large files on a user machine, it's possible\n> > that some of these large files still get fetched for some reasons\n> > (like checking out old branches) over time.\n> >\n> > In this case the repo size could grow too much for no good reason and a\n> > way to filter out some objects would be useful to remove the unneeded\n> > large files.\n>\n> Makes sense.\n>\n> If we repack without these objects, when the repository has a\n> promisor remote, we should be able to rely on that remote to supply\n> them on demand, once we need them again, no?\n\nI think in theory, yes, but this patch series (at least up to this\npoint) does not seem to implement that functionality by marking the\nrelevant remote(s) as promisors, if they weren't already.\n\n> [...] It does smell somewhat similar to the cruft packs but not\n> really (the choice over there is between exploding to loose and\n> keeping in a pack, and never involves loss of objects).\n\nIndeed. `pack-objects`'s `--stdin-packs` and `--cruft` work similarly,\nand I believe that we could use `--stdin-packs` here instead of having\nto store the list of objects which don't meet the filter's spec. IOW, I\nthink that this similarity is no coincidence...\n\nThanks,\nTaylor\n"},{"id":"478634","messageId":"ZJLjvhnh0lSRCVzG@nand.local","threadId":"59867","inReplyTo":"20230614192541.1599256-9-christian.couder@gmail.com","subject":"Re: [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T11:49:18Z","receivedAt":"2023-06-21T11:49:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 14, 2023 at 09:25:40PM +0200, Christian Couder wrote:\n> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> index aa29c7e648..070dd22610 100644\n> --- a/Documentation/git-repack.txt\n> +++ b/Documentation/git-repack.txt\n> @@ -148,6 +148,12 @@ depth is 4095.\n> \tresulting packfile and put them into a separate packfile. See\n> \tlinkgit:git-rev-list[1] for valid `` forms.\n>\n> +--filter-to=::\n> +\tWrite the pack containing filtered out objects to the\n> +\tdirectory ``. This can be used for putting the pack on a\n> +\tseparate object directory that is accessed through the Git\n> +\talternates mechanism. Only useful with `--filter`.\n\nHere you say \"only useful with --filter\", but...\n\n> @@ -1073,8 +1077,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\tstrvec_push(&cmd.args, \"--incremental\");\n> \t}\n>\n> -\tif (po_args.filter)\n> -\t\tprepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n> +\tif (po_args.filter) {\n> +\t\tif (!filter_to)\n> +\t\t\tfilter_to = packtmp;\n> +\t\tprepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);\n> +\t}\n\nWould you want an \"} else if (filter_to)\" here to die and show the usage\nmessage, since --filter-to needs --filter? Or maybe it should imply\n--filter-to.\n\nThanks,\nTaylor\n"},{"id":"478635","messageId":"ZJLk+IMOHRrlpNho@nand.local","threadId":"59867","inReplyTo":"CAP8UFD1cAMg-54YTmW6N9sbi3KKs2OmPRz7zshWDGkStgiZnUQ@mail.gmail.com","subject":"Re: [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T11:54:32Z","receivedAt":"2023-06-21T11:54:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 21, 2023 at 01:11:38PM +0200, Christian Couder wrote:\n> On Wed, Jun 21, 2023 at 12:52 PM Taylor Blau wrote:\n> >\n> > On Thu, Jun 15, 2023 at 03:50:17PM -0700, Junio C Hamano wrote:\n>\n> > > Makes sense. It is a bit sad that we have to accumulate everything\n> > > until the end at which time we have to dump the accumulated in bulk,\n> > > but that is a current limitation of list-objects-filter API and not\n> > > within the scope of this change. We may in the longer term want to\n> > > see if we can make the collection of filtered-out objects streamable\n> > > by replacing the .omits object array with a callback function, or do\n> > > something along that line.\n> >\n> > Hmm. I think it is possible to use something like `git pack-objects`'s\n> > `--stdin-packs` mode to accomplish this without needing to keep track of\n> > the set of discarded objects (i.e. those which don't match the filter).\n> >\n> > IIUC, the set of objects which don't match the filter is the same as the\n> > set of all objects in packs beforehand, differenced with the set of\n> > objects that shows up in the pack containing objects which *do* match\n> > the filter.\n> >\n> > If you mark all of the \"before\" packs with `-` in the input to\n> > `--stdin-packs`, and then pass along the pack containing the filtered\n> > set without `-` (to indicate that the resulting pack should not contain\n> > any objects which appear in that pack), I think you would end up with\n> > the set of non-matching objects.\n>\n> I agree that it can be done like this, but I am not sure it's very\n> efficient to do it like this. When we create the pack with filtered\n> out objects, we know the set of objects we filtered out, so it doesn't\n> seem efficient to make `git pack-objects --stdin-packs` read more\n> packfiles or their indexes than necessary and compute that set of\n> objects again.\n\nDiscovering that an object appears in one of the packs whose objects\nwe're excluding from the resulting pack is extremely cheap, since we\nhave a cache of just those packs (see the call to\n`has_object_kept_pack()` in `want_found_object()`).\n\nI'd think that in small cases the performance is probably worse, though\nthe working set would be small enough for the differences to not matter\nthat much in absolute terms.\n\nIn cases with many objects, storing all of their OIDs in memory would be\na pain.\n\nThanks,\nTaylor\n"},{"id":"478637","messageId":"CAP8UFD388DOTqueok7fRF+=-+s_PNria1oB18AEed3gjdRrqJA@mail.gmail.com","threadId":"59867","inReplyTo":"ZJLjvhnh0lSRCVzG@nand.local","subject":"Re: [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-21T12:08:38Z","receivedAt":"2023-06-21T12:08:59Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 1:49 PM Taylor Blau wrote:\n>\n> On Wed, Jun 14, 2023 at 09:25:40PM +0200, Christian Couder wrote:\n\n> > +--filter-to=::\n> > + Write the pack containing filtered out objects to the\n> > + directory ``. This can be used for putting the pack on a\n> > + separate object directory that is accessed through the Git\n> > + alternates mechanism. Only useful with `--filter`.\n>\n> Here you say \"only useful with --filter\", but...\n>\n> > @@ -1073,8 +1077,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> > strvec_push(&cmd.args, \"--incremental\");\n> > }\n> >\n> > - if (po_args.filter)\n> > - prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n> > + if (po_args.filter) {\n> > + if (!filter_to)\n> > + filter_to = packtmp;\n> > + prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);\n> > + }\n>\n> Would you want an \"} else if (filter_to)\" here to die and show the usage\n> message, since --filter-to needs --filter? Or maybe it should imply\n> --filter-to.\n\nIn the doc for --expire-to= there is \"Only useful with `--cruft\n-d`\" and I don't think there is a check to see if --cruft and -d have\nbeen passed when --expire-to is passed. So I am not sure if it's\nbetter to be consistent with --expire-to or not.\n"},{"id":"478640","messageId":"ZJLsMK73ae8Htec4@nand.local","threadId":"59867","inReplyTo":"CAP8UFD388DOTqueok7fRF+=-+s_PNria1oB18AEed3gjdRrqJA@mail.gmail.com","subject":"Re: [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T12:25:20Z","receivedAt":"2023-06-21T12:25:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 21, 2023 at 02:08:38PM +0200, Christian Couder wrote:\n> > > @@ -1073,8 +1077,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> > > strvec_push(&cmd.args, \"--incremental\");\n> > > }\n> > >\n> > > - if (po_args.filter)\n> > > - prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n> > > + if (po_args.filter) {\n> > > + if (!filter_to)\n> > > + filter_to = packtmp;\n> > > + prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);\n> > > + }\n> >\n> > Would you want an \"} else if (filter_to)\" here to die and show the usage\n> > message, since --filter-to needs --filter? Or maybe it should imply\n> > --filter-to.\n>\n> In the doc for --expire-to= there is \"Only useful with `--cruft\n> -d`\" and I don't think there is a check to see if --cruft and -d have\n> been passed when --expire-to is passed. So I am not sure if it's\n> better to be consistent with --expire-to or not.\n\nTBH, I don't think that my decision at the time to silently accept\n--expire-to without --cruft was the right one. It should at least\nrequire --cruft, or imply it. It doesn't make a ton of sense to use\nwithout -d, but doing so is OK, so I wouldn't consider that a failing\ncondition.\n\nIn other words, I would be fine with something like:\n\n--- 8< ---\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 0541c3ce15..1890f283ee 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -866,6 +866,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE)))\n \t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"-A\");\n\n+\t/* --expire-to implies cruft */\n+\tif (expire_to)\n+\t\tpack_everything |= PACK_CRUFT;\n+\n \tif (pack_everything & PACK_CRUFT) {\n \t\tpack_everything |= ALL_INTO_ONE;\n\n--- >8 ---\n\nBut that sounds like a good candidate for some #leftoverbits.\n\nIn the meantime, I would be absolutely fine with deviating from the\nexisting behavior of --expire-to w.r.t --cruft.\n\nThanks,\nTaylor\n"},{"id":"478642","messageId":"CAP8UFD3864uUjb0vR+B7xETJTFJoWdEqA5Gdyr42Lg3t8Auk=Q@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqmt10s0cw.fsf@gitster.g","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-21T14:40:10Z","receivedAt":"2023-06-21T14:40:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jun 16, 2023 at 2:43 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > After cloning with --filter=, for example to avoid\n> > getting unneeded large files on a user machine, it's possible\n> > that some of these large files still get fetched for some reasons\n> > (like checking out old branches) over time.\n> >\n> > In this case the repo size could grow too much for no good reason and a\n> > way to filter out some objects would be useful to remove the unneeded\n> > large files.\n>\n> Makes sense.\n>\n> If we repack without these objects, when the repository has a\n> promisor remote, we should be able to rely on that remote to supply\n> them on demand, once we need them again, no?\n\nYeah, sure.\n\n> > Deleting objects right away could corrupt a repo though,...\n>\n> Hmph, could you elaborate why it is the case? Isn't it the whole\n> point to have promisor remote and use a lazy clone with the --filter\n> option, so that objects that _ought_ to exist from connectivity's\n> point of view _are_ allowed to be missing because the promisor\n> promises to make them available on-demand?\n>\n> Side note: I think I know the answer. While trying to remove\n> UNNEEDED large files, doing so may discard NEEDED large\n> files when done carelessly (e.g. the file may have been\n> created locally and haven't been pushed back).\n\nYeah, right.\n\n> But (1) if\n> that is the problem, perhaps we should be more careful in\n> the first place?\n\nYeah, but earlier when we implemented `repack --filter=...` that was\nremoving objects, saying that one should be very careful in the docs\nand implementing safeguards didn't seem to be safe enough for\nreviewers. Reviewers said that the feature would anyway provide a too\neasy way for users to shoot their own foot.\n\n> (2) if it inherently is impossible to tell\n> which ones are unneeded reliably, the reason why it is\n> impossible, and the reason why \"try sifting into two bins,\n> one that we _think_ are unneeded and another for the rest,\n> and verify what we _thought_ are unneeded are all available\n> from the promisor remote\" is the best we can do, must be\n> described, I think.\n\nYou mean described in the `repack --filter=` doc? Yeah, I can describe\nthis use case in the doc, but see below.\n\n> > ... so it might be\n> > better to put those objects into a separate packfile instead of\n> > deleting them. The separate pack could then be removed after checking\n> > that all the objects in it are still available on a promisor remote it\n> > can access.\n>\n> Surely, sifting the objects into two bins (i.e. those that we\n> wouldn't have received if we cloned from the promisor remote just\n> now, which are prunable, and those that we cannot lose because the\n> promisor remote would not have them, e.g. we created them and have\n> not pushed them to the remote yet) without removing anything would\n> be safe, but if the result of such sifting must be verified, doesn't\n> it indicate that the sifting step was buggy or misdesigned?\n\nIt might indicate that we prefer to be safe, do things in different\nsteps and not provide an easy way for users to shoot their own foot.\nFor example it seems pretty safe to do things like this:\n\n 1) put all the objects we think should be on the promisor remote in\na separate packfile\n 2) start checking that each object in that packfile is available on\nthe promisor remote\n 3) if an object in that packfile isn't on the promisor remote, try\nto send it there\n 4) if we couldn't send the object, error out\n 5) if we haven't errored out after checking all the objects in the\npackfile, it means all these objects are now available from the\npromisor remote and we can safely delete the packfile\n\nThe above steps can be done while new objects are created on the repo,\nor fetched, or pushed into the repo. And, at least for now, it would\nbe done by a custom script, so users writing and installing it should\nknow what they are doing and would hopefully not complain that we\nprovided an easy way for them to shoot their foot.\n\nIf we don't even document the above in the --filter=... doc, it makes\nit even less likely that they will do this and that their script might\nbe wrong. So even if I could document it in version 2, I am not sure I\nshould.\n\n> It does\n> not sound like a very good justification to save them in a separate\n> packfile. It does smell somewhat similar to the cruft packs but not\n> really (the choice over there is between exploding to loose and\n> keeping in a pack, and never involves loss of objects).\n\nIf we are still worried about possible loss of objects, I am Ok with\nnot talking at all about use cases involving possible loss of objects.\n\n> > Also splitting a packfile into 2 packs depending on a filter could be\n> > useful in other usecases. For example some large blobs might take a lot\n> > of precious space on fast storage while they are rarely accessed, and\n> > it could make sense to move them in a separate cheaper, though slower,\n> > storage.\n>\n> This one, outside the context of partial clone client, does make\n> tons of sense.\n\nOk, so perhaps it is enough to justify this feature and patch series.\nAnd I can just avoid talking about other use cases at all?\n\n> I guess what I suspect is that this option, while it would be very\n> useful for the \"in other usecases\" scenario above, may not become\n> all that useful in the \"our lazy clone got bloated and we want to\n> trim objects we know we can retrieve from the promisor remote again\n> if necessary\" scenario, until the repack machinery learns to use an\n> extra piece of information (namely \"these are objects that we can\n> fetch from the promisor remote\") at the same time.\n\nYeah, perhaps we should wait for a command or a repack option or some\nhelper scripts to be able to perform steps 2) to 4) or 2) to 5) above\nbefore talking about use cases involving a promisor remote.\n\nOn the other hand, it's possible to imagine other steps than the steps\n2) to 4) described above. For example, if we want to repack on a\nserver where new large blobs can hardly be created and where there is\na receive hook that automatically sends all the large blobs to a\npromisor remote as soon as they are received, we might not need steps\n3) and 4) to send objects to the promisor remote. Just checking that\nthey are on the promisor remote might be enough.\n\nAlso even if we think we should have features covering all the 5\nsteps, should we cover all the ways blobs could be sent to the\npromisor remote as part of step 3)? Some people or server platforms\nmight want to use git for that purpose, but others might prefer for\nexample FTP or plain HTTP(S) so that a transfer can be restarted if it\nfails.\n\nSo should we really wait until we have all possible such use cases\ncovered by some features or scripts, or not? When does it become Ok to\ntalk about this? And then how much is it Ok to talk about this?\n\n> > This commit implements a new `--filter=` option in\n> > `git repack` that moves filtered out objects into a separate pack.\n> >\n> > This is done by reading filtered out objects from `git pack-objects`'s\n> > output and piping them into a separate `git pack-objects` process that\n> > will put them into a separate packfile.\n>\n> So, for example, you may say \"no blobs\" in the filter, and while\n> packing the local repository with the filter, resulting in a pack\n> that exclude all blobs, we will learn what blob objects we did not\n> pack into that packfile. We can pack them into a separate one, and\n> most of the blobs are what we could retrieve again from the promisor\n> remote, but some of the blobs are what we locally created ourselves\n> and haven't pushed back to the promisor remote yet. Now what? My\n> earlier suspicion that this mechanism may not be all that useful for\n> the \"slim bloated lazy clone\" comes from that I cannot think of a\n> good answer to this \"Now what?\" question---my naive solution would\n> involve enumerating the objects in that \"separate packfile\" that is\n> a mixture of precious ones and expendable ones, and then learning\n> which ones are precious, and creating a new pack that is a subset of\n> that \"separate packfile\" with only the precious ones. But if I do\n> so, I do not think we need this new mechanism that seems to go only\n> the half-way.\n\nI hope the above 2) to 5) steps and related explanations are a good\nanswer to the \"Now what?\" question.\n\nThanks,\nChristian.\n"},{"id":"478643","messageId":"CAP8UFD0PY283_C3K8N1cE=Rc98W7CfMArD5kzSnO-6yBqM3sTw@mail.gmail.com","threadId":"59867","inReplyTo":"ZJLdBHNFPfxHp3C0@nand.local","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-21T15:04:48Z","receivedAt":"2023-06-21T15:10:41Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 1:20 PM Taylor Blau wrote:\n>\n> On Thu, Jun 15, 2023 at 05:43:27PM -0700, Junio C Hamano wrote:\n> > Christian Couder writes:\n> >\n> > > After cloning with --filter=, for example to avoid\n> > > getting unneeded large files on a user machine, it's possible\n> > > that some of these large files still get fetched for some reasons\n> > > (like checking out old branches) over time.\n> > >\n> > > In this case the repo size could grow too much for no good reason and a\n> > > way to filter out some objects would be useful to remove the unneeded\n> > > large files.\n> >\n> > Makes sense.\n> >\n> > If we repack without these objects, when the repository has a\n> > promisor remote, we should be able to rely on that remote to supply\n> > them on demand, once we need them again, no?\n>\n> I think in theory, yes, but this patch series (at least up to this\n> point) does not seem to implement that functionality by marking the\n> relevant remote(s) as promisors, if they weren't already.\n\nYeah, it's not part of this patch series to implement all the features\nthat could be useful in the case of promisor remotes. This patch\nseries only hopes to implement a `repack --filter=...` option that can\nhelp in a number of different use cases. I'm open to opinions about\nwhether or not the doc and commit messages should talk, and how much,\nabout use cases related to promisor remotes.\n\n> > [...] It does smell somewhat similar to the cruft packs but not\n> > really (the choice over there is between exploding to loose and\n> > keeping in a pack, and never involves loss of objects).\n>\n> Indeed. `pack-objects`'s `--stdin-packs` and `--cruft` work similarly,\n> and I believe that we could use `--stdin-packs` here instead of having\n> to store the list of objects which don't meet the filter's spec. IOW, I\n> think that this similarity is no coincidence...\n\nYeah, I agree that we could use `--stdin-packs` to implement `repack\n--filter=...`. I am just not sure it's the best path forward\nperformance wise in the long run. So others' opinions are welcome\nabout that.\n\nAlso, as Junio said, this patch series is not responsible for the fact\nthat traverse_commit_list_filtered() stores oids into an oidset\ninstead of using a callback function. Fixing this would likely avoid\naccumulating oids in memory. And creating a packfile by sending oids\ninto pack-objects is something that is already done by\nrepack_promisor_objects(). So even if `--filter=...` is not reusing\n`--stdin-packs`, it is still reusing a lot of existing mechanisms.\n\nThanks,\nChristian.\n"},{"id":"478647","messageId":"xmqqedm4ix3c.fsf@gitster.g","threadId":"59867","inReplyTo":"ZJLsMK73ae8Htec4@nand.local","subject":"Re: [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-21T16:44:39Z","receivedAt":"2023-06-21T16:45:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> In other words, I would be fine with something like:\n>\n> --- 8< ---\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 0541c3ce15..1890f283ee 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -866,6 +866,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE)))\n> \t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"-A\");\n>\n> +\t/* --expire-to implies cruft */\n> +\tif (expire_to)\n> +\t\tpack_everything |= PACK_CRUFT;\n> +\n> \tif (pack_everything & PACK_CRUFT) {\n> \t\tpack_everything |= ALL_INTO_ONE;\n>\n> --- >8 ---\n>\n> But that sounds like a good candidate for some #leftoverbits.\n\nIt does. Thanks.\n\n"},{"id":"478648","messageId":"xmqqy1kchi3l.fsf@gitster.g","threadId":"59867","inReplyTo":"CAP8UFD3864uUjb0vR+B7xETJTFJoWdEqA5Gdyr42Lg3t8Auk=Q@mail.gmail.com","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-21T16:53:50Z","receivedAt":"2023-06-21T16:53:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> It might indicate that we prefer to be safe, do things in different\n> steps and not provide an easy way for users to shoot their own foot.\n> For example it seems pretty safe to do things like this:\n>\n> 1) put all the objects we think should be on the promisor remote in\n> a separate packfile\n> 2) start checking that each object in that packfile is available on\n> the promisor remote\n> 3) if an object in that packfile isn't on the promisor remote, try\n> to send it there\n> 4) if we couldn't send the object, error out\n> 5) if we haven't errored out after checking all the objects in the\n> packfile, it means all these objects are now available from the\n> promisor remote and we can safely delete the packfile\n\nI may be missing something, but to me, the above sound more like a\ntail wagging the dog.\n\nInstead of saying \"while repacking, we'll create the new pack with\nthe objects that we suspect that we cannot re-fetch from the\npromisor (allowing false positives for safety), and store the rest\nin a backup pack (that can immediately be discarded)\", the above\nsays \"while repacking, we'll create the new pack with objects that\nmatch the filter, and store the rest to another pack\". But because\nthe object selection criteria used in the latter is not something\nwith practical/useful meaning, in other words, it does not exactly\nmatch what we want, we fill the gaps between what we want (i.e. sift\nthe objects into \"refetchable\" and \"other\" bins) and what we\nhappened to have implemented (i.e. sift the objects into \"match\nfilter\" and \"other\" bints) by sending the objects that we _should_\nhave included in the new pack (i.e. \"not refetchable\") to the\npromisor to make them refetchable.\n\nI do not know what to think about that. I do not think there is\neven a way to guarantee that the push done for 3) will always be\ntaken and still leave the resulting promisor usable (e.g. we can\nmake them connected by coming up with a random new ref to point\nthese \"we are sending these only because we failed to include them\nin the set of objects we should consider local\" objects, but then\nhow would we avoid bloating the refs at the promisor remote side\n(which now has become a \"dumping ground\", rather than holding the\nobjects needed for histories that project participants care about).\n\nAs an argument to salvage this series as (one of the possible\ningredients to) a solution to \"slim down a bloated lazy clone\"\nproblem, it sounds a bit weak.\n\nThanks.\n"},{"id":"478675","messageId":"CAP8UFD30Kx_vYfdV4ipPPXNVr76pKshjTUQGcJfkUvG_+KD3zQ@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqy1kchi3l.fsf@gitster.g","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-06-22T08:39:09Z","receivedAt":"2023-06-22T08:39:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 6:53 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > It might indicate that we prefer to be safe, do things in different\n> > steps and not provide an easy way for users to shoot their own foot.\n> > For example it seems pretty safe to do things like this:\n> >\n> > 1) put all the objects we think should be on the promisor remote in\n> > a separate packfile\n> > 2) start checking that each object in that packfile is available on\n> > the promisor remote\n> > 3) if an object in that packfile isn't on the promisor remote, try\n> > to send it there\n> > 4) if we couldn't send the object, error out\n> > 5) if we haven't errored out after checking all the objects in the\n> > packfile, it means all these objects are now available from the\n> > promisor remote and we can safely delete the packfile\n>\n> I may be missing something, but to me, the above sound more like a\n> tail wagging the dog.\n>\n> Instead of saying \"while repacking, we'll create the new pack with\n> the objects that we suspect that we cannot re-fetch from the\n> promisor (allowing false positives for safety), and store the rest\n> in a backup pack (that can immediately be discarded)\",\n\nThis might be a good idea, but what if users prefer to send to a\npromisor remote the objects that should be on that promisor remote as\nsoon as possible, instead of keeping them on the local machine where\nthey take up possibly valuable space for no good reason?\n\nMy point is that there are a lot of different strategies that people\noperating with a promisor remote could adopt, so it's better to\niteratively give them building blocks that can help them instead of\ntrying to find and implement right away the best solution for a\nspecial use case or for every use case.\n\n> the above\n> says \"while repacking, we'll create the new pack with objects that\n> match the filter, and store the rest to another pack\". But because\n> the object selection criteria used in the latter is not something\n> with practical/useful meaning, in other words, it does not exactly\n> match what we want,\n\n\"What we want\" depends on the strategy chosen to manage objects on\npromisor remotes and I am not sure that the strategy you mention is\nalways better than the example strategy I talked about. For some users\nit might be better for others it might not.\n\n> we fill the gaps between what we want (i.e. sift\n> the objects into \"refetchable\" and \"other\" bins) and what we\n> happened to have implemented (i.e. sift the objects into \"match\n> filter\" and \"other\" bints) by sending the objects that we _should_\n> have included in the new pack (i.e. \"not refetchable\") to the\n> promisor to make them refetchable.\n>\n> I do not know what to think about that. I do not think there is\n> even a way to guarantee that the push done for 3) will always be\n> taken and still leave the resulting promisor usable (e.g. we can\n> make them connected by coming up with a random new ref to point\n> these \"we are sending these only because we failed to include them\n> in the set of objects we should consider local\" objects, but then\n> how would we avoid bloating the refs at the promisor remote side\n> (which now has become a \"dumping ground\", rather than holding the\n> objects needed for histories that project participants care about).\n\nThere are some configurations where users never want to delete any git\nobject. In those cases it doesn't matter if the promisor remote is a\n\"dumping ground\". Users might just want a promisor remote to keep all\nthe large files that have ever been pushed into the repo, to save more\nprecious space on the machine hosting the regular repo.\n\nThere are configurations where users can have garanties that the push\ndone for 3) will work with a very high probability so that the example\nstrategy I talked about can work reliably enough.\n\nThe example strategy I talked about is just one example where having\nrepack --filter work like it does in this patch series can be useful\nand safe. I don't pretend that it is always the best strategy and that\nsome users might not prefer another better strategy for them. If\nthat's the case perhaps they can implement another different option\nthat just checks that filtered out objects are indeed available from a\npromisor remote and then just omit these objects from the resulting\npack. In fact I would have nothing against such an option, and I might\neven implement it myself one day (no promise though).\n\nRight now they have nearly no helpful command (except perhaps using\npack-objects directly), so even if this is not the best possible help\nin all use cases, I am just saying that this can be useful in _some_\ncases.\n\n> As an argument to salvage this series as (one of the possible\n> ingredients to) a solution to \"slim down a bloated lazy clone\"\n> problem, it sounds a bit weak.\n\nI don't quite agree, but anyway I will use this argument less in\nversion 2 of this patch series, and I will talk more about the\nargument that `repack --filter=...` allows users to put packfiles\ncontaining some kinds of objects (like large blobs or all the blobs)\non cheaper disks (when not using a promisor remote).\n\nThanks.\n"},{"id":"478680","messageId":"ZJQq64eOkOxh8Qyt@nand.local","threadId":"59867","inReplyTo":"CAP8UFD0PY283_C3K8N1cE=Rc98W7CfMArD5kzSnO-6yBqM3sTw@mail.gmail.com","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-22T11:05:15Z","receivedAt":"2023-06-22T11:05:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 21, 2023 at 05:04:48PM +0200, Christian Couder wrote:\n> On Wed, Jun 21, 2023 at 1:20 PM Taylor Blau wrote:\n> >\n> > On Thu, Jun 15, 2023 at 05:43:27PM -0700, Junio C Hamano wrote:\n> > > Christian Couder writes:\n> > >\n> > > > After cloning with --filter=, for example to avoid\n> > > > getting unneeded large files on a user machine, it's possible\n> > > > that some of these large files still get fetched for some reasons\n> > > > (like checking out old branches) over time.\n> > > >\n> > > > In this case the repo size could grow too much for no good reason and a\n> > > > way to filter out some objects would be useful to remove the unneeded\n> > > > large files.\n> > >\n> > > Makes sense.\n> > >\n> > > If we repack without these objects, when the repository has a\n> > > promisor remote, we should be able to rely on that remote to supply\n> > > them on demand, once we need them again, no?\n> >\n> > I think in theory, yes, but this patch series (at least up to this\n> > point) does not seem to implement that functionality by marking the\n> > relevant remote(s) as promisors, if they weren't already.\n>\n> Yeah, it's not part of this patch series to implement all the features\n> that could be useful in the case of promisor remotes. This patch\n> series only hopes to implement a `repack --filter=...` option that can\n> help in a number of different use cases. I'm open to opinions about\n> whether or not the doc and commit messages should talk, and how much,\n> about use cases related to promisor remotes.\n>\n> > > [...] It does smell somewhat similar to the cruft packs but not\n> > > really (the choice over there is between exploding to loose and\n> > > keeping in a pack, and never involves loss of objects).\n> >\n> > Indeed. `pack-objects`'s `--stdin-packs` and `--cruft` work similarly,\n> > and I believe that we could use `--stdin-packs` here instead of having\n> > to store the list of objects which don't meet the filter's spec. IOW, I\n> > think that this similarity is no coincidence...\n>\n> Yeah, I agree that we could use `--stdin-packs` to implement `repack\n> --filter=...`. I am just not sure it's the best path forward\n> performance wise in the long run. So others' opinions are welcome\n> about that.\n\nI think it would almost certainly have comparable performance in most\ncases, and significantly better performance in large repositories. IIUC,\nthe current system has to remember the OID of every object which did not\npass the filter, and then construct a pack containing just those\nobjects.\n\nIt would be nice from a memory-savings perspective to not have to\nremember these OIDs. But it also just seems error prone to me to do so:\nwhat if we lose an OID, or reorder the list?\n\nI dunno. I feel pretty strongly that implementing this in terms of:\n\n - Write a filtered pack.\n - Construct the list of existing packs (marked with '-') and the\n filtered pack.\n - Pass that as input to `git pack-objects --stdin-packs`\n - If '-d' given, delete any existing pack(s).\n\nThanks,\nTaylor\n"},{"id":"478693","messageId":"xmqqa5wrfivc.fsf@gitster.g","threadId":"59867","inReplyTo":"CAP8UFD30Kx_vYfdV4ipPPXNVr76pKshjTUQGcJfkUvG_+KD3zQ@mail.gmail.com","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-22T18:32:23Z","receivedAt":"2023-06-22T18:32:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> I may be missing something, but to me, the above sound more like a\n>> tail wagging the dog.\n> ...\n> This might be a good idea, but what if users prefer to send to a\n> promisor remote the objects that should be on that promisor remote as\n> soon as possible, instead of keeping them on the local machine where\n> they take up possibly valuable space for no good reason?\n> ...\n> There are some configurations where users never want to delete any git\n> object. In those cases it doesn't matter if the promisor remote is a\n> \"dumping ground\".\n\nWhen one says \"everything\" in these sentences, I doubt one\nnecessarily means \"everything\". A topic one works on will have\niterations that is never pushed out, a topic one started may not\neven get to the state that is pushable to the central server. But\nthe objects that need to support such a topic (and its historical\nversions in its reflog) would need to be retained until they are\nexpired.\n\nCertainly, by pushing even such objects, you can say \"here is a pack\nwith filter=blob:none, and because I sent every blob every time I\ncreate locally to the promisor, I can always refetch what is not in\nthem by definition\".\n\nBut is that a good use of everybody's resources? The key phrase in\nwhat I said was \"... and still leve the resulting promisor usable\".\n\nThe promisor remote is in the unfortunate and unenviable position\nthat it cannot garbage collect anything because there may be\nsomebody who is still depending on such an object nobody planned to\nuse, but there is no mechanism to let it find out which ones are in\nactive use (or if you added some recently that I am forgetting, it\nwould change the equation---please remind me if that is the case).\n\nSo I would imagine that it would be fairly high in the priority list\nof server operators and project leads to make sure their promisor\nremotes do not become a true \"dumping ground\".\n\nFor \"trim a bloated lazy clone\" problem, I suspect that you would\nneed to know what is currently re-fetchable from the promisor and\ndrop those objects from your local repository, and the computation\nof what is currently re-fetchable would certainly involve the filter\nspecification you had with the promisor. The remote-tracking\nbranches you have for the promisor would serve as the other source\nof input to perform the computation.\n\nFor \"partition local and complete object store\" problem, using\nfilter specification to sift the objects into two bins (those that\nmatch and the rest), as the code changes in the series implements,\nmay be a useful mechansim. I briefly had to wonder if partitioning\ninto two (and not arbitrary number N) bins is sufficient, but did\nnot think of a scenario where we would benefit from 3 bins more than\nhaving 2 bins offhand.\n\nThanks.\n"},{"id":"479182","messageId":"20230705060812.2865188-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v2 1/8] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:05Z","receivedAt":"2023-07-05T06:08:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n t/t5317-pack-objects-filter-objects.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex a9995a932c..583270a85f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -298,8 +298,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3c4db66478..614721684a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4388,12 +4388,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..2ff3eef9a3 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n \t! grep blob verify_result\n '\n \n+test_expect_success 'verify blob:none packfile without --stdout' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n+\t! grep blob verify_result\n+'\n+\n test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n \tgit -C r1 verify-pack -v ../all.pack >verify_result &&\n \tgrep -E \"commit|tree\" verify_result |\n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479183","messageId":"20230705060812.2865188-1-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230614192541.1599256-1-christian.couder@gmail.com","subject":"[PATCH v2 0/8] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:04Z","receivedAt":"2023-07-05T06:08:46Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"# Intro\n\nLast year, John Cai sent 2 versions of a patch series to implement\n`git repack --filter=` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\n# Use cases for the new feature\n\nThis could be useful for example for the following purposes:\n\n 1) As a way for servers to save storage costs by for example moving\n large blobs, or all the blobs, or all the blobs in inactive\n repos, to separate storage (while still making them accessible\n using for example the alternates mechanism).\n\n 2) As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n 3) As a way for clients to reclaim some space when they cloned with\n a filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a\n separate directory or storage, then check that everything works\n well, and then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\nAlso since this version 2 of this new series, commit messages, don't\nmention uses cases like 2) or 3) above, as people have different\nopinions on how it should be done. How it should be done could depend\na lot on the way promisor remotes are used, the software and hardware\nsetups used, etc, so it seems more difficult to \"sell\" this series by\ntalking about such use cases. As use case 1) seems simpler and more\nappealing, it makes more sense to only talk about it in the commit\nmessages.\n\n# Changes since version 1\n\nThanks to Junio and Taylor who reviewed version 1! The changes are the\nfollowing:\n\n- I think that in the long run it might have been better for\n performance reasons to implement the `--filter=...` in `git repack`\n the way it was done in version 1.\n\n (It was done by first implementing `git pack-objects\n --print-filtered` to print objects omitted by a `--filter=...`\n option, then having `git repack --filter=...` launch such a command,\n read filtered out objects from it, and pipe them into a separate\n `git pack-objects` process that would put them into a separate\n packfile.)\n\n Anyway our list-objects-filter API is currently not well suited for\n that as it doesn't have a way to output the oids of the filtered out\n objects while the filtering is happening. So version 1 had to get\n them from an oid_set afterwards, which could use a lot of memory.\n\n As I don't want to have to work on improving the list-objects-filter\n API right now, I think it's better for now to just do as Taylor\n suggested, and as what the cruft code is already doing, which is to\n use the `--stdin-pack` option of `git pack-objects` and to pass it\n packfile names, some prefixed with '^', to let it compute what\n should be in the packfile(s) containing the filtered out objects and\n then to create that(/those) packfile(s).\n\n So patch 5/8 in version 2 which implements the\n `--filter=` option in `git repack` is very different\n from patch 6/9 which implemented it in version 1.\n\n- By doing it this way, this version gets rid of patch 2/9 in version\n 1 which added the `--print-filtered` option to pack-objects.\n\n- We also get rid of patch 4/9 in version 1 which refactored\n piping an oid to a command.\n\n- On the other hand, we can refactor the code which finds the pack\n prefix into a new small function as this will be needed by the new\n code implementing `--filter=`, so patch 4/8 in version\n 2 is new.\n\n- As suggested by Taylor, a small test has been added to verify that\n `--filter=blob:none` can work without `--stdout` in patch 1/8\n (previously 1/9).\n\n- In patch 3/8 (previously 5/9) which refactors the code to finish a\n pack-objects command into a new function, the `is_local` bit is not\n computed inside the new function anymore as suggested by both Junio\n and Taylor.\n\n- The commit messages of patch 5/8 (previously 6/9) and patch 7/8\n (previously 8/9) that implement the `--filter=` and\n `--filter-to=` options respectively have been changed to not\n talk about use cases related to promisor remotes as explained\n towards the end of the \"Use cases for the new feature\" section\n above.\n\n- Also in patch 5/8 (previously 6/9) that implements the\n `--filter=`option, the documentation for this section\n has been improved, suggesting to use it on bare repos along with -a\n and -d to get the best possible filtering.\n\n- And in patch 7/8 (previously 8/9) that implements the\n `--filter-to=` option, a new test has been added to check that\n `--filter=` and `--filter-to=` work well with\n the `--max-pack-size=...` option as suggested by Taylor.\n\n- Also in patch 7/8 (previously 8/9) we now check that\n `--filter=` has been passed and error out if not, as it\n doesn't make sense to use `--filter-to=` without\n `--filter=`. This was suggested by Taylor.\n\n- To avoid small merge conflicts in the `git gc` doc, this series has\n been rebased onto 9748a68200 (The sixth batch, 2023-06-29).\n\n# Commit overview\n\n* 1/8 pack-objects: allow `--filter` without `--stdout`\n\n This patch is the same as the first patch in the previous series and\n in v1. To be able to later repack with a filter we need `git\n pack-objects` to write packfiles when it's filtering instead of just\n writing the pack without the filtered out objects to stdout.\n\n* 2/8 t/helper: add 'find-pack' test-tool\n\n For testing `git repack --filter=...` that we are going to\n implement, it's useful to have a test helper that can tell which\n packfiles contain a specific object. No change in this patch\n compared to v1.\n\n* 3/8 repack: refactor finishing pack-objects command\n\n This is a small refactoring creating a new useful function, so that\n `git repack --filter=...` will be able to reuse it. The change\n compared to v1 is that the `is_local` bit is not computed inside the\n new function anymore.\n\n* 4/8 repack: refactor finding pack prefix\n\n This is a new patch with a small refactoring creating a small\n function that will be reused in the next patch.\n\n* 5/8 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. As explained\n above it works differently than in v1. It now uses one `git\n pack-objects` process with the `--filter` option. And then another\n `git pack-objects` process with the `--stdin-packs` option. Also the\n documentation of the new option has been improved compared to v1.\n\n* 6/8 gc: add `gc.repackFilter` config option\n\n This is a gc config option so that `git gc` can also repack using a\n filter and put the filtered out objects into a separate packfile. No\n changes compared to v1.\n\n* 7/8 repack: implement `--filter-to` for storing filtered out objects\n\n For some use cases, it's interesting to create the packfile that\n contains the filtered out objects into a separate location. This is\n similar to the `--expire-to` option for cruft packfiles. Since\n version 1 we now check that `--filter=` has been passed\n as using `--filter-to` without it doesn't make sense. Also a new\n test has been added to check that these options work well with\n `--max-pack-size=...`.\n\n* 8/8 gc: add `gc.repackFilterTo` config option\n\n This allows specifying the location of the packfile that contains\n the filtered out objects when using `gc.repackFilter`. No change\n since v1.\n\n# Range-diff\n\n 1: f4e1cc24d2 ! 1: 0bd1ad3071 pack-objects: allow `--filter` without `--stdout`\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const\n \n if (stdin_packs && use_internal_rev_list)\n die(_(\"cannot use internal rev list with --stdin-packs\"));\n +\n + ## t/t5317-pack-objects-filter-objects.sh ##\n +@@ t/t5317-pack-objects-filter-objects.sh: test_expect_success 'verify blob:none packfile has no blobs' '\n + ! grep blob verify_result\n + '\n + \n ++test_expect_success 'verify blob:none packfile without --stdout' '\n ++ git -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n ++ HEAD\n ++ EOF\n ++ git -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n ++ ! grep blob verify_result\n ++'\n ++\n + test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n + git -C r1 verify-pack -v ../all.pack >verify_result &&\n + grep -E \"commit|tree\" verify_result |\n 2: 8cf3db088e < -: ---------- pack-objects: add `--print-filtered` to print omitted objects\n 3: 2f3b16281c = 2: e49cd723c7 t/helper: add 'find-pack' test-tool\n 4: 0021a5e3bb < -: ---------- repack: refactor piping an oid to a command\n 5: dce5087cc3 ! 3: 3f87772ea6 repack: refactor finishing pack-objects command\n @@ builtin/repack.c: static void remove_redundant_bitmaps(struct string_list *inclu\n \n +static int finish_pack_objects_cmd(struct child_process *cmd,\n + struct string_list *names,\n -+ const char *destination)\n ++ int local)\n +{\n -+ int local = 1;\n + FILE *out;\n + struct strbuf line = STRBUF_INIT;\n +\n -+ if (destination) {\n -+ const char *scratch;\n -+ local = skip_prefix(destination, packdir, &scratch);\n -+ }\n -+\n + out = xfdopen(cmd->out, \"r\");\n + while (strbuf_getline_lf(&line, out) != EOF) {\n + struct string_list_item *item;\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *ar\n - FILE *in, *out;\n + FILE *in;\n int ret;\n -- const char *scratch;\n -- int local = skip_prefix(destination, packdir, &scratch);\n - \n - prepare_pack_objects(&cmd, args, destination);\n - \n + const char *scratch;\n + int local = skip_prefix(destination, packdir, &scratch);\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *args,\n fprintf(in, \"%s.pack\\n\", item->string);\n fclose(in);\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *ar\n - strbuf_release(&line);\n -\n - return finish_command(&cmd);\n -+ return finish_pack_objects_cmd(&cmd, names, destination);\n ++ return finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n - strbuf_release(&line);\n - fclose(out);\n - ret = finish_command(&cmd);\n -+ ret = finish_pack_objects_cmd(&cmd, &names, NULL);\n ++ ret = finish_pack_objects_cmd(&cmd, &names, 1);\n if (ret)\n goto cleanup;\n \n 6: fedde52ca1 < -: ---------- repack: add `--filter=` option\n -: ---------- > 4: 9997efaf33 repack: refactor finding pack prefix\n -: ---------- > 5: da27ecb91b repack: add `--filter=` option\n 7: 6ebd274334 ! 6: 49e4a184b4 gc: add `gc.repackFilter` config option\n @@ Commit message\n Signed-off-by: Christian Couder \n \n ## Documentation/config/gc.txt ##\n -@@ Documentation/config/gc.txt: or rebase occurring. Since these changes are not part of the current\n - project most users will want to expire them sooner, which is why the\n - default is more aggressive than `gc.reflogExpire`.\n +@@ Documentation/config/gc.txt: Multiple hooks are supported, but all must exit successfully, else the\n + operation (either generating a cruft pack or unpacking unreachable\n + objects) will be halted.\n \n +gc.repackFilter::\n + When repacking, use the specified filter to move certain\n 8: 5d68501b1f ! 7: 243c93aad3 repack: implement `--filter-to` for storing filtered out objects\n @@ Commit message\n accessible if, for example, the Git alternates mechanism is used to\n point to it.\n \n - If users want to remove a pack that contains filtered out objects after\n - checking that they are all already on a promisor remote, creating the\n - pack in a different directory makes it easier to do so.\n + While at it, as an example to show that `--filter` and `--filter-to`\n + work well with other options, let's also add a test to check that these\n + options work well with `--max-pack-size`.\n \n Signed-off-by: Christian Couder \n \n + repack: add test with --max-pack-size\n +\n ## Documentation/git-repack.txt ##\n @@ Documentation/git-repack.txt: depth is 4095.\n - resulting packfile and put them into a separate packfile. See\n - linkgit:git-rev-list[1] for valid `` forms.\n + this option. See linkgit:git-rev-list[1] for valid\n + `` forms.\n \n +--filter-to=::\n + Write the pack containing filtered out objects to the\n @@ Documentation/git-repack.txt: depth is 4095.\n Write a reachability bitmap index as part of the repack. This\n \n ## builtin/repack.c ##\n -@@ builtin/repack.c: static void prepare_pack_filtered_cmd(struct child_process *cmd,\n - }\n - \n - static void finish_pack_filtered_cmd(struct child_process *cmd,\n -- struct string_list *names)\n -+ struct string_list *names,\n -+ const char *destination)\n - {\n - if (cmd->in == -1) {\n - /* No packed objects; cmd was never started */\n -@@ builtin/repack.c: static void finish_pack_filtered_cmd(struct child_process *cmd,\n - \n - close(cmd->in);\n - \n -- if (finish_pack_objects_cmd(cmd, names, NULL, NULL))\n -+ if (finish_pack_objects_cmd(cmd, names, destination, NULL))\n - die(_(\"could not finish pack-objects to pack filtered objects\"));\n - }\n - \n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n + int write_midx = 0;\n const char *cruft_expiration = NULL;\n const char *expire_to = NULL;\n - struct child_process pack_filtered_cmd = CHILD_PROCESS_INIT;\n + const char *filter_to = NULL;\n \n struct option builtin_repack_options[] = {\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n strvec_push(&cmd.args, \"--incremental\");\n }\n \n -- if (po_args.filter)\n -- prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n -+ if (po_args.filter) {\n -+ if (!filter_to)\n -+ filter_to = packtmp;\n -+ prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);\n -+ }\n - \n ++ if (filter_to && !po_args.filter)\n ++ die(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n ++\n if (geometry)\n cmd.in = -1;\n + else\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n }\n \n - if (po_args.filter)\n -- finish_pack_filtered_cmd(&pack_filtered_cmd, &names);\n -+ finish_pack_filtered_cmd(&pack_filtered_cmd, &names, filter_to);\n - \n - string_list_sort(&names);\n - \n + if (po_args.filter) {\n ++ if (!filter_to)\n ++ filter_to = packtmp;\n ++\n + ret = write_filtered_pack(&po_args,\n +- packtmp,\n ++ filter_to,\n + find_pack_prefix(),\n + &names,\n + &existing_nonkept_packs,\n \n ## t/t7700-repack.sh ##\n @@ t/t7700-repack.sh: test_expect_success 'repacking with a filter works' '\n @@ t/t7700-repack.sh: test_expect_success 'repacking with a filter works' '\n + blob_content=$(git -C bare.git show $blob_hash) &&\n + test \"$blob_content\" = \"content1\"\n +'\n ++\n ++test_expect_success '--filter works with --max-pack-size' '\n ++ rm -rf filtered.git &&\n ++ git init --bare filtered.git &&\n ++ git init max-pack-size &&\n ++ (\n ++ cd max-pack-size &&\n ++ test_commit base &&\n ++ # two blobs which exceed the maximum pack size\n ++ test-tool genrandom foo 1048576 >foo &&\n ++ git hash-object -w foo &&\n ++ test-tool genrandom bar 1048576 >bar &&\n ++ git hash-object -w bar &&\n ++ git add foo bar &&\n ++ git commit -m \"adding foo and bar\"\n ++ ) &&\n ++ git clone --no-local --bare max-pack-size max-pack-size.git &&\n ++ (\n ++ cd max-pack-size.git &&\n ++ git -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n ++ --max-pack-size=1M \\\n ++ --filter-to=../filtered.git/objects/pack/pack &&\n ++ echo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n ++\n ++ # Check that the 3 blobs are in different packfiles in filtered.git\n ++ test_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n ++ test_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n ++ foo_pack=$(test-tool find-pack HEAD:foo) &&\n ++ bar_pack=$(test-tool find-pack HEAD:bar) &&\n ++ base_pack=$(test-tool find-pack HEAD:base.t) &&\n ++ test \"$foo_pack\" != \"$bar_pack\" &&\n ++ test \"$foo_pack\" != \"$base_pack\" &&\n ++ test \"$bar_pack\" != \"$base_pack\" &&\n ++ for pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n ++ do\n ++ case \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n ++ done\n ++ )\n ++'\n +\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n 9: ae45d9845e = 8: 8cb3faa74c gc: add `gc.repackFilterTo` config option\n\n\nChristian Couder (8):\n pack-objects: allow `--filter` without `--stdout`\n t/helper: add 'find-pack' test-tool\n repack: refactor finishing pack-objects command\n repack: refactor finding pack prefix\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 11 ++\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 15 +++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 162 ++++++++++++++++++-------\n t/helper/test-find-pack.c | 35 ++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5317-pack-objects-filter-objects.sh | 8 ++\n t/t6500-gc.sh | 23 ++++\n t/t7700-repack.sh | 82 +++++++++++++\n 13 files changed, 311 insertions(+), 50 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n\n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479184","messageId":"20230705060812.2865188-3-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v2 2/8] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:06Z","receivedAt":"2023-07-05T06:08:50Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In a following commit, we will make it possible to separate objects in\ndifferent packfiles depending on a filter.\n\nTo make sure that the right objects are in the right packs, let's add a\nnew test-tool that can display which packfile(s) a given object is in.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 35 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 38 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n\ndiff --git a/Makefile b/Makefile\nindex fb541dedc9..14ee0c45d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..1928fe7329\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ */\n+\n+static const char *find_pack_usage = \"\\n\"\n+\" test-tool find-pack \";\n+\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\n+\tsetup_git_directory();\n+\n+\tif (argc != 2)\n+\t\tusage(find_pack_usage);\n+\n+\tif (repo_get_oid(the_repository, argv[1], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[1]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tif (find_pack_entry_one(oid.hash, p))\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex abe8a785eb..41da40c296 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ea2672436c..411dbf2db4 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479185","messageId":"20230705060812.2865188-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v2 3/8] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:07Z","receivedAt":"2023-07-05T06:08:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -713,9 +743,8 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct string_list *existing_kept_packs)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n \tconst char *scratch;\n \tint local = skip_prefix(destination, packdir, &scratch);\n@@ -759,27 +788,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -790,10 +799,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1024,18 +1031,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479186","messageId":"20230705060812.2865188-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v2 4/8] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:08Z","receivedAt":"2023-07-05T06:08:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new find_pack_prefix() to refactor code that handles finding\nthe pack prefix from the packtmp and packdir global variables, as we are\ngoing to need this feature again in following commit.\n\nSigned-off-by: Christian Couder ` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:09Z","receivedAt":"2023-07-05T06:08:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new option puts the objects specified by `` into a\nseparate packfile.\n\nThis could be useful if, for example, some large blobs take a lot of\nprecious space on fast storage while they are rarely accessed. It could\nmake sense to move them into a separate cheaper, though slower, storage.\n\nIn other use cases it might make sense to put all the blobs into\nseparate storage.\n\nThis is done by running two `git pack-objects` commands. The first one\nis run with `--filter=`, using the specified filter. It\npacks objects while omitting the objects specified by the filter.\nThen another `git pack-objects` command is launched using\n`--stdin-packs`. We pass it all the previously existing packs into its\nstdin, so that it will pack all the objects in the previously existing\npacks. But we also pass into its stdin, the pack created by the previous\n`git pack-objects --filter=` command as well as the kept\npacks, all prefixed with '^', so that the objects in these packs will be\nomitted from the resulting pack. The result is that only the objects\nfiltered out by the first `git pack-objects` command are in the pack\nresulting from the second `git pack-objects` command.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 9 +++++\n builtin/repack.c | 67 ++++++++++++++++++++++++++++++++++++\n t/t7700-repack.sh | 16 +++++++++\n 3 files changed, 92 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..d702553033 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,15 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. Note\n+\tthat objects used in the working directory are not filtered\n+\tout. So for the split to fully work, it's best to perform it\n+\tin a bare repo and to use the `-a` and `-d` options along with\n+\tthis option. See linkgit:git-rev-list[1] for valid\n+\t`` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 4e5afee8d8..e2661b956c 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -54,6 +54,7 @@ struct pack_objects_args {\n \tconst char *depth;\n \tconst char *threads;\n \tconst char *max_pack_size;\n+\tconst char *filter;\n \tint no_reuse_delta;\n \tint no_reuse_object;\n \tint quiet;\n@@ -174,6 +175,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--threads=%s\", args->threads);\n \tif (args->max_pack_size)\n \t\tstrvec_pushf(&cmd->args, \"--max-pack-size=%s\", args->max_pack_size);\n+\tif (args->filter)\n+\t\tstrvec_pushf(&cmd->args, \"--filter=%s\", args->filter);\n \tif (args->no_reuse_delta)\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n \tif (args->no_reuse_object)\n@@ -734,6 +737,57 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n+static int write_filtered_pack(const struct pack_objects_args *args,\n+\t\t\t const char *destination,\n+\t\t\t const char *pack_prefix,\n+\t\t\t struct string_list *names,\n+\t\t\t struct string_list *existing_packs,\n+\t\t\t struct string_list *existing_kept_packs)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\tconst char *scratch;\n+\tint local = skip_prefix(destination, packdir, &scratch);\n+\n+\t/* We need to copy 'args' to modify it */\n+\tstruct pack_objects_args new_args = *args;\n+\n+\t/* No need to filter again */\n+\tnew_args.filter = NULL;\n+\n+\tprepare_pack_objects(&cmd, &new_args, destination);\n+\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\n+\tcmd.in = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * names has a confusing double use: it both provides the list\n+\t * of just-written new packs, and accepts the name of the\n+\t * filtered pack we are writing.\n+\t *\n+\t * By the time it is read here, it contains only the pack(s)\n+\t * that were just written, which is exactly the set of packs we\n+\t * want to consider kept.\n+\t */\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, names)\n+\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n+\tfor_each_string_list_item(item, existing_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tfor_each_string_list_item(item, existing_kept_packs)\n+\t\tfprintf(in, \"^%s.pack\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -866,6 +920,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_STRING(0, \"filter\", &po_args.filter, N_(\"args\"),\n+\t\t\t\tN_(\"object filtering\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -1105,6 +1161,17 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter) {\n+\t\tret = write_filtered_pack(&po_args,\n+\t\t\t\t\t packtmp,\n+\t\t\t\t\t find_pack_prefix(),\n+\t\t\t\t\t &names,\n+\t\t\t\t\t &existing_nonkept_packs,\n+\t\t\t\t\t &existing_kept_packs);\n+\t\tif (ret)\n+\t\t\tgoto cleanup;\n+\t}\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex af79266c58..66589e4217 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -293,6 +293,22 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n+\ttest -n \"$commit_pack\" &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -n \"$blob_pack\" &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479188","messageId":"20230705060812.2865188-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v2 6/8] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:10Z","receivedAt":"2023-07-05T06:09:00Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 12 ++++++++++++\n 3 files changed, 23 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex ca47eb2008..2153bde7ac 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the\n operation (either generating a cruft pack or unpacking unreachable\n objects) will be halted.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 91eec7703a..046147fdcc 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -171,6 +172,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -356,6 +359,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..5b89faf505 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,18 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none -c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479189","messageId":"20230705060812.2865188-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v2 8/8] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:12Z","receivedAt":"2023-07-05T06:09:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 6 ++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 2153bde7ac..0e32007502 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -150,6 +150,12 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. See the\n+\t`--filter-to=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 046147fdcc..f31167be6d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -63,6 +63,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -173,6 +174,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -362,6 +364,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 5b89faf505..37056a824b 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -214,6 +213,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479190","messageId":"20230705060812.2865188-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:08:11Z","receivedAt":"2023-07-05T06:09:05Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage. Even in a different directory this pack can be\naccessible if, for example, the Git alternates mechanism is used to\npoint to it.\n\nWhile at it, as an example to show that `--filter` and `--filter-to`\nwork well with other options, let's also add a test to check that these\noptions work well with `--max-pack-size`.\n\nSigned-off-by: Christian Couder \n\nrepack: add test with --max-pack-size\n---\n Documentation/git-repack.txt | 6 ++++\n builtin/repack.c | 11 +++++-\n t/t7700-repack.sh | 66 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex d702553033..396a91b9ac 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -152,6 +152,12 @@ depth is 4095.\n \tthis option. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. This can be used for putting the pack on a\n+\tseparate object directory that is accessed through the Git\n+\talternates mechanism. Only useful with `--filter`.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex e2661b956c..5695f9734d 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -879,6 +879,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -932,6 +933,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1075,6 +1078,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (filter_to && !po_args.filter)\n+\t\tdie(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n+\n \tif (geometry)\n \t\tcmd.in = -1;\n \telse\n@@ -1162,8 +1168,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\n \t\tret = write_filtered_pack(&po_args,\n-\t\t\t\t\t packtmp,\n+\t\t\t\t\t filter_to,\n \t\t\t\t\t find_pack_prefix(),\n \t\t\t\t\t &names,\n \t\t\t\t\t &existing_nonkept_packs,\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 66589e4217..a96c1635b2 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -309,6 +309,72 @@ test_expect_success 'repacking with a filter works' '\n \ttest \"$blob_pack2\" = \"$blob_pack\"\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n+\ttest -n \"$commit_pack\" &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -z \"$blob_pack\" &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack $blob_hash) &&\n+\ttest -n \"$blob_pack\" &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -n \"$blob_pack\" &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n+test_expect_success '--filter works with --max-pack-size' '\n+\trm -rf filtered.git &&\n+\tgit init --bare filtered.git &&\n+\tgit init max-pack-size &&\n+\t(\n+\t\tcd max-pack-size &&\n+\t\ttest_commit base &&\n+\t\t# two blobs which exceed the maximum pack size\n+\t\ttest-tool genrandom foo 1048576 >foo &&\n+\t\tgit hash-object -w foo &&\n+\t\ttest-tool genrandom bar 1048576 >bar &&\n+\t\tgit hash-object -w bar &&\n+\t\tgit add foo bar &&\n+\t\tgit commit -m \"adding foo and bar\"\n+\t) &&\n+\tgit clone --no-local --bare max-pack-size max-pack-size.git &&\n+\t(\n+\t\tcd max-pack-size.git &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--max-pack-size=1M \\\n+\t\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\t\techo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n+\n+\t\t# Check that the 3 blobs are in different packfiles in filtered.git\n+\t\ttest_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n+\t\tfoo_pack=$(test-tool find-pack HEAD:foo) &&\n+\t\tbar_pack=$(test-tool find-pack HEAD:bar) &&\n+\t\tbase_pack=$(test-tool find-pack HEAD:base.t) &&\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" != \"$base_pack\" &&\n+\t\ttest \"$bar_pack\" != \"$base_pack\" &&\n+\t\tfor pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n+\t\tdo\n+\t\t\tcase \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n+\t\tdone\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.41.0.244.g8cb3faa74c\n\n"},{"id":"479193","messageId":"CAP8UFD2=7AJfw4hyhFttqDOFQ+1TxRq5JpUNEpKsGk4b3VJ+HQ@mail.gmail.com","threadId":"59867","inReplyTo":"ZJLVzaQKBrgCARAG@nand.local","subject":"Re: [PATCH 1/9] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:16:31Z","receivedAt":"2023-07-05T06:16:48Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 12:49 PM Taylor Blau wrote:\n>\n> On Wed, Jun 14, 2023 at 09:25:33PM +0200, Christian Couder wrote:\n> > 9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\n> > taught `git pack-objects` to use `--filter`, but required the use of\n> > `--stdout` since a partial clone mechanism was not yet in place to\n> > handle missing objects. Since then, changes like 9e27beaa23\n> > (promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\n> > and others added support to dynamically fetch objects that were missing.\n> >\n> > Even without a promisor remote, filtering out objects can also be useful\n> > if we can put the filtered out objects in a separate pack, and in this\n> > case it also makes sense for pack-objects to write the packfile directly\n> > to an actual file rather than on stdout.\n> >\n> > Remove the `--stdout` requirement when using `--filter`, so that in a\n> > follow-up commit, repack can pass `--filter` to pack-objects to omit\n> > certain objects from the resulting packfile.\n>\n> Makes sense.\n>\n> Is there any situation in which using --stdout with --filter would be a\n> potential foot-gun? I am not as familiar with the partial clone\n> mechanism as others CC'd, so I have no idea one way or the other.\n\nThis patch allows `--filter` without `--stdout`, so `--stdout` with\n`--filter` was already allowed before this patch and is still allowed\nafter it.\n\nBesides, using `--stdout` or not using it is just a convenience. It\ndoesn't change much what users can do with `git pack-objects`.\n\n> If it is unsafe in certain situations (or, at the very least, could\n> produce surprising behavior), it may be worthwhile to only allow\n> `--filter= --stdout` with some kind of\n> `--filter-to-stdout-is-ok` flag to indicate that the caller knows what\n> they are doing.\n\n`git pack-objects` with `--filter` can be unsafe in some cases if\nusers do stupid things with packfiles afterwards, but allowing it to\nrun without --stdout doesn't significantly change that. In any case by\nitself it doesn't delete any data. It just creates a pack containing\nfewer objects.\n\nAnyway I added a small test that checks that --filter can be used\nwithout --stdout.\n"},{"id":"479194","messageId":"CAP8UFD2W8fC=h=s7x04hdNS5KPcVdya0UM1Ozsr26CdHU+NrKw@mail.gmail.com","threadId":"59867","inReplyTo":"ZJLjvhnh0lSRCVzG@nand.local","subject":"Re: [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T06:19:40Z","receivedAt":"2023-07-05T06:19:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 1:49 PM Taylor Blau wrote:\n>\n> On Wed, Jun 14, 2023 at 09:25:40PM +0200, Christian Couder wrote:\n> > diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> > index aa29c7e648..070dd22610 100644\n> > --- a/Documentation/git-repack.txt\n> > +++ b/Documentation/git-repack.txt\n> > @@ -148,6 +148,12 @@ depth is 4095.\n> > resulting packfile and put them into a separate packfile. See\n> > linkgit:git-rev-list[1] for valid `` forms.\n> >\n> > +--filter-to=::\n> > + Write the pack containing filtered out objects to the\n> > + directory ``. This can be used for putting the pack on a\n> > + separate object directory that is accessed through the Git\n> > + alternates mechanism. Only useful with `--filter`.\n>\n> Here you say \"only useful with --filter\", but...\n>\n> > @@ -1073,8 +1077,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> > strvec_push(&cmd.args, \"--incremental\");\n> > }\n> >\n> > - if (po_args.filter)\n> > - prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);\n> > + if (po_args.filter) {\n> > + if (!filter_to)\n> > + filter_to = packtmp;\n> > + prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);\n> > + }\n>\n> Would you want an \"} else if (filter_to)\" here to die and show the usage\n> message, since --filter-to needs --filter? Or maybe it should imply\n> --filter-to.\n\nI have added such a check in the version 2 I just sent. We now die()\nif --filter-to is used without --filter.\n"},{"id":"479196","messageId":"CAP8UFD1oOiH71NfzgNhE_aCMHT8U8mCNADoAfpghxgXzEN30iw@mail.gmail.com","threadId":"59867","inReplyTo":"ZJLcS78DDOjn6MzP@nand.local","subject":"Re: [PATCH 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-05T07:18:54Z","receivedAt":"2023-07-05T07:19:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 21, 2023 at 1:17 PM Taylor Blau wrote:\n>\n> On Wed, Jun 14, 2023 at 09:25:38PM +0200, Christian Couder wrote:\n> > ---\n> > Documentation/git-repack.txt | 5 +++\n> > builtin/repack.c | 75 ++++++++++++++++++++++++++++++++++--\n> > t/t7700-repack.sh | 16 ++++++++\n> > 3 files changed, 93 insertions(+), 3 deletions(-)\n>\n> Having read through the implementation in the repack builtin, I am\n> almost certain that my suggestion earlier in the thread to implement\n> this in terms of 'git pack-objects --filter' and 'git pack-objects\n> --stdin-packs' would work.\n\nYeah, it works, thanks. That's what is used in the version 2 I just sent.\n\n> > diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> > index 4017157949..aa29c7e648 100644\n> > --- a/Documentation/git-repack.txt\n> > +++ b/Documentation/git-repack.txt\n> > @@ -143,6 +143,11 @@ depth is 4095.\n> > a larger and slower repository; see the discussion in\n> > `pack.packSizeLimit`.\n> >\n> > +--filter=::\n> > + Remove objects matching the filter specification from the\n> > + resulting packfile and put them into a separate packfile. See\n> > + linkgit:git-rev-list[1] for valid `` forms.\n> > +\n>\n> This documentation leaves me with a handful of questions about how it\n> interacts with other options.\n\nI have improved this doc in version 2. It now says:\n\n--filter=::\n Remove objects matching the filter specification from the\n resulting packfile and put them into a separate packfile. Note\n that objects used in the working directory are not filtered\n out. So for the split to fully work, it's best to perform it\n in a bare repo and to use the `-a` and `-d` options along with\n this option. See linkgit:git-rev-list[1] for valid\n `` forms.\n\nFor comparison the doc about --cruft is:\n\n--cruft::\n Same as `-a`, unless `-d` is used. Then any unreachable objects\n are packed into a separate cruft pack. Unreachable objects can\n be pruned using the normal expiry rules with the next `git gc`\n invocation (see linkgit:git-gc[1]). Incompatible with `-k`.\n\n> Here are some:\n>\n> - What happens when you pass it with \"-d\"? Does it delete objects that\n> didn't match the filter? Leave them alone?\n\nWith or without \"-d\", no object is deleted, objects are just put into\n2 different packfiles depending on whether or not they match the\nfilter.\n\n> If the latter, should\n> this combination be declared invalid instead of silently ignoring\n> the user's request to delete redundant packs?\n\n\"-d\" is indeed about removing redundant packs. If you want to split\nthe objects into different packs according to a filter, then you\nprobably don't want to keep packs that contain both kinds of objects\nthat you want to separate into different packs, so the doc now\nrecommends using \"-d\" (along with \"-a\") when using \"--filter\". That's\nalso what the tests are using.\n\nI am not sure it's worth erroring out when \"-d\", or \"-a\", is not used.\nPerhaps there are some use cases where it's interesting to keep old\npacks, or, in case of not using \"-a\", to split only loose objects into\nseparate packs, but not objects from existing packs.\n\n> - What happens with --max-pack-size? Does the filtered pack get split\n> into multiple packs (as I think we would expect from such a\n> combination)?\n\nYes, in version 2 I have added a test (in the commit introducing\n--filter-to) that checks that objects that are filtered out and that\nare larger than --max-pack-size get split into their own packs.\n\n> - What about with `--cruft`? Does it split the cruft pack into two\n> based on whether or not the unreachable object(s) matched or didn't\n> match the filter?\n\nI am not sure as I don't know well how cruft works. I would need to\ncheck the code and/or test it.\n\n> - What happens when passed with \"--geometric\"? I don't think there is\n> a sensible interpretation (at least, I can't think of what it would\n> mean to do \"--filter= --geometric=\" off the top of my\n> head).\n\nNot sure either, as I also don't know well how --geometric works.\n\n> - What about with \"--write-bitmap-index\"? Do we write one bitmap\n> index? Two? If the latter, do we combine the packs into a MIDX\n> before writing the bitmap? Should we?\n\nIn the tests, repack --filter is used with \"-c\nrepack.writebitmaps=false\" as otherwise the bitmap writing code tends\nto complain that a pack is not complete or something like that. I\nthink it would make sense to have options that would make the bitmap\nwriting code write a regular bitmap index if all the objects are in a\nsingle pack, but a MIDX if they are in multiple packs. I don't know\nwhat --cruft or --max-pack-size are doing about this, but I think such\na feature could be useful in case those options are used.\n\nIn the meantime I would be Ok with disallowing using both\n--write-bitmap-index and --filter. The issue is that writing a bitmap\nindex is the default behavior, even without --write-bitmap-index, so\nit might be a bit more complex, but I plan to do something about it in\nversion 3.\n\n> I think it may be worth spelling out answers to some of these questions\n> in the documentation, and codifying those answers in the form of tests.\n\nIn version 2, the doc has been improved and now it seems comparable to\nthe --cruft doc. About the tests, I added one in version 2 related to\n--max-pack-size, and I am open to adding a few others, but I don't\nthink it's a good idea to add a lot of them. It just doesn't scale\nwhen commands have more than a few options.\n\nThe new code is reusing existing features (like --stdin-packs) that\nare already used by other code (like --cruft code). It is not adding\nnew intricate mechanisms, nor a lot of new code. It is not changing\ncode used by other features except for a few clean refactorings. This\nshould give a good indication that it will work well with other\nfeatures.\n\n> This makes me wonder whether or not this option should belong in repack\n> at all, or whether there should be some new special-purpose builtin that\n> is designed to split existing pack(s) based on whether or not they meet\n> some filter criteria.\n\n--cruft is also splitting existing packs based on some criteria. So if\nwe go this way, perhaps --cruft code should be moved first to this new\nspecial purpose builtin?\n"},{"id":"479215","messageId":"xmqqh6qi1byn.fsf@gitster.g","threadId":"59867","inReplyTo":"20230705060812.2865188-6-christian.couder@gmail.com","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-05T17:53:52Z","receivedAt":"2023-07-05T17:54:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> This could be useful if, for example, some large blobs take a lot of\n> precious space on fast storage while they are rarely accessed. It could\n> make sense to move them into a separate cheaper, though slower, storage.\n>\n> In other use cases it might make sense to put all the blobs into\n> separate storage.\n\nMinor nit. Aren't the above two the same use case?\n\n> This is done by running two `git pack-objects` commands. The first one\n> is run with `--filter=`, using the specified filter. It\n> packs objects while omitting the objects specified by the filter.\n> Then another `git pack-objects` command is launched using\n> `--stdin-packs`. We pass it all the previously existing packs into its\n> stdin, so that it will pack all the objects in the previously existing\n> packs. But we also pass into its stdin, the pack created by the previous\n> `git pack-objects --filter=` command as well as the kept\n> packs, all prefixed with '^', so that the objects in these packs will be\n> omitted from the resulting pack.\n\nWhen I started reading the paragraph, the first question that came\nto my mind was if these two pack-objects processes can and should be\nrun in parallel, which is answered in the part near the end of the\nparagraph. It may be a good idea to start the paragraph with \"by\nrunning `git pack-objects` command twice in a row\" or something to\nmake it clear that one should (and cannot be) run before the other\ncompletes.\n\nIn fact, isn't the call site of write_filtered_pack() in this patch\na bit too early? The subprocess that runs with \"--stdin-packs\" is\nstarted and told about the names of the pack we are going to create,\nand it does not start processing until it reads everything (i.e. we\nrun fclose(in) in the write_filtered_pack() function), but the loop\nover \"names\" string list in the caller that moves the tempfiles to\ntheir final filenames comes after the call to close_object_store()\nwe see in the post context of the call to write_filtered_pack() that\nis new in this patch.\n\nThe \"--stdin-packs\" one is told to exclude objects that appear in\nthese packs, so if the main process is a bit slow to finalize the\npackfiles it created (and told the \"--stdin-packs\" process about),\nit will not lead to repository corruption---just some objects are\nincluded in the packfiles \"--stdin-packs\" one creates even though\nthey do not have to. So it does not sound like a huge problem to\nme, but still it somehow looks wrong. Am I misreading the code?\n\nThanks.\n"},{"id":"479221","messageId":"xmqqcz161b3s.fsf@gitster.g","threadId":"59867","inReplyTo":"20230705060812.2865188-6-christian.couder@gmail.com","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-05T18:12:23Z","receivedAt":"2023-07-05T18:12:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> +--filter=::\n> +\tRemove objects matching the filter specification from the\n> +\tresulting packfile and put them into a separate packfile. Note\n> +\tthat objects used in the working directory are not filtered\n> +\tout. So for the split to fully work, it's best to perform it\n> +\tin a bare repo and to use the `-a` and `-d` options along with\n> +\tthis option. See linkgit:git-rev-list[1] for valid\n> +\t`` forms.\n\nAfter running the command with this option once, we will have two\npackfiles, one with objects that match and the other with objects\nthat do not match the filter spec. Then what is the next step for\nthe user of this feature? Moving the former to a slower storage\nwas cited as a motivation for the feature, but can the user tell\nwhich one of these two packfiles is the one that consists of the\nfiltered out objects? If there is no mechansim to do so, shouldn't\nwe have one to make this feature more usable?\n\nAt the level of \"pack-objects\" command, we report the new packfiles\nso that the user does not have to take \"ls .git/objects/pack\" before\nand after the operation to compare and learn which ones are new.\nI do not think \"repack\" that is a Porcelain should do such a\nreporting on its standard output, but that means either the feature\nshould probably be done at the plumbing level (i.e. \"pack-objects\"),\nor the marking of the new packfiles needs to be done in a way that\ntools can later find them out, e.g. on the filesystem, similar to\nthe way \".keep\" marker tells which ones are not to be repacked, etc.\n"},{"id":"479223","messageId":"xmqq5y6y1agl.fsf@gitster.g","threadId":"59867","inReplyTo":"20230705060812.2865188-8-christian.couder@gmail.com","subject":"Re: [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-05T18:26:18Z","receivedAt":"2023-07-05T18:27:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> A previous commit has implemented `git repack --filter=` to\n> allow users to filter out some objects from the main pack and move them\n> into a new different pack.\n\nOK, this sidesteps the question I had on an earlier step rather\nnicely. Instead of having to find out which ones are to be moved\naway, just generating them in a separate location would be more\nstraight forward.\n\nThe implementation does not seem to restrict where --filter-to\ndirectory can be placed, but shouldn't it make sure that it is one\nof the already specified alternates directories? Otherwise the user\nwill end up corrupting the repository, no?\n"},{"id":"479789","messageId":"20230724085909.3831831-1-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230705060812.2865188-1-christian.couder@gmail.com","subject":"[PATCH v3 0/8] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:01Z","receivedAt":"2023-07-24T08:59:41Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"# Intro\n\nLast year, John Cai sent 2 versions of a patch series to implement\n`git repack --filter=` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\n# Use cases for the new feature\n\nThis could be useful for example for the following purposes:\n\n 1) As a way for servers to save storage costs by for example moving\n large blobs, or all the blobs, or all the blobs in inactive\n repos, to separate storage (while still making them accessible\n using for example the alternates mechanism).\n\n 2) As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n 3) As a way for clients to reclaim some space when they cloned with\n a filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a\n separate directory or storage, then check that everything works\n well, and then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\nAlso since version 2 of this new series, commit messages, don't\nmention uses cases like 2) or 3) above, as people have different\nopinions on how it should be done. How it should be done could depend\na lot on the way promisor remotes are used, the software and hardware\nsetups used, etc, so it seems more difficult to \"sell\" this series by\ntalking about such use cases. As use case 1) seems simpler and more\nappealing, it makes more sense to only talk about it in the commit\nmessages.\n\n# Changes since version 2\n\nThanks to Junio who reviewed both version 1 and 2, and to Taylor who\nreviewed version 1! The changes are the following:\n\n- In patch 5/8, which introduces `--filter=` option, some\n explanations about how to find which new packfile contains the\n filtered out objects have been added to the commit message following\n Junio's comments.\n\n- In patch 5/8, it was clarified in the commit message that `git\n pack-objects` is run twice in row (and not in parallel) to implement\n the new option according to Junio's comments.\n\n- In patch 5/8 also, the documentaion of the new option says that\n `--no-write-bitmap-index` (or the ++ `repack.writebitmaps` config\n option set to `false`) should be used along with the option as\n otherwise writing bitmap index will fail. And a corresponding new\n test called '--filter fails with --write-bitmap-index' has been\n added to t/t7700-repack.sh. This should address Taylor's comments\n about v1 that were not addressed by v2.\n\n- In patch 7/8, which implements the `--filter-to=` option, the\n commit message now recommends using Git alternates mechanism before\n this option is used to make sure the directory specified by the new\n option is accessible by the repo as it could otherwise corrupt the\n repo. It also says that in some cases it might not be necessary to\n use such a mechanism, which is why the feature doesn't check that\n directory specified is accessible. The documentation of the new\n option also loudly warns that the repo could be corrupted if the Git\n alternates mechanism, and has a new link to that mechanism's\n documentation. This is to address Junio's comments.\n\n- In patch 8/8, which implements the `gc.repackFilterTo` config\n option, a similar loud warning has been added, and similar doc\n changes have been made, to the documentation of the new config\n option (which corresponds to the `--filter-to=` command line\n option).\n\n# Commit overview\n\n* 1/8 pack-objects: allow `--filter` without `--stdout`\n\n This patch is the same as in v1 and v2. To be able to later repack\n with a filter we need `git pack-objects` to write packfiles when\n it's filtering instead of just writing the pack without the filtered\n out objects to stdout.\n\n* 2/8 t/helper: add 'find-pack' test-tool\n\n No change in this patch compared to v1 and v2. For testing `git\n repack --filter=...` that we are going to implement, it's useful to\n have a test helper that can tell which packfiles contain a specific\n object.\n\n* 3/8 repack: refactor finishing pack-objects command\n\n No change in this patch compared to v2. This is a small refactoring\n creating a new useful function, so that `git repack --filter=...`\n will be able to reuse it.\n\n* 4/8 repack: refactor finding pack prefix\n\n No change in this patch compared to v2. This is another small\n refactoring creating a small function that will be reused in the\n next patch.\n\n* 5/8 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. It uses one\n `git pack-objects` process with the `--filter` option. And then\n another `git pack-objects` process with the `--stdin-packs`\n option. Only the commit message, documentation and tests have been\n changed a bit since v2.\n\n* 6/8 gc: add `gc.repackFilter` config option\n\n No change in this patch compared to v2 and v1. This is a gc config\n option so that `git gc` can also repack using a filter and put the\n filtered out objects into a separate packfile.\n\n* 7/8 repack: implement `--filter-to` for storing filtered out objects\n\n For some use cases, it's interesting to create the packfile that\n contains the filtered out objects into a separate location. This is\n similar to the `--expire-to` option for cruft packfiles. Only the\n commit message and the documentation have changed since version\n 2. They now explain and discuss the risks of using this option\n without making sure the specified directory is not accessible by the\n repo.\n\n* 8/8 gc: add `gc.repackFilterTo` config option\n\n This allows specifying the location of the packfile that contains\n the filtered out objects when using `gc.repackFilter`. As with the\n previous commit, since v2, the doc now explain and discuss the risks\n of using this option without making sure the specified directory is\n not accessible by the repo.\n\n# Range-diff since v2\n\n1: 0bd1ad3071 = 1: 4d75a1d7c3 pack-objects: allow `--filter` without `--stdout`\n2: e49cd723c7 = 2: fdf9b6e8cc t/helper: add 'find-pack' test-tool\n3: 3f87772ea6 = 3: e7cfdebc78 repack: refactor finishing pack-objects command\n4: 9997efaf33 = 4: 9c51063795 repack: refactor finding pack prefix\n5: da27ecb91b ! 5: a90e8045c3 repack: add `--filter=` option\n @@ Commit message\n This new option puts the objects specified by `` into a\n separate packfile.\n \n - This could be useful if, for example, some large blobs take a lot of\n + This could be useful if, for example, some large blobs take up a lot of\n precious space on fast storage while they are rarely accessed. It could\n make sense to move them into a separate cheaper, though slower, storage.\n \n In other use cases it might make sense to put all the blobs into\n separate storage.\n \n - This is done by running two `git pack-objects` commands. The first one\n - is run with `--filter=`, using the specified filter. It\n - packs objects while omitting the objects specified by the filter.\n - Then another `git pack-objects` command is launched using\n + It's possible to find which new packfile contains the filtered out\n + objects using one of the following:\n +\n + - `git verify-pack -v ...`,\n + - `test-tool find-pack ...`, which a previous commit added,\n + - `--filter-to=`, which a following commit will add to specify\n + where the pack containing the filtered out objects will be.\n +\n + This feature is implemented by running `git pack-objects` twice in a\n + row. The first command is run with `--filter=`, using the\n + specified filter. It packs objects while omitting the objects specified\n + by the filter. Then another `git pack-objects` command is launched using\n `--stdin-packs`. We pass it all the previously existing packs into its\n stdin, so that it will pack all the objects in the previously existing\n packs. But we also pass into its stdin, the pack created by the previous\n @@ Documentation/git-repack.txt: depth is 4095.\n + that objects used in the working directory are not filtered\n + out. So for the split to fully work, it's best to perform it\n + in a bare repo and to use the `-a` and `-d` options along with\n -+ this option. See linkgit:git-rev-list[1] for valid\n -+ `` forms.\n ++ this option. Also `--no-write-bitmap-index` (or the\n ++ `repack.writebitmaps` config option set to `false`) should be\n ++ used otherwise writing bitmap index will fail, as it supposes\n ++ a single packfile containing all the objects. See\n ++ linkgit:git-rev-list[1] for valid `` forms.\n +\n -b::\n --write-bitmap-index::\n @@ t/t7700-repack.sh: test_expect_success 'auto-bitmaps do not complain if unavaila\n + blob_pack2=$(test-tool -C bare.git find-pack HEAD:file2) &&\n + test \"$blob_pack2\" = \"$blob_pack\"\n +'\n ++\n ++test_expect_success '--filter fails with --write-bitmap-index' '\n ++ test_must_fail git -C bare.git repack -a -d --write-bitmap-index \\\n ++ --filter=blob:none &&\n ++\n ++ git -C bare.git repack -a -d --no-write-bitmap-index \\\n ++ --filter=blob:none\n ++'\n +\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n6: 49e4a184b4 = 6: 335b7f614d gc: add `gc.repackFilter` config option\n7: 243c93aad3 ! 7: b1be7f60b7 repack: implement `--filter-to` for storing filtered out objects\n @@ Commit message\n It would be nice if this new different pack could be created in a\n different directory than the regular pack. This would make it possible\n to move large blobs into a pack on a different kind of storage, for\n - example cheaper storage. Even in a different directory this pack can be\n - accessible if, for example, the Git alternates mechanism is used to\n - point to it.\n + example cheaper storage.\n +\n + Even in a different directory, this pack can be accessible if, for\n + example, the Git alternates mechanism is used to point to it. In fact\n + not using the Git alternates mechanism can corrupt a repo as the\n + generated pack containing the filtered objects might not be accessible\n + from the repo any more. So setting up the Git alternates mechanism\n + should be done before using this feature if the user wants the repo to\n + be fully usable while this feature is used.\n +\n + In some cases, like when a repo has just been cloned or when there is no\n + other activity in the repo, it's Ok to setup the Git alternates\n + mechanism afterwards though. It's also Ok to just inspect the generated\n + packfile containing the filtered objects and then just move it into the\n + '.git/objects/pack/' directory manually. That's why it's not necessary\n + for this command to check that the Git alternates mechanism has been\n + already setup.\n \n While at it, as an example to show that `--filter` and `--filter-to`\n work well with other options, let's also add a test to check that these\n @@ Commit message\n \n ## Documentation/git-repack.txt ##\n @@ Documentation/git-repack.txt: depth is 4095.\n - this option. See linkgit:git-rev-list[1] for valid\n - `` forms.\n + a single packfile containing all the objects. See\n + linkgit:git-rev-list[1] for valid `` forms.\n \n +--filter-to=::\n + Write the pack containing filtered out objects to the\n -+ directory ``. This can be used for putting the pack on a\n -+ separate object directory that is accessed through the Git\n -+ alternates mechanism. Only useful with `--filter`.\n ++ directory ``. Only useful with `--filter`. This can be\n ++ used for putting the pack on a separate object directory that\n ++ is accessed through the Git alternates mechanism. **WARNING:**\n ++ If the packfile containing the filtered out objects is not\n ++ accessible, the repo could be considered corrupt by Git as it\n ++ migh not be able to access the objects in that packfile. See\n ++ the `objects` and `objects/info/alternates` sections of\n ++ linkgit:gitrepository-layout[5].\n +\n -b::\n --write-bitmap-index::\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n &existing_nonkept_packs,\n \n ## t/t7700-repack.sh ##\n -@@ t/t7700-repack.sh: test_expect_success 'repacking with a filter works' '\n - test \"$blob_pack2\" = \"$blob_pack\"\n +@@ t/t7700-repack.sh: test_expect_success '--filter fails with --write-bitmap-index' '\n + --filter=blob:none\n '\n \n +test_expect_success '--filter-to stores filtered out objects' '\n8: 8cb3faa74c ! 8: ed66511823 gc: add `gc.repackFilterTo` config option\n @@ Documentation/config/gc.txt: gc.repackFilter::\n +gc.repackFilterTo::\n + When repacking and using a filter, see `gc.repackFilter`, the\n + specified location will be used to create the packfile\n -+ containing the filtered out objects. See the\n -+ `--filter-to=` option of linkgit:git-repack[1].\n ++ containing the filtered out objects. **WARNING:** The\n ++ specified location should be accessible, using for example the\n ++ Git alternates mechanism, otherwise the repo could be\n ++ considered corrupt by Git as it migh not be able to access the\n ++ objects in that packfile. See the `--filter-to=` option\n ++ of linkgit:git-repack[1] and the `objects/info/alternates`\n ++ section of linkgit:gitrepository-layout[5].\n +\n gc.rerereResolved::\n Records of conflicted merge you resolved earlier are\n\n\nChristian Couder (8):\n pack-objects: allow `--filter` without `--stdout`\n t/helper: add 'find-pack' test-tool\n repack: refactor finishing pack-objects command\n repack: refactor finding pack prefix\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 16 +++\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 23 ++++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 162 ++++++++++++++++++-------\n t/helper/test-find-pack.c | 35 ++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5317-pack-objects-filter-objects.sh | 8 ++\n t/t6500-gc.sh | 23 ++++\n t/t7700-repack.sh | 90 ++++++++++++++\n 13 files changed, 332 insertions(+), 50 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n\n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479790","messageId":"20230724085909.3831831-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v3 1/8] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:02Z","receivedAt":"2023-07-24T08:59:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n t/t5317-pack-objects-filter-objects.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex a9995a932c..583270a85f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -298,8 +298,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 06b33d49e9..7fca27ffbe 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4387,12 +4387,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..2ff3eef9a3 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n \t! grep blob verify_result\n '\n \n+test_expect_success 'verify blob:none packfile without --stdout' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n+\t! grep blob verify_result\n+'\n+\n test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n \tgit -C r1 verify-pack -v ../all.pack >verify_result &&\n \tgrep -E \"commit|tree\" verify_result |\n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479791","messageId":"20230724085909.3831831-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v3 3/8] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:04Z","receivedAt":"2023-07-24T08:59:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -705,9 +735,8 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct string_list *existing_kept_packs)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n \tconst char *scratch;\n \tint local = skip_prefix(destination, packdir, &scratch);\n@@ -751,27 +780,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -782,10 +791,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1016,18 +1023,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479792","messageId":"20230724085909.3831831-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v3 4/8] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:05Z","receivedAt":"2023-07-24T08:59:56Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new find_pack_prefix() to refactor code that handles finding\nthe pack prefix from the packtmp and packdir global variables, as we are\ngoing to need this feature again in following commit.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 35 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 38 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n\ndiff --git a/Makefile b/Makefile\nindex fb541dedc9..14ee0c45d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..1928fe7329\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ */\n+\n+static const char *find_pack_usage = \"\\n\"\n+\" test-tool find-pack \";\n+\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\n+\tsetup_git_directory();\n+\n+\tif (argc != 2)\n+\t\tusage(find_pack_usage);\n+\n+\tif (repo_get_oid(the_repository, argv[1], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[1]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tif (find_pack_entry_one(oid.hash, p))\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex abe8a785eb..41da40c296 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ea2672436c..411dbf2db4 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479794","messageId":"20230724085909.3831831-6-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v3 5/8] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:06Z","receivedAt":"2023-07-24T08:59:59Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new option puts the objects specified by `` into a\nseparate packfile.\n\nThis could be useful if, for example, some large blobs take up a lot of\nprecious space on fast storage while they are rarely accessed. It could\nmake sense to move them into a separate cheaper, though slower, storage.\n\nIn other use cases it might make sense to put all the blobs into\nseparate storage.\n\nIt's possible to find which new packfile contains the filtered out\nobjects using one of the following:\n\n - `git verify-pack -v ...`,\n - `test-tool find-pack ...`, which a previous commit added,\n - `--filter-to=`, which a following commit will add to specify\n where the pack containing the filtered out objects will be.\n\nThis feature is implemented by running `git pack-objects` twice in a\nrow. The first command is run with `--filter=`, using the\nspecified filter. It packs objects while omitting the objects specified\nby the filter. Then another `git pack-objects` command is launched using\n`--stdin-packs`. We pass it all the previously existing packs into its\nstdin, so that it will pack all the objects in the previously existing\npacks. But we also pass into its stdin, the pack created by the previous\n`git pack-objects --filter=` command as well as the kept\npacks, all prefixed with '^', so that the objects in these packs will be\nomitted from the resulting pack. The result is that only the objects\nfiltered out by the first `git pack-objects` command are in the pack\nresulting from the second `git pack-objects` command.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 12 +++++++\n builtin/repack.c | 67 ++++++++++++++++++++++++++++++++++++\n t/t7700-repack.sh | 24 +++++++++++++\n 3 files changed, 103 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..6d5bec7716 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,18 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. Note\n+\tthat objects used in the working directory are not filtered\n+\tout. So for the split to fully work, it's best to perform it\n+\tin a bare repo and to use the `-a` and `-d` options along with\n+\tthis option. Also `--no-write-bitmap-index` (or the\n+\t`repack.writebitmaps` config option set to `false`) should be\n+\tused otherwise writing bitmap index will fail, as it supposes\n+\ta single packfile containing all the objects. See\n+\tlinkgit:git-rev-list[1] for valid `` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 21e3b89f27..2c81b7738e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -53,6 +53,7 @@ struct pack_objects_args {\n \tconst char *depth;\n \tconst char *threads;\n \tconst char *max_pack_size;\n+\tconst char *filter;\n \tint no_reuse_delta;\n \tint no_reuse_object;\n \tint quiet;\n@@ -166,6 +167,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--threads=%s\", args->threads);\n \tif (args->max_pack_size)\n \t\tstrvec_pushf(&cmd->args, \"--max-pack-size=%s\", args->max_pack_size);\n+\tif (args->filter)\n+\t\tstrvec_pushf(&cmd->args, \"--filter=%s\", args->filter);\n \tif (args->no_reuse_delta)\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n \tif (args->no_reuse_object)\n@@ -726,6 +729,57 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n+static int write_filtered_pack(const struct pack_objects_args *args,\n+\t\t\t const char *destination,\n+\t\t\t const char *pack_prefix,\n+\t\t\t struct string_list *names,\n+\t\t\t struct string_list *existing_packs,\n+\t\t\t struct string_list *existing_kept_packs)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\tconst char *scratch;\n+\tint local = skip_prefix(destination, packdir, &scratch);\n+\n+\t/* We need to copy 'args' to modify it */\n+\tstruct pack_objects_args new_args = *args;\n+\n+\t/* No need to filter again */\n+\tnew_args.filter = NULL;\n+\n+\tprepare_pack_objects(&cmd, &new_args, destination);\n+\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\n+\tcmd.in = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * names has a confusing double use: it both provides the list\n+\t * of just-written new packs, and accepts the name of the\n+\t * filtered pack we are writing.\n+\t *\n+\t * By the time it is read here, it contains only the pack(s)\n+\t * that were just written, which is exactly the set of packs we\n+\t * want to consider kept.\n+\t */\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, names)\n+\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n+\tfor_each_string_list_item(item, existing_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tfor_each_string_list_item(item, existing_kept_packs)\n+\t\tfprintf(in, \"^%s.pack\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -858,6 +912,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_STRING(0, \"filter\", &po_args.filter, N_(\"args\"),\n+\t\t\t\tN_(\"object filtering\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -1097,6 +1153,17 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter) {\n+\t\tret = write_filtered_pack(&po_args,\n+\t\t\t\t\t packtmp,\n+\t\t\t\t\t find_pack_prefix(),\n+\t\t\t\t\t &names,\n+\t\t\t\t\t &existing_nonkept_packs,\n+\t\t\t\t\t &existing_kept_packs);\n+\t\tif (ret)\n+\t\t\tgoto cleanup;\n+\t}\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 27b66807cd..0a2c73bca7 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -327,6 +327,30 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n+\ttest -n \"$commit_pack\" &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -n \"$blob_pack\" &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+\ttest_must_fail git -C bare.git repack -a -d --write-bitmap-index \\\n+\t\t--filter=blob:none &&\n+\n+\tgit -C bare.git repack -a -d --no-write-bitmap-index \\\n+\t\t--filter=blob:none\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479795","messageId":"20230724085909.3831831-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v3 6/8] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:07Z","receivedAt":"2023-07-24T09:00:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 12 ++++++++++++\n 3 files changed, 23 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex ca47eb2008..2153bde7ac 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the\n operation (either generating a cruft pack or unpacking unreachable\n objects) will be halted.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 19d73067aa..9b0984f301 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -61,6 +61,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -170,6 +171,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -355,6 +358,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..5b89faf505 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,18 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none -c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479796","messageId":"20230724085909.3831831-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v3 8/8] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:09Z","receivedAt":"2023-07-24T09:00:07Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 11 +++++++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 2153bde7ac..466466d6cc 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -150,6 +150,17 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. **WARNING:** The\n+\tspecified location should be accessible, using for example the\n+\tGit alternates mechanism, otherwise the repo could be\n+\tconsidered corrupt by Git as it migh not be able to access the\n+\tobjects in that packfile. See the `--filter-to=` option\n+\tof linkgit:git-repack[1] and the `objects/info/alternates`\n+\tsection of linkgit:gitrepository-layout[5].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9b0984f301..1b7c775d94 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -172,6 +173,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -361,6 +363,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 5b89faf505..37056a824b 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -214,6 +213,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479797","messageId":"20230724085909.3831831-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v3 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T08:59:08Z","receivedAt":"2023-07-24T09:00:19Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage.\n\nEven in a different directory, this pack can be accessible if, for\nexample, the Git alternates mechanism is used to point to it. In fact\nnot using the Git alternates mechanism can corrupt a repo as the\ngenerated pack containing the filtered objects might not be accessible\nfrom the repo any more. So setting up the Git alternates mechanism\nshould be done before using this feature if the user wants the repo to\nbe fully usable while this feature is used.\n\nIn some cases, like when a repo has just been cloned or when there is no\nother activity in the repo, it's Ok to setup the Git alternates\nmechanism afterwards though. It's also Ok to just inspect the generated\npackfile containing the filtered objects and then just move it into the\n'.git/objects/pack/' directory manually. That's why it's not necessary\nfor this command to check that the Git alternates mechanism has been\nalready setup.\n\nWhile at it, as an example to show that `--filter` and `--filter-to`\nwork well with other options, let's also add a test to check that these\noptions work well with `--max-pack-size`.\n\nSigned-off-by: Christian Couder \n\nrepack: add test with --max-pack-size\n---\n Documentation/git-repack.txt | 11 ++++++\n builtin/repack.c | 11 +++++-\n t/t7700-repack.sh | 66 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 87 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 6d5bec7716..c0fbb0ed0c 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -155,6 +155,17 @@ depth is 4095.\n \ta single packfile containing all the objects. See\n \tlinkgit:git-rev-list[1] for valid `` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. Only useful with `--filter`. This can be\n+\tused for putting the pack on a separate object directory that\n+\tis accessed through the Git alternates mechanism. **WARNING:**\n+\tIf the packfile containing the filtered out objects is not\n+\taccessible, the repo could be considered corrupt by Git as it\n+\tmigh not be able to access the objects in that packfile. See\n+\tthe `objects` and `objects/info/alternates` sections of\n+\tlinkgit:gitrepository-layout[5].\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 2c81b7738e..626284191b 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -871,6 +871,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -924,6 +925,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1067,6 +1070,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (filter_to && !po_args.filter)\n+\t\tdie(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n+\n \tif (geometry)\n \t\tcmd.in = -1;\n \telse\n@@ -1154,8 +1160,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\n \t\tret = write_filtered_pack(&po_args,\n-\t\t\t\t\t packtmp,\n+\t\t\t\t\t filter_to,\n \t\t\t\t\t find_pack_prefix(),\n \t\t\t\t\t &names,\n \t\t\t\t\t &existing_nonkept_packs,\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 0a2c73bca7..2bf237ba3a 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -351,6 +351,72 @@ test_expect_success '--filter fails with --write-bitmap-index' '\n \t\t--filter=blob:none\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n+\ttest -n \"$commit_pack\" &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -z \"$blob_pack\" &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack $blob_hash) &&\n+\ttest -n \"$blob_pack\" &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n+\ttest -n \"$blob_pack\" &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n+test_expect_success '--filter works with --max-pack-size' '\n+\trm -rf filtered.git &&\n+\tgit init --bare filtered.git &&\n+\tgit init max-pack-size &&\n+\t(\n+\t\tcd max-pack-size &&\n+\t\ttest_commit base &&\n+\t\t# two blobs which exceed the maximum pack size\n+\t\ttest-tool genrandom foo 1048576 >foo &&\n+\t\tgit hash-object -w foo &&\n+\t\ttest-tool genrandom bar 1048576 >bar &&\n+\t\tgit hash-object -w bar &&\n+\t\tgit add foo bar &&\n+\t\tgit commit -m \"adding foo and bar\"\n+\t) &&\n+\tgit clone --no-local --bare max-pack-size max-pack-size.git &&\n+\t(\n+\t\tcd max-pack-size.git &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--max-pack-size=1M \\\n+\t\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\t\techo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n+\n+\t\t# Check that the 3 blobs are in different packfiles in filtered.git\n+\t\ttest_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n+\t\tfoo_pack=$(test-tool find-pack HEAD:foo) &&\n+\t\tbar_pack=$(test-tool find-pack HEAD:bar) &&\n+\t\tbase_pack=$(test-tool find-pack HEAD:base.t) &&\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" != \"$base_pack\" &&\n+\t\ttest \"$bar_pack\" != \"$base_pack\" &&\n+\t\tfor pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n+\t\tdo\n+\t\t\tcase \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n+\t\tdone\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.41.0.384.ged66511823\n\n"},{"id":"479798","messageId":"CAP8UFD1tqzp744j0KORw-zcgOn6Tufm4Kk3yct3vHGY29pbm-w@mail.gmail.com","threadId":"59867","inReplyTo":"xmqq5y6y1agl.fsf@gitster.g","subject":"Re: [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T09:00:56Z","receivedAt":"2023-07-24T09:01:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 5, 2023 at 8:26 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > A previous commit has implemented `git repack --filter=` to\n> > allow users to filter out some objects from the main pack and move them\n> > into a new different pack.\n>\n> OK, this sidesteps the question I had on an earlier step rather\n> nicely. Instead of having to find out which ones are to be moved\n> away, just generating them in a separate location would be more\n> straight forward.\n>\n> The implementation does not seem to restrict where --filter-to\n> directory can be placed, but shouldn't it make sure that it is one\n> of the already specified alternates directories? Otherwise the user\n> will end up corrupting the repository, no?\n\nI don't think it should make sure that the implementation should\nrestrict where the --filter-to directory can be placed.\n\nIn version 3, that I just sent, I have written the following in the\ncommit message to explain this:\n\n\"\n Even in a different directory, this pack can be accessible if, for\n example, the Git alternates mechanism is used to point to it. In fact\n not using the Git alternates mechanism can corrupt a repo as the\n generated pack containing the filtered objects might not be accessible\n from the repo any more. So setting up the Git alternates mechanism\n should be done before using this feature if the user wants the repo to\n be fully usable while this feature is used.\n\n In some cases, like when a repo has just been cloned or when there is no\n other activity in the repo, it's Ok to setup the Git alternates\n mechanism afterwards though. It's also Ok to just inspect the generated\n packfile containing the filtered objects and then just move it into the\n '.git/objects/pack/' directory manually. That's why it's not necessary\n for this command to check that the Git alternates mechanism has been\n already setup.\n\"\n\nI haven't mentioned cases related to promisor remotes, but I think in\nsome of those cases the feature can be very useful too while there is\nno need to check that the Git alternates mechanism has been set up.\n\nIn version 3, the doc for the --filter-to option and the corresponding\ngc.repackFilterTo config flag look like this:\n\n+--filter-to=::\n+ Write the pack containing filtered out objects to the\n+ directory ``. Only useful with `--filter`. This can be\n+ used for putting the pack on a separate object directory that\n+ is accessed through the Git alternates mechanism. **WARNING:**\n+ If the packfile containing the filtered out objects is not\n+ accessible, the repo could be considered corrupt by Git as it\n+ migh not be able to access the objects in that packfile. See\n+ the `objects` and `objects/info/alternates` sections of\n+ linkgit:gitrepository-layout[5].\n\n+gc.repackFilterTo::\n+ When repacking and using a filter, see `gc.repackFilter`, the\n+ specified location will be used to create the packfile\n+ containing the filtered out objects. **WARNING:** The\n+ specified location should be accessible, using for example the\n+ Git alternates mechanism, otherwise the repo could be\n+ considered corrupt by Git as it might not be able to access the\n+ objects in that packfile. See the `--filter-to=` option\n+ of linkgit:git-repack[1] and the `objects/info/alternates`\n+ section of linkgit:gitrepository-layout[5].\n\nSo they warn about possible issues with the feature and link to some\nrelevant doc.\n\nNow if we think that it's not enough, I would implement a check in the\ncode that would warn users loudly if the directory specified by those\noptions is not accessible using the Git alternates mechanism. It would\nbe annoying I think that it would be too restrictive to error out in\nthat case though.\n"},{"id":"479799","messageId":"CAP8UFD0aa+EZQ2Q=C2WjWrNL9desg-KLLjOKS8BUBR4DS1ytsQ@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqh6qi1byn.fsf@gitster.g","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T09:01:48Z","receivedAt":"2023-07-24T09:02:37Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 5, 2023 at 7:53 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > This could be useful if, for example, some large blobs take a lot of\n> > precious space on fast storage while they are rarely accessed. It could\n> > make sense to move them into a separate cheaper, though slower, storage.\n> >\n> > In other use cases it might make sense to put all the blobs into\n> > separate storage.\n>\n> Minor nit. Aren't the above two the same use case?\n\nIn the first case only some large blobs are moved to slower storage\nand in the other case all the blobs are moved to slower storage. So\nyeah the use cases are very similar. Not sure if and how I can improve\nthe above wording though.\n\n> > This is done by running two `git pack-objects` commands. The first one\n> > is run with `--filter=`, using the specified filter. It\n> > packs objects while omitting the objects specified by the filter.\n> > Then another `git pack-objects` command is launched using\n> > `--stdin-packs`. We pass it all the previously existing packs into its\n> > stdin, so that it will pack all the objects in the previously existing\n> > packs. But we also pass into its stdin, the pack created by the previous\n> > `git pack-objects --filter=` command as well as the kept\n> > packs, all prefixed with '^', so that the objects in these packs will be\n> > omitted from the resulting pack.\n>\n> When I started reading the paragraph, the first question that came\n> to my mind was if these two pack-objects processes can and should be\n> run in parallel, which is answered in the part near the end of the\n> paragraph. It may be a good idea to start the paragraph with \"by\n> running `git pack-objects` command twice in a row\" or something to\n> make it clear that one should (and cannot be) run before the other\n> completes.\n\nOk, in version 3 that I just sent, that paragraph starts with:\n\n\"\n This is done by running `git pack-objects` twice in a row. The first\n command is run with `--filter=`, using the specified\n filter.\n\"\n\n> In fact, isn't the call site of write_filtered_pack() in this patch\n> a bit too early? The subprocess that runs with \"--stdin-packs\" is\n> started and told about the names of the pack we are going to create,\n> and it does not start processing until it reads everything (i.e. we\n> run fclose(in) in the write_filtered_pack() function), but the loop\n> over \"names\" string list in the caller that moves the tempfiles to\n> their final filenames comes after the call to close_object_store()\n> we see in the post context of the call to write_filtered_pack() that\n> is new in this patch.\n\nI think it can work if the call to write_filtered_pack() is either\nbefore the call to close_object_store() or after it. It would just use\nthe tempfiles with their temporary name in the first case and with\ntheir final name in the second case.\n\nwrite_filtered_pack() is very similar to write_cruft_pack() which is\ncalled before the call to close_object_store(), so I prefer to keep it\nbefore that call too, if possible, for consistency.\n\n> The \"--stdin-packs\" one is told to exclude objects that appear in\n> these packs, so if the main process is a bit slow to finalize the\n> packfiles it created (and told the \"--stdin-packs\" process about),\n> it will not lead to repository corruption---just some objects are\n> included in the packfiles \"--stdin-packs\" one creates even though\n> they do not have to. So it does not sound like a huge problem to\n> me, but still it somehow looks wrong. Am I misreading the code?\n\nI would have thought that as finish_pack_objects_cmd() calls\nfinish_command() the first pack-objects command (which is called\nwithout --stdout) should be completely finished and the packfiles\nfully created when write_filtered_pack() (or write_cruft_pack()) is\ncalled, even if the object store is not closed, but you might be\nright.\n\nPerhaps this could be dealt with separately though, as I think we\nmight want to fix write_cruft_pack() first then.\n"},{"id":"479800","messageId":"CAP8UFD2TvKYsAsJJ-L=5Mc6k1QnzL9D1qL=-Wy9ByTMupdzoZA@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqcz161b3s.fsf@gitster.g","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-24T09:02:21Z","receivedAt":"2023-07-24T09:02:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 5, 2023 at 8:12 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > +--filter=::\n> > + Remove objects matching the filter specification from the\n> > + resulting packfile and put them into a separate packfile. Note\n> > + that objects used in the working directory are not filtered\n> > + out. So for the split to fully work, it's best to perform it\n> > + in a bare repo and to use the `-a` and `-d` options along with\n> > + this option. See linkgit:git-rev-list[1] for valid\n> > + `` forms.\n>\n> After running the command with this option once, we will have two\n> packfiles, one with objects that match and the other with objects\n> that do not match the filter spec. Then what is the next step for\n> the user of this feature? Moving the former to a slower storage\n> was cited as a motivation for the feature, but can the user tell\n> which one of these two packfiles is the one that consists of the\n> filtered out objects? If there is no mechansim to do so, shouldn't\n> we have one to make this feature more usable?\n>\n> At the level of \"pack-objects\" command, we report the new packfiles\n> so that the user does not have to take \"ls .git/objects/pack\" before\n> and after the operation to compare and learn which ones are new.\n> I do not think \"repack\" that is a Porcelain should do such a\n> reporting on its standard output, but that means either the feature\n> should probably be done at the plumbing level (i.e. \"pack-objects\"),\n> or the marking of the new packfiles needs to be done in a way that\n> tools can later find them out, e.g. on the filesystem, similar to\n> the way \".keep\" marker tells which ones are not to be repacked, etc.\n\nI think commands like `git verify-pack -v ...` can already tell a bit\nabout the content of a packfile.\n\nAlso this patch series adds `test-tool find-pack` which can help too.\nIt could maybe be converted into a new `git verify-pack --find-pack`\noption if users want it.\n\nThen, as you later found out, there is the --filter-to= option\nadded later by this series.\n\nTo clarify this, I have added the following to the commit message in\nthe version 3 I just sent:\n\n\"\n It's possible to find which new packfile contains the filtered out\n objects using one of the following:\n\n - `git verify-pack -v ...`,\n - `test-tool find-pack ...`, which a previous commit added,\n - `--filter-to=`, which a following commit will add to specify\n where the pack containing the filtered out objects will be.\n\"\n"},{"id":"479823","messageId":"xmqqy1j5p40f.fsf@gitster.g","threadId":"59867","inReplyTo":"CAP8UFD1tqzp744j0KORw-zcgOn6Tufm4Kk3yct3vHGY29pbm-w@mail.gmail.com","subject":"Re: [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-24T18:18:24Z","receivedAt":"2023-07-24T18:18:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> In version 3, the doc for the --filter-to option and the corresponding\n> gc.repackFilterTo config flag look like this:\n>\n> +--filter-to=::\n> + Write the pack containing filtered out objects to the\n> + directory ``. Only useful with `--filter`. This can be\n> + used for putting the pack on a separate object directory that\n> + is accessed through the Git alternates mechanism. **WARNING:**\n> + If the packfile containing the filtered out objects is not\n> + accessible, the repo could be considered corrupt by Git as it\n\n\"could be considered\" -> \"can become\".\n\n> + migh not be able to access the objects in that packfile. See\n\n\"migh\" -> \"might\".\n\n> + the `objects` and `objects/info/alternates` sections of\n> + linkgit:gitrepository-layout[5].\n>\n> +gc.repackFilterTo::\n> + When repacking and using a filter, see `gc.repackFilter`, the\n> + specified location will be used to create the packfile\n> + containing the filtered out objects. **WARNING:** The\n> + specified location should be accessible, using for example the\n> + Git alternates mechanism, otherwise the repo could be\n> + considered corrupt by Git as it might not be able to access the\n> + objects in that packfile. See the `--filter-to=` option\n> + of linkgit:git-repack[1] and the `objects/info/alternates`\n> + section of linkgit:gitrepository-layout[5].\n>\n> So they warn about possible issues with the feature and link to some\n> relevant doc.\n\nIn all other parts of the system, we tend to avoid such an \"unsafe\nby default\" desgin, especially when the risk is known before there\nis an implementation, and instead allow an explicit end-user action\n(ranging from command line option to interactive confirmation) to\nopt-into more risky behaviour. Should we consider --filter-to as\nsuch an \"always risky and prone to repository corruption\" option\n(just like \"--hard\" to \"reset\" is always loses changes in the\nworking tree without warning)?\n\nI am OK with that myself, but others may disagree.\n\nCome to think of it, we haven't seen much reviews from those other\nthan Taylor. Are folks content with the direction this series is\ngoing in general?\n\nThanks.\n\n"},{"id":"479824","messageId":"xmqqila9p3j8.fsf@gitster.g","threadId":"59867","inReplyTo":"CAP8UFD0aa+EZQ2Q=C2WjWrNL9desg-KLLjOKS8BUBR4DS1ytsQ@mail.gmail.com","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-24T18:28:43Z","receivedAt":"2023-07-24T18:29:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> Minor nit. Aren't the above two the same use case?\n>\n> In the first case only some large blobs are moved to slower storage\n> and in the other case all the blobs are moved to slower storage. So\n> yeah the use cases are very similar. Not sure if and how I can improve\n> the above wording though.\n\nJust by removing one or the other, it would be quite improved, no?\nMoving away some blobs could move away all or just a selected\nsubset.\n\n> I think it can work if the call to write_filtered_pack() is either\n> before the call to close_object_store() or after it. It would just use\n> the tempfiles with their temporary name in the first case and with\n> their final name in the second case.\n>\n> write_filtered_pack() is very similar to write_cruft_pack() which is\n> called before the call to close_object_store(), so I prefer to keep it\n> before that call too, if possible, for consistency.\n\nAs long as the set-up is not racy, either would be OK, as the names\nare not recorded in the end result.\n\nIf the upstream tells the downstream the temporary's name and then\nfinializes the temporary to the final name before the downstream\nreacts to the input, however, then by the time downstream starts\nworking on the file, the file may not exist under its original,\ntemporary name, and that kind of race was what I was worried about.\n\n> Perhaps this could be dealt with separately though, as I think we\n> might want to fix write_cruft_pack() first then.\n\nSorry, I am not understanding this quite well. Do you mean we\nshould add one more known-to-be-racy-and-incorrect codepath because\nthere is already a codepath that needs to be fixed anyway?\n\nIf write_cruft_pack() has a similar issue, then yeah, let's fix that\nfirst (testing might be tricky for any racy bugs). And let's use\nthe same technique as used to fix it in this series, too, so that we\ndo not reintroduce a similar bug due to racy setup.\n\nThanks.\n\n"},{"id":"479838","messageId":"CAFLLRpL8yUb5eSBwCuEcHkgZEZcAdugua0robVUNGUE6Eik8Bw@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqy1j5p40f.fsf@gitster.g","subject":"Re: [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Robert Coup","fromEmail":"robert.coup@koordinates.com","sentAt":"2023-07-25T13:41:35Z","receivedAt":"2023-07-25T13:41:57Z","isPatch":true,"sender":{"key":"robert.coup@koordinates.com","avatar":"https://gravatar.com/avatar/d1a87d63ffb562b791992d8a119ebbdd742e703109d23333ca3fca51306ee95c?d=mp&s=160"},"body":"Hi Junio,\n\nOn Mon, 24 Jul 2023 at 19:18, Junio C Hamano wrote:\n> Come to think of it, we haven't seen much reviews from those other\n> than Taylor. Are folks content with the direction this series is\n> going in general?\n\nFor what it's worth I have a medium-term plan similar to Gitlab's with\nrespect to moving chunks of repositories onto lower cost storage media\n& to promisor remotes. Like others, I wasn't at all sure about the\noriginal approach (and commented at the time). What Christian is\nproposing here seems much cleaner, is usable without complex\ngymnastics or safety equipment, and provides a better building block\nfor future work.\n\nRob :)\n"},{"id":"479839","messageId":"CAP8UFD3b6gCog5P7WKzTuPQV2Lhf51=xO7ys+W7o0pGewJMcFg@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqila9p3j8.fsf@gitster.g","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-25T15:22:46Z","receivedAt":"2023-07-25T15:23:05Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jul 24, 2023 at 8:28 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> >> Minor nit. Aren't the above two the same use case?\n> >\n> > In the first case only some large blobs are moved to slower storage\n> > and in the other case all the blobs are moved to slower storage. So\n> > yeah the use cases are very similar. Not sure if and how I can improve\n> > the above wording though.\n>\n> Just by removing one or the other, it would be quite improved, no?\n> Moving away some blobs could move away all or just a selected\n> subset.\n\nOk, I have done that in my current version.\n\n> > I think it can work if the call to write_filtered_pack() is either\n> > before the call to close_object_store() or after it. It would just use\n> > the tempfiles with their temporary name in the first case and with\n> > their final name in the second case.\n> >\n> > write_filtered_pack() is very similar to write_cruft_pack() which is\n> > called before the call to close_object_store(), so I prefer to keep it\n> > before that call too, if possible, for consistency.\n>\n> As long as the set-up is not racy, either would be OK, as the names\n> are not recorded in the end result.\n>\n> If the upstream tells the downstream the temporary's name and then\n> finializes the temporary to the final name before the downstream\n> reacts to the input,\n\nIt doesn't seem to me that it's what happens.\n\nWe have the following order:\n\n - finish_pack_objects_cmd() is called for the first pack-objects\nprocess. It populates the 'names' string_list with the temporary name\nof the packfile it generated (which doesn't contain the filtered out\nobjects) and calls finish_command() to finish the first pack-objects\nprocess. So as far as I understand nothing can be written anymore to\nthe packfile when finish_pack_objects_cmd() returns.\n\n - write_filtered_pack() is called. It starts the second pack-objects\nprocess and passes it the temporary name of the packfile that was just\nwritten, taking it from the 'names' string_list. It then calls\nfinish_pack_objects_cmd() for the second process which populates the\n'names' string_list with the temporary name of the packfile created by\nthe second process and finishes the second process. So nothing can\nthen be written in the second packfile anymore.\n\n - close_object_store() is called which renames the packfiles from\nthe 'names' string_list giving them their final name.\n\nSo the final names are given only once both processes are finished and\nboth packfiles have been fully written.\n\n> however, then by the time downstream starts\n> working on the file, the file may not exist under its original,\n> temporary name, and that kind of race was what I was worried about.\n>\n> > Perhaps this could be dealt with separately though, as I think we\n> > might want to fix write_cruft_pack() first then.\n>\n> Sorry, I am not understanding this quite well. Do you mean we\n> should add one more known-to-be-racy-and-incorrect codepath because\n> there is already a codepath that needs to be fixed anyway?\n\nNo.\n\n> If write_cruft_pack() has a similar issue, then yeah, let's fix that\n> first (testing might be tricky for any racy bugs). And let's use\n> the same technique as used to fix it in this series, too, so that we\n> do not reintroduce a similar bug due to racy setup.\n\nYeah, that's what I mean.\n\nI am not sure the race actually exists though. I have tried to explain\nwhy it seems to me that things look correct, but from previous\nexperience I know that you are very often right, and I might have\nmissed something.\n\nThanks.\n"},{"id":"479841","messageId":"CAP8UFD0YzYR0y7st3CXbed9OdzbxRKK3wy1TzFGWxx+m3DfiDw@mail.gmail.com","threadId":"59867","inReplyTo":"xmqqy1j5p40f.fsf@gitster.g","subject":"Re: [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-07-25T15:45:10Z","receivedAt":"2023-07-25T15:45:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jul 24, 2023 at 8:18 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > In version 3, the doc for the --filter-to option and the corresponding\n> > gc.repackFilterTo config flag look like this:\n> >\n> > +--filter-to=::\n> > + Write the pack containing filtered out objects to the\n> > + directory ``. Only useful with `--filter`. This can be\n> > + used for putting the pack on a separate object directory that\n> > + is accessed through the Git alternates mechanism. **WARNING:**\n> > + If the packfile containing the filtered out objects is not\n> > + accessible, the repo could be considered corrupt by Git as it\n>\n> \"could be considered\" -> \"can become\".\n>\n> > + migh not be able to access the objects in that packfile. See\n>\n> \"migh\" -> \"might\".\n>\n> > + the `objects` and `objects/info/alternates` sections of\n> > + linkgit:gitrepository-layout[5].\n\nThanks for catching these, they are fixed in my current version.\n"},{"id":"479844","messageId":"xmqqr0owlyud.fsf@gitster.g","threadId":"59867","inReplyTo":"CAFLLRpL8yUb5eSBwCuEcHkgZEZcAdugua0robVUNGUE6Eik8Bw@mail.gmail.com","subject":"Re: [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-25T16:50:34Z","receivedAt":"2023-07-25T16:52:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robert Coup writes:\n\n> ... Like others, I wasn't at all sure about the\n> original approach (and commented at the time). What Christian is\n> proposing here seems much cleaner, is usable without complex\n> gymnastics or safety equipment, and provides a better building block\n> for future work.\n\nNice to hear a positive feedback [*].\n\nThanks.\n\n[Footnote]\n\n * Of course negative ones as long as they are consturctive are also\n welcome;-)\n"},{"id":"479845","messageId":"xmqqcz0fnbs8.fsf@gitster.g","threadId":"59867","inReplyTo":"CAP8UFD3b6gCog5P7WKzTuPQV2Lhf51=xO7ys+W7o0pGewJMcFg@mail.gmail.com","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-25T17:25:43Z","receivedAt":"2023-07-25T17:25:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> We have the following order:\n>\n> - finish_pack_objects_cmd() is called for the first pack-objects\n> process. It populates the 'names' string_list with the temporary name\n> of the packfile it generated (which doesn't contain the filtered out\n> objects) and calls finish_command() to finish the first pack-objects\n> process. So as far as I understand nothing can be written anymore to\n> the packfile when finish_pack_objects_cmd() returns.\n>\n> - write_filtered_pack() is called. It starts the second pack-objects\n> process and passes it the temporary name of the packfile that was just\n> written, taking it from the 'names' string_list. It then calls\n> finish_pack_objects_cmd() for the second process which populates the\n> 'names' string_list with the temporary name of the packfile created by\n> the second process and finishes the second process. So nothing can\n> then be written in the second packfile anymore.\n>\n> - close_object_store() is called which renames the packfiles from\n> the 'names' string_list giving them their final name.\n\n\"which renames\" -> \"and then we enter a loop to rename\" and\nclose_object_store() itself and its callees do not do much, but yes,\nyou are right. As finish_pack_objects_cmd() is synchronous, there\ncannot be such race as a feared (if we were feeding the pack objects\nprocess that collects the objects that would have filtered out with\nthe final packfile paths, and if we were only renaming them to the\nfinal paths after that close_object_store() call, then the process\nwould want to see the final names that are not there yet, but that's\nnot a race but a bug that would reliably trigger).\n\n> So the final names are given only once both processes are finished and\n> both packfiles have been fully written.\n\nThanks for walking through the codepaths involved. We are good\nthen.\n"},{"id":"479867","messageId":"ZMBO5A0WxWystYyv@nand.local","threadId":"59867","inReplyTo":"20230724085909.3831831-2-christian.couder@gmail.com","subject":"Re: [PATCH v3 1/8] pack-objects: allow `--filter` without `--stdout`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-25T22:38:28Z","receivedAt":"2023-07-25T22:45:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 24, 2023 at 10:59:02AM +0200, Christian Couder wrote:\n> diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\n> index b26d476c64..2ff3eef9a3 100755\n> --- a/t/t5317-pack-objects-filter-objects.sh\n> +++ b/t/t5317-pack-objects-filter-objects.sh\n> @@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n> \t! grep blob verify_result\n> '\n>\n> +test_expect_success 'verify blob:none packfile without --stdout' '\n> +\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n> +\tHEAD\n> +\tEOF\n> +\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n> +\t! grep blob verify_result\n> +'\n\nJust a couple of style nits here. It's a little strange (for me, at\nleast) to see the heredoc into a git process. I wonder if it might be\nclearer to write something like:\n\n echo HEAD >in &&\n git -C r1 pack-objects --revs --filter=blob:none $packdir/pack Create a new finish_pack_objects_cmd() to refactor duplicated code\n> that handles reading the packfile names from the output of a\n> `git pack-objects` command and putting it into a string_list, as well as\n> calling finish_command().\n>\n> While at it, beautify a code comment a bit in the new function.\n\nEverything here looks good to me. Thanks for cleaning this up into its\nown function and DRY-ing things up a little bit.\n\n> Signed-off-by: Christian Couder ---\n> builtin/repack.c | 70 +++++++++++++++++++++++-------------------------\n> 1 file changed, 33 insertions(+), 37 deletions(-)\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index aea5ca9d44..96af2d1caf 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -696,6 +696,36 @@ static void remove_redundant_bitmaps(struct string_list *include,\n> \tstrbuf_release(&path);\n> }\n>\n> +static int finish_pack_objects_cmd(struct child_process *cmd,\n> +\t\t\t\t struct string_list *names,\n> +\t\t\t\t int local)\n\nI'm glad to see \"local\" in the arguments list ;-). I think that the\nimplementation came out nice and clean here.\n\nThanks,\nTaylor\n"},{"id":"479870","messageId":"ZMBQ97ucRg9gO4gW@nand.local","threadId":"59867","inReplyTo":"20230724085909.3831831-5-christian.couder@gmail.com","subject":"Re: [PATCH v3 4/8] repack: refactor finding pack prefix","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-25T22:47:19Z","receivedAt":"2023-07-25T22:51:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 24, 2023 at 10:59:05AM +0200, Christian Couder wrote:\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 96af2d1caf..21e3b89f27 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -783,6 +783,17 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n> \treturn finish_pack_objects_cmd(&cmd, names, local);\n> }\n>\n> +static const char *find_pack_prefix(void)\n> +{\n> +\tconst char *pack_prefix;\n> +\tif (!skip_prefix(packtmp, packdir, &pack_prefix))\n\nI wonder if this might be a good opportunity to pass \"packtmp\" and\n\"packdir\" as arguments to the function. I know that these are globals,\nbut it at least nudges us in the right direction away from adding more\nglobal variables.\n\nThanks,\nTaylor\n"},{"id":"479871","messageId":"ZMBQRRppsadqa8Ua@nand.local","threadId":"59867","inReplyTo":"20230724085909.3831831-3-christian.couder@gmail.com","subject":"Re: [PATCH v3 2/8] t/helper: add 'find-pack' test-tool","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-25T22:44:21Z","receivedAt":"2023-07-25T22:52:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 24, 2023 at 10:59:03AM +0200, Christian Couder wrote:\n> ---\n> Makefile | 1 +\n> t/helper/test-find-pack.c | 35 +++++++++++++++++++++++++++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> 4 files changed, 38 insertions(+)\n> create mode 100645 t/helper/test-find-pack.c\n\nEverything that you wrote here seems reasonable to me, and the\nimplementation of the new test tool is very straightforward.\n\nI'm pretty sure that everything here is correct, and we'll implicitly\ntest the behavior of the new helper in following patches.\n\nThat said, I think that it might be prudent here to \"test the tests\" and\nwrite a simple test script that exercises this test helper over a more\ntrivial case. There is definitely prior art for testing our helpers\ndirectly in the t00?? tests.\n\nAmong the test helpers that I can think of off the top of my head, I\nthink a good handful of them have tests:\n\n - t0011-hashmap.sh\n - t0015-hash.sh\n - t0016-oidmap.sh\n - t0019-json-writer.sh\n - t0052-simple-ipc.sh\n - t0060-path-utils.sh\n - t0061-run-command.sh\n - t0063-string-list.sh\n - t0064-oid-array.sh\n - t0066-dir-iterator.sh\n - t0095-bloom.sh\n\nI would definitely recommend adding a test here, too. Like I said\nearlier, I think that you are implicitly testing the new behavior here,\nbut it's going to happen in much more complicated environments than\nsomething you could construct synthetically here.\n\nThanks,\nTaylor\n"},{"id":"479873","messageId":"ZMBU+SALVQthOgC7@nand.local","threadId":"59867","inReplyTo":"20230724085909.3831831-6-christian.couder@gmail.com","subject":"Re: [PATCH v3 5/8] repack: add `--filter=` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-25T23:04:25Z","receivedAt":"2023-07-25T23:04:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 24, 2023 at 10:59:06AM +0200, Christian Couder wrote:\n> This feature is implemented by running `git pack-objects` twice in a\n> row. The first command is run with `--filter=`, using the\n> specified filter. It packs objects while omitting the objects specified\n> by the filter. Then another `git pack-objects` command is launched using\n> `--stdin-packs`. We pass it all the previously existing packs into its\n> stdin, so that it will pack all the objects in the previously existing\n> packs. But we also pass into its stdin, the pack created by the previous\n> `git pack-objects --filter=` command as well as the kept\n> packs, all prefixed with '^', so that the objects in these packs will be\n> omitted from the resulting pack. The result is that only the objects\n> filtered out by the first `git pack-objects` command are in the pack\n> resulting from the second `git pack-objects` command.\n\nVery nice. I appreciate you taking my suggestion here; I'm hopeful that\nit simplified things and resulted in fewer new lines of code.\n\n> Signed-off-by: John Cai \n> Signed-off-by: Christian Couder \n> ---\n> Documentation/git-repack.txt | 12 +++++++\n> builtin/repack.c | 67 ++++++++++++++++++++++++++++++++++++\n> t/t7700-repack.sh | 24 +++++++++++++\n> 3 files changed, 103 insertions(+)\n>\n> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> index 4017157949..6d5bec7716 100644\n> --- a/Documentation/git-repack.txt\n> +++ b/Documentation/git-repack.txt\n> @@ -143,6 +143,18 @@ depth is 4095.\n> \ta larger and slower repository; see the discussion in\n> \t`pack.packSizeLimit`.\n>\n> +--filter=::\n> +\tRemove objects matching the filter specification from the\n> +\tresulting packfile and put them into a separate packfile. Note\n> +\tthat objects used in the working directory are not filtered\n> +\tout. So for the split to fully work, it's best to perform it\n> +\tin a bare repo and to use the `-a` and `-d` options along with\n> +\tthis option. Also `--no-write-bitmap-index` (or the\n> +\t`repack.writebitmaps` config option set to `false`) should be\n> +\tused otherwise writing bitmap index will fail, as it supposes\n> +\ta single packfile containing all the objects. See\n> +\tlinkgit:git-rev-list[1] for valid `` forms.\n> +\n> -b::\n> --write-bitmap-index::\n> \tWrite a reachability bitmap index as part of the repack. This\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 21e3b89f27..2c81b7738e 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -53,6 +53,7 @@ struct pack_objects_args {\n> \tconst char *depth;\n> \tconst char *threads;\n> \tconst char *max_pack_size;\n> +\tconst char *filter;\n> \tint no_reuse_delta;\n> \tint no_reuse_object;\n> \tint quiet;\n> @@ -166,6 +167,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n> \t\tstrvec_pushf(&cmd->args, \"--threads=%s\", args->threads);\n> \tif (args->max_pack_size)\n> \t\tstrvec_pushf(&cmd->args, \"--max-pack-size=%s\", args->max_pack_size);\n> +\tif (args->filter)\n> +\t\tstrvec_pushf(&cmd->args, \"--filter=%s\", args->filter);\n> \tif (args->no_reuse_delta)\n> \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n> \tif (args->no_reuse_object)\n> @@ -726,6 +729,57 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n> \treturn finish_command(cmd);\n> }\n>\n> +static int write_filtered_pack(const struct pack_objects_args *args,\n> +\t\t\t const char *destination,\n> +\t\t\t const char *pack_prefix,\n> +\t\t\t struct string_list *names,\n> +\t\t\t struct string_list *existing_packs,\n> +\t\t\t struct string_list *existing_kept_packs)\n> +{\n> +\tstruct child_process cmd = CHILD_PROCESS_INIT;\n> +\tstruct string_list_item *item;\n> +\tFILE *in;\n> +\tint ret;\n> +\tconst char *scratch;\n> +\tint local = skip_prefix(destination, packdir, &scratch);\n> +\n> +\t/* We need to copy 'args' to modify it */\n> +\tstruct pack_objects_args new_args = *args;\n> +\n> +\t/* No need to filter again */\n> +\tnew_args.filter = NULL;\n> +\n> +\tprepare_pack_objects(&cmd, &new_args, destination);\n> +\n> +\tstrvec_push(&cmd.args, \"--stdin-packs\");\n> +\n> +\tcmd.in = -1;\n> +\n> +\tret = start_command(&cmd);\n> +\tif (ret)\n> +\t\treturn ret;\n\n\n> +\t/*\n> +\t * names has a confusing double use: it both provides the list\n> +\t * of just-written new packs, and accepts the name of the\n> +\t * filtered pack we are writing.\n> +\t *\n> +\t * By the time it is read here, it contains only the pack(s)\n> +\t * that were just written, which is exactly the set of packs we\n> +\t * want to consider kept.\n> +\t */\n\nI think that this comment partially comes from the cruft pack code,\nwhere we use the `names` string list both to reference existing packs at\nthe start of the repack, and to keep track of the pack we just wrote (to\nexclude its contents from the cruft pack).\n\nBut I think we only write into \"names\" via finish_pack_objects_cmd() to\nrecord the name of the pack we just wrote containing objects which\ndidn't meet the filter's conditions.\n\nSo I think that leaving this comment in is OK, but TBH I was on the\nfence when I wrote that back in f9825d1cf75 (builtin/repack.c: support\ngenerating a cruft pack, 2022-05-20), so I would just as soon drop it.\n\n> +\tin = xfdopen(cmd.in, \"w\");\n> +\tfor_each_string_list_item(item, names)\n> +\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n> +\tfor_each_string_list_item(item, existing_packs)\n> +\t\tfprintf(in, \"%s.pack\\n\", item->string);\n\n> +\tfor_each_string_list_item(item, existing_kept_packs)\n> +\t\tfprintf(in, \"^%s.pack\\n\", item->string);\n\nI think we may only want to do this if `honor_pack_keep` is zero.\nOtherwise we'd avoid packing objects that appear in kept packs, even if\nthe caller told us to include objects found in kept packs.\n\n> +\tfclose(in);\n> +\n> +\treturn finish_pack_objects_cmd(&cmd, names, local);\n> +}\n> +\n> static int write_cruft_pack(const struct pack_objects_args *args,\n> \t\t\t const char *destination,\n> \t\t\t const char *pack_prefix,\n> @@ -858,6 +912,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\t\t\tN_(\"limits the maximum number of threads\")),\n> \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n> \t\t\t\tN_(\"maximum size of each packfile\")),\n> +\t\tOPT_STRING(0, \"filter\", &po_args.filter, N_(\"args\"),\n> +\t\t\t\tN_(\"object filtering\")),\n\nI suppose we're storing the filter as a string here because we're just\ngoing to pass it down to pack-objects directly. That part makes sense,\nbut I think we are producing subtly inconsistent behavior when\nspecifying multiple --filter options.\n\nIIRC, passing --filter more than once down to pack-objects produces a\nfilter whose objects match all of the individually specified\nsub-filters. But IIUC, using OPT_STRING here means that later\n`--filter`'s override earlier ones.\n\nSo I think at minimum we'd want to store the filter arguments in a\nstrvec. But I would probably just as soon parse them into a bona-fide\nlist_objects_filter_options struct, and then reconstruct the arguments\nto pack-objects based on that.\n\n> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n> index 27b66807cd..0a2c73bca7 100755\n> --- a/t/t7700-repack.sh\n> +++ b/t/t7700-repack.sh\n> @@ -327,6 +327,30 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n> \ttest_must_be_empty actual\n> '\n>\n> +test_expect_success 'repacking with a filter works' '\n> +\tgit -C bare.git repack -a -d &&\n> +\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n\nHuh! I never knew about the test_stdout_line_count function, I thought\nthat we always just had test_line_count. Neat!\n\n> +\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n> +\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n> +\tcommit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n> +\ttest -n \"$commit_pack\" &&\n\nI wonder if the test-tool itself should exit with a non-zero code if it\ncan't find the given object in any pack. It would at least allow us to\ndrop the \"test -n $foo\" after every invocation of the test-helper in\nthis test.\n\nArguably callers may want to ensure that an object doesn't exist in any\npack, and this would be inconvenient for them, since they'd have to\nwrite something like:\n\n test_must_fail test-tool find-pack $obj\n\nbut I think a more direct test like\n\n test_must_fail git cat-file -t $obj\n\nwould do just as well.\n\n> +\tblob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n> +\ttest -n \"$blob_pack\" &&\n> +\ttest \"$commit_pack\" != \"$blob_pack\" &&\n> +\ttree_pack=$(test-tool -C bare.git find-pack HEAD^{tree}) &&\n> +\ttest \"$tree_pack\" = \"$commit_pack\" &&\n> +\tblob_pack2=$(test-tool -C bare.git find-pack HEAD:file2) &&\n> +\ttest \"$blob_pack2\" = \"$blob_pack\"\n> +'\n\nThis all looks good, but I think there are a couple of more things that\nwe'd want to test for here:\n\n - That the list of all objects appears the same before and after all\n of the repacking. I think that this is tested implicitly already in\n your test, but having it written down explicitly would harden this\n against regressions that cause us to inadvertently delete an object\n we shouldn't have.\n\n (FWIW, I think this would be limited to running something like \"git\n cat-file --batch-check='%(objectname)' --batch-all-objects\" before\n and after all of the repacking, and ensuring that the two test_cmp\n without failure).\n\n - Another thing that I don't think we're testing here is that objects\n that *don't* match the filter don't appear in one of the filtered\n packs. I think we'd probably want to assert on the exact contents of\n the pack by dumping the list of objects into a file like \"expect\",\n and then dumping the actual set of objects with \"git show-index\n <$idx | cut -d' ' -f2\" or something.\n\nAnother thought from the OPT_STRING business above is that we probably\nwant to test this with non-trivial filter arguments. There are probably\na handful of interesting cases here, like passing `--no-filter`, passing\n`--filter` multiple times, passing invalid values for `--filter`, etc.\n\n> +test_expect_success '--filter fails with --write-bitmap-index' '\n> +\ttest_must_fail git -C bare.git repack -a -d --write-bitmap-index \\\n> +\t\t--filter=blob:none &&\n\nDo we want to ensure that we get the exit code corresponding with\nshowing the usage text? I could go either way, but I do think that we\nshould grep through the output on stderr to ensure that we get the\nappropriate error message.\n\n> +\tgit -C bare.git repack -a -d --no-write-bitmap-index \\\n> +\t\t--filter=blob:none\n\nI don't think that this test is adding anything that the above\n\"repacking with a filter works\" test isn't covering already.\n\nThanks,\nTaylor\n"},{"id":"479874","messageId":"ZMBVzWoFQCUCSTwE@nand.local","threadId":"59867","inReplyTo":"20230724085909.3831831-7-christian.couder@gmail.com","subject":"Re: [PATCH v3 6/8] gc: add `gc.repackFilter` config option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-25T23:07:57Z","receivedAt":"2023-07-25T23:08:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 24, 2023 at 10:59:07AM +0200, Christian Couder wrote:\n> A previous commit has implemented `git repack --filter=` to\n> allow users to filter out some objects from the main pack and move them\n> into a new different pack.\n>\n> Users might want to perform such a cleanup regularly at the same time as\n> they perform other repacks and cleanups, so as part of `git gc`.\n>\n> Let's allow them to configure a for that purpose using a\n> new gc.repackFilter config option.\n\nMakes sense.\n\n> Now when `git gc` will perform a repack with a configured\n> through this option and not empty, the repack process will be passed a\n> corresponding `--filter=` argument.\n\nI may be missing something, but what happens if the user has configured\ngc.repackFilter, but passes additional filters over the command-line\narguments? I'm not sure whether these should be AND'd with the existing\nfilters in config, or if they should reset them to zero, or something\nelse.\n\nRegardless, I think it would be beneficial to users if we spelled this\nout in git-gc(1) instead of just this patch message here.\n\n> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\n> index 69509d0c11..5b89faf505 100755\n> --- a/t/t6500-gc.sh\n> +++ b/t/t6500-gc.sh\n> @@ -202,6 +202,18 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n> \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n> '\n>\n> +test_expect_success 'gc.repackFilter launches repack with a filter' '\n> +\ttest_when_finished \"rm -rf bare.git\" &&\n> +\tgit clone --no-local --bare . bare.git &&\n> +\n> +\tgit -C bare.git -c gc.cruftPacks=false gc &&\n> +\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n> +\n> +\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none -c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n\nNit: can we wrap this across multiple lines?\n\n> +\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n> +\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n> +'\n\nI think the `test_subcommand` helper might work here, and it would allow\nyou to avoid writing a long grep invocation.\n\nThanks,\nTaylor\n"},{"id":"479875","messageId":"xmqqo7jzh9mh.fsf@gitster.g","threadId":"59867","inReplyTo":"xmqqcz0fnbs8.fsf@gitster.g","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-25T23:08:54Z","receivedAt":"2023-07-25T23:09:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Thanks for walking through the codepaths involved. We are good\n> then.\n\nSorry, but not so fast.\n\nhttps://github.com/git/git/actions/runs/5661445152 (seen with this topic)\nhttps://github.com/git/git/actions/runs/5662517690 (seen w/o this topic)\n\nThe former fails t7700 in the linux-TEST-vars job, while the latter\npasses the same job.\n\nThanks.\n\n"},{"id":"479876","messageId":"ZMBWZFoeEXcn5fXZ@nand.local","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"Re: [PATCH v3 0/8] Repack objects into separate packfiles based on a filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-25T23:10:28Z","receivedAt":"2023-07-25T23:10:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 24, 2023 at 10:59:01AM +0200, Christian Couder wrote:\n> # Changes since version 2\n>\n> Thanks to Junio who reviewed both version 1 and 2, and to Taylor who\n> reviewed version 1! The changes are the following:\n\nApologies for not getting to the second version sooner! It fell off of\nmy post-vacation to-do list, and I'm only just getting to the third\nround.\n\nOverall I am happy with the direction here and think that this is on\nthe right track. The major points (using --stdin-packs to implement the\ncomplementary pack, the behavior of --filter-to, etc.) all look good to\nme.\n\nBut I think there are a handful of smaller issues that we may want to at\nleast discuss first. If so, I think that a handful of them merit a\nreroll. But I imagine that that rerolled version would be ready to get\npicked up.\n\nThanks,\nTaylor\n"},{"id":"479878","messageId":"xmqqcz0fh7o0.fsf@gitster.g","threadId":"59867","inReplyTo":"ZMBO5A0WxWystYyv@nand.local","subject":"Re: [PATCH v3 1/8] pack-objects: allow `--filter` without `--stdout`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-25T23:51:11Z","receivedAt":"2023-07-25T23:51:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Mon, Jul 24, 2023 at 10:59:02AM +0200, Christian Couder wrote:\n>> diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\n>> index b26d476c64..2ff3eef9a3 100755\n>> --- a/t/t5317-pack-objects-filter-objects.sh\n>> +++ b/t/t5317-pack-objects-filter-objects.sh\n>> @@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n>> \t! grep blob verify_result\n>> '\n>>\n>> +test_expect_success 'verify blob:none packfile without --stdout' '\n>> +\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n>> +\tHEAD\n>> +\tEOF\n>> +\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n>> +\t! grep blob verify_result\n>> +'\n>\n> Just a couple of style nits here. It's a little strange (for me, at\n> least) to see the heredoc into a git process.\n\nFWIW, \"git\" is after all no different from any other command and\nredirecting here-doc, especially with an option like \"--stdin\" is in\neffect, is perfectly sensible, I think. Preparing an input in a\nfile (e.g. \"cat >file < ... I am less certain\n> about redirecting the output into a file \"packhash\", only to cat it back\n> out.\n\nBut that would make the syntax awkward. Do you mean something along\nthis line?\n\n\tvar=$(git ... <<-EOF\n\t\there text\n\tEOF\n\t) &&\n\tgit ... mypackname-$var.pack &&\n\t...\n\nSomehow here-doc and $(command subsitution) does not visually mix\nwell.\n\nAlso, $var will not be inspectable when running this test under \"-i\n-v\", so it hurts debuggability without taking the output in a\ntemporary file. You could do \"-x\", of course, but that would make\neverything ultra verbose, so...\n"},{"id":"480280","messageId":"20230808082608.582319-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:07Z","receivedAt":"2023-08-08T16:16:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage.\n\nEven in a different directory, this pack can be accessible if, for\nexample, the Git alternates mechanism is used to point to it. In fact\nnot using the Git alternates mechanism can corrupt a repo as the\ngenerated pack containing the filtered objects might not be accessible\nfrom the repo any more. So setting up the Git alternates mechanism\nshould be done before using this feature if the user wants the repo to\nbe fully usable while this feature is used.\n\nIn some cases, like when a repo has just been cloned or when there is no\nother activity in the repo, it's Ok to setup the Git alternates\nmechanism afterwards though. It's also Ok to just inspect the generated\npackfile containing the filtered objects and then just move it into the\n'.git/objects/pack/' directory manually. That's why it's not necessary\nfor this command to check that the Git alternates mechanism has been\nalready setup.\n\nWhile at it, as an example to show that `--filter` and `--filter-to`\nwork well with other options, let's also add a test to check that these\noptions work well with `--max-pack-size`.\n\nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 11 +++++++\n builtin/repack.c | 10 +++++-\n t/t7700-repack.sh | 62 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 6d5bec7716..8545a32667 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -155,6 +155,17 @@ depth is 4095.\n \ta single packfile containing all the objects. See\n \tlinkgit:git-rev-list[1] for valid `` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. Only useful with `--filter`. This can be\n+\tused for putting the pack on a separate object directory that\n+\tis accessed through the Git alternates mechanism. **WARNING:**\n+\tIf the packfile containing the filtered out objects is not\n+\taccessible, the repo can become corrupt as it might not be\n+\tpossible to access the objects in that packfile. See the\n+\t`objects` and `objects/info/alternates` sections of\n+\tlinkgit:gitrepository-layout[5].\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 876c115cdc..f5bc650c1e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -870,6 +870,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -922,6 +923,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1071,6 +1074,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (po_args.filter_options.choice)\n \t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n \t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\telse if (filter_to)\n+\t\tdie(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n \n \tif (geometry)\n \t\tcmd.in = -1;\n@@ -1159,8 +1164,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter_options.choice) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\n \t\tret = write_filtered_pack(&po_args,\n-\t\t\t\t\t packtmp,\n+\t\t\t\t\t filter_to,\n \t\t\t\t\t find_pack_prefix(packdir, packtmp),\n \t\t\t\t\t &keep_pack_list,\n \t\t\t\t\t &names,\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 5d3e53134c..9b1e189a62 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -461,6 +461,68 @@ test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n \t)\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 0 HEAD:file1) &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack -c 1 $blob_hash) &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n+test_expect_success '--filter works with --max-pack-size' '\n+\trm -rf filtered.git &&\n+\tgit init --bare filtered.git &&\n+\tgit init max-pack-size &&\n+\t(\n+\t\tcd max-pack-size &&\n+\t\ttest_commit base &&\n+\t\t# two blobs which exceed the maximum pack size\n+\t\ttest-tool genrandom foo 1048576 >foo &&\n+\t\tgit hash-object -w foo &&\n+\t\ttest-tool genrandom bar 1048576 >bar &&\n+\t\tgit hash-object -w bar &&\n+\t\tgit add foo bar &&\n+\t\tgit commit -m \"adding foo and bar\"\n+\t) &&\n+\tgit clone --no-local --bare max-pack-size max-pack-size.git &&\n+\t(\n+\t\tcd max-pack-size.git &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--max-pack-size=1M \\\n+\t\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\t\techo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n+\n+\t\t# Check that the 3 blobs are in different packfiles in filtered.git\n+\t\ttest_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar) &&\n+\t\tbase_pack=$(test-tool find-pack -c 1 HEAD:base.t) &&\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" != \"$base_pack\" &&\n+\t\ttest \"$bar_pack\" != \"$base_pack\" &&\n+\t\tfor pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n+\t\tdo\n+\t\t\tcase \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n+\t\tdone\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480286","messageId":"CAP8UFD1Z3O5Wf8D9dYZdgPuOWgo7NbQsHAWifkPUGE=Eo1U7EQ@mail.gmail.com","threadId":"59867","inReplyTo":"ZMBVzWoFQCUCSTwE@nand.local","subject":"Re: [PATCH v3 6/8] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:38:26Z","receivedAt":"2023-08-08T16:40:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 26, 2023 at 1:07 AM Taylor Blau wrote:\n>\n> On Mon, Jul 24, 2023 at 10:59:07AM +0200, Christian Couder wrote:\n> > A previous commit has implemented `git repack --filter=` to\n> > allow users to filter out some objects from the main pack and move them\n> > into a new different pack.\n> >\n> > Users might want to perform such a cleanup regularly at the same time as\n> > they perform other repacks and cleanups, so as part of `git gc`.\n> >\n> > Let's allow them to configure a for that purpose using a\n> > new gc.repackFilter config option.\n>\n> Makes sense.\n>\n> > Now when `git gc` will perform a repack with a configured\n> > through this option and not empty, the repack process will be passed a\n> > corresponding `--filter=` argument.\n>\n> I may be missing something, but what happens if the user has configured\n> gc.repackFilter, but passes additional filters over the command-line\n> arguments? I'm not sure whether these should be AND'd with the existing\n> filters in config, or if they should reset them to zero, or something\n> else.\n\n`git gc` doesn't recognize `--filter=<...>` arguments, only `git\nrepack` is being teached to recognize it in this patch series. So I\ndon't see how there could be multiple such arguments on the command\nline when `git gc` is used.\n\nAlso in version 4 `git repack` can be passed many such arguments\nanyway. So I think we are good.\n\nWe could support multiple gc.repackFilter config options, but on the\nother hand using something like\n`combine:++...` should work, as the content\nof the option is passed as-is to the command line. So we can leave\nthat improvement for later if people don't like the `combine:...` and\nare interested in it.\n\n> Regardless, I think it would be beneficial to users if we spelled this\n> out in git-gc(1) instead of just this patch message here.\n\nI am not sure what should be spelled out. I think we refer people to\nthe `repack --filter=...` option which in turn refers to the `rev-list\n--filter=...` which contains a good amount of documentation about how\n`--filter=...` works, including the fact that `combine:...` can be\nused and that multiple `--filter=...` options can be passed.\n\n> > diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\n> > index 69509d0c11..5b89faf505 100755\n> > --- a/t/t6500-gc.sh\n> > +++ b/t/t6500-gc.sh\n> > @@ -202,6 +202,18 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n> > grep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n> > '\n> >\n> > +test_expect_success 'gc.repackFilter launches repack with a filter' '\n> > + test_when_finished \"rm -rf bare.git\" &&\n> > + git clone --no-local --bare . bare.git &&\n> > +\n> > + git -C bare.git -c gc.cruftPacks=false gc &&\n> > + test_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n> > +\n> > + GIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none -c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n>\n> Nit: can we wrap this across multiple lines?\n\nDone in version 4.\n\n> > + test_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n> > + grep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n> > +'\n>\n> I think the `test_subcommand` helper might work here, and it would allow\n> you to avoid writing a long grep invocation.\n\nOther tests related to gc.reflogExpire above use a grep invocation\nsimilar to this one, while `test_subcommand` isn't used in the test\nscript, so I think the grep invocation makes the whole script a bit\neasier to understand.\n\nThanks.\n"},{"id":"480287","messageId":"20230808082608.582319-6-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 5/8] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:05Z","receivedAt":"2023-08-08T16:40:40Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new option puts the objects specified by `` into a\nseparate packfile.\n\nThis could be useful if, for example, some blobs take up a lot of\nprecious space on fast storage while they are rarely accessed. It could\nmake sense to move them into a separate cheaper, though slower, storage.\n\nIt's possible to find which new packfile contains the filtered out\nobjects using one of the following:\n\n - `git verify-pack -v ...`,\n - `test-tool find-pack ...`, which a previous commit added,\n - `--filter-to=`, which a following commit will add to specify\n where the pack containing the filtered out objects will be.\n\nThis feature is implemented by running `git pack-objects` twice in a\nrow. The first command is run with `--filter=`, using the\nspecified filter. It packs objects while omitting the objects specified\nby the filter. Then another `git pack-objects` command is launched using\n`--stdin-packs`. We pass it all the previously existing packs into its\nstdin, so that it will pack all the objects in the previously existing\npacks. But we also pass into its stdin, the pack created by the previous\n`git pack-objects --filter=` command as well as the kept\npacks, all prefixed with '^', so that the objects in these packs will be\nomitted from the resulting pack. The result is that only the objects\nfiltered out by the first `git pack-objects` command are in the pack\nresulting from the second `git pack-objects` command.\n\nAs the interactions with kept packs are a bit tricky, a few related\ntests are added.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 12 ++++\n builtin/repack.c | 75 ++++++++++++++++++++\n t/t7700-repack.sh | 134 +++++++++++++++++++++++++++++++++++\n 3 files changed, 221 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..6d5bec7716 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,18 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. Note\n+\tthat objects used in the working directory are not filtered\n+\tout. So for the split to fully work, it's best to perform it\n+\tin a bare repo and to use the `-a` and `-d` options along with\n+\tthis option. Also `--no-write-bitmap-index` (or the\n+\t`repack.writebitmaps` config option set to `false`) should be\n+\tused otherwise writing bitmap index will fail, as it supposes\n+\ta single packfile containing all the objects. See\n+\tlinkgit:git-rev-list[1] for valid `` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 4e40f4c04e..876c115cdc 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -21,6 +21,7 @@\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n #include \"refs.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define ALL_INTO_ONE 1\n #define LOOSEN_UNREACHABLE 2\n@@ -57,6 +58,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tstruct list_objects_filter_options filter_options;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -726,6 +728,57 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n+static int write_filtered_pack(const struct pack_objects_args *args,\n+\t\t\t const char *destination,\n+\t\t\t const char *pack_prefix,\n+\t\t\t struct string_list *keep_pack_list,\n+\t\t\t struct string_list *names,\n+\t\t\t struct string_list *existing_packs,\n+\t\t\t struct string_list *existing_kept_packs)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret, i;\n+\tconst char *caret;\n+\tconst char *scratch;\n+\tint local = skip_prefix(destination, packdir, &scratch);\n+\n+\tprepare_pack_objects(&cmd, args, destination);\n+\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\n+\tif (!pack_kept_objects)\n+\t\tstrvec_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list->nr; i++)\n+\t\tstrvec_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t keep_pack_list->items[i].string);\n+\n+\tcmd.in = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Here 'names' contains only the pack(s) that were just\n+\t * written, which is exactly the packs we want to keep. Also\n+\t * 'existing_kept_packs' already contains the packs in\n+\t * 'keep_pack_list'.\n+\t */\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, names)\n+\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n+\tfor_each_string_list_item(item, existing_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tcaret = pack_kept_objects ? \"\" : \"^\";\n+\tfor_each_string_list_item(item, existing_kept_packs)\n+\t\tfprintf(in, \"%s%s.pack\\n\", caret, item->string);\n+\tfclose(in);\n+\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -858,6 +911,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&po_args.filter_options),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -871,6 +925,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tOPT_END()\n \t};\n \n+\tlist_objects_filter_init(&po_args.filter_options);\n+\tlist_objects_filter_init(&cruft_po_args.filter_options);\n+\n \tgit_config(repack_config, &cruft_po_args);\n \n \targc = parse_options(argc, argv, prefix, builtin_repack_options,\n@@ -1011,6 +1068,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (po_args.filter_options.choice)\n+\t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\n \tif (geometry)\n \t\tcmd.in = -1;\n \telse\n@@ -1097,6 +1158,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter_options.choice) {\n+\t\tret = write_filtered_pack(&po_args,\n+\t\t\t\t\t packtmp,\n+\t\t\t\t\t find_pack_prefix(packdir, packtmp),\n+\t\t\t\t\t &keep_pack_list,\n+\t\t\t\t\t &names,\n+\t\t\t\t\t &existing_nonkept_packs,\n+\t\t\t\t\t &existing_kept_packs);\n+\t\tif (ret)\n+\t\t\tgoto cleanup;\n+\t}\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\n@@ -1231,6 +1304,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&existing_nonkept_packs, 0);\n \tstring_list_clear(&existing_kept_packs, 0);\n \tclear_pack_geometry(geometry);\n+\tlist_objects_filter_release(&po_args.filter_options);\n+\tlist_objects_filter_release(&cruft_po_args.filter_options);\n \n \treturn ret;\n }\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 27b66807cd..5d3e53134c 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -327,6 +327,140 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack -c 1 HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack -c 1 HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 test_must_fail git -C bare.git repack \\\n+\t\t-a -d --write-bitmap-index --filter=blob:none\n+'\n+\n+test_expect_success 'repacking with two filters works' '\n+\tgit init two-filters &&\n+\t(\n+\t\tcd two-filters &&\n+\t\tmkdir subdir &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit subdir_bar subdir/bar &&\n+\t\ttest_commit subdir_baz subdir/baz\n+\t) &&\n+\tgit clone --no-local --bare two-filters two-filters.git &&\n+\t(\n+\t\tcd two-filters.git &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/*.pack &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d \\\n+\t\t\t--filter=blob:none --filter=tree:1 &&\n+\t\ttest_stdout_line_count = 2 ls objects/pack/*.pack &&\n+\t\tcommit_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\t\tblob_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\troot_tree_pack=$(test-tool find-pack -c 1 HEAD^{tree}) &&\n+\t\tsubdir_tree_hash=$(git ls-tree --object-only HEAD -- subdir) &&\n+\t\tsubdir_tree_pack=$(test-tool find-pack -c 1 \"$subdir_tree_hash\") &&\n+\n+\t\t# Root tree and subdir tree are not in the same packfiles\n+\t\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\t\ttest \"$commit_pack\" = \"$root_tree_pack\" &&\n+\t\ttest \"$blob_pack\" = \"$subdir_tree_pack\"\n+\t)\n+'\n+\n+prepare_for_keep_packs () {\n+\tgit init keep-packs &&\n+\t(\n+\t\tcd keep-packs &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit bar\n+\t) &&\n+\tgit clone --no-local --bare keep-packs keep-packs.git &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\t# Create two packs\n+\t\t# The first pack will contain all of the objects except one blob\n+\t\tgit rev-list --objects --all >objs &&\n+\t\tgrep -v \"bar.t\" objs | git pack-objects pack &&\n+\t\t# The second pack will contain the excluded object and be kept\n+\t\tpackid=$(grep \"bar.t\" objs | git pack-objects pack) &&\n+\t\t>pack-$packid.keep &&\n+\n+\t\t# Replace the existing pack with the 2 new ones\n+\t\trm -f objects/pack/pack* &&\n+\t\tmv pack-* objects/pack/\n+\t)\n+}\n+\n+test_expect_success '--filter works with .keep packs' '\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack_1=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\t# Object bar is still only in the old .keep pack\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$bar_pack_1\" = \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack_1\" &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\" &&\n+\t\ttest \"$bar_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n+test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n+\trm -rf keep-packs keep-packs.git &&\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--pack-kept-objects &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\ttest-tool find-pack -c 2 HEAD:bar.t >bar_pack_1 &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\t# Object bar is in both the old .keep pack and the new\n+\t\t# pack that contained the filtered out objects\n+\t\tgrep \"$bar_pack\" bar_pack_1 &&\n+\t\tgrep \"$foo_pack_1\" bar_pack_1 &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480288","messageId":"20230808082608.582319-1-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230724085909.3831831-1-christian.couder@gmail.com","subject":"[PATCH v4 0/8] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:00Z","receivedAt":"2023-08-08T16:48:37Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"# Intro\n\nLast year, John Cai sent 2 versions of a patch series to implement\n`git repack --filter=` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\n# Use cases for the new feature\n\nThis could be useful for example for the following purposes:\n\n 1) As a way for servers to save storage costs by for example moving\n large blobs, or all the blobs, or all the blobs in inactive\n repos, to separate storage (while still making them accessible\n using for example the alternates mechanism).\n\n 2) As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n 3) As a way for clients to reclaim some space when they cloned with\n a filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a\n separate directory or storage, then check that everything works\n well, and then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\nAlso since version 2 of this new series, commit messages, don't\nmention uses cases like 2) or 3) above, as people have different\nopinions on how it should be done. How it should be done could depend\na lot on the way promisor remotes are used, the software and hardware\nsetups used, etc, so it seems more difficult to \"sell\" this series by\ntalking about such use cases. As use case 1) seems simpler and more\nappealing, it makes more sense to only talk about it in the commit\nmessages.\n\n# Changes since version 3\n\nThanks to Junio who reviewed both version 1, 2 and 3, and to Taylor\nwho reviewed version 1 and 3! The changes are the following:\n\n- In patch 2/8, which introduces `test-tool find-pack`, a new\n `--check-count ` option has been added to check the number of\n packfiles an object is in. To keep things simple and extendable, the\n parse-options API is now used to parse arguments and options.\n\n- Also in patch 2/8, a test script 't0080-find-pack.sh' has been\n introduced to test `test-tool find-pack`, as suggested by Taylor.\n\n- In patch 4/8, which refactors code into a find_pack_prefix()\n function, this function has been changed to accept a `packdir` and a\n `packtmp` argument, instead of using the global variables with the\n same names, as suggested by Taylor.\n\n- In patch 5/8, which introduces `--filter=` option, a\n `struct list_objects_filter_option` and some related functions and\n macros are now used to handle these options, instead of a character\n string. This allows more than one `--filter=` option to\n be passed, and a new test has been added to check that this works,\n as suggested by Taylor.\n\n- In patch 5/8, some changes have been made to better handle kept\n packfiles and related tests have been added to check that this works\n well, as suggested by Taylor.\n\n- In patch 5/8, a comment about the 'names' variable has been\n shortened a lot and improved a bit with additional useful\n information, as suggested by Taylor.\n\n- Also in patch 5/8, tests have been improved and shortened by using\n the new `--check-count ` option of `test-tool find-pack`.\n\n- Also in patch 5/8, the test that checks that `--filter=...` fails\n with `--write-bitmap-index` has been changed to use\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 which should fix a CI test\n that sets this variable to 1. This test has also been simplified by\n removing a useless call to `repack --filter=...` as suggested by\n Taylor.\n\n- Also in patch 5/8, the commit message has been improved a bit and\n now only talks about the use case of moving some blobs that take up\n precious space to a cheaper storage, as suggested by Junio.\n\n- In patch 6/8, which implements the `gc.repackFilter` config option,\n a line in the tests that was too long has been split over 2 lines,\n as suggested by Taylor.\n\n- In patch 7/8, which implements the `--filter-to=` option, the\n documentation of that option talking about possible corruption has\n been clarified a bit, as suggested by Junio.\n\n- Also in patch 7/8, tests have been improved and shortened by using\n the new `--check-count ` option of `test-tool find-pack`.\n\n# Commit overview\n\n* 1/8 pack-objects: allow `--filter` without `--stdout`\n\n This patch is the same as in v1, v2 and v3. To be able to later\n repack with a filter we need `git pack-objects` to write packfiles\n when it's filtering instead of just writing the pack without the\n filtered out objects to stdout.\n\n* 2/8 t/helper: add 'find-pack' test-tool\n\n For testing `git repack --filter=...` that we are going to\n implement, it's useful to have a test helper that can tell which\n packfiles contain a specific object. Since v3 the new\n `--check-count ` option has been added, and tests have been added\n in a new 't0080-find-pack.sh' test script.\n\n* 3/8 repack: refactor finishing pack-objects command\n\n No change in this patch compared to v2 and v3. This is a small\n refactoring creating a new useful function, so that `git repack\n --filter=...` will be able to reuse it.\n\n* 4/8 repack: refactor finding pack prefix\n\n This is another small refactoring creating a small function that\n will be reused in the next patch. Since v3 the new function\n introduced in this patch has been changed to accept a `packdir` and\n a `packtmp` argument, instead of using the global variables with the\n same names.\n\n* 5/8 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. It uses one\n `git pack-objects` process with the `--filter` option. And then\n another `git pack-objects` process with the `--stdin-packs`\n option. A lot of changes have been made since v3:\n\n - The `list_objects_filter_option` struct and some related\n functions and macros are used to handle the new\n `--filter=` option. A new test has been added to\n check that using multiple such options works.\n\n - Handling of kept packfiles has been improved and related tests\n have been added.\n\n - A comment about the 'names' variable has been shortened a lot\n and improved a bit.\n\n - Tests have been improved and shortened by using the new\n `--check-count ` option of `test-tool find-pack`.\n\n - The test that checks that `--filter=...` fails with\n `--write-bitmap-index` has been improved to pass a CI test and\n shortened.\n\n - The commit message has been improved a bit.\n\n* 6/8 gc: add `gc.repackFilter` config option\n\n This is a gc config option so that `git gc` can also repack using a\n filter and put the filtered out objects into a separate\n packfile. Since v3, a line in the tests that was too long has been\n split over 2 lines.\n\n* 7/8 repack: implement `--filter-to` for storing filtered out objects\n\n For some use cases, it's interesting to create the packfile that\n contains the filtered out objects into a separate location. This is\n similar to the `--expire-to` option for cruft packfiles. Since v3,\n documentation of that option talking about possible corruption has\n been clarified a bit, and tests have been improved and shortened by\n using the new `--check-count ` option of `test-tool find-pack`.\n\n* 8/8 gc: add `gc.repackFilterTo` config option\n\n No change in this patch compared to v3. This allows specifying the\n location of the packfile that contains the filtered out objects when\n using `gc.repackFilter`.\n\n# Range-diff since v3\n\n(Sorry, but the range-diff doesn't show changes in patches 2/8 and 5/8\nas there has been a lot of changes in them. Instead it shows that the\nold commit has been removed and a new one added.)\n\n1: 4d75a1d7c3 = 1: 4d75a1d7c3 pack-objects: allow `--filter` without `--stdout`\n2: fdf9b6e8cc < -: ---------- t/helper: add 'find-pack' test-tool\n-: ---------- > 2: 0bf9f53158 t/helper: add 'find-pack' test-tool\n3: e7cfdebc78 = 3: 54060d775e repack: refactor finishing pack-objects command\n4: 9c51063795 ! 4: 948ea541ae repack: refactor finding pack prefix\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *ar\n return finish_pack_objects_cmd(&cmd, names, local);\n }\n \n -+static const char *find_pack_prefix(void)\n ++static const char *find_pack_prefix(char *packdir, char *packtmp)\n +{\n + const char *pack_prefix;\n + if (!skip_prefix(packtmp, packdir, &pack_prefix))\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n - packtmp, packdir);\n - if (*pack_prefix == '/')\n - pack_prefix++;\n -+ const char *pack_prefix = find_pack_prefix();\n ++ const char *pack_prefix = find_pack_prefix(packdir, packtmp);\n \n if (!cruft_po_args.window)\n cruft_po_args.window = po_args.window;\n5: a90e8045c3 < -: ---------- repack: add `--filter=` option\n-: ---------- > 5: 0635425289 repack: add `--filter=` option\n6: 335b7f614d ! 6: bf8be2c812 gc: add `gc.repackFilter` config option\n @@ t/t6500-gc.sh: test_expect_success 'one of gc.reflogExpire{Unreachable,}=never d\n + git -C bare.git -c gc.cruftPacks=false gc &&\n + test_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n +\n -+ GIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none -c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n ++ GIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none \\\n ++ -c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n + test_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n + grep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n +'\n7: b1be7f60b7 ! 7: abe7526222 repack: implement `--filter-to` for storing filtered out objects\n @@ Commit message\n \n Signed-off-by: Christian Couder \n \n - repack: add test with --max-pack-size\n -\n ## Documentation/git-repack.txt ##\n @@ Documentation/git-repack.txt: depth is 4095.\n a single packfile containing all the objects. See\n @@ Documentation/git-repack.txt: depth is 4095.\n + used for putting the pack on a separate object directory that\n + is accessed through the Git alternates mechanism. **WARNING:**\n + If the packfile containing the filtered out objects is not\n -+ accessible, the repo could be considered corrupt by Git as it\n -+ migh not be able to access the objects in that packfile. See\n -+ the `objects` and `objects/info/alternates` sections of\n ++ accessible, the repo can become corrupt as it might not be\n ++ possible to access the objects in that packfile. See the\n ++ `objects` and `objects/info/alternates` sections of\n + linkgit:gitrepository-layout[5].\n +\n -b::\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n };\n \n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n - strvec_push(&cmd.args, \"--incremental\");\n - }\n - \n -+ if (filter_to && !po_args.filter)\n + if (po_args.filter_options.choice)\n + strvec_pushf(&cmd.args, \"--filter=%s\",\n + expand_list_objects_filter_spec(&po_args.filter_options));\n ++ else if (filter_to)\n + die(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n -+\n + \n if (geometry)\n cmd.in = -1;\n - else\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n }\n \n - if (po_args.filter) {\n + if (po_args.filter_options.choice) {\n + if (!filter_to)\n + filter_to = packtmp;\n +\n ret = write_filtered_pack(&po_args,\n - packtmp,\n + filter_to,\n - find_pack_prefix(),\n + find_pack_prefix(packdir, packtmp),\n + &keep_pack_list,\n &names,\n - &existing_nonkept_packs,\n \n ## t/t7700-repack.sh ##\n -@@ t/t7700-repack.sh: test_expect_success '--filter fails with --write-bitmap-index' '\n - --filter=blob:none\n +@@ t/t7700-repack.sh: test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n + )\n '\n \n +test_expect_success '--filter-to stores filtered out objects' '\n @@ t/t7700-repack.sh: test_expect_success '--filter fails with --write-bitmap-index\n + test_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n + test_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n +\n -+ commit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n -+ test -n \"$commit_pack\" &&\n -+ blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n -+ test -z \"$blob_pack\" &&\n ++ commit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n ++ blob_pack=$(test-tool -C bare.git find-pack -c 0 HEAD:file1) &&\n + blob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n + test -n \"$blob_hash\" &&\n -+ blob_pack=$(test-tool -C filtered.git find-pack $blob_hash) &&\n -+ test -n \"$blob_pack\" &&\n ++ blob_pack=$(test-tool -C filtered.git find-pack -c 1 $blob_hash) &&\n +\n + echo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n -+ blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n -+ test -n \"$blob_pack\" &&\n ++ blob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n + blob_content=$(git -C bare.git show $blob_hash) &&\n + test \"$blob_content\" = \"content1\"\n +'\n @@ t/t7700-repack.sh: test_expect_success '--filter fails with --write-bitmap-index\n + # Check that the 3 blobs are in different packfiles in filtered.git\n + test_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n + test_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n -+ foo_pack=$(test-tool find-pack HEAD:foo) &&\n -+ bar_pack=$(test-tool find-pack HEAD:bar) &&\n -+ base_pack=$(test-tool find-pack HEAD:base.t) &&\n ++ foo_pack=$(test-tool find-pack -c 1 HEAD:foo) &&\n ++ bar_pack=$(test-tool find-pack -c 1 HEAD:bar) &&\n ++ base_pack=$(test-tool find-pack -c 1 HEAD:base.t) &&\n + test \"$foo_pack\" != \"$bar_pack\" &&\n + test \"$foo_pack\" != \"$base_pack\" &&\n + test \"$bar_pack\" != \"$base_pack\" &&\n8: ed66511823 = 8: ccdc858f73 gc: add `gc.repackFilterTo` config option\n\n\nChristian Couder (8):\n pack-objects: allow `--filter` without `--stdout`\n t/helper: add 'find-pack' test-tool\n repack: refactor finishing pack-objects command\n repack: refactor finding pack prefix\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 16 ++\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 23 +++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 169 +++++++++++++++------\n t/helper/test-find-pack.c | 50 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 +++++++++++\n t/t5317-pack-objects-filter-objects.sh | 8 +\n t/t6500-gc.sh | 24 +++\n t/t7700-repack.sh | 196 +++++++++++++++++++++++++\n 14 files changed, 543 insertions(+), 50 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480289","messageId":"CAP8UFD3BiZQ4-M91sc6p2wBFrzPQQfja_xKpT26BcBS7OVA4MA@mail.gmail.com","threadId":"59867","inReplyTo":"ZMBU+SALVQthOgC7@nand.local","subject":"Re: [PATCH v3 5/8] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:34:25Z","receivedAt":"2023-08-08T17:00:37Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 26, 2023 at 1:04 AM Taylor Blau wrote:\n>\n> On Mon, Jul 24, 2023 at 10:59:06AM +0200, Christian Couder wrote:\n\n> > + /*\n> > + * names has a confusing double use: it both provides the list\n> > + * of just-written new packs, and accepts the name of the\n> > + * filtered pack we are writing.\n> > + *\n> > + * By the time it is read here, it contains only the pack(s)\n> > + * that were just written, which is exactly the set of packs we\n> > + * want to consider kept.\n> > + */\n>\n> I think that this comment partially comes from the cruft pack code,\n> where we use the `names` string list both to reference existing packs at\n> the start of the repack, and to keep track of the pack we just wrote (to\n> exclude its contents from the cruft pack).\n>\n> But I think we only write into \"names\" via finish_pack_objects_cmd() to\n> record the name of the pack we just wrote containing objects which\n> didn't meet the filter's conditions.\n>\n> So I think that leaving this comment in is OK, but TBH I was on the\n> fence when I wrote that back in f9825d1cf75 (builtin/repack.c: support\n> generating a cruft pack, 2022-05-20), so I would just as soon drop it.\n\nI made the comment smaller in version 4.\n\n> > + in = xfdopen(cmd.in, \"w\");\n> > + for_each_string_list_item(item, names)\n> > + fprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n> > + for_each_string_list_item(item, existing_packs)\n> > + fprintf(in, \"%s.pack\\n\", item->string);\n>\n> > + for_each_string_list_item(item, existing_kept_packs)\n> > + fprintf(in, \"^%s.pack\\n\", item->string);\n>\n> I think we may only want to do this if `honor_pack_keep` is zero.\n> Otherwise we'd avoid packing objects that appear in kept packs, even if\n> the caller told us to include objects found in kept packs.\n\nIn version 4 I have made changes to better support kept packfiles and\nrelated options, including adding tests.\n\n> > @@ -858,6 +912,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> > N_(\"limits the maximum number of threads\")),\n> > OPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n> > N_(\"maximum size of each packfile\")),\n> > + OPT_STRING(0, \"filter\", &po_args.filter, N_(\"args\"),\n> > + N_(\"object filtering\")),\n>\n> I suppose we're storing the filter as a string here because we're just\n> going to pass it down to pack-objects directly. That part makes sense,\n> but I think we are producing subtly inconsistent behavior when\n> specifying multiple --filter options.\n>\n> IIRC, passing --filter more than once down to pack-objects produces a\n> filter whose objects match all of the individually specified\n> sub-filters. But IIUC, using OPT_STRING here means that later\n> `--filter`'s override earlier ones.\n>\n> So I think at minimum we'd want to store the filter arguments in a\n> strvec. But I would probably just as soon parse them into a bona-fide\n> list_objects_filter_options struct, and then reconstruct the arguments\n> to pack-objects based on that.\n\nIn version 4 a `list_objects_filter_options` struct is now used, and\nthere is a test to check that more than one `--filter=`\noption is supported.\n\n> > + git -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n> > + test_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n> > + commit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n> > + test -n \"$commit_pack\" &&\n>\n> I wonder if the test-tool itself should exit with a non-zero code if it\n> can't find the given object in any pack. It would at least allow us to\n> drop the \"test -n $foo\" after every invocation of the test-helper in\n> this test.\n>\n> Arguably callers may want to ensure that an object doesn't exist in any\n> pack, and this would be inconvenient for them, since they'd have to\n> write something like:\n>\n> test_must_fail test-tool find-pack $obj\n>\n> but I think a more direct test like\n>\n> test_must_fail git cat-file -t $obj\n>\n> would do just as well.\n\nThanks for these suggestions, but I prefered to add the `--check-count\n` option to `test-tool find-pack` in version 4.\n\nThis way `--check-count 0` or `-c 0` for short can be used to check\nthat an object is in no packfile, though it could be for example in a\npromisor remote or a loose object file. It's also nice to be able to\ncheck that an object is in exactly 2 packfiles in some cases.\n\n> > + blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&\n> > + test -n \"$blob_pack\" &&\n> > + test \"$commit_pack\" != \"$blob_pack\" &&\n> > + tree_pack=$(test-tool -C bare.git find-pack HEAD^{tree}) &&\n> > + test \"$tree_pack\" = \"$commit_pack\" &&\n> > + blob_pack2=$(test-tool -C bare.git find-pack HEAD:file2) &&\n> > + test \"$blob_pack2\" = \"$blob_pack\"\n> > +'\n>\n> This all looks good, but I think there are a couple of more things that\n> we'd want to test for here:\n>\n> - That the list of all objects appears the same before and after all\n> of the repacking. I think that this is tested implicitly already in\n> your test, but having it written down explicitly would harden this\n> against regressions that cause us to inadvertently delete an object\n> we shouldn't have.\n\nI don't think we need to test this. `git pack-objects\n--filter=` already existed before this series and is\ntested elsewhere. We can trust that command and its tests, and just\ncheck that we used it correctly by checking that only a few objects\nare in the right packfiles.\n\n> (FWIW, I think this would be limited to running something like \"git\n> cat-file --batch-check='%(objectname)' --batch-all-objects\" before\n> and after all of the repacking, and ensuring that the two test_cmp\n> without failure).\n\nI agree that it would not be difficult to do. I just think it's not necessary.\n\n> - Another thing that I don't think we're testing here is that objects\n> that *don't* match the filter don't appear in one of the filtered\n> packs.\n\nIn version 4 we do test that for some objects, as `test-tool find-pack\n-c 1 $object` would error out if the object is in more than one\npackfile.\n\n> I think we'd probably want to assert on the exact contents of\n> the pack by dumping the list of objects into a file like \"expect\",\n> and then dumping the actual set of objects with \"git show-index\n> <$idx | cut -d' ' -f2\" or something.\n>\n> Another thought from the OPT_STRING business above is that we probably\n> want to test this with non-trivial filter arguments. There are probably\n> a handful of interesting cases here, like passing `--no-filter`, passing\n> `--filter` multiple times, passing invalid values for `--filter`, etc.\n\nIn version 4 there is one test passing `--filter=...` multiple times.\nI think this is enough, as the `list_objects_filter_options` struct\nand related functions and mechanisms are tested elsewhere already.\n\n> > +test_expect_success '--filter fails with --write-bitmap-index' '\n> > + test_must_fail git -C bare.git repack -a -d --write-bitmap-index \\\n> > + --filter=blob:none &&\n>\n> Do we want to ensure that we get the exit code corresponding with\n> showing the usage text? I could go either way, but I do think that we\n> should grep through the output on stderr to ensure that we get the\n> appropriate error message.\n\nI am not sure that testing the exit code and the stderr output is\nalways needed. Here I think that this test is more for documentation\npurposes than really enforcing something important. In fact if the\nbehavior would change and `--write-bitmap-index` would understand that\nit should write an MIDX instead of a regular index, that behavior\nchange could be considered in some ways as an improvement and we would\nonly need to remove 'test_must_fail' here.\n\n> > + git -C bare.git repack -a -d --no-write-bitmap-index \\\n> > + --filter=blob:none\n>\n> I don't think that this test is adding anything that the above\n> \"repacking with a filter works\" test isn't covering already.\n\nOk, I have removed it in version 4.\n\nThanks!\n"},{"id":"480295","messageId":"20230808082608.582319-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 4/8] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:04Z","receivedAt":"2023-08-08T17:39:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new find_pack_prefix() to refactor code that handles finding\nthe pack prefix from the packtmp and packdir global variables, as we are\ngoing to need this feature again in following commit.\n\nSigned-off-by: Christian Couder ` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:45:48Z","receivedAt":"2023-08-08T17:42:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 26, 2023 at 1:09 AM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> > Thanks for walking through the codepaths involved. We are good\n> > then.\n>\n> Sorry, but not so fast.\n>\n> https://github.com/git/git/actions/runs/5661445152 (seen with this topic)\n> https://github.com/git/git/actions/runs/5662517690 (seen w/o this topic)\n>\n> The former fails t7700 in the linux-TEST-vars job, while the latter\n> passes the same job.\n\nI think this was because I added the following test:\n\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+ test_must_fail git -C bare.git repack -a -d --write-bitmap-index \\\n+ --filter=blob:none &&\n+\n+ git -C bare.git repack -a -d --no-write-bitmap-index \\\n+ --filter=blob:none\n+'\n\nwhich fails because in the linux-TEST-vars job the\nGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP env variable is set to 1 and\nthis counteracts the `--write-bitmap-index` option.\n\nI have tried to fix it like this:\n\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 test_must_fail git -C\nbare.git repack \\\n+ -a -d --write-bitmap-index --filter=blob:none\n+'\n\nbut I haven't been able to check that this works on CI as all the job\nseems to fail these days before they even start:\n\nhttps://github.com/chriscool/git/actions/runs/5791544404/job/15696524676\n\nThanks!\n"},{"id":"480297","messageId":"CAP8UFD3_XOtY-cetnO6sPSj3N-EpF+eq836Vv0v8jPDTSkVupA@mail.gmail.com","threadId":"59867","inReplyTo":"ZMBQRRppsadqa8Ua@nand.local","subject":"Re: [PATCH v3 2/8] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:28:31Z","receivedAt":"2023-08-08T18:03:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 26, 2023 at 12:44 AM Taylor Blau wrote:\n>\n> On Mon, Jul 24, 2023 at 10:59:03AM +0200, Christian Couder wrote:\n> > ---\n> > Makefile | 1 +\n> > t/helper/test-find-pack.c | 35 +++++++++++++++++++++++++++++++++++\n> > t/helper/test-tool.c | 1 +\n> > t/helper/test-tool.h | 1 +\n> > 4 files changed, 38 insertions(+)\n> > create mode 100645 t/helper/test-find-pack.c\n>\n> Everything that you wrote here seems reasonable to me, and the\n> implementation of the new test tool is very straightforward.\n>\n> I'm pretty sure that everything here is correct, and we'll implicitly\n> test the behavior of the new helper in following patches.\n>\n> That said, I think that it might be prudent here to \"test the tests\" and\n> write a simple test script that exercises this test helper over a more\n> trivial case. There is definitely prior art for testing our helpers\n> directly in the t00?? tests.\n\nOk, I have written a new t0080-find-pack.sh test script for this in\nthe version 4 I just sent.\n\nI have also changed `test-tool find-pack` so that it now accepts a\n`--check-count ` option. This addresses some of your comments on\nanother patch in the previous version of this series. As the code is\nnow a bit more complex, there is more justification for a test script.\n\nThanks.\n"},{"id":"480298","messageId":"20230808082608.582319-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 3/8] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:03Z","receivedAt":"2023-08-08T18:09:16Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -705,9 +735,8 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct string_list *existing_kept_packs)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n \tconst char *scratch;\n \tint local = skip_prefix(destination, packdir, &scratch);\n@@ -751,27 +780,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -782,10 +791,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1016,18 +1023,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480300","messageId":"20230808082608.582319-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 6/8] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:06Z","receivedAt":"2023-08-08T18:24:07Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex ca47eb2008..2153bde7ac 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the\n operation (either generating a cruft pack or unpacking unreachable\n objects) will be halted.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 19d73067aa..9b0984f301 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -61,6 +61,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -170,6 +171,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -355,6 +358,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..232e403b66 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,19 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480307","messageId":"CAP8UFD0gojKUhZv82d-CVt9nPL+0UWCr+nV7+5_UFykCDuM9WA@mail.gmail.com","threadId":"59867","inReplyTo":"ZMBQ97ucRg9gO4gW@nand.local","subject":"Re: [PATCH v3 4/8] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:29:02Z","receivedAt":"2023-08-08T19:01:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 26, 2023 at 12:47 AM Taylor Blau wrote:\n>\n> On Mon, Jul 24, 2023 at 10:59:05AM +0200, Christian Couder wrote:\n> > diff --git a/builtin/repack.c b/builtin/repack.c\n> > index 96af2d1caf..21e3b89f27 100644\n> > --- a/builtin/repack.c\n> > +++ b/builtin/repack.c\n> > @@ -783,6 +783,17 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n> > return finish_pack_objects_cmd(&cmd, names, local);\n> > }\n> >\n> > +static const char *find_pack_prefix(void)\n> > +{\n> > + const char *pack_prefix;\n> > + if (!skip_prefix(packtmp, packdir, &pack_prefix))\n>\n> I wonder if this might be a good opportunity to pass \"packtmp\" and\n> \"packdir\" as arguments to the function. I know that these are globals,\n> but it at least nudges us in the right direction away from adding more\n> global variables.\n\nI have changed this in the version 4 I just sent. Now \"packtmp\" and\n\"packdir\" are passed as arguments to the function as you suggest.\nThanks.\n"},{"id":"480308","messageId":"20230808082608.582319-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 8/8] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:08Z","receivedAt":"2023-08-08T19:05:46Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 11 +++++++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 2153bde7ac..466466d6cc 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -150,6 +150,17 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. **WARNING:** The\n+\tspecified location should be accessible, using for example the\n+\tGit alternates mechanism, otherwise the repo could be\n+\tconsidered corrupt by Git as it migh not be able to access the\n+\tobjects in that packfile. See the `--filter-to=` option\n+\tof linkgit:git-repack[1] and the `objects/info/alternates`\n+\tsection of linkgit:gitrepository-layout[5].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9b0984f301..1b7c775d94 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -172,6 +173,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -361,6 +363,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 232e403b66..e412cf8daf 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -215,6 +214,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480317","messageId":"20230808082608.582319-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 1/8] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:01Z","receivedAt":"2023-08-08T19:51:32Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n t/t5317-pack-objects-filter-objects.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex a9995a932c..583270a85f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -298,8 +298,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d2a162d528..000ebec7ab 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4400,12 +4400,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..2ff3eef9a3 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n \t! grep blob verify_result\n '\n \n+test_expect_success 'verify blob:none packfile without --stdout' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n+\t! grep blob verify_result\n+'\n+\n test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n \tgit -C r1 verify-pack -v ../all.pack >verify_result &&\n \tgrep -E \"commit|tree\" verify_result |\n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480333","messageId":"20230808082608.582319-3-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v4 2/8] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-08T08:26:02Z","receivedAt":"2023-08-08T20:41:55Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In a following commit, we will make it possible to separate objects in\ndifferent packfiles depending on a filter.\n\nTo make sure that the right objects are in the right packs, let's add a\nnew test-tool that can display which packfile(s) a given object is in.\n\nLet's also make it possible to check if a given object is in the\nexpected number of packfiles with a `--check-count ` option.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 50 ++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 +++++++++++++++++++++++++++++++++++++++\n 5 files changed, 135 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\ndiff --git a/Makefile b/Makefile\nindex fb541dedc9..14ee0c45d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..4b9e09ce25\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,50 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"parse-options.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ *\n+ * If '--check-count ' is passed, then error out if the number of\n+ * packfiles containing the object is not .\n+ */\n+\n+static const char *find_pack_usage[] = {\n+\t\"test-tool find-pack [--check-count ] \",\n+\tNULL\n+};\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\tint count = -1, actual_count = 0;\n+\tconst char *prefix = setup_git_directory();\n+\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER('c', \"check-count\", &count, \"expected number of packs\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, find_pack_usage, 0);\n+\tif (argc != 1)\n+\t\tusage(find_pack_usage[0]);\n+\n+\tif (repo_get_oid(the_repository, argv[0], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[0]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tif (find_pack_entry_one(oid.hash, p)) {\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t\t\tactual_count++;\n+\t\t}\n+\n+\tif (count > -1 && count != actual_count)\n+\t\tdie (\"bad packfile count %d instead of %d\", actual_count, count);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex abe8a785eb..41da40c296 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ea2672436c..411dbf2db4 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\ndiff --git a/t/t0080-find-pack.sh b/t/t0080-find-pack.sh\nnew file mode 100755\nindex 0000000000..67b11216a3\n--- /dev/null\n+++ b/t/t0080-find-pack.sh\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+test_description='test `test-tool find-pack`'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit one &&\n+\ttest_commit two &&\n+\ttest_commit three &&\n+\ttest_commit four &&\n+\ttest_commit five\n+'\n+\n+test_expect_success 'repack everything into a single packfile' '\n+\tgit repack -a -d --no-write-bitmap-index &&\n+\n+\thead_commit_pack=$(test-tool find-pack HEAD) &&\n+\thead_tree_pack=$(test-tool find-pack HEAD^{tree}) &&\n+\tone_pack=$(test-tool find-pack HEAD:one.t) &&\n+\tthree_pack=$(test-tool find-pack HEAD:three.t) &&\n+\told_commit_pack=$(test-tool find-pack HEAD~4) &&\n+\n+\ttest-tool find-pack --check-count 1 HEAD &&\n+\ttest-tool find-pack --check-count=1 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count=0 HEAD:one.t &&\n+\t! test-tool find-pack -c 2 HEAD:one.t &&\n+\ttest-tool find-pack -c 1 HEAD:three.t &&\n+\n+\t# Packfile exists at the right path\n+\tcase \"$head_commit_pack\" in\n+\t\t\".git/objects/pack/pack-\"*\".pack\") true ;;\n+\t\t*) false ;;\n+\tesac &&\n+\ttest -f \"$head_commit_pack\" &&\n+\n+\t# Everything is in the same pack\n+\ttest \"$head_commit_pack\" = \"$head_tree_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$one_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$three_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$old_commit_pack\"\n+'\n+\n+test_expect_success 'add more packfiles' '\n+\tgit rev-parse HEAD^{tree} HEAD:two.t HEAD:four.t >objects &&\n+\tgit pack-objects .git/objects/pack/mypackname1 >packhash1 objects &&\n+\tgit pack-objects .git/objects/pack/mypackname2 >packhash2 head_tree_packs &&\n+\tgrep \"$head_commit_pack\" head_tree_packs &&\n+\tgrep mypackname1 head_tree_packs &&\n+\t! grep mypackname2 head_tree_packs &&\n+\ttest-tool find-pack --check-count 2 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count 1 HEAD^{tree} &&\n+\n+\t# HEAD:five.t is also in 2 packfiles\n+\ttest-tool find-pack HEAD:five.t >five_packs &&\n+\tgrep \"$head_commit_pack\" five_packs &&\n+\t! grep mypackname1 five_packs &&\n+\tgrep mypackname2 five_packs &&\n+\ttest-tool find-pack -c 2 HEAD:five.t &&\n+\t! test-tool find-pack --check-count=0 HEAD:five.t\n+'\n+\n+test_expect_success 'add more commits (as loose objects)' '\n+\ttest_commit six &&\n+\ttest_commit seven &&\n+\n+\ttest -z \"$(test-tool find-pack HEAD)\" &&\n+\ttest -z \"$(test-tool find-pack HEAD:six.t)\" &&\n+\ttest-tool find-pack --check-count 0 HEAD &&\n+\ttest-tool find-pack -c 0 HEAD:six.t &&\n+\t! test-tool find-pack -c 1 HEAD:seven.t\n+'\n+\n+test_done\n-- \n2.42.0.rc0.8.g76fac86b0e\n\n"},{"id":"480399","messageId":"ZNP5RNJgTU698ra6@nand.local","threadId":"59867","inReplyTo":"CAP8UFD1eX8JMd91Say_sC7h_V08oRq32Wu9RM+SFtAQnhRPO2w@mail.gmail.com","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-09T20:38:28Z","receivedAt":"2023-08-09T20:39:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 08, 2023 at 10:45:48AM +0200, Christian Couder wrote:\n> On Wed, Jul 26, 2023 at 1:09 AM Junio C Hamano wrote:\n> >\n> > Junio C Hamano writes:\n> >\n> > > Thanks for walking through the codepaths involved. We are good\n> > > then.\n> >\n> > Sorry, but not so fast.\n> >\n> > https://github.com/git/git/actions/runs/5661445152 (seen with this topic)\n> > https://github.com/git/git/actions/runs/5662517690 (seen w/o this topic)\n> >\n> > The former fails t7700 in the linux-TEST-vars job, while the latter\n> > passes the same job.\n>\n> I think this was because I added the following test:\n>\n> +test_expect_success '--filter fails with --write-bitmap-index' '\n> + test_must_fail git -C bare.git repack -a -d --write-bitmap-index \\\n> + --filter=blob:none &&\n> +\n> + git -C bare.git repack -a -d --no-write-bitmap-index \\\n> + --filter=blob:none\n> +'\n>\n> which fails because in the linux-TEST-vars job the\n> GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP env variable is set to 1 and\n> this counteracts the `--write-bitmap-index` option.\n\nMakes sense. That linux-TEST-vars job always seems to get me, too.\n\n(As an aside, and definitely not related to your patch here, I wonder if\nwe should consider dropping some of the older TEST variables that belong\nto features that we no longer consider experimental).\n\n> I have tried to fix it like this:\n>\n> +test_expect_success '--filter fails with --write-bitmap-index' '\n> + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 test_must_fail git -C\n> bare.git repack \\\n> + -a -d --write-bitmap-index --filter=blob:none\n> +'\n>\n> but I haven't been able to check that this works on CI as all the job\n> seems to fail these days before they even start:\n\nI think the cannonical way to do this is with env, like so:\n\n test_must_fail env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n git -C bare.git repack -ad --write-bitmap-index --filter=blob:none 2>err\n\nThanks,\nTaylor\n"},{"id":"480402","messageId":"ZNQBOQwaVsRL4suR@nand.local","threadId":"59867","inReplyTo":"CAP8UFD3BiZQ4-M91sc6p2wBFrzPQQfja_xKpT26BcBS7OVA4MA@mail.gmail.com","subject":"Re: [PATCH v3 5/8] repack: add `--filter=` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-09T21:12:25Z","receivedAt":"2023-08-09T21:12:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 08, 2023 at 10:34:25AM +0200, Christian Couder wrote:\n> > > + git -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n> > > + test_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n> > > + commit_pack=$(test-tool -C bare.git find-pack HEAD) &&\n> > > + test -n \"$commit_pack\" &&\n> >\n> > I wonder if the test-tool itself should exit with a non-zero code if it\n> > can't find the given object in any pack. It would at least allow us to\n> > drop the \"test -n $foo\" after every invocation of the test-helper in\n> > this test.\n> >\n> > Arguably callers may want to ensure that an object doesn't exist in any\n> > pack, and this would be inconvenient for them, since they'd have to\n> > write something like:\n> >\n> > test_must_fail test-tool find-pack $obj\n> >\n> > but I think a more direct test like\n> >\n> > test_must_fail git cat-file -t $obj\n> >\n> > would do just as well.\n>\n> Thanks for these suggestions, but I prefered to add the `--check-count\n> ` option to `test-tool find-pack` in version 4.\n>\n> This way `--check-count 0` or `-c 0` for short can be used to check\n> that an object is in no packfile, though it could be for example in a\n> promisor remote or a loose object file. It's also nice to be able to\n> check that an object is in exactly 2 packfiles in some cases.\n\n\"--check-count 0\" is a nice approach, thanks!\n\n> > This all looks good, but I think there are a couple of more things that\n> > we'd want to test for here:\n> >\n> > - That the list of all objects appears the same before and after all\n> > of the repacking. I think that this is tested implicitly already in\n> > your test, but having it written down explicitly would harden this\n> > against regressions that cause us to inadvertently delete an object\n> > we shouldn't have.\n>\n> I don't think we need to test this. `git pack-objects\n> --filter=` already existed before this series and is\n> tested elsewhere. We can trust that command and its tests, and just\n> check that we used it correctly by checking that only a few objects\n> are in the right packfiles.\n\nYeah, I don't think we should be worried about whether or not\npack-objects is doing the right thing here: I agree that we have\nsufficient coverage for that elsewhere throughout the test suite. I was\nmore concerned at catching bugs or regressions at the 'repack' layer.\n\nBut you're more familiar with these changes than I am, so I trust your\njudgement.\n\n> > > +test_expect_success '--filter fails with --write-bitmap-index' '\n> > > + test_must_fail git -C bare.git repack -a -d --write-bitmap-index \\\n> > > + --filter=blob:none &&\n> >\n> > Do we want to ensure that we get the exit code corresponding with\n> > showing the usage text? I could go either way, but I do think that we\n> > should grep through the output on stderr to ensure that we get the\n> > appropriate error message.\n>\n> I am not sure that testing the exit code and the stderr output is\n> always needed. Here I think that this test is more for documentation\n> purposes than really enforcing something important. In fact if the\n> behavior would change and `--write-bitmap-index` would understand that\n> it should write an MIDX instead of a regular index, that behavior\n> change could be considered in some ways as an improvement and we would\n> only need to remove 'test_must_fail' here.\n\nI don't feel that strongly about it, TBH, I think I was more commenting\non that we seem to have many of these tests that go\n\n test_must_fail git 2>err &&\n grep \"appropriate error message\" err\n\nthroughout the suite. I don't feel strongly enough to suggest that we\nadd more for this specific purpose.\n\nThanks,\nTaylor\n"},{"id":"480404","messageId":"ZNQCBVXhGMhQcgu1@nand.local","threadId":"59867","inReplyTo":"CAP8UFD1Z3O5Wf8D9dYZdgPuOWgo7NbQsHAWifkPUGE=Eo1U7EQ@mail.gmail.com","subject":"Re: [PATCH v3 6/8] gc: add `gc.repackFilter` config option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-09T21:15:49Z","receivedAt":"2023-08-09T21:15:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 08, 2023 at 10:38:26AM +0200, Christian Couder wrote:\n> > I may be missing something, but what happens if the user has configured\n> > gc.repackFilter, but passes additional filters over the command-line\n> > arguments? I'm not sure whether these should be AND'd with the existing\n> > filters in config, or if they should reset them to zero, or something\n> > else.\n>\n> `git gc` doesn't recognize `--filter=<...>` arguments, only `git\n> repack` is being teached to recognize it in this patch series. So I\n> don't see how there could be multiple such arguments on the command\n> line when `git gc` is used.\n>\n> Also in version 4 `git repack` can be passed many such arguments\n> anyway. So I think we are good.\n\nAh, thanks. Sorry for the misunderstanding :-).\n\n> We could support multiple gc.repackFilter config options, but on the\n> other hand using something like\n> `combine:++...` should work, as the content\n> of the option is passed as-is to the command line. So we can leave\n> that improvement for later if people don't like the `combine:...` and\n> are interested in it.\n\nI agree. To me it seems like there are probably relatively few people\nwho would want to specify a multi-valued configuration directly when\nthey could just use the \"combine\" trick you suggest. In either case, I\nagree that it can be done on top later.\n\nThanks,\nTaylor\n"},{"id":"480405","messageId":"ZNQCvEJif4/VooyD@nand.local","threadId":"59867","inReplyTo":"20230808082608.582319-3-christian.couder@gmail.com","subject":"Re: [PATCH v4 2/8] t/helper: add 'find-pack' test-tool","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-09T21:18:52Z","receivedAt":"2023-08-09T21:18:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 08, 2023 at 10:26:02AM +0200, Christian Couder wrote:\n> +\tif (count > -1 && count != actual_count)\n> +\t\tdie (\"bad packfile count %d instead of %d\", actual_count, count);\n\nI think there is an extra space between \"die\" and the opening\nparenthesis. But obviously not worth a reroll here.\n\n> diff --git a/t/t0080-find-pack.sh b/t/t0080-find-pack.sh\n> new file mode 100755\n> index 0000000000..67b11216a3\n> --- /dev/null\n> +++ b/t/t0080-find-pack.sh\n> @@ -0,0 +1,82 @@\n\nThese new tests look great, thanks for adding them :-).\n\nThanks,\nTaylor\n"},{"id":"480406","messageId":"ZNQDFmyX6Egl9hRu@nand.local","threadId":"59867","inReplyTo":"20230808082608.582319-5-christian.couder@gmail.com","subject":"Re: [PATCH v4 4/8] repack: refactor finding pack prefix","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-09T21:20:22Z","receivedAt":"2023-08-09T21:22:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 08, 2023 at 10:26:04AM +0200, Christian Couder wrote:\n> Create a new find_pack_prefix() to refactor code that handles finding\n> the pack prefix from the packtmp and packdir global variables, as we are\n> going to need this feature again in following commit.\n>\n> Signed-off-by: Christian Couder ---\n> builtin/repack.c | 18 ++++++++++++------\n> 1 file changed, 12 insertions(+), 6 deletions(-)\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 96af2d1caf..4e40f4c04e 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -783,6 +783,17 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n> \treturn finish_pack_objects_cmd(&cmd, names, local);\n> }\n>\n> +static const char *find_pack_prefix(char *packdir, char *packtmp)\n\nI'm definitely nitpicking here, but I think that both of these could be\n\"const\" to indicate that we're not modifying \"packdir\" or \"packtmp\".\n\nBut again, definitely not worth a reroll.\n\nThanks,\nTaylor\n"},{"id":"480407","messageId":"ZNQH6EMKqbuUzEhs@nand.local","threadId":"59867","inReplyTo":"20230808082608.582319-6-christian.couder@gmail.com","subject":"Re: [PATCH v4 5/8] repack: add `--filter=` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-09T21:40:56Z","receivedAt":"2023-08-09T21:41:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 08, 2023 at 10:26:05AM +0200, Christian Couder wrote:\n> @@ -871,6 +925,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\tOPT_END()\n> \t};\n>\n> +\tlist_objects_filter_init(&po_args.fturailter_options);\n> +\tlist_objects_filter_init(&cruft_po_args.filter_options);\n\nInitializing `po_args`'s `filter_options` makes sense to me, but do we\never use the cruft_po_args ones? From looking at this patch, I don't\nthink we do.\n\nInitializing them and then calling list_objects_filter_release() on it\nisn't wrong, but I can't tell whether initializing these is necessary\nor not in the first place.\n\n> +test_expect_success '--filter fails with --write-bitmap-index' '\n> +\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 test_must_fail git -C bare.git repack \\\n> +\t\t-a -d --write-bitmap-index --filter=blob:none\n> +'\n\nI can't remember off-hand why, but I am pretty sure that we usually\nwrite \"test_must_fail env ...\" before the command-line arguments instead\nof setting the environment variable outside of the test_must_fail call.\n\nI *think* that this is the same issue as the single-shot environment\nvariable assignment before function call thing that we see in some\nshells.\n\nRegardless, I wonder if we should be catching this --filter +\n--write-bitmap-index thing earlier. The error message I get when\nrunning this is:\n\n warning: Failed to write bitmap index. Packfile doesn't have full closure (object ac3e272b72bbf89def8657766b855d0656630ed4 is missing)\n fatal: failed to write bitmap index\n\nWhich comes from deep within the pack-bitmap-write.c internals\n(specifically in a failing call to `find_object_pos()`).\n\nI don't think that's wrong per-se, but I wonder if catching the\ncombination earlier would allow us to carry on writing the pack even if\nthe caller erroneously specified that they wanted a bitmap, similar to\nhow we handle that combination with other options (see the comment in\nbuiltin/pack-objects.c that starts with \"'hard reasons not to use\nbitmaps [...]'\").\n\nI doubt we'd see many naturally occurring instances of users running\n\"git repack\" with both the --filter spec option and\n--write-bitmap-index. But, I think that it would come up more often in\nbare repositories, where writing reachability bitmaps is the default\nstate during repacking unless specified otherwise.\n\nI suspect doing something like:\n\n--- 8< ---\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 000ebec7ab..d75d122a86 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4431,7 +4431,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tuse_bitmap_index = use_bitmap_index_default;\n\n \t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n-\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow(the_repository))\n+\tif (!use_internal_rev_list ||\n+\t (!pack_to_stdout && write_bitmap_index) ||\n+\t is_repository_shallow(the_repository) ||\n+\t filter_options.choice)\n \t\tuse_bitmap_index = 0;\n\n \tif (pack_to_stdout || !rev_list_all)\n--- >8 ---\n\nwould do the trick (perhaps with a warning(), and the corresponding test\nmodification, but I think I could go either way on the warning() since\nthere isn't one there currently.)\n\nI looked through the rest of the tests, and they all looked good to me,\nthanks.\n\nThanks,\nTaylor\n"},{"id":"480409","messageId":"ZNQI5vpcoXbkkPWa@nand.local","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"Re: [PATCH v4 0/8] Repack objects into separate packfiles based on a filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-09T21:45:10Z","receivedAt":"2023-08-09T21:45:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 08, 2023 at 10:26:00AM +0200, Christian Couder wrote:\n> # Changes since version 3\n>\n> Thanks to Junio who reviewed both version 1, 2 and 3, and to Taylor\n> who reviewed version 1 and 3! The changes are the following:\n\nI took a look through the range-diff as well as the patches themselves\nagain (skimming through the last three, which are much more\nstraightforward than the preceding ones).\n\nEverything looks good to me here, and I think that this version is ready\nto get picked up once we're on the other side of 2.42.\n\nI left a couple of comments throughout, but none of them merit a reroll\non their own. I think there are a couple of things we could easily\nignore (marking parameters as \"const\", etc.), and a couple of things\nthat we should probably take a look at after the dust has settled here.\n\nWe *may* want to fix up the test_must_fail invocation that has the\nenvironment variable on the left-hand side instead of using\n\"test_must_fail env\", but I don't know for sure.\n\nI do think that we should take another look at disabling the bitmap\nmachinery when given `--filter`, but I think that, too, can be done in\nanother series.\n\nThanks again for being so patient with all of my review comments. I hope\nit wasn't too big of a pain; this area feels very fragile (to me, at\nleast) so I wanted to give it an extra careful set of eyes.\n\nThanks,\nTaylor\n"},{"id":"480410","messageId":"xmqq350rlvzf.fsf@gitster.g","threadId":"59867","inReplyTo":"ZNQI5vpcoXbkkPWa@nand.local","subject":"Re: [PATCH v4 0/8] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-09T21:57:40Z","receivedAt":"2023-08-09T21:57:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> We *may* want to fix up the test_must_fail invocation that has the\n> environment variable on the left-hand side instead of using\n> \"test_must_fail env\", but I don't know for sure.\n\nAh, that is a show-stopper bug. We must fix it, but the necessary\nchange should be trivial.\n\n> Thanks again for being so patient with all of my review comments. I hope\n> it wasn't too big of a pain; this area feels very fragile (to me, at\n> least) so I wanted to give it an extra careful set of eyes.\n\nThanks for writing and reviewing.\n\n"},{"id":"480411","messageId":"xmqqy1ijkezm.fsf@gitster.g","threadId":"59867","inReplyTo":"CAP8UFD1eX8JMd91Say_sC7h_V08oRq32Wu9RM+SFtAQnhRPO2w@mail.gmail.com","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-09T22:50:05Z","receivedAt":"2023-08-09T22:50:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> but I haven't been able to check that this works on CI as all the job\n> seems to fail these days before they even start:\n\nThis remark made me worried, but (luckily) it does not seem to be\nthe case for other topics cooking in 'next' or queued in 'seen'.\n\nIt does seem that this topic directly queued on 'master' by itself\nwithout any other topics do break the CI quite badly. Almost\nnothing passes:\n\n https://github.com/git/git/actions/runs/5812873987\n\nbut it may be something as silly as failing test lint. I didn't\ncheck very closely.\n\nThanks.\n"},{"id":"480412","messageId":"xmqqttt7kcqi.fsf@gitster.g","threadId":"59867","inReplyTo":"xmqqy1ijkezm.fsf@gitster.g","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-09T23:38:45Z","receivedAt":"2023-08-09T23:38:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> It does seem that this topic directly queued on 'master' by itself\n> without any other topics do break the CI quite badly. Almost\n> nothing passes:\n>\n> https://github.com/git/git/actions/runs/5812873987\n>\n> but it may be something as silly as failing test lint. I didn't\n> check very closely.\n\nAnd a bit further digging reveals that it is the case.\n\nhttps://github.com/git/git/actions/runs/5812873987/job/15759211568#step:4:787\n\nLocally you should be able to reproduce it by\n\n make\n make -C t test-lint\n\nbefore sending your patches out.\n\nI've queued a squashable fix-up on top of the topic.\n\nThanks.\n\n---\n t/t7700-repack.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 9b1e189a62..48e92aa6f7 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -342,8 +342,9 @@ test_expect_success 'repacking with a filter works' '\n '\n \n test_expect_success '--filter fails with --write-bitmap-index' '\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 test_must_fail git -C bare.git repack \\\n-\t\t-a -d --write-bitmap-index --filter=blob:none\n+\ttest_must_fail \\\n+\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n '\n \n test_expect_success 'repacking with two filters works' '\n-- \n2.42.0-rc1\n\n"},{"id":"480413","messageId":"20230810001008.GA795985@coredump.intra.peff.net","threadId":"59867","inReplyTo":"xmqqttt7kcqi.fsf@gitster.g","subject":"Re: [PATCH v2 5/8] repack: add `--filter=` option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-08-10T00:10:08Z","receivedAt":"2023-08-10T00:10:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 09, 2023 at 04:38:45PM -0700, Junio C Hamano wrote:\n\n> Locally you should be able to reproduce it by\n> \n> make\n> make -C t test-lint\n> \n> before sending your patches out.\n\nThis shouldn't be necessary. \"make test\" is supposed to run test-lint\n(and does catch this case for me). Provided you haven't overridden that\nby setting TEST_LINT to some more limited set.\n\n-Peff\n"},{"id":"480608","messageId":"20230812000011.1227371-1-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230808082608.582319-1-christian.couder@gmail.com","subject":"[PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:03Z","receivedAt":"2023-08-12T00:00:37Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"# Intro\n\nLast year, John Cai sent 2 versions of a patch series to implement\n`git repack --filter=` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\n# Use cases for the new feature\n\nThis could be useful for example for the following purposes:\n\n 1) As a way for servers to save storage costs by for example moving\n large blobs, or all the blobs, or all the blobs in inactive\n repos, to separate storage (while still making them accessible\n using for example the alternates mechanism).\n\n 2) As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n 3) As a way for clients to reclaim some space when they cloned with\n a filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a\n separate directory or storage, then check that everything works\n well, and then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\nAlso since version 2 of this new series, commit messages, don't\nmention uses cases like 2) or 3) above, as people have different\nopinions on how it should be done. How it should be done could depend\na lot on the way promisor remotes are used, the software and hardware\nsetups used, etc, so it seems more difficult to \"sell\" this series by\ntalking about such use cases. As use case 1) seems simpler and more\nappealing, it makes more sense to only talk about it in the commit\nmessages.\n\n# Changes since version 4\n\nThanks to Junio who reviewed versions 1, 2, 3 and 4, and to Taylor who\nreviewed version 1, 3 and 4! Thanks also to Robert Coup who\nparticipated in the discussions related to version 2 and Peff who\nparticipated in the discussions related to version 4. The changes are\nthe following:\n\n- In patch 2/8, which introduces `test-tool find-pack`, a spurious\n space character has been removed between 'die' and '(', as suggested\n by Taylor.\n\n- In patch 4/8, which refactors code into a find_pack_prefix()\n function, this function has been changed so that the `packdir` and\n `packtmp` arguments are now 'const', as suggested by Taylor.\n\n- In patch 5/8, which introduces `--filter=` option, the\n `filter_options` member of the 'cruft_po_args' variable is not\n initialized and freed anymore, as this member is actually unused.\n\n- Also in patch 5/8, the '--filter fails with --write-bitmap-index'\n test has been changed to use `test_must_fail env` to fix failures\n with the 'test-lint' Makefile target, as suggested by Junio and\n Taylor. (Junio's 'SQUASH???' patch was squashed into that patch.)\n\n- Also the series was rebased on top of v2.42.0-rc1 as it will likely\n be merged after v2.42.0 will be released and Junio's\n cc/repack-sift-filtered-objects-to-separate-pack branch is based on\n top of v2.42.0-rc0.\n\n# Commit overview\n\n* 1/8 pack-objects: allow `--filter` without `--stdout`\n\n This patch is the same as in v1, v2, v3 and v4. To be able to later\n repack with a filter we need `git pack-objects` to write packfiles\n when it's filtering instead of just writing the pack without the\n filtered out objects to stdout.\n\n* 2/8 t/helper: add 'find-pack' test-tool\n\n For testing `git repack --filter=...` that we are going to\n implement, it's useful to have a test helper that can tell which\n packfiles contain a specific object. Since v4 only a space character\n has been removed between a function name and the following '(' to\n comply with our style guide.\n\n* 3/8 repack: refactor finishing pack-objects command\n\n No change in this patch compared to v2, v3 and v4. This is a small\n refactoring creating a new useful function, so that `git repack\n --filter=...` will be able to reuse it.\n\n* 4/8 repack: refactor finding pack prefix\n\n This is another small refactoring creating a small function that\n will be reused in the next patch. Since v4 the new function has been\n changed so that its `packdir` and `packtmp` argument are now const.\n\n* 5/8 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. It uses one\n `git pack-objects` process with the `--filter` option. And then\n another `git pack-objects` process with the `--stdin-packs`\n option. A few changes have been made since v4:\n\n - The `filter_options` member of the 'cruft_po_args' variable is\n not initialized and freed anymore, as this member is actually\n unused.\n\n - The test that checks that `--filter=...` fails with\n `--write-bitmap-index` has been changed to use `test_must_fail\n env` to fix failures with the 'test-lint' Makefile target.\n\n* 6/8 gc: add `gc.repackFilter` config option\n\n No change in this patch compared to v4. This is a gc config option\n so that `git gc` can also repack using a filter and put the filtered\n out objects into a separate packfile.\n\n* 7/8 repack: implement `--filter-to` for storing filtered out objects\n\n No change in this patch compared to v4. For some use cases, it's\n interesting to create the packfile that contains the filtered out\n objects into a separate location. This is similar to the\n `--expire-to` option for cruft packfiles.\n\n* 8/8 gc: add `gc.repackFilterTo` config option\n\n No change in this patch compared to v3 and v4. This allows\n specifying the location of the packfile that contains the filtered\n out objects when using `gc.repackFilter`.\n\n# Range-diff since v4\n\n1: 09fd23c7d0 = 1: bbcc368876 pack-objects: allow `--filter` without `--stdout`\n2: c75010d20c ! 2: f1b80e5728 t/helper: add 'find-pack' test-tool\n @@ t/helper/test-find-pack.c (new)\n + }\n +\n + if (count > -1 && count != actual_count)\n -+ die (\"bad packfile count %d instead of %d\", actual_count, count);\n ++ die(\"bad packfile count %d instead of %d\", actual_count, count);\n +\n + return 0;\n +}\n3: 28221861a0 = 3: ffecc73960 repack: refactor finishing pack-objects command\n4: 41d4faf62b ! 4: 6c2f381a88 repack: refactor finding pack prefix\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *ar\n return finish_pack_objects_cmd(&cmd, names, local);\n }\n \n -+static const char *find_pack_prefix(char *packdir, char *packtmp)\n ++static const char *find_pack_prefix(const char *packdir, const char *packtmp)\n +{\n + const char *pack_prefix;\n + if (!skip_prefix(packtmp, packdir, &pack_prefix))\n5: a929572b96 ! 5: 134700c2ce repack: add `--filter=` option\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n };\n \n + list_objects_filter_init(&po_args.filter_options);\n -+ list_objects_filter_init(&cruft_po_args.filter_options);\n +\n git_config(repack_config, &cruft_po_args);\n \n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n string_list_clear(&existing_kept_packs, 0);\n clear_pack_geometry(geometry);\n + list_objects_filter_release(&po_args.filter_options);\n -+ list_objects_filter_release(&cruft_po_args.filter_options);\n \n return ret;\n }\n @@ t/t7700-repack.sh: test_expect_success 'auto-bitmaps do not complain if unavaila\n +'\n +\n +test_expect_success '--filter fails with --write-bitmap-index' '\n -+ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 test_must_fail git -C bare.git repack \\\n -+ -a -d --write-bitmap-index --filter=blob:none\n ++ test_must_fail \\\n ++ env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n ++ git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n +'\n +\n +test_expect_success 'repacking with two filters works' '\n6: a22a560d74 = 6: d3365c7b48 gc: add `gc.repackFilter` config option\n7: 387b427fed = 7: 9a09382cd1 repack: implement `--filter-to` for storing filtered out objects\n8: 76fac86b0e = 8: a52e3a71db gc: add `gc.repackFilterTo` config option\n\n\nChristian Couder (8):\n pack-objects: allow `--filter` without `--stdout`\n t/helper: add 'find-pack' test-tool\n repack: refactor finishing pack-objects command\n repack: refactor finding pack prefix\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 16 ++\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 23 +++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 167 +++++++++++++++------\n t/helper/test-find-pack.c | 50 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 ++++++++++\n t/t5317-pack-objects-filter-objects.sh | 8 +\n t/t6500-gc.sh | 24 +++\n t/t7700-repack.sh | 197 +++++++++++++++++++++++++\n 14 files changed, 542 insertions(+), 50 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480609","messageId":"20230812000011.1227371-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"[PATCH v5 1/8] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:04Z","receivedAt":"2023-08-12T00:00:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n t/t5317-pack-objects-filter-objects.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex a9995a932c..583270a85f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -298,8 +298,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d2a162d528..000ebec7ab 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4400,12 +4400,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..2ff3eef9a3 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n \t! grep blob verify_result\n '\n \n+test_expect_success 'verify blob:none packfile without --stdout' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n+\t! grep blob verify_result\n+'\n+\n test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n \tgit -C r1 verify-pack -v ../all.pack >verify_result &&\n \tgrep -E \"commit|tree\" verify_result |\n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480610","messageId":"20230812000011.1227371-3-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"[PATCH v5 2/8] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:05Z","receivedAt":"2023-08-12T00:00:46Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In a following commit, we will make it possible to separate objects in\ndifferent packfiles depending on a filter.\n\nTo make sure that the right objects are in the right packs, let's add a\nnew test-tool that can display which packfile(s) a given object is in.\n\nLet's also make it possible to check if a given object is in the\nexpected number of packfiles with a `--check-count ` option.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 50 ++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 +++++++++++++++++++++++++++++++++++++++\n 5 files changed, 135 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\ndiff --git a/Makefile b/Makefile\nindex ace3e5a506..2534c831e8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..e8bd793e58\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,50 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"parse-options.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ *\n+ * If '--check-count ' is passed, then error out if the number of\n+ * packfiles containing the object is not .\n+ */\n+\n+static const char *find_pack_usage[] = {\n+\t\"test-tool find-pack [--check-count ] \",\n+\tNULL\n+};\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\tint count = -1, actual_count = 0;\n+\tconst char *prefix = setup_git_directory();\n+\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER('c', \"check-count\", &count, \"expected number of packs\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, find_pack_usage, 0);\n+\tif (argc != 1)\n+\t\tusage(find_pack_usage[0]);\n+\n+\tif (repo_get_oid(the_repository, argv[0], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[0]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tif (find_pack_entry_one(oid.hash, p)) {\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t\t\tactual_count++;\n+\t\t}\n+\n+\tif (count > -1 && count != actual_count)\n+\t\tdie(\"bad packfile count %d instead of %d\", actual_count, count);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex abe8a785eb..41da40c296 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ea2672436c..411dbf2db4 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\ndiff --git a/t/t0080-find-pack.sh b/t/t0080-find-pack.sh\nnew file mode 100755\nindex 0000000000..67b11216a3\n--- /dev/null\n+++ b/t/t0080-find-pack.sh\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+test_description='test `test-tool find-pack`'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit one &&\n+\ttest_commit two &&\n+\ttest_commit three &&\n+\ttest_commit four &&\n+\ttest_commit five\n+'\n+\n+test_expect_success 'repack everything into a single packfile' '\n+\tgit repack -a -d --no-write-bitmap-index &&\n+\n+\thead_commit_pack=$(test-tool find-pack HEAD) &&\n+\thead_tree_pack=$(test-tool find-pack HEAD^{tree}) &&\n+\tone_pack=$(test-tool find-pack HEAD:one.t) &&\n+\tthree_pack=$(test-tool find-pack HEAD:three.t) &&\n+\told_commit_pack=$(test-tool find-pack HEAD~4) &&\n+\n+\ttest-tool find-pack --check-count 1 HEAD &&\n+\ttest-tool find-pack --check-count=1 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count=0 HEAD:one.t &&\n+\t! test-tool find-pack -c 2 HEAD:one.t &&\n+\ttest-tool find-pack -c 1 HEAD:three.t &&\n+\n+\t# Packfile exists at the right path\n+\tcase \"$head_commit_pack\" in\n+\t\t\".git/objects/pack/pack-\"*\".pack\") true ;;\n+\t\t*) false ;;\n+\tesac &&\n+\ttest -f \"$head_commit_pack\" &&\n+\n+\t# Everything is in the same pack\n+\ttest \"$head_commit_pack\" = \"$head_tree_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$one_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$three_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$old_commit_pack\"\n+'\n+\n+test_expect_success 'add more packfiles' '\n+\tgit rev-parse HEAD^{tree} HEAD:two.t HEAD:four.t >objects &&\n+\tgit pack-objects .git/objects/pack/mypackname1 >packhash1 objects &&\n+\tgit pack-objects .git/objects/pack/mypackname2 >packhash2 head_tree_packs &&\n+\tgrep \"$head_commit_pack\" head_tree_packs &&\n+\tgrep mypackname1 head_tree_packs &&\n+\t! grep mypackname2 head_tree_packs &&\n+\ttest-tool find-pack --check-count 2 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count 1 HEAD^{tree} &&\n+\n+\t# HEAD:five.t is also in 2 packfiles\n+\ttest-tool find-pack HEAD:five.t >five_packs &&\n+\tgrep \"$head_commit_pack\" five_packs &&\n+\t! grep mypackname1 five_packs &&\n+\tgrep mypackname2 five_packs &&\n+\ttest-tool find-pack -c 2 HEAD:five.t &&\n+\t! test-tool find-pack --check-count=0 HEAD:five.t\n+'\n+\n+test_expect_success 'add more commits (as loose objects)' '\n+\ttest_commit six &&\n+\ttest_commit seven &&\n+\n+\ttest -z \"$(test-tool find-pack HEAD)\" &&\n+\ttest -z \"$(test-tool find-pack HEAD:six.t)\" &&\n+\ttest-tool find-pack --check-count 0 HEAD &&\n+\ttest-tool find-pack -c 0 HEAD:six.t &&\n+\t! test-tool find-pack -c 1 HEAD:seven.t\n+'\n+\n+test_done\n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480611","messageId":"20230812000011.1227371-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"[PATCH v5 3/8] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:06Z","receivedAt":"2023-08-12T00:00:48Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -705,9 +735,8 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct string_list *existing_kept_packs)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n \tconst char *scratch;\n \tint local = skip_prefix(destination, packdir, &scratch);\n@@ -751,27 +780,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -782,10 +791,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1016,18 +1023,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480612","messageId":"20230812000011.1227371-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"[PATCH v5 4/8] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:07Z","receivedAt":"2023-08-12T00:00:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new find_pack_prefix() to refactor code that handles finding\nthe pack prefix from the packtmp and packdir global variables, as we are\ngoing to need this feature again in following commit.\n\nSigned-off-by: Christian Couder ` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:08Z","receivedAt":"2023-08-12T00:00:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new option puts the objects specified by `` into a\nseparate packfile.\n\nThis could be useful if, for example, some blobs take up a lot of\nprecious space on fast storage while they are rarely accessed. It could\nmake sense to move them into a separate cheaper, though slower, storage.\n\nIt's possible to find which new packfile contains the filtered out\nobjects using one of the following:\n\n - `git verify-pack -v ...`,\n - `test-tool find-pack ...`, which a previous commit added,\n - `--filter-to=`, which a following commit will add to specify\n where the pack containing the filtered out objects will be.\n\nThis feature is implemented by running `git pack-objects` twice in a\nrow. The first command is run with `--filter=`, using the\nspecified filter. It packs objects while omitting the objects specified\nby the filter. Then another `git pack-objects` command is launched using\n`--stdin-packs`. We pass it all the previously existing packs into its\nstdin, so that it will pack all the objects in the previously existing\npacks. But we also pass into its stdin, the pack created by the previous\n`git pack-objects --filter=` command as well as the kept\npacks, all prefixed with '^', so that the objects in these packs will be\nomitted from the resulting pack. The result is that only the objects\nfiltered out by the first `git pack-objects` command are in the pack\nresulting from the second `git pack-objects` command.\n\nAs the interactions with kept packs are a bit tricky, a few related\ntests are added.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 12 ++++\n builtin/repack.c | 73 +++++++++++++++++++\n t/t7700-repack.sh | 135 +++++++++++++++++++++++++++++++++++\n 3 files changed, 220 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..6d5bec7716 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,18 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. Note\n+\tthat objects used in the working directory are not filtered\n+\tout. So for the split to fully work, it's best to perform it\n+\tin a bare repo and to use the `-a` and `-d` options along with\n+\tthis option. Also `--no-write-bitmap-index` (or the\n+\t`repack.writebitmaps` config option set to `false`) should be\n+\tused otherwise writing bitmap index will fail, as it supposes\n+\ta single packfile containing all the objects. See\n+\tlinkgit:git-rev-list[1] for valid `` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 825da1caca..c672387ab9 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -21,6 +21,7 @@\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n #include \"refs.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define ALL_INTO_ONE 1\n #define LOOSEN_UNREACHABLE 2\n@@ -57,6 +58,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tstruct list_objects_filter_options filter_options;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -726,6 +728,57 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n+static int write_filtered_pack(const struct pack_objects_args *args,\n+\t\t\t const char *destination,\n+\t\t\t const char *pack_prefix,\n+\t\t\t struct string_list *keep_pack_list,\n+\t\t\t struct string_list *names,\n+\t\t\t struct string_list *existing_packs,\n+\t\t\t struct string_list *existing_kept_packs)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret, i;\n+\tconst char *caret;\n+\tconst char *scratch;\n+\tint local = skip_prefix(destination, packdir, &scratch);\n+\n+\tprepare_pack_objects(&cmd, args, destination);\n+\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\n+\tif (!pack_kept_objects)\n+\t\tstrvec_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list->nr; i++)\n+\t\tstrvec_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t keep_pack_list->items[i].string);\n+\n+\tcmd.in = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Here 'names' contains only the pack(s) that were just\n+\t * written, which is exactly the packs we want to keep. Also\n+\t * 'existing_kept_packs' already contains the packs in\n+\t * 'keep_pack_list'.\n+\t */\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, names)\n+\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n+\tfor_each_string_list_item(item, existing_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tcaret = pack_kept_objects ? \"\" : \"^\";\n+\tfor_each_string_list_item(item, existing_kept_packs)\n+\t\tfprintf(in, \"%s%s.pack\\n\", caret, item->string);\n+\tfclose(in);\n+\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -858,6 +911,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&po_args.filter_options),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -871,6 +925,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tOPT_END()\n \t};\n \n+\tlist_objects_filter_init(&po_args.filter_options);\n+\n \tgit_config(repack_config, &cruft_po_args);\n \n \targc = parse_options(argc, argv, prefix, builtin_repack_options,\n@@ -1011,6 +1067,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (po_args.filter_options.choice)\n+\t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\n \tif (geometry)\n \t\tcmd.in = -1;\n \telse\n@@ -1097,6 +1157,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter_options.choice) {\n+\t\tret = write_filtered_pack(&po_args,\n+\t\t\t\t\t packtmp,\n+\t\t\t\t\t find_pack_prefix(packdir, packtmp),\n+\t\t\t\t\t &keep_pack_list,\n+\t\t\t\t\t &names,\n+\t\t\t\t\t &existing_nonkept_packs,\n+\t\t\t\t\t &existing_kept_packs);\n+\t\tif (ret)\n+\t\t\tgoto cleanup;\n+\t}\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\n@@ -1231,6 +1303,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&existing_nonkept_packs, 0);\n \tstring_list_clear(&existing_kept_packs, 0);\n \tclear_pack_geometry(geometry);\n+\tlist_objects_filter_release(&po_args.filter_options);\n \n \treturn ret;\n }\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 27b66807cd..39e89445fd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -327,6 +327,141 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack -c 1 HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack -c 1 HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+\ttest_must_fail \\\n+\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n+'\n+\n+test_expect_success 'repacking with two filters works' '\n+\tgit init two-filters &&\n+\t(\n+\t\tcd two-filters &&\n+\t\tmkdir subdir &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit subdir_bar subdir/bar &&\n+\t\ttest_commit subdir_baz subdir/baz\n+\t) &&\n+\tgit clone --no-local --bare two-filters two-filters.git &&\n+\t(\n+\t\tcd two-filters.git &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/*.pack &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d \\\n+\t\t\t--filter=blob:none --filter=tree:1 &&\n+\t\ttest_stdout_line_count = 2 ls objects/pack/*.pack &&\n+\t\tcommit_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\t\tblob_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\troot_tree_pack=$(test-tool find-pack -c 1 HEAD^{tree}) &&\n+\t\tsubdir_tree_hash=$(git ls-tree --object-only HEAD -- subdir) &&\n+\t\tsubdir_tree_pack=$(test-tool find-pack -c 1 \"$subdir_tree_hash\") &&\n+\n+\t\t# Root tree and subdir tree are not in the same packfiles\n+\t\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\t\ttest \"$commit_pack\" = \"$root_tree_pack\" &&\n+\t\ttest \"$blob_pack\" = \"$subdir_tree_pack\"\n+\t)\n+'\n+\n+prepare_for_keep_packs () {\n+\tgit init keep-packs &&\n+\t(\n+\t\tcd keep-packs &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit bar\n+\t) &&\n+\tgit clone --no-local --bare keep-packs keep-packs.git &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\t# Create two packs\n+\t\t# The first pack will contain all of the objects except one blob\n+\t\tgit rev-list --objects --all >objs &&\n+\t\tgrep -v \"bar.t\" objs | git pack-objects pack &&\n+\t\t# The second pack will contain the excluded object and be kept\n+\t\tpackid=$(grep \"bar.t\" objs | git pack-objects pack) &&\n+\t\t>pack-$packid.keep &&\n+\n+\t\t# Replace the existing pack with the 2 new ones\n+\t\trm -f objects/pack/pack* &&\n+\t\tmv pack-* objects/pack/\n+\t)\n+}\n+\n+test_expect_success '--filter works with .keep packs' '\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack_1=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\t# Object bar is still only in the old .keep pack\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$bar_pack_1\" = \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack_1\" &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\" &&\n+\t\ttest \"$bar_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n+test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n+\trm -rf keep-packs keep-packs.git &&\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--pack-kept-objects &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\ttest-tool find-pack -c 2 HEAD:bar.t >bar_pack_1 &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\t# Object bar is in both the old .keep pack and the new\n+\t\t# pack that contained the filtered out objects\n+\t\tgrep \"$bar_pack\" bar_pack_1 &&\n+\t\tgrep \"$foo_pack_1\" bar_pack_1 &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480614","messageId":"20230812000011.1227371-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"[PATCH v5 6/8] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:09Z","receivedAt":"2023-08-12T00:01:01Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex ca47eb2008..2153bde7ac 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the\n operation (either generating a cruft pack or unpacking unreachable\n objects) will be halted.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 19d73067aa..9b0984f301 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -61,6 +61,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -170,6 +171,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -355,6 +358,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..232e403b66 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,19 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480615","messageId":"20230812000011.1227371-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"[PATCH v5 7/8] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:10Z","receivedAt":"2023-08-12T00:01:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage.\n\nEven in a different directory, this pack can be accessible if, for\nexample, the Git alternates mechanism is used to point to it. In fact\nnot using the Git alternates mechanism can corrupt a repo as the\ngenerated pack containing the filtered objects might not be accessible\nfrom the repo any more. So setting up the Git alternates mechanism\nshould be done before using this feature if the user wants the repo to\nbe fully usable while this feature is used.\n\nIn some cases, like when a repo has just been cloned or when there is no\nother activity in the repo, it's Ok to setup the Git alternates\nmechanism afterwards though. It's also Ok to just inspect the generated\npackfile containing the filtered objects and then just move it into the\n'.git/objects/pack/' directory manually. That's why it's not necessary\nfor this command to check that the Git alternates mechanism has been\nalready setup.\n\nWhile at it, as an example to show that `--filter` and `--filter-to`\nwork well with other options, let's also add a test to check that these\noptions work well with `--max-pack-size`.\n\nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 11 +++++++\n builtin/repack.c | 10 +++++-\n t/t7700-repack.sh | 62 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 6d5bec7716..8545a32667 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -155,6 +155,17 @@ depth is 4095.\n \ta single packfile containing all the objects. See\n \tlinkgit:git-rev-list[1] for valid `` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. Only useful with `--filter`. This can be\n+\tused for putting the pack on a separate object directory that\n+\tis accessed through the Git alternates mechanism. **WARNING:**\n+\tIf the packfile containing the filtered out objects is not\n+\taccessible, the repo can become corrupt as it might not be\n+\tpossible to access the objects in that packfile. See the\n+\t`objects` and `objects/info/alternates` sections of\n+\tlinkgit:gitrepository-layout[5].\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c672387ab9..c396029ec9 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -870,6 +870,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -922,6 +923,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1070,6 +1073,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (po_args.filter_options.choice)\n \t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n \t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\telse if (filter_to)\n+\t\tdie(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n \n \tif (geometry)\n \t\tcmd.in = -1;\n@@ -1158,8 +1163,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter_options.choice) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\n \t\tret = write_filtered_pack(&po_args,\n-\t\t\t\t\t packtmp,\n+\t\t\t\t\t filter_to,\n \t\t\t\t\t find_pack_prefix(packdir, packtmp),\n \t\t\t\t\t &keep_pack_list,\n \t\t\t\t\t &names,\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 39e89445fd..48e92aa6f7 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -462,6 +462,68 @@ test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n \t)\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 0 HEAD:file1) &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack -c 1 $blob_hash) &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n+test_expect_success '--filter works with --max-pack-size' '\n+\trm -rf filtered.git &&\n+\tgit init --bare filtered.git &&\n+\tgit init max-pack-size &&\n+\t(\n+\t\tcd max-pack-size &&\n+\t\ttest_commit base &&\n+\t\t# two blobs which exceed the maximum pack size\n+\t\ttest-tool genrandom foo 1048576 >foo &&\n+\t\tgit hash-object -w foo &&\n+\t\ttest-tool genrandom bar 1048576 >bar &&\n+\t\tgit hash-object -w bar &&\n+\t\tgit add foo bar &&\n+\t\tgit commit -m \"adding foo and bar\"\n+\t) &&\n+\tgit clone --no-local --bare max-pack-size max-pack-size.git &&\n+\t(\n+\t\tcd max-pack-size.git &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--max-pack-size=1M \\\n+\t\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\t\techo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n+\n+\t\t# Check that the 3 blobs are in different packfiles in filtered.git\n+\t\ttest_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar) &&\n+\t\tbase_pack=$(test-tool find-pack -c 1 HEAD:base.t) &&\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" != \"$base_pack\" &&\n+\t\ttest \"$bar_pack\" != \"$base_pack\" &&\n+\t\tfor pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n+\t\tdo\n+\t\t\tcase \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n+\t\tdone\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480616","messageId":"20230812000011.1227371-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"[PATCH v5 8/8] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:00:11Z","receivedAt":"2023-08-12T00:01:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 11 +++++++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 2153bde7ac..466466d6cc 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -150,6 +150,17 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. **WARNING:** The\n+\tspecified location should be accessible, using for example the\n+\tGit alternates mechanism, otherwise the repo could be\n+\tconsidered corrupt by Git as it migh not be able to access the\n+\tobjects in that packfile. See the `--filter-to=` option\n+\tof linkgit:git-repack[1] and the `objects/info/alternates`\n+\tsection of linkgit:gitrepository-layout[5].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9b0984f301..1b7c775d94 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -172,6 +173,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -361,6 +363,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 232e403b66..e412cf8daf 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -215,6 +214,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.rc1.8.ga52e3a71db\n\n"},{"id":"480617","messageId":"CAP8UFD1HhvaAV2=_tYXQOsi57qq6U6Hp1-LEBCaM-9n-mmKHrg@mail.gmail.com","threadId":"59867","inReplyTo":"ZNQI5vpcoXbkkPWa@nand.local","subject":"Re: [PATCH v4 0/8] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-08-12T00:12:50Z","receivedAt":"2023-08-12T00:13:06Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Aug 9, 2023 at 11:45 PM Taylor Blau wrote:\n\n> I took a look through the range-diff as well as the patches themselves\n> again (skimming through the last three, which are much more\n> straightforward than the preceding ones).\n>\n> Everything looks good to me here, and I think that this version is ready\n> to get picked up once we're on the other side of 2.42.\n\nThanks again for your review!\n\n> I left a couple of comments throughout, but none of them merit a reroll\n> on their own. I think there are a couple of things we could easily\n> ignore (marking parameters as \"const\", etc.), and a couple of things\n> that we should probably take a look at after the dust has settled here.\n\nThe version 5 I just sent should fix all the small things that you\nfound in your review.\n\n> We *may* want to fix up the test_must_fail invocation that has the\n> environment variable on the left-hand side instead of using\n> \"test_must_fail env\", but I don't know for sure.\n\nThis is fixed by squashing Junio's 'SQUASH???' commit in version 5.\n\n> I do think that we should take another look at disabling the bitmap\n> machinery when given `--filter`, but I think that, too, can be done in\n> another series.\n\nI agree. I plan to do it later when this is merged. I think it would\nmake it easier to use the new --filter feature, but it would require\nchanges in code, tests and documentation, which can be done later.\n\nThanks,\nChristian.\n"},{"id":"480666","messageId":"xmqqv8dhjfgm.fsf@gitster.g","threadId":"59867","inReplyTo":"20230812000011.1227371-1-christian.couder@gmail.com","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-15T00:51:05Z","receivedAt":"2023-08-15T00:56:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> # Changes since version 4\n>\n> Thanks to Junio who reviewed versions 1, 2, 3 and 4, and to Taylor who\n> reviewed version 1, 3 and 4! Thanks also to Robert Coup who\n> participated in the discussions related to version 2 and Peff who\n> participated in the discussions related to version 4. The changes are\n> the following:\n>\n> - In patch 2/8, which introduces `test-tool find-pack`, a spurious\n> space character has been removed between 'die' and '(', as suggested\n> by Taylor.\n>\n> - In patch 4/8, which refactors code into a find_pack_prefix()\n> function, this function has been changed so that the `packdir` and\n> `packtmp` arguments are now 'const', as suggested by Taylor.\n>\n> - In patch 5/8, which introduces `--filter=` option, the\n> `filter_options` member of the 'cruft_po_args' variable is not\n> initialized and freed anymore, as this member is actually unused.\n>\n> - Also in patch 5/8, the '--filter fails with --write-bitmap-index'\n> test has been changed to use `test_must_fail env` to fix failures\n> with the 'test-lint' Makefile target, as suggested by Junio and\n> Taylor. (Junio's 'SQUASH???' patch was squashed into that patch.)\n\nThanks. I do not recall if the previous version with SQUASH??? passed\nthe tests or not, but this round seems to be breaking the exact test\nwe had trouble with with the previous round:\n\n https://github.com/git/git/actions/runs/5850998716/job/15861158252#step:4:1822\n\nThe symptom looks like that \"test_must_fail env\" test is not\nfailing. Ring a bell?\n\nThanks.\n"},{"id":"480687","messageId":"ZNvxg7BVJ+C5XFY4@nand.local","threadId":"59867","inReplyTo":"xmqqv8dhjfgm.fsf@gitster.g","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-15T21:43:31Z","receivedAt":"2023-08-15T21:44:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Aug 14, 2023 at 05:51:05PM -0700, Junio C Hamano wrote:\n> Thanks. I do not recall if the previous version with SQUASH??? passed\n> the tests or not, but this round seems to be breaking the exact test\n> we had trouble with with the previous round:\n>\n> https://github.com/git/git/actions/runs/5850998716/job/15861158252#step:4:1822\n>\n> The symptom looks like that \"test_must_fail env\" test is not\n> failing. Ring a bell?\n\nThat does ring a bell for me, but this is a different failure than\nbefore, IIRC.\n\nThis time we're expecting to fail writing a bitmap during a filtered\nrepack, but we succeed. I was wondering in [1] whether or not we should\nbe catching this bad combination of options more eagerly than relying on\nthe pack-bitmap machinery to notice that we're missing a reachability\nclosure.\n\nI think the reason that this succeeds is that we already have a bitmap,\nand it likely reuses all of the existing bitmaps before discovering that\nthe pack we wrote doesn't contain all objects. So doing this \"fixes\" the\nimmediate issue:\n\n--- 8< ---\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 48e92aa6f7..e5134d3451 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -342,6 +342,7 @@ test_expect_success 'repacking with a filter works' '\n '\n\n test_expect_success '--filter fails with --write-bitmap-index' '\n+\trm -f bare.git/objects/pack/*.bitmap &&\n \ttest_must_fail \\\n \t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n \t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n--- >8 ---\n\nbut I wonder if a more complete fix would be something like:\n\n--- 8< ---\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c396029ec9..f021349c4e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -48,6 +48,11 @@ static const char incremental_bitmap_conflict_error[] = N_(\n \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n );\n\n+static const char filtered_bitmap_conflict_error[] = N_(\n+\"Filtered repacks are incompatible with bitmap indexes. Use\\n\"\n+\"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n+);\n+\n struct pack_objects_args {\n \tconst char *window;\n \tconst char *window_memory;\n@@ -953,7 +958,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n\n \tif (write_bitmaps < 0) {\n \t\tif (!write_midx &&\n-\t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n+\t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()) &&\n+\t\t !po_args.filter_options.choice)\n \t\t\twrite_bitmaps = 0;\n \t} else if (write_bitmaps &&\n \t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n@@ -966,6 +972,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n\n+\tif (write_bitmaps && po_args.filter_options.choice)\n+\t\tdie(_(filtered_bitmap_conflict_error));\n+\n \tif (write_bitmaps && po_args.local && has_alt_odb(the_repository)) {\n \t\t/*\n \t\t * When asked to do a local repack, but we have\n--- >8 ---\n\nwould be preferable.\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/ZNQH6EMKqbuUzEhs@nand.local/\n"},{"id":"480691","messageId":"xmqqjztwgcnc.fsf@gitster.g","threadId":"59867","inReplyTo":"ZNvxg7BVJ+C5XFY4@nand.local","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-15T22:32:23Z","receivedAt":"2023-08-15T22:33:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> I think the reason that this succeeds is that we already have a bitmap,\n> and it likely reuses all of the existing bitmaps before discovering that\n> the pack we wrote doesn't contain all objects.\n\nNow I am confused.\n\nWe were asked to write bitmap index when we are going to create an\nincomplete pack, and the packfile we generate with the filter will\nnot have full set of objects, and generating a bitmap with such an\nincomplete knowledge of what objects are reachable from what would\nbe a disaster, so we should turn it off. But the posted patch\nlacked such a \"we should abort when bitmap is asked to be written\nwhile filtering\" logic.\n\nThen what were we expecting for the test to fail for?\n\n> but I wonder if a more complete fix would be something like:\n> ...\n> @@ -966,6 +972,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n> \t\tdie(_(incremental_bitmap_conflict_error));\n>\n> +\tif (write_bitmaps && po_args.filter_options.choice)\n> +\t\tdie(_(filtered_bitmap_conflict_error));\n> +\n\nIt sounds like the most direct fix.\n"},{"id":"480695","messageId":"ZNwFlcS3SOS9h77N@nand.local","threadId":"59867","inReplyTo":"xmqqjztwgcnc.fsf@gitster.g","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-15T23:09:09Z","receivedAt":"2023-08-15T23:10:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 15, 2023 at 03:32:23PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > I think the reason that this succeeds is that we already have a bitmap,\n> > and it likely reuses all of the existing bitmaps before discovering that\n> > the pack we wrote doesn't contain all objects.\n>\n> Now I am confused.\n>\n> We were asked to write bitmap index when we are going to create an\n> incomplete pack, and the packfile we generate with the filter will\n> not have full set of objects, and generating a bitmap with such an\n> incomplete knowledge of what objects are reachable from what would\n> be a disaster, so we should turn it off. But the posted patch\n> lacked such a \"we should abort when bitmap is asked to be written\n> while filtering\" logic.\n\nI was similarly confused, and started writing a patch to detect when we\nsee objects in one bitmap but not the other when remapping. But we\nalready handle that case, see the call to `rebuild_bitmap()` from\n`fill_bitmap_commit()` in pack-bitmap-write.c.\n\nSo I don't think we'd ever end up reusing an existing bitmap that refers\nto objects that we don't have.\n\nBut something is definitely strange here. The bitmap generated by this\ntest claims to have three commits:\n\n $ ~/src/git/t/helper/test-tool bitmap list-commits\n 95a9e53327b06212dcf98bd44794b0e2b913deab\n 3677360288c631b6b2e1f0e1f081b1e518605e9f\n 6f105e6234717c52e9b117b08840926910a68314\n\n...but none of them actually appear to exist in the bitmap:\n\n $ git rev-list --test-bitmap 95a9e53327b06212dcf98bd44794b0e2b913deab\n Bitmap v1 test (3 entries loaded)\n Found bitmap for '95a9e53327b06212dcf98bd44794b0e2b913deab'. 64 bits / 8b3b6ee7 checksum\n fatal: object not in bitmap: 'ac3e272b72bbf89def8657766b855d0656630ed4'\n\nI think what's going on here is that we attempt to create bitmaps for\nall three of those commits. We then try and reuse the existing bitmaps,\nbut fail, because we are missing some objects.\n\nSo then we try and generate the bitmap from scratch, and when we get\ndown to fill_bitmap_tree() we look up the bit position of the tree\nitself, and find a non-zero answer, indicating that we have already\nmarked that tree.\n\nAnd fill_bitmap_tree() correctly assumes that if we have marked the bit\ncorresponding to the tree, that everything reachable from that tree has\nalso been marked. So we never try and locate the bit position for the\nblob, since we already think that we have a blob marked in the resulting\nbitmap!\n\nBut why is that tree marked in the first place? It's because we attempt\nto rebuild the bitmap from the existing .bitmap file, but fail part of\nthe way through (when we look up the first blob object in the reposition\ntable). But that happens *after* we see the tree object, so its bit\nposition is marked, even though we didn't rebuild a complete bitmap.\n\nI don't think this matters outside of filtered repacks, but it would be\na serious bug to not catch this earlier up like suggested in the\n(quoted) patch below.\n\n> > but I wonder if a more complete fix would be something like:\n> > ...\n> > @@ -966,6 +972,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> > \tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n> > \t\tdie(_(incremental_bitmap_conflict_error));\n> >\n> > +\tif (write_bitmaps && po_args.filter_options.choice)\n> > +\t\tdie(_(filtered_bitmap_conflict_error));\n> > +\n>\n> It sounds like the most direct fix.\n\nI agree.\n\nI think that we would be OK to not change the implementation of\nrebuild_bitmap(), or its caller in fill_bitmap_commit(), since this only\nbites us when bitmapping a filtered pack, and we should catch that case\nwell before getting this deep into the bitmap code.\n\nBut it does seem suspect that we rebuild right into ent->bitmap, so we\nmay want to consider doing something like:\n\n--- >8 ---\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f6757c3cbf..f4ecdf8b0e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -413,15 +413,19 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n\n \t\tif (old_bitmap && mapping) {\n \t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct bitmap *remapped = bitmap_new();\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n \t\t\t * parents or the tree for this commit.\n \t\t\t */\n-\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap)) {\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n+\t\t\t\tbitmap_or(ent->bitmap, remapped);\n+\t\t\t\tbitmap_free(remapped);\n \t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tbitmap_free(remapped);\n \t\t}\n\n \t\t/*\n--- 8< ---\n\non top.\n\nApplying that patch and then rerunning the tests with the appropriate\nTEST variables causes the 'git repack' to fail as expected, ensuring\nthat the containing test passes.\n\nThanks,\nTaylor\n"},{"id":"480696","messageId":"xmqqfs4jhp3p.fsf@gitster.g","threadId":"59867","inReplyTo":"ZNwFlcS3SOS9h77N@nand.local","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-15T23:18:02Z","receivedAt":"2023-08-15T23:18:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> But why is that tree marked in the first place? It's because we attempt\n> to rebuild the bitmap from the existing .bitmap file, but fail part of\n> the way through (when we look up the first blob object in the reposition\n> table). But that happens *after* we see the tree object, so its bit\n> position is marked, even though we didn't rebuild a complete bitmap.\n\nSo, there is another bug lurking, other than the lack of \"combining\nfiltered repack and bitmaps are explicitly forbidden\" logic? We see\nthe tree object, we immediately mark it as \"done\" even we are not,\nthen we finish in failure and the \"done\" mark is left behind? Do we\nneed two bits, \"under review\" and \"done\", or something then?\n\n> But it does seem suspect that we rebuild right into ent->bitmap, so we\n> may want to consider doing something like:\n>\n> --- >8 ---\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index f6757c3cbf..f4ecdf8b0e 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -413,15 +413,19 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n>\n> \t\tif (old_bitmap && mapping) {\n> \t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n> +\t\t\tstruct bitmap *remapped = bitmap_new();\n> \t\t\t/*\n> \t\t\t * If this commit has an old bitmap, then translate that\n> \t\t\t * bitmap and add its bits to this one. No need to walk\n> \t\t\t * parents or the tree for this commit.\n> \t\t\t */\n> -\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap)) {\n> +\t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n> +\t\t\t\tbitmap_or(ent->bitmap, remapped);\n> +\t\t\t\tbitmap_free(remapped);\n> \t\t\t\treused_bitmaps_nr++;\n> \t\t\t\tcontinue;\n> \t\t\t}\n> +\t\t\tbitmap_free(remapped);\n> \t\t}\n>\n> \t\t/*\n> --- 8< ---\n>\n> on top.\n>\n> Applying that patch and then rerunning the tests with the appropriate\n> TEST variables causes the 'git repack' to fail as expected, ensuring\n> that the containing test passes.\n\nInteresting.\n"},{"id":"480699","messageId":"ZNwaoGBNrpdzlPUq@nand.local","threadId":"59867","inReplyTo":"xmqqfs4jhp3p.fsf@gitster.g","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-16T00:38:56Z","receivedAt":"2023-08-16T00:39:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 15, 2023 at 04:18:02PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > But why is that tree marked in the first place? It's because we attempt\n> > to rebuild the bitmap from the existing .bitmap file, but fail part of\n> > the way through (when we look up the first blob object in the reposition\n> > table). But that happens *after* we see the tree object, so its bit\n> > position is marked, even though we didn't rebuild a complete bitmap.\n>\n> So, there is another bug lurking, other than the lack of \"combining\n> filtered repack and bitmaps are explicitly forbidden\" logic?\n\nI think that there is a bug lurking in the sense of trying to reuse\nbitmaps when covering a pack that doesn't have reachability closure in\nthis particular scenario.\n\nBut there are no \"blessed\" use-cases for doing this. So I think that we\nshould indeed fix this, but I am not immediately concerned here.\n\n> We see the tree object, we immediately mark it as \"done\" even we are\n> not, then we finish in failure and the \"done\" mark is left behind? Do\n> we need two bits, \"under review\" and \"done\", or something then?\n\nNo; we can either reuse a complete bitmap or not. So it's fine to OR\nall of the (permuted) bits into ent->bitmap, but it's not OK to fill in\njust part of them.\n\nThanks,\nTaylor\n"},{"id":"480720","messageId":"xmqqil9egb65.fsf@gitster.g","threadId":"59867","inReplyTo":"ZNwaoGBNrpdzlPUq@nand.local","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-16T17:16:34Z","receivedAt":"2023-08-16T17:17:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> I think that there is a bug lurking in the sense of trying to reuse\n> bitmaps when covering a pack that doesn't have reachability closure in\n> this particular scenario.\n>\n> But there are no \"blessed\" use-cases for doing this. So I think that we\n> should indeed fix this, but I am not immediately concerned here.\n\nOK.\n\n> No; we can either reuse a complete bitmap or not. So it's fine to OR\n> all of the (permuted) bits into ent->bitmap, but it's not OK to fill in\n> just part of them.\n\nSounds sane.\n\n"},{"id":"481672","messageId":"CAP8UFD2kFjcD29kCp4j1AceCs=mr+_vN+sT5AVnoQNJBaoFt8A@mail.gmail.com","threadId":"59867","inReplyTo":"ZNwFlcS3SOS9h77N@nand.local","subject":"Re: [PATCH v5 0/8] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:20:27Z","receivedAt":"2023-09-11T21:38:05Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Aug 16, 2023 at 1:09 AM Taylor Blau wrote:\n>\n> On Tue, Aug 15, 2023 at 03:32:23PM -0700, Junio C Hamano wrote:\n> > Taylor Blau writes:\n\n> > > but I wonder if a more complete fix would be something like:\n> > > ...\n> > > @@ -966,6 +972,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> > > if (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n> > > die(_(incremental_bitmap_conflict_error));\n> > >\n> > > + if (write_bitmaps && po_args.filter_options.choice)\n> > > + die(_(filtered_bitmap_conflict_error));\n> > > +\n> >\n> > It sounds like the most direct fix.\n\nI would be Ok with such a fix, if we think that we don't want to fix\nthe underlying issue, or if we think that fixing the underlying issue\nis not enough...\n\n> I agree.\n>\n> I think that we would be OK to not change the implementation of\n> rebuild_bitmap(), or its caller in fill_bitmap_commit(), since this only\n> bites us when bitmapping a filtered pack, and we should catch that case\n> well before getting this deep into the bitmap code.\n>\n> But it does seem suspect that we rebuild right into ent->bitmap, so we\n> may want to consider doing something like:\n>\n> --- >8 ---\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index f6757c3cbf..f4ecdf8b0e 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -413,15 +413,19 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n>\n> if (old_bitmap && mapping) {\n> struct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n> + struct bitmap *remapped = bitmap_new();\n> /*\n> * If this commit has an old bitmap, then translate that\n> * bitmap and add its bits to this one. No need to walk\n> * parents or the tree for this commit.\n> */\n> - if (old && !rebuild_bitmap(mapping, old, ent->bitmap)) {\n> + if (old && !rebuild_bitmap(mapping, old, remapped)) {\n> + bitmap_or(ent->bitmap, remapped);\n> + bitmap_free(remapped);\n> reused_bitmaps_nr++;\n> continue;\n> }\n> + bitmap_free(remapped);\n> }\n>\n> /*\n> --- 8< ---\n>\n> on top.\n>\n> Applying that patch and then rerunning the tests with the appropriate\n> TEST variables causes the 'git repack' to fail as expected, ensuring\n> that the containing test passes.\n\n...however I think that fixing this underlying issue is important, as\nit might cause other tricky issues in the future, for example if other\nbitmap code is copying or reusing this code.\n\nSo I just sent a version 6 of this series with this change in a new\npatch. I hope my explanations in the commit message are good enough.\n\nThanks for finding the cause of the CI test failures and suggesting this fix,\nChristian.\n"},{"id":"481675","messageId":"20230911150618.129737-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 4/9] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:13Z","receivedAt":"2023-09-11T21:38:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new find_pack_prefix() to refactor code that handles finding\nthe pack prefix from the packtmp and packdir global variables, as we are\ngoing to need this feature again in following commit.\n\nSigned-off-by: Christian Couder ` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\n# Use cases for the new feature\n\nThis could be useful for example for the following purposes:\n\n 1) As a way for servers to save storage costs by for example moving\n large blobs, or all the blobs, or all the blobs in inactive\n repos, to separate storage (while still making them accessible\n using for example the alternates mechanism).\n\n 2) As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n 3) As a way for clients to reclaim some space when they cloned with\n a filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a\n separate directory or storage, then check that everything works\n well, and then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\nAlso since version 2 of this new series, commit messages, don't\nmention uses cases like 2) or 3) above, as people have different\nopinions on how it should be done. How it should be done could depend\na lot on the way promisor remotes are used, the software and hardware\nsetups used, etc, so it seems more difficult to \"sell\" this series by\ntalking about such use cases. As use case 1) seems simpler and more\nappealing, it makes more sense to only talk about it in the commit\nmessages.\n\n# Changes since version 5\n\nThanks to Junio who reviewed or commented on versions 1, 2, 3, 4 and\n5, and to Taylor who reviewed or commented on version 1, 3, 4 and 5!\nThanks also to Robert Coup who participated in the discussions related\nto version 2 and Peff who participated in the discussions related to\nversion 4. There is only the following code change since version 5:\n\n- Patch 5/9 (pack-bitmap-write: rebuild using new bitmap when\n remapping) is new. It fixes a bitmap rebuilding issue that wasn't\n triggered previously but got triggered by this series and caused CI\n tests to fail. The patch is taken from a suggestion by Taylor in:\n\n https://lore.kernel.org/git/ZNwFlcS3SOS9h77N@nand.local/\n\n I checked that CI tests now passes in:\n\n https://github.com/chriscool/git/actions/runs/6122146278\n\n (There is a failure on 'win test (5)' with \"failed: t7527.17\n directory changes to a file\", but it looks like it's not related to\n the previous issue and also not related to to this series at all.)\n\nAnother change is that this series has been rebased on top of\n94e83dcf5b (The seventh batch, 2023-09-07) to fix a few conflicts\nrelated to changes in the geometry code, as can be seen in the\nshort range-diff below.\n\n# Commit overview\n\n(No changes in any of the patches compared to version 5, except that\npatch 5/9 is new.)\n\n* 1/9 pack-objects: allow `--filter` without `--stdout`\n\n To be able to later repack with a filter we need `git pack-objects`\n to write packfiles when it's filtering instead of just writing the\n pack without the filtered out objects to stdout.\n\n* 2/9 t/helper: add 'find-pack' test-tool\n\n For testing `git repack --filter=...` that we are going to\n implement, it's useful to have a test helper that can tell which\n packfiles contain a specific object.\n\n* 3/9 repack: refactor finishing pack-objects command\n\n This is a small refactoring creating a new useful function, so that\n `git repack --filter=...` will be able to reuse it.\n\n* 4/9 repack: refactor finding pack prefix\n\n This is another small refactoring creating a small function that\n will be reused in the next patch.\n\n* 5/9 pack-bitmap-write: rebuild using new bitmap when remapping\n\n This patch is new in version 6. It fixes an issue when bitmaps are\n rebuilt that was revealed by this series, and caused a CI test to\n fail.\n\n* 6/9 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. It uses one\n `git pack-objects` process with the `--filter` option. And then\n another `git pack-objects` process with the `--stdin-packs`\n option.\n \n* 7/9 gc: add `gc.repackFilter` config option\n\n This is a gc config option so that `git gc` can also repack using a\n filter and put the filtered out objects into a separate packfile.\n\n* 8/9 repack: implement `--filter-to` for storing filtered out objects\n\n For some use cases, it's interesting to create the packfile that\n contains the filtered out objects into a separate location. This is\n similar to the `--expire-to` option for cruft packfiles.\n\n* 9/9 gc: add `gc.repackFilterTo` config option\n\n This allows specifying the location of the packfile that contains\n the filtered out objects when using `gc.repackFilter`.\n\n# Range-diff since v5\n\n 1: bbcc368876 = 1: da931b5082 pack-objects: allow `--filter` without `--stdout`\n 2: f1b80e5728 = 2: 10504b3699 t/helper: add 'find-pack' test-tool\n 3: ffecc73960 ! 3: ee12eb8ad7 repack: refactor finishing pack-objects command\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *ar\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n struct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n struct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n - struct pack_geometry *geometry = NULL;\n + struct pack_geometry geometry = { 0 };\n - struct strbuf line = STRBUF_INIT;\n struct tempfile *refs_snapshot = NULL;\n int i, ext, ret;\n 4: 6c2f381a88 = 4: d197e0c370 repack: refactor finding pack prefix\n -: ---------- > 5: abeef5fbad pack-bitmap-write: rebuild using new bitmap when remapping\n 5: 134700c2ce ! 6: 31ca2579d3 repack: add `--filter=` option\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n + strvec_pushf(&cmd.args, \"--filter=%s\",\n + expand_list_objects_filter_spec(&po_args.filter_options));\n +\n - if (geometry)\n + if (geometry.split_factor)\n cmd.in = -1;\n else\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n string_list_clear(&existing_nonkept_packs, 0);\n string_list_clear(&existing_kept_packs, 0);\n - clear_pack_geometry(geometry);\n + free_pack_geometry(&geometry);\n + list_objects_filter_release(&po_args.filter_options);\n \n return ret;\n 6: d3365c7b48 = 7: fa70ae85f2 gc: add `gc.repackFilter` config option\n 7: 9a09382cd1 ! 8: e01ea3dd70 repack: implement `--filter-to` for storing filtered out objects\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n + else if (filter_to)\n + die(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n \n - if (geometry)\n + if (geometry.split_factor)\n cmd.in = -1;\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n }\n 8: a52e3a71db = 9: d6ff314189 gc: add `gc.repackFilterTo` config option\n\n\nChristian Couder (9):\n pack-objects: allow `--filter` without `--stdout`\n t/helper: add 'find-pack' test-tool\n repack: refactor finishing pack-objects command\n repack: refactor finding pack prefix\n pack-bitmap-write: rebuild using new bitmap when remapping\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 16 ++\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 23 +++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 167 +++++++++++++++------\n pack-bitmap-write.c | 6 +-\n t/helper/test-find-pack.c | 50 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 ++++++++++\n t/t5317-pack-objects-filter-objects.sh | 8 +\n t/t6500-gc.sh | 24 +++\n t/t7700-repack.sh | 197 +++++++++++++++++++++++++\n 15 files changed, 547 insertions(+), 51 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481682","messageId":"20230911150618.129737-6-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 5/9] pack-bitmap-write: rebuild using new bitmap when remapping","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:14Z","receivedAt":"2023-09-11T21:38:38Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"`git repack` is about to learn a new `--filter=` option and\nwe will want to check that this option is incompatible with\n`--write-bitmap-index`.\n\nUnfortunately it appears that a test like:\n\ntest_expect_success '--filter fails with --write-bitmap-index' '\n test_must_fail \\\n env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n'\n\nsometimes fail because when rebuilding bitmaps, it appears that we are\nreusing existing bitmap information. So instead of detecting that some\nobjects are missing and erroring out as it should, the\n`git repack --write-bitmap-index --filter=...` command succeeds.\n\nLet's fix that by making sure we rebuild bitmaps using new bitmaps\ninstead of existing ones.\n\nHelped-by: Taylor Blau \nSigned-off-by: Christian Couder \n---\n pack-bitmap-write.c | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f6757c3cbf..f4ecdf8b0e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -413,15 +413,19 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \n \t\tif (old_bitmap && mapping) {\n \t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct bitmap *remapped = bitmap_new();\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n \t\t\t * parents or the tree for this commit.\n \t\t\t */\n-\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap)) {\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n+\t\t\t\tbitmap_or(ent->bitmap, remapped);\n+\t\t\t\tbitmap_free(remapped);\n \t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tbitmap_free(remapped);\n \t\t}\n \n \t\t/*\n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481684","messageId":"20230911150618.129737-3-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 2/9] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:11Z","receivedAt":"2023-09-11T21:38:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In a following commit, we will make it possible to separate objects in\ndifferent packfiles depending on a filter.\n\nTo make sure that the right objects are in the right packs, let's add a\nnew test-tool that can display which packfile(s) a given object is in.\n\nLet's also make it possible to check if a given object is in the\nexpected number of packfiles with a `--check-count ` option.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 50 ++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 +++++++++++++++++++++++++++++++++++++++\n 5 files changed, 135 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\ndiff --git a/Makefile b/Makefile\nindex 5776309365..742b76998e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..e8bd793e58\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,50 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"parse-options.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ *\n+ * If '--check-count ' is passed, then error out if the number of\n+ * packfiles containing the object is not .\n+ */\n+\n+static const char *find_pack_usage[] = {\n+\t\"test-tool find-pack [--check-count ] \",\n+\tNULL\n+};\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\tint count = -1, actual_count = 0;\n+\tconst char *prefix = setup_git_directory();\n+\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER('c', \"check-count\", &count, \"expected number of packs\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, find_pack_usage, 0);\n+\tif (argc != 1)\n+\t\tusage(find_pack_usage[0]);\n+\n+\tif (repo_get_oid(the_repository, argv[0], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[0]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tif (find_pack_entry_one(oid.hash, p)) {\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t\t\tactual_count++;\n+\t\t}\n+\n+\tif (count > -1 && count != actual_count)\n+\t\tdie(\"bad packfile count %d instead of %d\", actual_count, count);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex abe8a785eb..41da40c296 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ea2672436c..411dbf2db4 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\ndiff --git a/t/t0080-find-pack.sh b/t/t0080-find-pack.sh\nnew file mode 100755\nindex 0000000000..67b11216a3\n--- /dev/null\n+++ b/t/t0080-find-pack.sh\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+test_description='test `test-tool find-pack`'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit one &&\n+\ttest_commit two &&\n+\ttest_commit three &&\n+\ttest_commit four &&\n+\ttest_commit five\n+'\n+\n+test_expect_success 'repack everything into a single packfile' '\n+\tgit repack -a -d --no-write-bitmap-index &&\n+\n+\thead_commit_pack=$(test-tool find-pack HEAD) &&\n+\thead_tree_pack=$(test-tool find-pack HEAD^{tree}) &&\n+\tone_pack=$(test-tool find-pack HEAD:one.t) &&\n+\tthree_pack=$(test-tool find-pack HEAD:three.t) &&\n+\told_commit_pack=$(test-tool find-pack HEAD~4) &&\n+\n+\ttest-tool find-pack --check-count 1 HEAD &&\n+\ttest-tool find-pack --check-count=1 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count=0 HEAD:one.t &&\n+\t! test-tool find-pack -c 2 HEAD:one.t &&\n+\ttest-tool find-pack -c 1 HEAD:three.t &&\n+\n+\t# Packfile exists at the right path\n+\tcase \"$head_commit_pack\" in\n+\t\t\".git/objects/pack/pack-\"*\".pack\") true ;;\n+\t\t*) false ;;\n+\tesac &&\n+\ttest -f \"$head_commit_pack\" &&\n+\n+\t# Everything is in the same pack\n+\ttest \"$head_commit_pack\" = \"$head_tree_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$one_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$three_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$old_commit_pack\"\n+'\n+\n+test_expect_success 'add more packfiles' '\n+\tgit rev-parse HEAD^{tree} HEAD:two.t HEAD:four.t >objects &&\n+\tgit pack-objects .git/objects/pack/mypackname1 >packhash1 objects &&\n+\tgit pack-objects .git/objects/pack/mypackname2 >packhash2 head_tree_packs &&\n+\tgrep \"$head_commit_pack\" head_tree_packs &&\n+\tgrep mypackname1 head_tree_packs &&\n+\t! grep mypackname2 head_tree_packs &&\n+\ttest-tool find-pack --check-count 2 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count 1 HEAD^{tree} &&\n+\n+\t# HEAD:five.t is also in 2 packfiles\n+\ttest-tool find-pack HEAD:five.t >five_packs &&\n+\tgrep \"$head_commit_pack\" five_packs &&\n+\t! grep mypackname1 five_packs &&\n+\tgrep mypackname2 five_packs &&\n+\ttest-tool find-pack -c 2 HEAD:five.t &&\n+\t! test-tool find-pack --check-count=0 HEAD:five.t\n+'\n+\n+test_expect_success 'add more commits (as loose objects)' '\n+\ttest_commit six &&\n+\ttest_commit seven &&\n+\n+\ttest -z \"$(test-tool find-pack HEAD)\" &&\n+\ttest -z \"$(test-tool find-pack HEAD:six.t)\" &&\n+\ttest-tool find-pack --check-count 0 HEAD &&\n+\ttest-tool find-pack -c 0 HEAD:six.t &&\n+\t! test-tool find-pack -c 1 HEAD:seven.t\n+'\n+\n+test_done\n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481687","messageId":"20230911150618.129737-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:15Z","receivedAt":"2023-09-11T21:38:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new option puts the objects specified by `` into a\nseparate packfile.\n\nThis could be useful if, for example, some blobs take up a lot of\nprecious space on fast storage while they are rarely accessed. It could\nmake sense to move them into a separate cheaper, though slower, storage.\n\nIt's possible to find which new packfile contains the filtered out\nobjects using one of the following:\n\n - `git verify-pack -v ...`,\n - `test-tool find-pack ...`, which a previous commit added,\n - `--filter-to=`, which a following commit will add to specify\n where the pack containing the filtered out objects will be.\n\nThis feature is implemented by running `git pack-objects` twice in a\nrow. The first command is run with `--filter=`, using the\nspecified filter. It packs objects while omitting the objects specified\nby the filter. Then another `git pack-objects` command is launched using\n`--stdin-packs`. We pass it all the previously existing packs into its\nstdin, so that it will pack all the objects in the previously existing\npacks. But we also pass into its stdin, the pack created by the previous\n`git pack-objects --filter=` command as well as the kept\npacks, all prefixed with '^', so that the objects in these packs will be\nomitted from the resulting pack. The result is that only the objects\nfiltered out by the first `git pack-objects` command are in the pack\nresulting from the second `git pack-objects` command.\n\nAs the interactions with kept packs are a bit tricky, a few related\ntests are added.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 12 ++++\n builtin/repack.c | 73 +++++++++++++++++++\n t/t7700-repack.sh | 135 +++++++++++++++++++++++++++++++++++\n 3 files changed, 220 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..6d5bec7716 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,18 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. Note\n+\tthat objects used in the working directory are not filtered\n+\tout. So for the split to fully work, it's best to perform it\n+\tin a bare repo and to use the `-a` and `-d` options along with\n+\tthis option. Also `--no-write-bitmap-index` (or the\n+\t`repack.writebitmaps` config option set to `false`) should be\n+\tused otherwise writing bitmap index will fail, as it supposes\n+\ta single packfile containing all the objects. See\n+\tlinkgit:git-rev-list[1] for valid `` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 8de3009b9f..ac70698a41 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -21,6 +21,7 @@\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n #include \"refs.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define ALL_INTO_ONE 1\n #define LOOSEN_UNREACHABLE 2\n@@ -57,6 +58,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tstruct list_objects_filter_options filter_options;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -725,6 +727,57 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n+static int write_filtered_pack(const struct pack_objects_args *args,\n+\t\t\t const char *destination,\n+\t\t\t const char *pack_prefix,\n+\t\t\t struct string_list *keep_pack_list,\n+\t\t\t struct string_list *names,\n+\t\t\t struct string_list *existing_packs,\n+\t\t\t struct string_list *existing_kept_packs)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret, i;\n+\tconst char *caret;\n+\tconst char *scratch;\n+\tint local = skip_prefix(destination, packdir, &scratch);\n+\n+\tprepare_pack_objects(&cmd, args, destination);\n+\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\n+\tif (!pack_kept_objects)\n+\t\tstrvec_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list->nr; i++)\n+\t\tstrvec_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t keep_pack_list->items[i].string);\n+\n+\tcmd.in = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Here 'names' contains only the pack(s) that were just\n+\t * written, which is exactly the packs we want to keep. Also\n+\t * 'existing_kept_packs' already contains the packs in\n+\t * 'keep_pack_list'.\n+\t */\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, names)\n+\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n+\tfor_each_string_list_item(item, existing_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tcaret = pack_kept_objects ? \"\" : \"^\";\n+\tfor_each_string_list_item(item, existing_kept_packs)\n+\t\tfprintf(in, \"%s%s.pack\\n\", caret, item->string);\n+\tfclose(in);\n+\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -855,6 +908,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&po_args.filter_options),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -868,6 +922,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tOPT_END()\n \t};\n \n+\tlist_objects_filter_init(&po_args.filter_options);\n+\n \tgit_config(repack_config, &cruft_po_args);\n \n \targc = parse_options(argc, argv, prefix, builtin_repack_options,\n@@ -1008,6 +1064,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (po_args.filter_options.choice)\n+\t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\n \tif (geometry.split_factor)\n \t\tcmd.in = -1;\n \telse\n@@ -1096,6 +1156,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter_options.choice) {\n+\t\tret = write_filtered_pack(&po_args,\n+\t\t\t\t\t packtmp,\n+\t\t\t\t\t find_pack_prefix(packdir, packtmp),\n+\t\t\t\t\t &keep_pack_list,\n+\t\t\t\t\t &names,\n+\t\t\t\t\t &existing_nonkept_packs,\n+\t\t\t\t\t &existing_kept_packs);\n+\t\tif (ret)\n+\t\t\tgoto cleanup;\n+\t}\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\n@@ -1230,6 +1302,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&existing_nonkept_packs, 0);\n \tstring_list_clear(&existing_kept_packs, 0);\n \tfree_pack_geometry(&geometry);\n+\tlist_objects_filter_release(&po_args.filter_options);\n \n \treturn ret;\n }\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 27b66807cd..39e89445fd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -327,6 +327,141 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack -c 1 HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack -c 1 HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+\ttest_must_fail \\\n+\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n+'\n+\n+test_expect_success 'repacking with two filters works' '\n+\tgit init two-filters &&\n+\t(\n+\t\tcd two-filters &&\n+\t\tmkdir subdir &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit subdir_bar subdir/bar &&\n+\t\ttest_commit subdir_baz subdir/baz\n+\t) &&\n+\tgit clone --no-local --bare two-filters two-filters.git &&\n+\t(\n+\t\tcd two-filters.git &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/*.pack &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d \\\n+\t\t\t--filter=blob:none --filter=tree:1 &&\n+\t\ttest_stdout_line_count = 2 ls objects/pack/*.pack &&\n+\t\tcommit_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\t\tblob_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\troot_tree_pack=$(test-tool find-pack -c 1 HEAD^{tree}) &&\n+\t\tsubdir_tree_hash=$(git ls-tree --object-only HEAD -- subdir) &&\n+\t\tsubdir_tree_pack=$(test-tool find-pack -c 1 \"$subdir_tree_hash\") &&\n+\n+\t\t# Root tree and subdir tree are not in the same packfiles\n+\t\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\t\ttest \"$commit_pack\" = \"$root_tree_pack\" &&\n+\t\ttest \"$blob_pack\" = \"$subdir_tree_pack\"\n+\t)\n+'\n+\n+prepare_for_keep_packs () {\n+\tgit init keep-packs &&\n+\t(\n+\t\tcd keep-packs &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit bar\n+\t) &&\n+\tgit clone --no-local --bare keep-packs keep-packs.git &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\t# Create two packs\n+\t\t# The first pack will contain all of the objects except one blob\n+\t\tgit rev-list --objects --all >objs &&\n+\t\tgrep -v \"bar.t\" objs | git pack-objects pack &&\n+\t\t# The second pack will contain the excluded object and be kept\n+\t\tpackid=$(grep \"bar.t\" objs | git pack-objects pack) &&\n+\t\t>pack-$packid.keep &&\n+\n+\t\t# Replace the existing pack with the 2 new ones\n+\t\trm -f objects/pack/pack* &&\n+\t\tmv pack-* objects/pack/\n+\t)\n+}\n+\n+test_expect_success '--filter works with .keep packs' '\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack_1=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\t# Object bar is still only in the old .keep pack\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$bar_pack_1\" = \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack_1\" &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\" &&\n+\t\ttest \"$bar_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n+test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n+\trm -rf keep-packs keep-packs.git &&\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--pack-kept-objects &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\ttest-tool find-pack -c 2 HEAD:bar.t >bar_pack_1 &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\t# Object bar is in both the old .keep pack and the new\n+\t\t# pack that contained the filtered out objects\n+\t\tgrep \"$bar_pack\" bar_pack_1 &&\n+\t\tgrep \"$foo_pack_1\" bar_pack_1 &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481695","messageId":"20230911150618.129737-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 1/9] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:10Z","receivedAt":"2023-09-11T21:39:16Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n t/t5317-pack-objects-filter-objects.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex dea7eacb0f..e32404c6aa 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -296,8 +296,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 72241bdca4..e3e1d11640 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4399,12 +4399,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..2ff3eef9a3 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n \t! grep blob verify_result\n '\n \n+test_expect_success 'verify blob:none packfile without --stdout' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n+\t! grep blob verify_result\n+'\n+\n test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n \tgit -C r1 verify-pack -v ../all.pack >verify_result &&\n \tgrep -E \"commit|tree\" verify_result |\n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481698","messageId":"20230911150618.129737-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:17Z","receivedAt":"2023-09-11T21:39:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage.\n\nEven in a different directory, this pack can be accessible if, for\nexample, the Git alternates mechanism is used to point to it. In fact\nnot using the Git alternates mechanism can corrupt a repo as the\ngenerated pack containing the filtered objects might not be accessible\nfrom the repo any more. So setting up the Git alternates mechanism\nshould be done before using this feature if the user wants the repo to\nbe fully usable while this feature is used.\n\nIn some cases, like when a repo has just been cloned or when there is no\nother activity in the repo, it's Ok to setup the Git alternates\nmechanism afterwards though. It's also Ok to just inspect the generated\npackfile containing the filtered objects and then just move it into the\n'.git/objects/pack/' directory manually. That's why it's not necessary\nfor this command to check that the Git alternates mechanism has been\nalready setup.\n\nWhile at it, as an example to show that `--filter` and `--filter-to`\nwork well with other options, let's also add a test to check that these\noptions work well with `--max-pack-size`.\n\nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 11 +++++++\n builtin/repack.c | 10 +++++-\n t/t7700-repack.sh | 62 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 6d5bec7716..8545a32667 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -155,6 +155,17 @@ depth is 4095.\n \ta single packfile containing all the objects. See\n \tlinkgit:git-rev-list[1] for valid `` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. Only useful with `--filter`. This can be\n+\tused for putting the pack on a separate object directory that\n+\tis accessed through the Git alternates mechanism. **WARNING:**\n+\tIf the packfile containing the filtered out objects is not\n+\taccessible, the repo can become corrupt as it might not be\n+\tpossible to access the objects in that packfile. See the\n+\t`objects` and `objects/info/alternates` sections of\n+\tlinkgit:gitrepository-layout[5].\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex ac70698a41..e0e1b52cf0 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -867,6 +867,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -919,6 +920,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1067,6 +1070,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (po_args.filter_options.choice)\n \t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n \t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\telse if (filter_to)\n+\t\tdie(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n \n \tif (geometry.split_factor)\n \t\tcmd.in = -1;\n@@ -1157,8 +1162,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter_options.choice) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\n \t\tret = write_filtered_pack(&po_args,\n-\t\t\t\t\t packtmp,\n+\t\t\t\t\t filter_to,\n \t\t\t\t\t find_pack_prefix(packdir, packtmp),\n \t\t\t\t\t &keep_pack_list,\n \t\t\t\t\t &names,\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 39e89445fd..48e92aa6f7 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -462,6 +462,68 @@ test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n \t)\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 0 HEAD:file1) &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack -c 1 $blob_hash) &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n+test_expect_success '--filter works with --max-pack-size' '\n+\trm -rf filtered.git &&\n+\tgit init --bare filtered.git &&\n+\tgit init max-pack-size &&\n+\t(\n+\t\tcd max-pack-size &&\n+\t\ttest_commit base &&\n+\t\t# two blobs which exceed the maximum pack size\n+\t\ttest-tool genrandom foo 1048576 >foo &&\n+\t\tgit hash-object -w foo &&\n+\t\ttest-tool genrandom bar 1048576 >bar &&\n+\t\tgit hash-object -w bar &&\n+\t\tgit add foo bar &&\n+\t\tgit commit -m \"adding foo and bar\"\n+\t) &&\n+\tgit clone --no-local --bare max-pack-size max-pack-size.git &&\n+\t(\n+\t\tcd max-pack-size.git &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--max-pack-size=1M \\\n+\t\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\t\techo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n+\n+\t\t# Check that the 3 blobs are in different packfiles in filtered.git\n+\t\ttest_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar) &&\n+\t\tbase_pack=$(test-tool find-pack -c 1 HEAD:base.t) &&\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" != \"$base_pack\" &&\n+\t\ttest \"$bar_pack\" != \"$base_pack\" &&\n+\t\tfor pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n+\t\tdo\n+\t\t\tcase \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n+\t\tdone\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481711","messageId":"20230911150618.129737-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 7/9] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:16Z","receivedAt":"2023-09-11T23:02:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex ca47eb2008..2153bde7ac 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the\n operation (either generating a cruft pack or unpacking unreachable\n objects) will be halted.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 369bd43fb2..607c0ac23e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -61,6 +61,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -170,6 +171,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -355,6 +358,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..232e403b66 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,19 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481712","messageId":"20230911150618.129737-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 3/9] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:12Z","receivedAt":"2023-09-11T23:02:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -704,9 +734,8 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct string_list *existing_kept_packs)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n \tconst char *scratch;\n \tint local = skip_prefix(destination, packdir, &scratch);\n@@ -749,27 +778,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -780,10 +789,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry geometry = { 0 };\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1013,18 +1020,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"481717","messageId":"20230911150618.129737-10-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v6 9/9] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-11T15:06:18Z","receivedAt":"2023-09-11T23:02:40Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 11 +++++++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 2153bde7ac..466466d6cc 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -150,6 +150,17 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. **WARNING:** The\n+\tspecified location should be accessible, using for example the\n+\tGit alternates mechanism, otherwise the repo could be\n+\tconsidered corrupt by Git as it migh not be able to access the\n+\tobjects in that packfile. See the `--filter-to=` option\n+\tof linkgit:git-repack[1] and the `objects/info/alternates`\n+\tsection of linkgit:gitrepository-layout[5].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 607c0ac23e..8aad103b45 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -172,6 +173,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -361,6 +363,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 232e403b66..e412cf8daf 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -215,6 +214,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.167.gd6ff314189\n\n"},{"id":"482271","messageId":"20230925152517.803579-1-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230911150618.129737-1-christian.couder@gmail.com","subject":"[PATCH v7 0/9] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:08Z","receivedAt":"2023-09-25T15:25:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"# Intro\n\nLast year, John Cai sent 2 versions of a patch series to implement\n`git repack --filter=` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\n# Use cases for the new feature\n\nThis could be useful for example for the following purposes:\n\n 1) As a way for servers to save storage costs by for example moving\n large blobs, or all the blobs, or all the blobs in inactive\n repos, to separate storage (while still making them accessible\n using for example the alternates mechanism).\n\n 2) As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n 3) As a way for clients to reclaim some space when they cloned with\n a filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a\n separate directory or storage, then check that everything works\n well, and then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\nAlso since version 2 of this new series, commit messages, don't\nmention uses cases like 2) or 3) above, as people have different\nopinions on how it should be done. How it should be done could depend\na lot on the way promisor remotes are used, the software and hardware\nsetups used, etc, so it seems more difficult to \"sell\" this series by\ntalking about such use cases. As use case 1) seems simpler and more\nappealing, it makes more sense to only talk about it in the commit\nmessages.\n\n# Changes since version 6\n\nThanks to Junio who reviewed or commented on versions 1, 2, 3, 4 and\n5, and to Taylor who reviewed or commented on version 1, 3, 4, 5 and\n6! Thanks also to Robert Coup who participated in the discussions\nrelated to version 2 and Peff who participated in the discussions\nrelated to version 4. There are only the following changes since\nversion 6:\n\n- This series has been rebased on top of bcb6cae296 (The twelfth\n batch, 2023-09-22) to fix conflicts with a `builtin/repack.c`\n refactoring patch series called tb/repack-existing-packs-cleanup by\n Taylor Blau that recently graduated to 'master':\n\n\thttps://lore.kernel.org/git/cover.1694632644.git.me@ttaylorr.com/\n\thttps://lore.kernel.org/git/xmqqil81wqkx.fsf@gitster.g/\n\n- Patch 6/9 (repack: add `--filter=` option) has been\n reworked to apply on top of the above mentioned patch series.\n Taylor even posted the fixup patch to apply to this series so that\n it works well on top of his series:\n \n https://lore.kernel.org/git/ZQNKkn0YYLUyN5Ih@nand.local/\n\nI checked that CI tests passes in:\n\nhttps://github.com/chriscool/git/actions/runs/6300816764\n\nAll jobs seem to have succeeded.\n\n# Commit overview\n\n(No changes in any of the patches compared to version 5, except on\npatch 6/9.)\n\n* 1/9 pack-objects: allow `--filter` without `--stdout`\n\n To be able to later repack with a filter we need `git pack-objects`\n to write packfiles when it's filtering instead of just writing the\n pack without the filtered out objects to stdout.\n\n* 2/9 t/helper: add 'find-pack' test-tool\n\n For testing `git repack --filter=...` that we are going to\n implement, it's useful to have a test helper that can tell which\n packfiles contain a specific object.\n\n* 3/9 repack: refactor finishing pack-objects command\n\n This is a small refactoring creating a new useful function, so that\n `git repack --filter=...` will be able to reuse it.\n\n* 4/9 repack: refactor finding pack prefix\n\n This is another small refactoring creating a small function that\n will be reused in the next patch.\n\n* 5/9 pack-bitmap-write: rebuild using new bitmap when remapping\n\n This patch is new in version 6. It fixes an issue when bitmaps are\n rebuilt that was revealed by this series, and caused a CI test to\n fail.\n\n* 6/9 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. It uses one\n `git pack-objects` process with the `--filter` option. And then\n another `git pack-objects` process with the `--stdin-packs`\n option. This is the only patch changed in version 7.\n \n* 7/9 gc: add `gc.repackFilter` config option\n\n This is a gc config option so that `git gc` can also repack using a\n filter and put the filtered out objects into a separate packfile.\n\n* 8/9 repack: implement `--filter-to` for storing filtered out objects\n\n For some use cases, it's interesting to create the packfile that\n contains the filtered out objects into a separate location. This is\n similar to the `--expire-to` option for cruft packfiles.\n\n* 9/9 gc: add `gc.repackFilterTo` config option\n\n This allows specifying the location of the packfile that contains\n the filtered out objects when using `gc.repackFilter`.\n\n# Range-diff since v6\n\n 1: da931b5082 = 1: eec0c09731 pack-objects: allow `--filter` without `--stdout`\n 2: 10504b3699 = 2: 19c8b8a4b9 t/helper: add 'find-pack' test-tool\n 3: ee12eb8ad7 ! 3: aaaf40bd5d repack: refactor finishing pack-objects command\n @@ builtin/repack.c: static void remove_redundant_bitmaps(struct string_list *inclu\n const char *destination,\n const char *pack_prefix,\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *args,\n - struct string_list *existing_kept_packs)\n + struct existing_packs *existing)\n {\n struct child_process cmd = CHILD_PROCESS_INIT;\n - struct strbuf line = STRBUF_INIT;\n @@ builtin/repack.c: static int write_cruft_pack(const struct pack_objects_args *ar\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n - struct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n - struct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n + struct string_list names = STRING_LIST_INIT_DUP;\n + struct existing_packs existing = EXISTING_PACKS_INIT;\n struct pack_geometry geometry = { 0 };\n - struct strbuf line = STRBUF_INIT;\n struct tempfile *refs_snapshot = NULL;\n 4: d197e0c370 = 4: 1eb6bc3f7e repack: refactor finding pack prefix\n 5: abeef5fbad = 5: b9159e1803 pack-bitmap-write: rebuild using new bitmap when remapping\n 6: 31ca2579d3 ! 6: f2f5bb54d3 repack: add `--filter=` option\n @@ Commit message\n As the interactions with kept packs are a bit tricky, a few related\n tests are added.\n \n + Helped-by: Taylor Blau \n Signed-off-by: John Cai \n Signed-off-by: Christian Couder \n \n @@ builtin/repack.c: static int finish_pack_objects_cmd(struct child_process *cmd,\n +static int write_filtered_pack(const struct pack_objects_args *args,\n + const char *destination,\n + const char *pack_prefix,\n -+ struct string_list *keep_pack_list,\n -+ struct string_list *names,\n -+ struct string_list *existing_packs,\n -+ struct string_list *existing_kept_packs)\n ++ struct existing_packs *existing,\n ++ struct string_list *names)\n +{\n + struct child_process cmd = CHILD_PROCESS_INIT;\n + struct string_list_item *item;\n + FILE *in;\n -+ int ret, i;\n ++ int ret;\n + const char *caret;\n + const char *scratch;\n + int local = skip_prefix(destination, packdir, &scratch);\n @@ builtin/repack.c: static int finish_pack_objects_cmd(struct child_process *cmd,\n +\n + if (!pack_kept_objects)\n + strvec_push(&cmd.args, \"--honor-pack-keep\");\n -+ for (i = 0; i < keep_pack_list->nr; i++)\n -+ strvec_pushf(&cmd.args, \"--keep-pack=%s\",\n -+ keep_pack_list->items[i].string);\n ++ for_each_string_list_item(item, &existing->kept_packs)\n ++ strvec_pushf(&cmd.args, \"--keep-pack=%s\", item->string);\n +\n + cmd.in = -1;\n +\n @@ builtin/repack.c: static int finish_pack_objects_cmd(struct child_process *cmd,\n + in = xfdopen(cmd.in, \"w\");\n + for_each_string_list_item(item, names)\n + fprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n -+ for_each_string_list_item(item, existing_packs)\n ++ for_each_string_list_item(item, &existing->non_kept_packs)\n ++ fprintf(in, \"%s.pack\\n\", item->string);\n ++ for_each_string_list_item(item, &existing->cruft_packs)\n + fprintf(in, \"%s.pack\\n\", item->string);\n + caret = pack_kept_objects ? \"\" : \"^\";\n -+ for_each_string_list_item(item, existing_kept_packs)\n ++ for_each_string_list_item(item, &existing->kept_packs)\n + fprintf(in, \"%s%s.pack\\n\", caret, item->string);\n + fclose(in);\n +\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n + ret = write_filtered_pack(&po_args,\n + packtmp,\n + find_pack_prefix(packdir, packtmp),\n -+ &keep_pack_list,\n -+ &names,\n -+ &existing_nonkept_packs,\n -+ &existing_kept_packs);\n ++ &existing,\n ++ &names);\n + if (ret)\n + goto cleanup;\n + }\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n \n close_object_store(the_repository->objects);\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n - string_list_clear(&existing_nonkept_packs, 0);\n - string_list_clear(&existing_kept_packs, 0);\n + string_list_clear(&names, 1);\n + existing_packs_release(&existing);\n free_pack_geometry(&geometry);\n + list_objects_filter_release(&po_args.filter_options);\n \n 7: fa70ae85f2 = 7: 7ea0307628 gc: add `gc.repackFilter` config option\n 8: e01ea3dd70 ! 8: 698647815b repack: implement `--filter-to` for storing filtered out objects\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n - packtmp,\n + filter_to,\n find_pack_prefix(packdir, packtmp),\n - &keep_pack_list,\n - &names,\n + &existing,\n + &names);\n \n ## t/t7700-repack.sh ##\n @@ t/t7700-repack.sh: test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n 9: d6ff314189 = 9: 57b2ba444c gc: add `gc.repackFilterTo` config option\n\n\nChristian Couder (9):\n pack-objects: allow `--filter` without `--stdout`\n t/helper: add 'find-pack' test-tool\n repack: refactor finishing pack-objects command\n repack: refactor finding pack prefix\n pack-bitmap-write: rebuild using new bitmap when remapping\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 16 ++\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 23 +++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 164 ++++++++++++++------\n pack-bitmap-write.c | 6 +-\n t/helper/test-find-pack.c | 50 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 ++++++++++\n t/t5317-pack-objects-filter-objects.sh | 8 +\n t/t6500-gc.sh | 24 +++\n t/t7700-repack.sh | 197 +++++++++++++++++++++++++\n 15 files changed, 544 insertions(+), 51 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482272","messageId":"20230925152517.803579-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 1/9] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:09Z","receivedAt":"2023-09-25T15:25:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n t/t5317-pack-objects-filter-objects.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex dea7eacb0f..e32404c6aa 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -296,8 +296,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6eb9756836..89a8b5a976 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4402,12 +4402,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..2ff3eef9a3 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n \t! grep blob verify_result\n '\n \n+test_expect_success 'verify blob:none packfile without --stdout' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n+\t! grep blob verify_result\n+'\n+\n test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n \tgit -C r1 verify-pack -v ../all.pack >verify_result &&\n \tgrep -E \"commit|tree\" verify_result |\n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482273","messageId":"20230925152517.803579-3-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 2/9] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:10Z","receivedAt":"2023-09-25T15:25:55Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In a following commit, we will make it possible to separate objects in\ndifferent packfiles depending on a filter.\n\nTo make sure that the right objects are in the right packs, let's add a\nnew test-tool that can display which packfile(s) a given object is in.\n\nLet's also make it possible to check if a given object is in the\nexpected number of packfiles with a `--check-count ` option.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 50 ++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0080-find-pack.sh | 82 +++++++++++++++++++++++++++++++++++++++\n 5 files changed, 135 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0080-find-pack.sh\n\ndiff --git a/Makefile b/Makefile\nindex 003e63b792..f267034d23 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..e8bd793e58\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,50 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"parse-options.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ *\n+ * If '--check-count ' is passed, then error out if the number of\n+ * packfiles containing the object is not .\n+ */\n+\n+static const char *find_pack_usage[] = {\n+\t\"test-tool find-pack [--check-count ] \",\n+\tNULL\n+};\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\tint count = -1, actual_count = 0;\n+\tconst char *prefix = setup_git_directory();\n+\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER('c', \"check-count\", &count, \"expected number of packs\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, find_pack_usage, 0);\n+\tif (argc != 1)\n+\t\tusage(find_pack_usage[0]);\n+\n+\tif (repo_get_oid(the_repository, argv[0], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[0]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tif (find_pack_entry_one(oid.hash, p)) {\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t\t\tactual_count++;\n+\t\t}\n+\n+\tif (count > -1 && count != actual_count)\n+\t\tdie(\"bad packfile count %d instead of %d\", actual_count, count);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 621ac3dd10..9010ac6de7 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex a641c3a81d..f134f96b97 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\ndiff --git a/t/t0080-find-pack.sh b/t/t0080-find-pack.sh\nnew file mode 100755\nindex 0000000000..67b11216a3\n--- /dev/null\n+++ b/t/t0080-find-pack.sh\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+test_description='test `test-tool find-pack`'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit one &&\n+\ttest_commit two &&\n+\ttest_commit three &&\n+\ttest_commit four &&\n+\ttest_commit five\n+'\n+\n+test_expect_success 'repack everything into a single packfile' '\n+\tgit repack -a -d --no-write-bitmap-index &&\n+\n+\thead_commit_pack=$(test-tool find-pack HEAD) &&\n+\thead_tree_pack=$(test-tool find-pack HEAD^{tree}) &&\n+\tone_pack=$(test-tool find-pack HEAD:one.t) &&\n+\tthree_pack=$(test-tool find-pack HEAD:three.t) &&\n+\told_commit_pack=$(test-tool find-pack HEAD~4) &&\n+\n+\ttest-tool find-pack --check-count 1 HEAD &&\n+\ttest-tool find-pack --check-count=1 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count=0 HEAD:one.t &&\n+\t! test-tool find-pack -c 2 HEAD:one.t &&\n+\ttest-tool find-pack -c 1 HEAD:three.t &&\n+\n+\t# Packfile exists at the right path\n+\tcase \"$head_commit_pack\" in\n+\t\t\".git/objects/pack/pack-\"*\".pack\") true ;;\n+\t\t*) false ;;\n+\tesac &&\n+\ttest -f \"$head_commit_pack\" &&\n+\n+\t# Everything is in the same pack\n+\ttest \"$head_commit_pack\" = \"$head_tree_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$one_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$three_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$old_commit_pack\"\n+'\n+\n+test_expect_success 'add more packfiles' '\n+\tgit rev-parse HEAD^{tree} HEAD:two.t HEAD:four.t >objects &&\n+\tgit pack-objects .git/objects/pack/mypackname1 >packhash1 objects &&\n+\tgit pack-objects .git/objects/pack/mypackname2 >packhash2 head_tree_packs &&\n+\tgrep \"$head_commit_pack\" head_tree_packs &&\n+\tgrep mypackname1 head_tree_packs &&\n+\t! grep mypackname2 head_tree_packs &&\n+\ttest-tool find-pack --check-count 2 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count 1 HEAD^{tree} &&\n+\n+\t# HEAD:five.t is also in 2 packfiles\n+\ttest-tool find-pack HEAD:five.t >five_packs &&\n+\tgrep \"$head_commit_pack\" five_packs &&\n+\t! grep mypackname1 five_packs &&\n+\tgrep mypackname2 five_packs &&\n+\ttest-tool find-pack -c 2 HEAD:five.t &&\n+\t! test-tool find-pack --check-count=0 HEAD:five.t\n+'\n+\n+test_expect_success 'add more commits (as loose objects)' '\n+\ttest_commit six &&\n+\ttest_commit seven &&\n+\n+\ttest -z \"$(test-tool find-pack HEAD)\" &&\n+\ttest -z \"$(test-tool find-pack HEAD:six.t)\" &&\n+\ttest-tool find-pack --check-count 0 HEAD &&\n+\ttest-tool find-pack -c 0 HEAD:six.t &&\n+\t! test-tool find-pack -c 1 HEAD:seven.t\n+'\n+\n+test_done\n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482274","messageId":"20230925152517.803579-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 3/9] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:11Z","receivedAt":"2023-09-25T15:25:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -814,9 +844,8 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct existing_packs *existing)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n \tconst char *scratch;\n \tint local = skip_prefix(destination, packdir, &scratch);\n@@ -861,27 +890,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -891,10 +900,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list names = STRING_LIST_INIT_DUP;\n \tstruct existing_packs existing = EXISTING_PACKS_INIT;\n \tstruct pack_geometry geometry = { 0 };\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1124,18 +1131,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482275","messageId":"20230925152517.803579-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 4/9] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:12Z","receivedAt":"2023-09-25T15:25:59Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new find_pack_prefix() to refactor code that handles finding\nthe pack prefix from the packtmp and packdir global variables, as we are\ngoing to need this feature again in following commit.\n\nSigned-off-by: Christian Couder ` option and\nwe will want to check that this option is incompatible with\n`--write-bitmap-index`.\n\nUnfortunately it appears that a test like:\n\ntest_expect_success '--filter fails with --write-bitmap-index' '\n test_must_fail \\\n env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n'\n\nsometimes fail because when rebuilding bitmaps, it appears that we are\nreusing existing bitmap information. So instead of detecting that some\nobjects are missing and erroring out as it should, the\n`git repack --write-bitmap-index --filter=...` command succeeds.\n\nLet's fix that by making sure we rebuild bitmaps using new bitmaps\ninstead of existing ones.\n\nHelped-by: Taylor Blau \nSigned-off-by: Christian Couder \n---\n pack-bitmap-write.c | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f6757c3cbf..f4ecdf8b0e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -413,15 +413,19 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \n \t\tif (old_bitmap && mapping) {\n \t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct bitmap *remapped = bitmap_new();\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n \t\t\t * parents or the tree for this commit.\n \t\t\t */\n-\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap)) {\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n+\t\t\t\tbitmap_or(ent->bitmap, remapped);\n+\t\t\t\tbitmap_free(remapped);\n \t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tbitmap_free(remapped);\n \t\t}\n \n \t\t/*\n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482277","messageId":"20230925152517.803579-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 7/9] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:15Z","receivedAt":"2023-09-25T15:26:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex ca47eb2008..2153bde7ac 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the\n operation (either generating a cruft pack or unpacking unreachable\n objects) will be halted.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 00192ae5d3..98148e98fe 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -61,6 +61,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -170,6 +171,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -355,6 +358,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..232e403b66 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,19 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482278","messageId":"20230925152517.803579-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:14Z","receivedAt":"2023-09-25T15:26:06Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new option puts the objects specified by `` into a\nseparate packfile.\n\nThis could be useful if, for example, some blobs take up a lot of\nprecious space on fast storage while they are rarely accessed. It could\nmake sense to move them into a separate cheaper, though slower, storage.\n\nIt's possible to find which new packfile contains the filtered out\nobjects using one of the following:\n\n - `git verify-pack -v ...`,\n - `test-tool find-pack ...`, which a previous commit added,\n - `--filter-to=`, which a following commit will add to specify\n where the pack containing the filtered out objects will be.\n\nThis feature is implemented by running `git pack-objects` twice in a\nrow. The first command is run with `--filter=`, using the\nspecified filter. It packs objects while omitting the objects specified\nby the filter. Then another `git pack-objects` command is launched using\n`--stdin-packs`. We pass it all the previously existing packs into its\nstdin, so that it will pack all the objects in the previously existing\npacks. But we also pass into its stdin, the pack created by the previous\n`git pack-objects --filter=` command as well as the kept\npacks, all prefixed with '^', so that the objects in these packs will be\nomitted from the resulting pack. The result is that only the objects\nfiltered out by the first `git pack-objects` command are in the pack\nresulting from the second `git pack-objects` command.\n\nAs the interactions with kept packs are a bit tricky, a few related\ntests are added.\n\nHelped-by: Taylor Blau \nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 12 ++++\n builtin/repack.c | 70 ++++++++++++++++++\n t/t7700-repack.sh | 135 +++++++++++++++++++++++++++++++++++\n 3 files changed, 217 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..6d5bec7716 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,18 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. Note\n+\tthat objects used in the working directory are not filtered\n+\tout. So for the split to fully work, it's best to perform it\n+\tin a bare repo and to use the `-a` and `-d` options along with\n+\tthis option. Also `--no-write-bitmap-index` (or the\n+\t`repack.writebitmaps` config option set to `false`) should be\n+\tused otherwise writing bitmap index will fail, as it supposes\n+\ta single packfile containing all the objects. See\n+\tlinkgit:git-rev-list[1] for valid `` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 9ef0044384..c7b564192f 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -21,6 +21,7 @@\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n #include \"refs.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define ALL_INTO_ONE 1\n #define LOOSEN_UNREACHABLE 2\n@@ -56,6 +57,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tstruct list_objects_filter_options filter_options;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -836,6 +838,56 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n+static int write_filtered_pack(const struct pack_objects_args *args,\n+\t\t\t const char *destination,\n+\t\t\t const char *pack_prefix,\n+\t\t\t struct existing_packs *existing,\n+\t\t\t struct string_list *names)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\tconst char *caret;\n+\tconst char *scratch;\n+\tint local = skip_prefix(destination, packdir, &scratch);\n+\n+\tprepare_pack_objects(&cmd, args, destination);\n+\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\n+\tif (!pack_kept_objects)\n+\t\tstrvec_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor_each_string_list_item(item, &existing->kept_packs)\n+\t\tstrvec_pushf(&cmd.args, \"--keep-pack=%s\", item->string);\n+\n+\tcmd.in = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Here 'names' contains only the pack(s) that were just\n+\t * written, which is exactly the packs we want to keep. Also\n+\t * 'existing_kept_packs' already contains the packs in\n+\t * 'keep_pack_list'.\n+\t */\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, names)\n+\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n+\tfor_each_string_list_item(item, &existing->non_kept_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tfor_each_string_list_item(item, &existing->cruft_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tcaret = pack_kept_objects ? \"\" : \"^\";\n+\tfor_each_string_list_item(item, &existing->kept_packs)\n+\t\tfprintf(in, \"%s%s.pack\\n\", caret, item->string);\n+\tfclose(in);\n+\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -966,6 +1018,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&po_args.filter_options),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -979,6 +1032,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tOPT_END()\n \t};\n \n+\tlist_objects_filter_init(&po_args.filter_options);\n+\n \tgit_config(repack_config, &cruft_po_args);\n \n \targc = parse_options(argc, argv, prefix, builtin_repack_options,\n@@ -1119,6 +1174,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (po_args.filter_options.choice)\n+\t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\n \tif (geometry.split_factor)\n \t\tcmd.in = -1;\n \telse\n@@ -1205,6 +1264,16 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter_options.choice) {\n+\t\tret = write_filtered_pack(&po_args,\n+\t\t\t\t\t packtmp,\n+\t\t\t\t\t find_pack_prefix(packdir, packtmp),\n+\t\t\t\t\t &existing,\n+\t\t\t\t\t &names);\n+\t\tif (ret)\n+\t\t\tgoto cleanup;\n+\t}\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\n@@ -1297,6 +1366,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&names, 1);\n \texisting_packs_release(&existing);\n \tfree_pack_geometry(&geometry);\n+\tlist_objects_filter_release(&po_args.filter_options);\n \n \treturn ret;\n }\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 27b66807cd..39e89445fd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -327,6 +327,141 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack -c 1 HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack -c 1 HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+\ttest_must_fail \\\n+\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n+'\n+\n+test_expect_success 'repacking with two filters works' '\n+\tgit init two-filters &&\n+\t(\n+\t\tcd two-filters &&\n+\t\tmkdir subdir &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit subdir_bar subdir/bar &&\n+\t\ttest_commit subdir_baz subdir/baz\n+\t) &&\n+\tgit clone --no-local --bare two-filters two-filters.git &&\n+\t(\n+\t\tcd two-filters.git &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/*.pack &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d \\\n+\t\t\t--filter=blob:none --filter=tree:1 &&\n+\t\ttest_stdout_line_count = 2 ls objects/pack/*.pack &&\n+\t\tcommit_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\t\tblob_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\troot_tree_pack=$(test-tool find-pack -c 1 HEAD^{tree}) &&\n+\t\tsubdir_tree_hash=$(git ls-tree --object-only HEAD -- subdir) &&\n+\t\tsubdir_tree_pack=$(test-tool find-pack -c 1 \"$subdir_tree_hash\") &&\n+\n+\t\t# Root tree and subdir tree are not in the same packfiles\n+\t\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\t\ttest \"$commit_pack\" = \"$root_tree_pack\" &&\n+\t\ttest \"$blob_pack\" = \"$subdir_tree_pack\"\n+\t)\n+'\n+\n+prepare_for_keep_packs () {\n+\tgit init keep-packs &&\n+\t(\n+\t\tcd keep-packs &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit bar\n+\t) &&\n+\tgit clone --no-local --bare keep-packs keep-packs.git &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\t# Create two packs\n+\t\t# The first pack will contain all of the objects except one blob\n+\t\tgit rev-list --objects --all >objs &&\n+\t\tgrep -v \"bar.t\" objs | git pack-objects pack &&\n+\t\t# The second pack will contain the excluded object and be kept\n+\t\tpackid=$(grep \"bar.t\" objs | git pack-objects pack) &&\n+\t\t>pack-$packid.keep &&\n+\n+\t\t# Replace the existing pack with the 2 new ones\n+\t\trm -f objects/pack/pack* &&\n+\t\tmv pack-* objects/pack/\n+\t)\n+}\n+\n+test_expect_success '--filter works with .keep packs' '\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack_1=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\t# Object bar is still only in the old .keep pack\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$bar_pack_1\" = \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack_1\" &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\" &&\n+\t\ttest \"$bar_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n+test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n+\trm -rf keep-packs keep-packs.git &&\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--pack-kept-objects &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\ttest-tool find-pack -c 2 HEAD:bar.t >bar_pack_1 &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\t# Object bar is in both the old .keep pack and the new\n+\t\t# pack that contained the filtered out objects\n+\t\tgrep \"$bar_pack\" bar_pack_1 &&\n+\t\tgrep \"$foo_pack_1\" bar_pack_1 &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482279","messageId":"20230925152517.803579-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:16Z","receivedAt":"2023-09-25T15:26:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage.\n\nEven in a different directory, this pack can be accessible if, for\nexample, the Git alternates mechanism is used to point to it. In fact\nnot using the Git alternates mechanism can corrupt a repo as the\ngenerated pack containing the filtered objects might not be accessible\nfrom the repo any more. So setting up the Git alternates mechanism\nshould be done before using this feature if the user wants the repo to\nbe fully usable while this feature is used.\n\nIn some cases, like when a repo has just been cloned or when there is no\nother activity in the repo, it's Ok to setup the Git alternates\nmechanism afterwards though. It's also Ok to just inspect the generated\npackfile containing the filtered objects and then just move it into the\n'.git/objects/pack/' directory manually. That's why it's not necessary\nfor this command to check that the Git alternates mechanism has been\nalready setup.\n\nWhile at it, as an example to show that `--filter` and `--filter-to`\nwork well with other options, let's also add a test to check that these\noptions work well with `--max-pack-size`.\n\nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 11 +++++++\n builtin/repack.c | 10 +++++-\n t/t7700-repack.sh | 62 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 6d5bec7716..8545a32667 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -155,6 +155,17 @@ depth is 4095.\n \ta single packfile containing all the objects. See\n \tlinkgit:git-rev-list[1] for valid `` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. Only useful with `--filter`. This can be\n+\tused for putting the pack on a separate object directory that\n+\tis accessed through the Git alternates mechanism. **WARNING:**\n+\tIf the packfile containing the filtered out objects is not\n+\taccessible, the repo can become corrupt as it might not be\n+\tpossible to access the objects in that packfile. See the\n+\t`objects` and `objects/info/alternates` sections of\n+\tlinkgit:gitrepository-layout[5].\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c7b564192f..db9277081d 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -977,6 +977,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -1029,6 +1030,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1177,6 +1180,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (po_args.filter_options.choice)\n \t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n \t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\telse if (filter_to)\n+\t\tdie(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n \n \tif (geometry.split_factor)\n \t\tcmd.in = -1;\n@@ -1265,8 +1270,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter_options.choice) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\n \t\tret = write_filtered_pack(&po_args,\n-\t\t\t\t\t packtmp,\n+\t\t\t\t\t filter_to,\n \t\t\t\t\t find_pack_prefix(packdir, packtmp),\n \t\t\t\t\t &existing,\n \t\t\t\t\t &names);\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 39e89445fd..48e92aa6f7 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -462,6 +462,68 @@ test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n \t)\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 0 HEAD:file1) &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack -c 1 $blob_hash) &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n+test_expect_success '--filter works with --max-pack-size' '\n+\trm -rf filtered.git &&\n+\tgit init --bare filtered.git &&\n+\tgit init max-pack-size &&\n+\t(\n+\t\tcd max-pack-size &&\n+\t\ttest_commit base &&\n+\t\t# two blobs which exceed the maximum pack size\n+\t\ttest-tool genrandom foo 1048576 >foo &&\n+\t\tgit hash-object -w foo &&\n+\t\ttest-tool genrandom bar 1048576 >bar &&\n+\t\tgit hash-object -w bar &&\n+\t\tgit add foo bar &&\n+\t\tgit commit -m \"adding foo and bar\"\n+\t) &&\n+\tgit clone --no-local --bare max-pack-size max-pack-size.git &&\n+\t(\n+\t\tcd max-pack-size.git &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--max-pack-size=1M \\\n+\t\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\t\techo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n+\n+\t\t# Check that the 3 blobs are in different packfiles in filtered.git\n+\t\ttest_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar) &&\n+\t\tbase_pack=$(test-tool find-pack -c 1 HEAD:base.t) &&\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" != \"$base_pack\" &&\n+\t\ttest \"$bar_pack\" != \"$base_pack\" &&\n+\t\tfor pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n+\t\tdo\n+\t\t\tcase \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n+\t\tdone\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482280","messageId":"20230925152517.803579-10-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v7 9/9] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-25T15:25:17Z","receivedAt":"2023-09-25T15:26:20Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 11 +++++++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 2153bde7ac..466466d6cc 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -150,6 +150,17 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. **WARNING:** The\n+\tspecified location should be accessible, using for example the\n+\tGit alternates mechanism, otherwise the repo could be\n+\tconsidered corrupt by Git as it migh not be able to access the\n+\tobjects in that packfile. See the `--filter-to=` option\n+\tof linkgit:git-repack[1] and the `objects/info/alternates`\n+\tsection of linkgit:gitrepository-layout[5].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 98148e98fe..68ca8d45bf 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -172,6 +173,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -361,6 +363,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 232e403b66..e412cf8daf 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -215,6 +214,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.279.g57b2ba444c\n\n"},{"id":"482302","messageId":"xmqqy1gurrpj.fsf@gitster.g","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"Re: [PATCH v7 0/9] Repack objects into separate packfiles based on a filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-25T19:14:00Z","receivedAt":"2023-09-25T19:14:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> # Changes since version 6\n>\n> Thanks to Junio who reviewed or commented on versions 1, 2, 3, 4 and\n> 5, and to Taylor who reviewed or commented on version 1, 3, 4, 5 and\n> 6! Thanks also to Robert Coup who participated in the discussions\n> related to version 2 and Peff who participated in the discussions\n> related to version 4. There are only the following changes since\n> version 6:\n>\n> - This series has been rebased on top of bcb6cae296 (The twelfth\n> batch, 2023-09-22) to fix conflicts with a `builtin/repack.c`\n> refactoring patch series called tb/repack-existing-packs-cleanup by\n> Taylor Blau that recently graduated to 'master':\n>\n> \thttps://lore.kernel.org/git/cover.1694632644.git.me@ttaylorr.com/\n> \thttps://lore.kernel.org/git/xmqqil81wqkx.fsf@gitster.g/\n>\n> - Patch 6/9 (repack: add `--filter=` option) has been\n> reworked to apply on top of the above mentioned patch series.\n> Taylor even posted the fixup patch to apply to this series so that\n> it works well on top of his series:\n> \n> https://lore.kernel.org/git/ZQNKkn0YYLUyN5Ih@nand.local/\n\nThanks, both, for working well together.\n\nWill replace and merge to 'seen'. Let's see others supporting the\nchange to chime in, and get it merged to 'next' soonish. I gave a\nquick cursory look and changes to rebuild on the \"existing packs\ncleanup\" topic all looked sensible.\n\n\n"},{"id":"482310","messageId":"ZRIMsdiXLqRQSJRh@nand.local","threadId":"59867","inReplyTo":"xmqqy1gurrpj.fsf@gitster.g","subject":"Re: [PATCH v7 0/9] Repack objects into separate packfiles based on a filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-09-25T22:41:53Z","receivedAt":"2023-09-25T22:41:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Sep 25, 2023 at 12:14:00PM -0700, Junio C Hamano wrote:\n> Thanks, both, for working well together.\n\nChristian made it easy to do so! ;-)\n\n> Will replace and merge to 'seen'. Let's see others supporting the\n> change to chime in, and get it merged to 'next' soonish. I gave a\n> quick cursory look and changes to rebuild on the \"existing packs\n> cleanup\" topic all looked sensible.\n\nSounds good. I took a look over the range-diff and the changes were as\nexpected. Having reviewed earlier rounds of this series in depth, I'm\ncomfortable merging this down whenever you are.\n\nThanks,\nTaylor\n"},{"id":"482532","messageId":"20231002165504.1325153-1-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20230925152517.803579-1-christian.couder@gmail.com","subject":"[PATCH v8 0/9] Repack objects into separate packfiles based on a filter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:54:55Z","receivedAt":"2023-10-02T16:55:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"# Intro\n\nLast year, John Cai sent 2 versions of a patch series to implement\n`git repack --filter=` and later I sent 4 versions of a\npatch series trying to do it a bit differently:\n\n - https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/\n - https://lore.kernel.org/git/20221012135114.294680-1-christian.couder@gmail.com/\n\nIn these patch series, the `--filter=` removed the\nfiltered out objects altogether which was considered very dangerous\neven though we implemented different safety checks in some of the\nlatter series.\n\nIn some discussions, it was mentioned that such a feature, or a\nsimilar feature in `git gc`, or in a new standalone command (perhaps\ncalled `git prune-filtered`), should put the filtered out objects into\na new packfile instead of deleting them.\n\nRecently there were internal discussions at GitLab about either moving\nblobs from inactive repos onto cheaper storage, or moving large blobs\nonto cheaper storage. This lead us to rethink at repacking using a\nfilter, but moving the filtered out objects into a separate packfile\ninstead of deleting them.\n\nSo here is a new patch series doing that while implementing the\n`--filter=` option in `git repack`.\n\n# Use cases for the new feature\n\nThis could be useful for example for the following purposes:\n\n 1) As a way for servers to save storage costs by for example moving\n large blobs, or all the blobs, or all the blobs in inactive\n repos, to separate storage (while still making them accessible\n using for example the alternates mechanism).\n\n 2) As a way to use partial clone on a Git server to offload large\n blobs to, for example, an http server, while using multiple\n promisor remotes (to be able to access everything) on the client\n side. (In this case the packfile that contains the filtered out\n object can be manualy removed after checking that all the objects\n it contains are available through the promisor remote.)\n\n 3) As a way for clients to reclaim some space when they cloned with\n a filter to save disk space but then fetched a lot of unwanted\n objects (for example when checking out old branches) and now want\n to remove these unwanted objects. (In this case they can first\n move the packfile that contains filtered out objects to a\n separate directory or storage, then check that everything works\n well, and then manually remove the packfile after some time.)\n\nAs the features and the code are quite different from those in the\nprevious series, I decided to start a new series instead of continuing\na previous one.\n\nAlso since version 2 of this new series, commit messages, don't\nmention uses cases like 2) or 3) above, as people have different\nopinions on how it should be done. How it should be done could depend\na lot on the way promisor remotes are used, the software and hardware\nsetups used, etc, so it seems more difficult to \"sell\" this series by\ntalking about such use cases. As use case 1) seems simpler and more\nappealing, it makes more sense to only talk about it in the commit\nmessages.\n\n# Changes since version 7\n\nThanks to Junio who reviewed or commented on nearly all the versions,\nand to Taylor who reviewed or commented on version 1, 3, 4, 5 and 6!\nThanks also to Robert Coup who participated in the discussions related\nto version 2 and Peff who participated in the discussions related to\nversion 4.\n\nThere are only the following changes since version 7:\n\n- This series has been rebased on top of 493f462273 (The thirteenth\n batch, 2023-09-29) to avoid possible conflicts with other series\n that could potentially conflict with this one.\n\n- Patch 2/9 (t/helper: add 'find-pack' test-tool) has been reworked to\n use the \"t0081\" test script number instead of \"t0080\" as the later\n is used by js/doc-unit-tests. I asked in:\n \n https://lore.kernel.org/git/CAP8UFD2YbYH5aZEG5NX8HLe9VeEQ+NhBfiZ9Mhy3UXTUrab3ug@mail.gmail.com/\n\n if someone thought another number was better, but got no answer.\n\nI checked that CI tests passes in:\n\nhttps://github.com/chriscool/git/actions/runs/6382343338\n\nAll jobs seem to have succeeded.\n\n# Commit overview\n\n(No changes in any of the patches compared to version 7, except on\npatch 2/9.)\n\n* 1/9 pack-objects: allow `--filter` without `--stdout`\n\n To be able to later repack with a filter we need `git pack-objects`\n to write packfiles when it's filtering instead of just writing the\n pack without the filtered out objects to stdout.\n\n* 2/9 t/helper: add 'find-pack' test-tool\n\n For testing `git repack --filter=...` that we are going to\n implement, it's useful to have a test helper that can tell which\n packfiles contain a specific object. The only change compared to v7\n is the change in test script number.\n\n* 3/9 repack: refactor finishing pack-objects command\n\n This is a small refactoring creating a new useful function, so that\n `git repack --filter=...` will be able to reuse it.\n\n* 4/9 repack: refactor finding pack prefix\n\n This is another small refactoring creating a small function that\n will be reused in the next patch.\n\n* 5/9 pack-bitmap-write: rebuild using new bitmap when remapping\n\n It fixes an issue when bitmaps are rebuilt that was revealed by this\n series, and caused a CI test to fail.\n\n* 6/9 repack: add `--filter=` option\n\n This actually adds the `--filter=` option. It uses one\n `git pack-objects` process with the `--filter` option. And then\n another `git pack-objects` process with the `--stdin-packs`\n option.\n \n* 7/9 gc: add `gc.repackFilter` config option\n\n This is a gc config option so that `git gc` can also repack using a\n filter and put the filtered out objects into a separate packfile.\n\n* 8/9 repack: implement `--filter-to` for storing filtered out objects\n\n For some use cases, it's interesting to create the packfile that\n contains the filtered out objects into a separate location. This is\n similar to the `--expire-to` option for cruft packfiles.\n\n* 9/9 gc: add `gc.repackFilterTo` config option\n\n This allows specifying the location of the packfile that contains\n the filtered out objects when using `gc.repackFilter`.\n\n# Range-diff since v7\n\n 1: eec0c09731 = 1: b23d216277 pack-objects: allow `--filter` without `--stdout`\n 2: 19c8b8a4b9 ! 2: 27e70ccf39 t/helper: add 'find-pack' test-tool\n @@ t/helper/test-tool.h: int cmd__dump_reftable(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n \n - ## t/t0080-find-pack.sh (new) ##\n + ## t/t0081-find-pack.sh (new) ##\n @@\n +#!/bin/sh\n +\n 3: aaaf40bd5d = 3: 7e692c4cfd repack: refactor finishing pack-objects command\n 4: 1eb6bc3f7e = 4: 227159ed4e repack: refactor finding pack prefix\n 5: b9159e1803 = 5: 79786eb5e1 pack-bitmap-write: rebuild using new bitmap when remapping\n 6: f2f5bb54d3 = 6: 205d33850e repack: add `--filter=` option\n 7: 7ea0307628 = 7: 16b1621169 gc: add `gc.repackFilter` config option\n 8: 698647815b = 8: 92a5ff7cc7 repack: implement `--filter-to` for storing filtered out objects\n 9: 57b2ba444c = 9: 5bfd918c90 gc: add `gc.repackFilterTo` config option\n\nChristian Couder (9):\n pack-objects: allow `--filter` without `--stdout`\n t/helper: add 'find-pack' test-tool\n repack: refactor finishing pack-objects command\n repack: refactor finding pack prefix\n pack-bitmap-write: rebuild using new bitmap when remapping\n repack: add `--filter=` option\n gc: add `gc.repackFilter` config option\n repack: implement `--filter-to` for storing filtered out objects\n gc: add `gc.repackFilterTo` config option\n\n Documentation/config/gc.txt | 16 ++\n Documentation/git-pack-objects.txt | 4 +-\n Documentation/git-repack.txt | 23 +++\n Makefile | 1 +\n builtin/gc.c | 10 ++\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 164 ++++++++++++++------\n pack-bitmap-write.c | 6 +-\n t/helper/test-find-pack.c | 50 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0081-find-pack.sh | 82 ++++++++++\n t/t5317-pack-objects-filter-objects.sh | 8 +\n t/t6500-gc.sh | 24 +++\n t/t7700-repack.sh | 197 +++++++++++++++++++++++++\n 15 files changed, 544 insertions(+), 51 deletions(-)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0081-find-pack.sh\n\n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482533","messageId":"20231002165504.1325153-2-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 1/9] pack-objects: allow `--filter` without `--stdout`","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:54:56Z","receivedAt":"2023-10-02T16:55:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21)\ntaught `git pack-objects` to use `--filter`, but required the use of\n`--stdout` since a partial clone mechanism was not yet in place to\nhandle missing objects. Since then, changes like 9e27beaa23\n(promisor-remote: implement promisor_remote_get_direct(), 2019-06-25)\nand others added support to dynamically fetch objects that were missing.\n\nEven without a promisor remote, filtering out objects can also be useful\nif we can put the filtered out objects in a separate pack, and in this\ncase it also makes sense for pack-objects to write the packfile directly\nto an actual file rather than on stdout.\n\nRemove the `--stdout` requirement when using `--filter`, so that in a\nfollow-up commit, repack can pass `--filter` to pack-objects to omit\ncertain objects from the resulting packfile.\n\nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-pack-objects.txt | 4 ++--\n builtin/pack-objects.c | 8 ++------\n t/t5317-pack-objects-filter-objects.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex dea7eacb0f..e32404c6aa 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -296,8 +296,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tnevertheless.\n \n --filter=::\n-\tRequires `--stdout`. Omits certain objects (usually blobs) from\n-\tthe resulting packfile. See linkgit:git-rev-list[1] for valid\n+\tOmits certain objects (usually blobs) from the resulting\n+\tpackfile. See linkgit:git-rev-list[1] for valid\n \t`` forms.\n \n --no-filter::\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6eb9756836..89a8b5a976 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4402,12 +4402,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (filter_options.choice) {\n-\t\tif (!pack_to_stdout)\n-\t\t\tdie(_(\"cannot use --filter without --stdout\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n-\t}\n+\tif (stdin_packs && filter_options.choice)\n+\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\ndiff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh\nindex b26d476c64..2ff3eef9a3 100755\n--- a/t/t5317-pack-objects-filter-objects.sh\n+++ b/t/t5317-pack-objects-filter-objects.sh\n@@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' '\n \t! grep blob verify_result\n '\n \n+test_expect_success 'verify blob:none packfile without --stdout' '\n+\tgit -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF &&\n+\tHEAD\n+\tEOF\n+\tgit -C r1 verify-pack -v \"mypackname-$(cat packhash).pack\" >verify_result &&\n+\t! grep blob verify_result\n+'\n+\n test_expect_success 'verify normal and blob:none packfiles have same commits/trees' '\n \tgit -C r1 verify-pack -v ../all.pack >verify_result &&\n \tgrep -E \"commit|tree\" verify_result |\n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482534","messageId":"20231002165504.1325153-3-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 2/9] t/helper: add 'find-pack' test-tool","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:54:57Z","receivedAt":"2023-10-02T16:55:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In a following commit, we will make it possible to separate objects in\ndifferent packfiles depending on a filter.\n\nTo make sure that the right objects are in the right packs, let's add a\nnew test-tool that can display which packfile(s) a given object is in.\n\nLet's also make it possible to check if a given object is in the\nexpected number of packfiles with a `--check-count ` option.\n\nSigned-off-by: Christian Couder \n---\n Makefile | 1 +\n t/helper/test-find-pack.c | 50 ++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0081-find-pack.sh | 82 +++++++++++++++++++++++++++++++++++++++\n 5 files changed, 135 insertions(+)\n create mode 100644 t/helper/test-find-pack.c\n create mode 100755 t/t0081-find-pack.sh\n\ndiff --git a/Makefile b/Makefile\nindex 003e63b792..f267034d23 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-env-helper.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-find-pack.o\n TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nnew file mode 100644\nindex 0000000000..e8bd793e58\n--- /dev/null\n+++ b/t/helper/test-find-pack.c\n@@ -0,0 +1,50 @@\n+#include \"test-tool.h\"\n+#include \"object-name.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"parse-options.h\"\n+#include \"setup.h\"\n+\n+/*\n+ * Display the path(s), one per line, of the packfile(s) containing\n+ * the given object.\n+ *\n+ * If '--check-count ' is passed, then error out if the number of\n+ * packfiles containing the object is not .\n+ */\n+\n+static const char *find_pack_usage[] = {\n+\t\"test-tool find-pack [--check-count ] \",\n+\tNULL\n+};\n+\n+int cmd__find_pack(int argc, const char **argv)\n+{\n+\tstruct object_id oid;\n+\tstruct packed_git *p;\n+\tint count = -1, actual_count = 0;\n+\tconst char *prefix = setup_git_directory();\n+\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER('c', \"check-count\", &count, \"expected number of packs\"),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, find_pack_usage, 0);\n+\tif (argc != 1)\n+\t\tusage(find_pack_usage[0]);\n+\n+\tif (repo_get_oid(the_repository, argv[0], &oid))\n+\t\tdie(\"cannot parse %s as an object name\", argv[0]);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tif (find_pack_entry_one(oid.hash, p)) {\n+\t\t\tprintf(\"%s\\n\", p->pack_name);\n+\t\t\tactual_count++;\n+\t\t}\n+\n+\tif (count > -1 && count != actual_count)\n+\t\tdie(\"bad packfile count %d instead of %d\", actual_count, count);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 621ac3dd10..9010ac6de7 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"env-helper\", cmd__env_helper },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"find-pack\", cmd__find_pack },\n \t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex a641c3a81d..f134f96b97 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv);\n int cmd__env_helper(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__find_pack(int argc, const char **argv);\n int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\ndiff --git a/t/t0081-find-pack.sh b/t/t0081-find-pack.sh\nnew file mode 100755\nindex 0000000000..67b11216a3\n--- /dev/null\n+++ b/t/t0081-find-pack.sh\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+test_description='test `test-tool find-pack`'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit one &&\n+\ttest_commit two &&\n+\ttest_commit three &&\n+\ttest_commit four &&\n+\ttest_commit five\n+'\n+\n+test_expect_success 'repack everything into a single packfile' '\n+\tgit repack -a -d --no-write-bitmap-index &&\n+\n+\thead_commit_pack=$(test-tool find-pack HEAD) &&\n+\thead_tree_pack=$(test-tool find-pack HEAD^{tree}) &&\n+\tone_pack=$(test-tool find-pack HEAD:one.t) &&\n+\tthree_pack=$(test-tool find-pack HEAD:three.t) &&\n+\told_commit_pack=$(test-tool find-pack HEAD~4) &&\n+\n+\ttest-tool find-pack --check-count 1 HEAD &&\n+\ttest-tool find-pack --check-count=1 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count=0 HEAD:one.t &&\n+\t! test-tool find-pack -c 2 HEAD:one.t &&\n+\ttest-tool find-pack -c 1 HEAD:three.t &&\n+\n+\t# Packfile exists at the right path\n+\tcase \"$head_commit_pack\" in\n+\t\t\".git/objects/pack/pack-\"*\".pack\") true ;;\n+\t\t*) false ;;\n+\tesac &&\n+\ttest -f \"$head_commit_pack\" &&\n+\n+\t# Everything is in the same pack\n+\ttest \"$head_commit_pack\" = \"$head_tree_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$one_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$three_pack\" &&\n+\ttest \"$head_commit_pack\" = \"$old_commit_pack\"\n+'\n+\n+test_expect_success 'add more packfiles' '\n+\tgit rev-parse HEAD^{tree} HEAD:two.t HEAD:four.t >objects &&\n+\tgit pack-objects .git/objects/pack/mypackname1 >packhash1 objects &&\n+\tgit pack-objects .git/objects/pack/mypackname2 >packhash2 head_tree_packs &&\n+\tgrep \"$head_commit_pack\" head_tree_packs &&\n+\tgrep mypackname1 head_tree_packs &&\n+\t! grep mypackname2 head_tree_packs &&\n+\ttest-tool find-pack --check-count 2 HEAD^{tree} &&\n+\t! test-tool find-pack --check-count 1 HEAD^{tree} &&\n+\n+\t# HEAD:five.t is also in 2 packfiles\n+\ttest-tool find-pack HEAD:five.t >five_packs &&\n+\tgrep \"$head_commit_pack\" five_packs &&\n+\t! grep mypackname1 five_packs &&\n+\tgrep mypackname2 five_packs &&\n+\ttest-tool find-pack -c 2 HEAD:five.t &&\n+\t! test-tool find-pack --check-count=0 HEAD:five.t\n+'\n+\n+test_expect_success 'add more commits (as loose objects)' '\n+\ttest_commit six &&\n+\ttest_commit seven &&\n+\n+\ttest -z \"$(test-tool find-pack HEAD)\" &&\n+\ttest -z \"$(test-tool find-pack HEAD:six.t)\" &&\n+\ttest-tool find-pack --check-count 0 HEAD &&\n+\ttest-tool find-pack -c 0 HEAD:six.t &&\n+\t! test-tool find-pack -c 1 HEAD:seven.t\n+'\n+\n+test_done\n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482535","messageId":"20231002165504.1325153-4-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 3/9] repack: refactor finishing pack-objects command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:54:58Z","receivedAt":"2023-10-02T16:55:44Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new finish_pack_objects_cmd() to refactor duplicated code\nthat handles reading the packfile names from the output of a\n`git pack-objects` command and putting it into a string_list, as well as\ncalling finish_command().\n\nWhile at it, beautify a code comment a bit in the new function.\n\nSigned-off-by: Christian Couder out, \"r\");\n+\twhile (strbuf_getline_lf(&line, out) != EOF) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (line.len != the_hash_algo->hexsz)\n+\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n+\t\t\t \"from pack-objects.\"));\n+\t\t/*\n+\t\t * Avoid putting packs written outside of the repository in the\n+\t\t * list of names.\n+\t\t */\n+\t\tif (local) {\n+\t\t\titem = string_list_append(names, line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t}\n+\t}\n+\tfclose(out);\n+\n+\tstrbuf_release(&line);\n+\n+\treturn finish_command(cmd);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -814,9 +844,8 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t struct existing_packs *existing)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in, *out;\n+\tFILE *in;\n \tint ret;\n \tconst char *scratch;\n \tint local = skip_prefix(destination, packdir, &scratch);\n@@ -861,27 +890,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only \"\n-\t\t\t \"from pack-objects.\"));\n-\t\t/*\n-\t\t * avoid putting packs written outside of the repository in the\n-\t\t * list of names\n-\t\t */\n-\t\tif (local) {\n-\t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n-\t\t}\n-\t}\n-\tfclose(out);\n-\n-\tstrbuf_release(&line);\n-\n-\treturn finish_command(&cmd);\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n }\n \n int cmd_repack(int argc, const char **argv, const char *prefix)\n@@ -891,10 +900,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list names = STRING_LIST_INIT_DUP;\n \tstruct existing_packs existing = EXISTING_PACKS_INIT;\n \tstruct pack_geometry geometry = { 0 };\n-\tstruct strbuf line = STRBUF_INIT;\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n-\tFILE *out;\n \tint show_progress;\n \n \t/* variables to be filled by option parsing */\n@@ -1124,18 +1131,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tout = xfdopen(cmd.out, \"r\");\n-\twhile (strbuf_getline_lf(&line, out) != EOF) {\n-\t\tstruct string_list_item *item;\n-\n-\t\tif (line.len != the_hash_algo->hexsz)\n-\t\t\tdie(_(\"repack: Expecting full hex object ID lines only from pack-objects.\"));\n-\t\titem = string_list_append(&names, line.buf);\n-\t\titem->util = populate_pack_exts(item->string);\n-\t}\n-\tstrbuf_release(&line);\n-\tfclose(out);\n-\tret = finish_command(&cmd);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n \tif (ret)\n \t\tgoto cleanup;\n \n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482538","messageId":"20231002165504.1325153-5-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 4/9] repack: refactor finding pack prefix","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:54:59Z","receivedAt":"2023-10-02T16:55:45Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Create a new find_pack_prefix() to refactor code that handles finding\nthe pack prefix from the packtmp and packdir global variables, as we are\ngoing to need this feature again in following commit.\n\nSigned-off-by: Christian Couder ` option and\nwe will want to check that this option is incompatible with\n`--write-bitmap-index`.\n\nUnfortunately it appears that a test like:\n\ntest_expect_success '--filter fails with --write-bitmap-index' '\n test_must_fail \\\n env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n'\n\nsometimes fail because when rebuilding bitmaps, it appears that we are\nreusing existing bitmap information. So instead of detecting that some\nobjects are missing and erroring out as it should, the\n`git repack --write-bitmap-index --filter=...` command succeeds.\n\nLet's fix that by making sure we rebuild bitmaps using new bitmaps\ninstead of existing ones.\n\nHelped-by: Taylor Blau \nSigned-off-by: Christian Couder \n---\n pack-bitmap-write.c | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f6757c3cbf..f4ecdf8b0e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -413,15 +413,19 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \n \t\tif (old_bitmap && mapping) {\n \t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct bitmap *remapped = bitmap_new();\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n \t\t\t * parents or the tree for this commit.\n \t\t\t */\n-\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap)) {\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n+\t\t\t\tbitmap_or(ent->bitmap, remapped);\n+\t\t\t\tbitmap_free(remapped);\n \t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tbitmap_free(remapped);\n \t\t}\n \n \t\t/*\n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482537","messageId":"20231002165504.1325153-8-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 7/9] gc: add `gc.repackFilter` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:55:02Z","receivedAt":"2023-10-02T16:55:49Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nUsers might want to perform such a cleanup regularly at the same time as\nthey perform other repacks and cleanups, so as part of `git gc`.\n\nLet's allow them to configure a for that purpose using a\nnew gc.repackFilter config option.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be passed a\ncorresponding `--filter=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 5 +++++\n builtin/gc.c | 6 ++++++\n t/t6500-gc.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex ca47eb2008..2153bde7ac 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the\n operation (either generating a cruft pack or unpacking unreachable\n objects) will be halted.\n \n+gc.repackFilter::\n+\tWhen repacking, use the specified filter to move certain\n+\tobjects into a separate packfile. See the\n+\t`--filter=` option of linkgit:git-repack[1].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 00192ae5d3..98148e98fe 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -61,6 +61,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static char *repack_filter;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -170,6 +171,8 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n+\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -355,6 +358,9 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n+\n+\tif (repack_filter && *repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 69509d0c11..232e403b66 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -202,6 +202,19 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n \tgrep -E \"^trace: (built-in|exec|run_command): git reflog expire --\" trace.out\n '\n \n+test_expect_success 'gc.repackFilter launches repack with a filter' '\n+\ttest_when_finished \"rm -rf bare.git\" &&\n+\tgit clone --no-local --bare . bare.git &&\n+\n+\tgit -C bare.git -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tGIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482539","messageId":"20231002165504.1325153-10-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 9/9] gc: add `gc.repackFilterTo` config option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:55:04Z","receivedAt":"2023-10-02T16:55:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit implemented the `gc.repackFilter` config option\nto specify a filter that should be used by `git gc` when\nperforming repacks.\n\nAnother previous commit has implemented\n`git repack --filter-to=` to specify the location of the\npackfile containing filtered out objects when using a filter.\n\nLet's implement the `gc.repackFilterTo` config option to specify\nthat location in the config when `gc.repackFilter` is used.\n\nNow when `git gc` will perform a repack with a configured\nthrough this option and not empty, the repack process will be\npassed a corresponding `--filter-to=` argument.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/gc.txt | 11 +++++++++++\n builtin/gc.c | 4 ++++\n t/t6500-gc.sh | 13 ++++++++++++-\n 3 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 2153bde7ac..466466d6cc 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -150,6 +150,17 @@ gc.repackFilter::\n \tobjects into a separate packfile. See the\n \t`--filter=` option of linkgit:git-repack[1].\n \n+gc.repackFilterTo::\n+\tWhen repacking and using a filter, see `gc.repackFilter`, the\n+\tspecified location will be used to create the packfile\n+\tcontaining the filtered out objects. **WARNING:** The\n+\tspecified location should be accessible, using for example the\n+\tGit alternates mechanism, otherwise the repo could be\n+\tconsidered corrupt by Git as it migh not be able to access the\n+\tobjects in that packfile. See the `--filter-to=` option\n+\tof linkgit:git-repack[1] and the `objects/info/alternates`\n+\tsection of linkgit:gitrepository-layout[5].\n+\n gc.rerereResolved::\n \tRecords of conflicted merge you resolved earlier are\n \tkept for this many days when 'git rerere gc' is run.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 98148e98fe..68ca8d45bf 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,6 +62,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static char *repack_filter;\n+static char *repack_filter_to;\n static unsigned long big_pack_threshold;\n static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n@@ -172,6 +173,7 @@ static void gc_config(void)\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -361,6 +363,8 @@ static void add_repack_all_option(struct string_list *keep_pack)\n \n \tif (repack_filter && *repack_filter)\n \t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n+\tif (repack_filter_to && *repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 232e403b66..e412cf8daf 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip \"e\n '\n \n test_expect_success 'gc.repackFilter launches repack with a filter' '\n-\ttest_when_finished \"rm -rf bare.git\" &&\n \tgit clone --no-local --bare . bare.git &&\n \n \tgit -C bare.git -c gc.cruftPacks=false gc &&\n@@ -215,6 +214,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' '\n \tgrep -E \"^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*\" trace.out\n '\n \n+test_expect_success 'gc.repackFilterTo store filtered out objects' '\n+\ttest_when_finished \"rm -rf bare.git filtered.git\" &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c gc.repackFilter=blob:none \\\n+\t\t-c gc.repackFilterTo=../filtered.git/objects/pack/pack \\\n+\t\t-c repack.writeBitmaps=false -c gc.cruftPacks=false gc &&\n+\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack\n+'\n+\n prepare_cruft_history () {\n \ttest_commit base &&\n \n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482540","messageId":"20231002165504.1325153-9-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 8/9] repack: implement `--filter-to` for storing filtered out objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:55:03Z","receivedAt":"2023-10-02T16:55:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit has implemented `git repack --filter=` to\nallow users to filter out some objects from the main pack and move them\ninto a new different pack.\n\nIt would be nice if this new different pack could be created in a\ndifferent directory than the regular pack. This would make it possible\nto move large blobs into a pack on a different kind of storage, for\nexample cheaper storage.\n\nEven in a different directory, this pack can be accessible if, for\nexample, the Git alternates mechanism is used to point to it. In fact\nnot using the Git alternates mechanism can corrupt a repo as the\ngenerated pack containing the filtered objects might not be accessible\nfrom the repo any more. So setting up the Git alternates mechanism\nshould be done before using this feature if the user wants the repo to\nbe fully usable while this feature is used.\n\nIn some cases, like when a repo has just been cloned or when there is no\nother activity in the repo, it's Ok to setup the Git alternates\nmechanism afterwards though. It's also Ok to just inspect the generated\npackfile containing the filtered objects and then just move it into the\n'.git/objects/pack/' directory manually. That's why it's not necessary\nfor this command to check that the Git alternates mechanism has been\nalready setup.\n\nWhile at it, as an example to show that `--filter` and `--filter-to`\nwork well with other options, let's also add a test to check that these\noptions work well with `--max-pack-size`.\n\nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 11 +++++++\n builtin/repack.c | 10 +++++-\n t/t7700-repack.sh | 62 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 6d5bec7716..8545a32667 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -155,6 +155,17 @@ depth is 4095.\n \ta single packfile containing all the objects. See\n \tlinkgit:git-rev-list[1] for valid `` forms.\n \n+--filter-to=::\n+\tWrite the pack containing filtered out objects to the\n+\tdirectory ``. Only useful with `--filter`. This can be\n+\tused for putting the pack on a separate object directory that\n+\tis accessed through the Git alternates mechanism. **WARNING:**\n+\tIf the packfile containing the filtered out objects is not\n+\taccessible, the repo can become corrupt as it might not be\n+\tpossible to access the objects in that packfile. See the\n+\t`objects` and `objects/info/alternates` sections of\n+\tlinkgit:gitrepository-layout[5].\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c7b564192f..db9277081d 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -977,6 +977,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint write_midx = 0;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n+\tconst char *filter_to = NULL;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -1029,6 +1030,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n+\t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n+\t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n \t\tOPT_END()\n \t};\n \n@@ -1177,6 +1180,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (po_args.filter_options.choice)\n \t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n \t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\telse if (filter_to)\n+\t\tdie(_(\"option '%s' can only be used along with '%s'\"), \"--filter-to\", \"--filter\");\n \n \tif (geometry.split_factor)\n \t\tcmd.in = -1;\n@@ -1265,8 +1270,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (po_args.filter_options.choice) {\n+\t\tif (!filter_to)\n+\t\t\tfilter_to = packtmp;\n+\n \t\tret = write_filtered_pack(&po_args,\n-\t\t\t\t\t packtmp,\n+\t\t\t\t\t filter_to,\n \t\t\t\t\t find_pack_prefix(packdir, packtmp),\n \t\t\t\t\t &existing,\n \t\t\t\t\t &names);\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 39e89445fd..48e92aa6f7 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -462,6 +462,68 @@ test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n \t)\n '\n \n+test_expect_success '--filter-to stores filtered out objects' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\n+\tgit init --bare filtered.git &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d \\\n+\t\t--filter=blob:none \\\n+\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&\n+\ttest_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&\n+\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 0 HEAD:file1) &&\n+\tblob_hash=$(git -C bare.git rev-parse HEAD:file1) &&\n+\ttest -n \"$blob_hash\" &&\n+\tblob_pack=$(test-tool -C filtered.git find-pack -c 1 $blob_hash) &&\n+\n+\techo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\tblob_content=$(git -C bare.git show $blob_hash) &&\n+\ttest \"$blob_content\" = \"content1\"\n+'\n+\n+test_expect_success '--filter works with --max-pack-size' '\n+\trm -rf filtered.git &&\n+\tgit init --bare filtered.git &&\n+\tgit init max-pack-size &&\n+\t(\n+\t\tcd max-pack-size &&\n+\t\ttest_commit base &&\n+\t\t# two blobs which exceed the maximum pack size\n+\t\ttest-tool genrandom foo 1048576 >foo &&\n+\t\tgit hash-object -w foo &&\n+\t\ttest-tool genrandom bar 1048576 >bar &&\n+\t\tgit hash-object -w bar &&\n+\t\tgit add foo bar &&\n+\t\tgit commit -m \"adding foo and bar\"\n+\t) &&\n+\tgit clone --no-local --bare max-pack-size max-pack-size.git &&\n+\t(\n+\t\tcd max-pack-size.git &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--max-pack-size=1M \\\n+\t\t\t--filter-to=../filtered.git/objects/pack/pack &&\n+\t\techo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates &&\n+\n+\t\t# Check that the 3 blobs are in different packfiles in filtered.git\n+\t\ttest_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/pack-*.pack &&\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar) &&\n+\t\tbase_pack=$(test-tool find-pack -c 1 HEAD:base.t) &&\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" != \"$base_pack\" &&\n+\t\ttest \"$bar_pack\" != \"$base_pack\" &&\n+\t\tfor pack in \"$foo_pack\" \"$bar_pack\" \"$base_pack\"\n+\t\tdo\n+\t\t\tcase \"$foo_pack\" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac\n+\t\tdone\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482541","messageId":"20231002165504.1325153-7-christian.couder@gmail.com","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"[PATCH v8 6/9] repack: add `--filter=` option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-10-02T16:55:01Z","receivedAt":"2023-10-02T16:55:55Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new option puts the objects specified by `` into a\nseparate packfile.\n\nThis could be useful if, for example, some blobs take up a lot of\nprecious space on fast storage while they are rarely accessed. It could\nmake sense to move them into a separate cheaper, though slower, storage.\n\nIt's possible to find which new packfile contains the filtered out\nobjects using one of the following:\n\n - `git verify-pack -v ...`,\n - `test-tool find-pack ...`, which a previous commit added,\n - `--filter-to=`, which a following commit will add to specify\n where the pack containing the filtered out objects will be.\n\nThis feature is implemented by running `git pack-objects` twice in a\nrow. The first command is run with `--filter=`, using the\nspecified filter. It packs objects while omitting the objects specified\nby the filter. Then another `git pack-objects` command is launched using\n`--stdin-packs`. We pass it all the previously existing packs into its\nstdin, so that it will pack all the objects in the previously existing\npacks. But we also pass into its stdin, the pack created by the previous\n`git pack-objects --filter=` command as well as the kept\npacks, all prefixed with '^', so that the objects in these packs will be\nomitted from the resulting pack. The result is that only the objects\nfiltered out by the first `git pack-objects` command are in the pack\nresulting from the second `git pack-objects` command.\n\nAs the interactions with kept packs are a bit tricky, a few related\ntests are added.\n\nHelped-by: Taylor Blau \nSigned-off-by: John Cai \nSigned-off-by: Christian Couder \n---\n Documentation/git-repack.txt | 12 ++++\n builtin/repack.c | 70 ++++++++++++++++++\n t/t7700-repack.sh | 135 +++++++++++++++++++++++++++++++++++\n 3 files changed, 217 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 4017157949..6d5bec7716 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -143,6 +143,18 @@ depth is 4095.\n \ta larger and slower repository; see the discussion in\n \t`pack.packSizeLimit`.\n \n+--filter=::\n+\tRemove objects matching the filter specification from the\n+\tresulting packfile and put them into a separate packfile. Note\n+\tthat objects used in the working directory are not filtered\n+\tout. So for the split to fully work, it's best to perform it\n+\tin a bare repo and to use the `-a` and `-d` options along with\n+\tthis option. Also `--no-write-bitmap-index` (or the\n+\t`repack.writebitmaps` config option set to `false`) should be\n+\tused otherwise writing bitmap index will fail, as it supposes\n+\ta single packfile containing all the objects. See\n+\tlinkgit:git-rev-list[1] for valid `` forms.\n+\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 9ef0044384..c7b564192f 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -21,6 +21,7 @@\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n #include \"refs.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define ALL_INTO_ONE 1\n #define LOOSEN_UNREACHABLE 2\n@@ -56,6 +57,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tstruct list_objects_filter_options filter_options;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -836,6 +838,56 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n+static int write_filtered_pack(const struct pack_objects_args *args,\n+\t\t\t const char *destination,\n+\t\t\t const char *pack_prefix,\n+\t\t\t struct existing_packs *existing,\n+\t\t\t struct string_list *names)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\tconst char *caret;\n+\tconst char *scratch;\n+\tint local = skip_prefix(destination, packdir, &scratch);\n+\n+\tprepare_pack_objects(&cmd, args, destination);\n+\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\n+\tif (!pack_kept_objects)\n+\t\tstrvec_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor_each_string_list_item(item, &existing->kept_packs)\n+\t\tstrvec_pushf(&cmd.args, \"--keep-pack=%s\", item->string);\n+\n+\tcmd.in = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Here 'names' contains only the pack(s) that were just\n+\t * written, which is exactly the packs we want to keep. Also\n+\t * 'existing_kept_packs' already contains the packs in\n+\t * 'keep_pack_list'.\n+\t */\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, names)\n+\t\tfprintf(in, \"^%s-%s.pack\\n\", pack_prefix, item->string);\n+\tfor_each_string_list_item(item, &existing->non_kept_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tfor_each_string_list_item(item, &existing->cruft_packs)\n+\t\tfprintf(in, \"%s.pack\\n\", item->string);\n+\tcaret = pack_kept_objects ? \"\" : \"^\";\n+\tfor_each_string_list_item(item, &existing->kept_packs)\n+\t\tfprintf(in, \"%s%s.pack\\n\", caret, item->string);\n+\tfclose(in);\n+\n+\treturn finish_pack_objects_cmd(&cmd, names, local);\n+}\n+\n static int write_cruft_pack(const struct pack_objects_args *args,\n \t\t\t const char *destination,\n \t\t\t const char *pack_prefix,\n@@ -966,6 +1018,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"limits the maximum number of threads\")),\n \t\tOPT_STRING(0, \"max-pack-size\", &po_args.max_pack_size, N_(\"bytes\"),\n \t\t\t\tN_(\"maximum size of each packfile\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&po_args.filter_options),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n@@ -979,6 +1032,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tOPT_END()\n \t};\n \n+\tlist_objects_filter_init(&po_args.filter_options);\n+\n \tgit_config(repack_config, &cruft_po_args);\n \n \targc = parse_options(argc, argv, prefix, builtin_repack_options,\n@@ -1119,6 +1174,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&cmd.args, \"--incremental\");\n \t}\n \n+\tif (po_args.filter_options.choice)\n+\t\tstrvec_pushf(&cmd.args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(&po_args.filter_options));\n+\n \tif (geometry.split_factor)\n \t\tcmd.in = -1;\n \telse\n@@ -1205,6 +1264,16 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (po_args.filter_options.choice) {\n+\t\tret = write_filtered_pack(&po_args,\n+\t\t\t\t\t packtmp,\n+\t\t\t\t\t find_pack_prefix(packdir, packtmp),\n+\t\t\t\t\t &existing,\n+\t\t\t\t\t &names);\n+\t\tif (ret)\n+\t\t\tgoto cleanup;\n+\t}\n+\n \tstring_list_sort(&names);\n \n \tclose_object_store(the_repository->objects);\n@@ -1297,6 +1366,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&names, 1);\n \texisting_packs_release(&existing);\n \tfree_pack_geometry(&geometry);\n+\tlist_objects_filter_release(&po_args.filter_options);\n \n \treturn ret;\n }\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 27b66807cd..39e89445fd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -327,6 +327,141 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'repacking with a filter works' '\n+\tgit -C bare.git repack -a -d &&\n+\ttest_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&\n+\tgit -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\ttest_stdout_line_count = 2 ls bare.git/objects/pack/*.pack &&\n+\tcommit_pack=$(test-tool -C bare.git find-pack -c 1 HEAD) &&\n+\tblob_pack=$(test-tool -C bare.git find-pack -c 1 HEAD:file1) &&\n+\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\ttree_pack=$(test-tool -C bare.git find-pack -c 1 HEAD^{tree}) &&\n+\ttest \"$tree_pack\" = \"$commit_pack\" &&\n+\tblob_pack2=$(test-tool -C bare.git find-pack -c 1 HEAD:file2) &&\n+\ttest \"$blob_pack2\" = \"$blob_pack\"\n+'\n+\n+test_expect_success '--filter fails with --write-bitmap-index' '\n+\ttest_must_fail \\\n+\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n+'\n+\n+test_expect_success 'repacking with two filters works' '\n+\tgit init two-filters &&\n+\t(\n+\t\tcd two-filters &&\n+\t\tmkdir subdir &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit subdir_bar subdir/bar &&\n+\t\ttest_commit subdir_baz subdir/baz\n+\t) &&\n+\tgit clone --no-local --bare two-filters two-filters.git &&\n+\t(\n+\t\tcd two-filters.git &&\n+\t\ttest_stdout_line_count = 1 ls objects/pack/*.pack &&\n+\t\tgit -c repack.writebitmaps=false repack -a -d \\\n+\t\t\t--filter=blob:none --filter=tree:1 &&\n+\t\ttest_stdout_line_count = 2 ls objects/pack/*.pack &&\n+\t\tcommit_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\t\tblob_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\troot_tree_pack=$(test-tool find-pack -c 1 HEAD^{tree}) &&\n+\t\tsubdir_tree_hash=$(git ls-tree --object-only HEAD -- subdir) &&\n+\t\tsubdir_tree_pack=$(test-tool find-pack -c 1 \"$subdir_tree_hash\") &&\n+\n+\t\t# Root tree and subdir tree are not in the same packfiles\n+\t\ttest \"$commit_pack\" != \"$blob_pack\" &&\n+\t\ttest \"$commit_pack\" = \"$root_tree_pack\" &&\n+\t\ttest \"$blob_pack\" = \"$subdir_tree_pack\"\n+\t)\n+'\n+\n+prepare_for_keep_packs () {\n+\tgit init keep-packs &&\n+\t(\n+\t\tcd keep-packs &&\n+\t\ttest_commit foo &&\n+\t\ttest_commit bar\n+\t) &&\n+\tgit clone --no-local --bare keep-packs keep-packs.git &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\t# Create two packs\n+\t\t# The first pack will contain all of the objects except one blob\n+\t\tgit rev-list --objects --all >objs &&\n+\t\tgrep -v \"bar.t\" objs | git pack-objects pack &&\n+\t\t# The second pack will contain the excluded object and be kept\n+\t\tpackid=$(grep \"bar.t\" objs | git pack-objects pack) &&\n+\t\t>pack-$packid.keep &&\n+\n+\t\t# Replace the existing pack with the 2 new ones\n+\t\trm -f objects/pack/pack* &&\n+\t\tmv pack-* objects/pack/\n+\t)\n+}\n+\n+test_expect_success '--filter works with .keep packs' '\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack_1=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\t# Object bar is still only in the old .keep pack\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$bar_pack_1\" = \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack_1\" &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\" &&\n+\t\ttest \"$bar_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n+test_expect_success '--filter works with --pack-kept-objects and .keep packs' '\n+\trm -rf keep-packs keep-packs.git &&\n+\tprepare_for_keep_packs &&\n+\t(\n+\t\tcd keep-packs.git &&\n+\n+\t\tfoo_pack=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\tbar_pack=$(test-tool find-pack -c 1 HEAD:bar.t) &&\n+\t\thead_pack=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack\" != \"$bar_pack\" &&\n+\t\ttest \"$foo_pack\" = \"$head_pack\" &&\n+\n+\t\tgit -c repack.writebitmaps=false repack -a -d --filter=blob:none \\\n+\t\t\t--pack-kept-objects &&\n+\n+\t\tfoo_pack_1=$(test-tool find-pack -c 1 HEAD:foo.t) &&\n+\t\ttest-tool find-pack -c 2 HEAD:bar.t >bar_pack_1 &&\n+\t\thead_pack_1=$(test-tool find-pack -c 1 HEAD) &&\n+\n+\t\ttest \"$foo_pack_1\" != \"$foo_pack\" &&\n+\t\ttest \"$foo_pack_1\" != \"$bar_pack\" &&\n+\t\ttest \"$head_pack_1\" != \"$head_pack\" &&\n+\n+\t\t# Object bar is in both the old .keep pack and the new\n+\t\t# pack that contained the filtered out objects\n+\t\tgrep \"$bar_pack\" bar_pack_1 &&\n+\t\tgrep \"$foo_pack_1\" bar_pack_1 &&\n+\t\ttest \"$foo_pack_1\" != \"$head_pack_1\"\n+\t)\n+'\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n-- \n2.42.0.305.g5bfd918c90\n\n"},{"id":"482544","messageId":"ZRsknb4NxNHTR21E@nand.local","threadId":"59867","inReplyTo":"20231002165504.1325153-1-christian.couder@gmail.com","subject":"Re: [PATCH v8 0/9] Repack objects into separate packfiles based on a filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T20:14:21Z","receivedAt":"2023-10-02T20:14:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 02, 2023 at 06:54:55PM +0200, Christian Couder wrote:\n> # Range-diff since v7\n>\n> 1: eec0c09731 = 1: b23d216277 pack-objects: allow `--filter` without `--stdout`\n> 2: 19c8b8a4b9 ! 2: 27e70ccf39 t/helper: add 'find-pack' test-tool\n> @@ t/helper/test-tool.h: int cmd__dump_reftable(int argc, const char **argv);\n> int cmd__genrandom(int argc, const char **argv);\n> int cmd__genzeros(int argc, const char **argv);\n>\n> - ## t/t0080-find-pack.sh (new) ##\n> + ## t/t0081-find-pack.sh (new) ##\n> @@\n> +#!/bin/sh\n> +\n> 3: aaaf40bd5d = 3: 7e692c4cfd repack: refactor finishing pack-objects command\n> 4: 1eb6bc3f7e = 4: 227159ed4e repack: refactor finding pack prefix\n> 5: b9159e1803 = 5: 79786eb5e1 pack-bitmap-write: rebuild using new bitmap when remapping\n> 6: f2f5bb54d3 = 6: 205d33850e repack: add `--filter=` option\n> 7: 7ea0307628 = 7: 16b1621169 gc: add `gc.repackFilter` config option\n> 8: 698647815b = 8: 92a5ff7cc7 repack: implement `--filter-to` for storing filtered out objects\n> 9: 57b2ba444c = 9: 5bfd918c90 gc: add `gc.repackFilterTo` config option\n\nThese all look just as good as v7 ;-).\n\nThanks,\nTaylor\n"}]}